Recover Corrupt/Missing OCR with No Backup - (Oracle 10g)

by Jeff Hunter, Sr. Database Administrator Contents

1. 2. 3. 4.

Overview Example Configuration Recover Corrupt/Missing OCR About the Author

Overview It happens. Not very often, but it can happen. You are faced with a corrupt or missing Oracle Cluster Registry (OCR) and have no backup to recover from. So, how can something like this occur? We know that the CRSD process is responsible for creating backup copies of the OCR every 4 hours from the master node in the CRS_home/cdata directory. These backups are meant to be used to recover the OCR from a lost or corrupt OCR file using the ocrconfig -restore command, so how is it possible to be in a situation where the OCR needs to be recovered and you have no viable backup? Well, consider a scenario where you add a node to the cluster and before the next backup (before 4 hours) you find the OCR has been corrupted. You may have forgotten to create a logical export of the OCR before adding the new node or worse yet, the logical export you took is also corrupt. In either case, you are left with a corrupt OCR and no recent backup. Talk about a bad day! Another possible scenario could be a shell script that wrongly deletes all available backups. Talk about an even worse day. In the event the OCR is corrupt on one node and all options to recover it have failed, one safe way to re-create the OCR (and consequently the voting disk) is to reinstall the Oracle Clusterware software. In order to accomplish this, a complete outage is required for the entire cluster throughout the duration of the re-install. The Oracle Clusterware software will need to be fully removed, the OCR and voting disks reformatted, all virtual IP addresses (VIPs) de-installed, and a complete reinstall of the Oracle Clusterware software will need to be performed. It should also be noted that any patches that were applied to the original clusterware install will need to be re-

applied. As you can see, having a backup of the OCR and voting disk can dramatically simplify the recovery of your system! A second and much more efficient method used to re-create the OCR (and consequently the voting disk as well) is to re-run the root.sh script from the primary node in the cluster. This is described in Doc ID: 399482.1 on the My Oracle Support web site. In my opinion, this method is quicker and much less intrusive than reinstalling Oracle Clusterware. Using root.sh to re-create the OCR/Voting Disk is the focus of this article. It is worth mentioning that only one of the two methods mentioned above needs to be performed in order to recover from a lost or corrupt OCR. In addition to recovering the OCR, either method could also be used to restore the SCLS directories from an accidental delete. These are internal only directories which are created by root.sh and on the Linux platform are located at /etc/oracle/scls_scr. If the SCLS directories are accidentally removed then they can only be created using the same methods used to re-create the OCR which is the focus of this article. There are two other critical files in Oracle Clusterware that if accidentally deleted, are a bit easier to recover from:

Voting Disk If there are multiple voting disks and one was accidentally deleted, then check if there are any backups of this voting disk. If there are no backups then we can add one using the crsctl add votedisk command.

Socket files in /tmp/.oracle or /var/tmp/.oracle If these files are accidentally deleted, then stop the Oracle Clusterware on that node and restart it again. This will recreate these socket files. If the socket files for cssd are deleted, then the Oracle Clusterware stack may not come down in which case the node has to be bounced.

Example Configuration The example configuration used in this article consists of a two-node RAC with a clustered database named racdb.idevelopment.info running Oracle RAC 10g Release 2 on the Linux x86 platform. The two node names are racnode1 and racnode2, each

hosting a single Oracle instance named racdb1 and racdb2 respectively. For a detailed guide on building the example clustered database environment, please see: Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 5.3 / iSCSI) The example Oracle Clusterware environment is configured with three mirrored voting disks and two mirrored OCR files all of which are located on an OCFS2 clustered file system. Note that the voting disk is owned by the oracle user in the oinstall group with 0644 permissions while the OCR file is owned by root in the oinstall group with 0640 permissions:
[oracle@racnode1 ~]$ ls -l /u02/oradata/racdb total 39840 -rw-r--r-- 1 oracle oinstall 10240000 Oct 9 -rw-r--r-- 1 oracle oinstall 10240000 Oct 9 -rw-r--r-- 1 oracle oinstall 10240000 Oct 9 drwxr-xr-x 2 oracle oinstall 3896 Aug 26 -rw-r----- 1 root oinstall 268644352 Oct 9 -rw-r----- 1 root oinstall 268644352 Oct 9

19:33 19:36 19:38 23:45 19:27 19:28

CSSFile CSSFile_mirror1 CSSFile_mirror2 dbs OCRFile OCRFile_mirror

Check Current OCR File

[oracle@racnode1 ~]$ ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name Device/File Name is as follows : : 2 : 262120 : 4676 : 257444 : 1513888898 : /u02/oradata/racdb/OCRFile Device/File integrity check succeeded : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check succeeded

Cluster registry integrity check succeeded

Check Current Voting Disk

[oracle@racnode1 ~]$ crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile 1. 0 /u02/oradata/racdb/CSSFile_mirror1 2. 0 /u02/oradata/racdb/CSSFile_mirror2 located 3 votedisk(s).

Network Settings
Oracle RAC Node 1 - (racnode1)

Devi IP Address Subnet ce eth0

Gateway Purpose

192.168.1.1 255.255.25 192.168. Connects racnode1 to the public network 51 5.0 1.1 192.168.2.1 255.255.25 51 5.0 192.168.3.1 255.255.25 51 5.0 Connects racnode1 to iSCSI shared storage (Openfiler). Connects racnode1 (interconnect) to racnode2 (racnode2-priv)

eth1

eth2

/etc/hosts 127.0.0.1 localhost.localdomain localhost

# Public Network - (eth0) 192.168.1.151 racnode1 192.168.1.152 racnode2 # Network Storage - (eth1) 192.168.2.151 racnode1-san 192.168.2.152 racnode2-san # Private Interconnect - (eth2) 192.168.3.151 racnode1-priv 192.168.3.152 racnode2-priv # Public Virtual IP (VIP) addresses - (eth0:1) 192.168.1.251 racnode1-vip 192.168.1.252 racnode2-vip # Private Storage Network for Openfiler - (eth1) 192.168.1.195 openfiler1 192.168.2.195 openfiler1-priv

Oracle RAC Node 2 - (racnode2)

Devi IP Address Subnet ce eth0 Gateway Purpose

192.168.1.1 255.255.25 192.168. Connects racnode2 to the public network 52 5.0 1.1 192.168.2.1 255.255.25 52 5.0 192.168.3.1 255.255.25 52 5.0 Connects racnode2 to iSCSI shared storage (Openfiler). Connects racnode2 (interconnect) to racnode1 (racnode1-priv)

eth1

eth2

/etc/hosts

127.0.0.1

localhost.localdomain localhost

# Public Network - (eth0) 192.168.1.151 racnode1 192.168.1.152 racnode2 # Network Storage - (eth1) 192.168.2.151 racnode1-san 192.168.2.152 racnode2-san # Private Interconnect - (eth2) 192.168.3.151 racnode1-priv 192.168.3.152 racnode2-priv # Public Virtual IP (VIP) addresses - (eth0:1) 192.168.1.251 racnode1-vip 192.168.1.252 racnode2-vip # Private Storage Network for Openfiler - (eth1) 192.168.1.195 openfiler1 192.168.2.195 openfiler1-priv

Recover Corrupt/Missing OCR To describe the steps required in recovering the OCR, it is assumed the current OCR has been accidentally deleted and no viable backups are available. It is also assumed the CRS stack was up and running on both nodes in the cluster at the time the OCR files were removed:
[root@racnode1 ~]# rm /u02/oradata/racdb/OCRFile [root@racnode1 ~]# rm /u02/oradata/racdb/OCRFile_mirror [root@racnode1 ~]# ps root 548 27171 oracle 575 566 root 1118 660 1000 -m 500 -f oracle 1277 749 [root@racnode2 ~]# ps oracle 674 673 root 815 27760 root 1201 827 1000 -m 500 -f oracle 1442 891 -ef | grep d.bin | grep -v grep 0 Oct09 ? 00:06:17 /u01/app/crs/bin/crsd.bin reboot 0 Oct09 ? 00:00:10 /u01/app/crs/bin/evmd.bin 0 Oct09 ? 00:00:00 /u01/app/crs/bin/oprocd.bin run -t 0 Oct09 ? 00:03:31 /u01/app/crs/bin/ocssd.bin

-ef | grep d.bin | grep -v grep 0 Oct09 ? 00:00:10 /u01/app/crs/bin/evmd.bin 0 Oct09 ? 00:06:12 /u01/app/crs/bin/crsd.bin reboot 0 Oct09 ? 00:00:00 /u01/app/crs/bin/oprocd.bin run -t 0 Oct09 ? 00:03:43 /u01/app/crs/bin/ocssd.bin

1. Shutdown Oracle Clusterware on All Nodes.

Although all OCR files have been lost or corrupted, the Oracle Clusterware daemons as well as the clustered database remain running. In this scenario, Oracle Clusterware and all managed resources need to be shut down in order to start the OCR recovery. Attempting to stop CRS using crsctl stop crs will fail given it cannot write to the now lost/corrupt OCR file:
[root@racnode1 ~]# crsctl stop crs OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

With the environment in this unstable state, shutdown all database instances from all nodes in the cluster and then reboot each node:
[oracle@racnode1 ~]$ sqlplus / as sysdba SQL> shutdown immediate [root@racnode1 ~]# reboot -----------------------------------------------[oracle@racnode2 ~]$ sqlplus / as sysdba SQL> shutdown immediate [root@racnode2 ~]# reboot

When the Oracle RAC nodes come back up, note that Oracle Clusterware will fail to start as a result of the lost/corrupt OCR file:
[root@racnode1 ~]# crs_stat -t CRS-0184: Cannot communicate with the CRS daemon. [root@racnode2 ~]# crs_stat -t CRS-0184: Cannot communicate with the CRS daemon.

2. Execute rootdelete.sh from All Nodes. The rootdelete.sh script can be found at $ORA_CRS_HOME/install/rootdelete.sh on all nodes in the cluster:
[root@racnode1 ~]# $ORA_CRS_HOME/install/rootdelete.sh Shutting down Oracle Cluster Ready Services (CRS): OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2] Shutdown has begun. The daemons should exit soon. Checking to see if Oracle CRS stack is down... Oracle CRS stack is not running. Oracle CRS stack is down now.

Removing script for Oracle Cluster Ready services Updating ocr file for downgrade Cleaning up SCR settings in '/etc/oracle/scls_scr' [root@racnode2 ~]# $ORA_CRS_HOME/install/rootdelete.sh Shutting down Oracle Cluster Ready Services (CRS): OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2] Shutdown has begun. The daemons should exit soon. Checking to see if Oracle CRS stack is down... Oracle CRS stack is not running. Oracle CRS stack is down now. Removing script for Oracle Cluster Ready services Updating ocr file for downgrade Cleaning up SCR settings in '/etc/oracle/scls_scr'

The "OCR initialization failed accessing OCR device" and PROC-26 errors can be safely ignored given the OCR is not available. The most important action is that the SCR entries are cleaned up. Keep in mind that if you have more than two nodes in your cluster, you need to run rootdelete.sh on all other nodes as well. 3. Run rootdeinstall.sh from the Primary Node. The primary node is the node where the Oracle Clusterware installation was performed on (which is typically node1). For the purpose of this example, I originally installed Oracle Clusterware from the machine racnode1 which is therefore the primary node. The rootdeinstall.sh script will clear out any old data from a raw storage device in preparation for the new OCR. If the OCR is on a clustered file system, a new OCR file(s) will be created with null data.
[root@racnode1 ~]# $ORA_CRS_HOME/install/rootdeinstall.sh Removing contents from OCR mirror device 2560+0 records in 2560+0 records out 10485760 bytes (10 MB) copied, 0.0513806 seconds, 204 MB/s Removing contents from OCR device 2560+0 records in 2560+0 records out 10485760 bytes (10 MB) copied, 0.0443477 seconds, 236 MB/s

4. Run root.sh from the Primary Node. (same node as above) Amoung several other tasks, this script will create the OCR and voting disk(s).

[root@racnode1 ~]# $ORA_CRS_HOME/root.sh Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node : node 1: racnode1 racnode1-priv racnode1 node 2: racnode2 racnode2-priv racnode2 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Now formatting voting device: /u02/oradata/racdb/CSSFile Now formatting voting device: /u02/oradata/racdb/CSSFile_mirror1 Now formatting voting device: /u02/oradata/racdb/CSSFile_mirror2 Format of 3 voting devices complete. Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. racnode1 CSS is inactive on these nodes. racnode2 Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.

5. Run root.sh from All Remaining Nodes.

[root@racnode2 ~]# $ORA_CRS_HOME/root.sh Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node : node 1: racnode1 racnode1-priv racnode1 node 2: racnode2 racnode2-priv racnode2 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. racnode1 racnode2 CSS is active on all nodes.

Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps Creating Creating Creating Starting Starting Starting Done. VIP GSD ONS VIP GSD ONS application application application application application application resource resource resource resource resource resource on on on on on on (2) (2) (2) (2) (2) (2) nodes... nodes... nodes... nodes... nodes... nodes...

6. Oracle 10.2.0.1 users should note that running root.sh on the last node will fail.

Most notably is the silent mode VIPCA configuration failing because of BUG 4437727 in 10.2.0.1. Refer to my article Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 5.3 / iSCSI)to workaround these errors. 7. The Oracle Clusterware and Oracle RAC software in my configuration were patched with 10.2.0.4 and therefore did not receive any errors during the running of root.sh on the last node. 8. Configure Server-Side ONS using racgons.
CRS_home/bin/racgons add_config hostname1:port hostname2:port [root@racnode1 ~]# $ORA_CRS_HOME/bin/racgons add_config racnode1:6200 racnode2:6200 [root@racnode1 ~]# $ORA_CRS_HOME/bin/onsctl ping Number of onsconfiguration retrieved, numcfg = 2 onscfg[0] {node = racnode1, port = 6200} Adding remote host racnode1:6200 onscfg[1] {node = racnode2, port = 6200} Adding remote host racnode2:6200 ons is running ...

9. Configure Network Interfaces for Clusterware. Log in as the owner of the Oracle Clusterware software which is typically the oracle user account and configure all network interfaces. The first step is to identify the current interfaces and IP addresses using oifcfg iflist. As discussed in the network settings section,eth0/192.168.1.0 is my public interface/network, eth1/192.168.2.0 is my iSCSI storage network and not used specifically for Oracle Clusterware, and eth2/192.168.3.0 is the cluster_interconnect interface/network.
[oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/oifcfg iflist eth0 192.168.1.0 <-- public interface

eth1 eth2

192.168.2.0 192.168.3.0

<-- not used <-- cluster interconnect

[oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/oifcfg setif -global eth0/192.168.1.0:public [oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/oifcfg setif -global eth2/192.168.3.0:cluster_interconnect [oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/oifcfg getif eth0 192.168.1.0 global public eth2 192.168.3.0 global cluster_interconnect

10.

Add TNS Listener using NETCA.

As the Oracle Clusterware software owner (typically oracle), add a cluster TNS listener configuration to OCR using netca. This may give errors if the listener.ora contains the entries already. If this is the case, move the listener.ora to /tmp from the$ORACLE_HOME/network/admin or from the $TNS_ADMIN directory if the TNS_ADMIN environmental is defined and then run netca. Add all the listeners that were added during the original Oracle Clusterware software installation.
[oracle@racnode1 ~]$ export DISPLAY=<X-Windows Terminal>:0 [oracle@racnode1 ~]$ mv $TNS_ADMIN/listener.ora /tmp/listener.ora.original [oracle@racnode2 ~]$ mv $TNS_ADMIN/listener.ora /tmp/listener.ora.original [oracle@racnode1 ~]$ netca &

11.

Add all Resources Back to OCR using srvctl.

As a final step, log in as the Oracle Clusterware software owner (typically oracle) and add all resources back to the OCR using the srvctlcommand. Please ensure that these commands are not run as the root user account. Add ASM INSTANCE(S) to OCR:
srvctl add asm -n <node_name> -i <asm_instance_name> -o <oracle_home> [oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/srvctl add asm -i +ASM1 -n racnode1 -o /u01/app/oracle/product/10.2.0/db_1 [oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/srvctl add asm -i +ASM2 -n racnode2 -o /u01/app/oracle/product/10.2.0/db_1

Add DATABASE to OCR:

srvctl add database -d <db_unique_name> -o <oracle_home>

[oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/srvctl add database -d racdb -o /u01/app/oracle/product/10.2.0/db_1

Add INSTANCE(S) to OCR:

srvctl add instance -d <db_unique_name> -i <instance_name> -n <node_name> [oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/srvctl add instance -d racdb -i racdb1 -n racnode1 [oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/srvctl add instance -d racdb -i racdb2 -n racnode2

Add SERVICE(S) to OCR:

srvctl add service -d <db_unique_name> -s <service_name> -r <preferred_list> -P <TAF_policy>

where

TAF_policy

is set to

NONE, BASIC,

or

PRECONNECT

[oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/srvctl add service -d racdb -s racdb_srvc -r racdb1,racdb2 -P BASIC

After completing the steps above, the OCR should have been successfully recreated. Bring up all of the resources that were added to the OCR and runcluvfy to verify the cluster configuration.
[oracle@racnode1 ~]$ $ORA_CRS_HOME/bin/crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....srvc.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....E1.lsnr application ONLINE ONLINE racnode1 ora....de1.gsd application ONLINE ONLINE racnode1 ora....de1.ons application ONLINE ONLINE racnode1 ora....de1.vip application ONLINE ONLINE racnode1 ora....SM2.asm application OFFLINE OFFLINE ora....E2.lsnr application ONLINE ONLINE racnode2 ora....de2.gsd application ONLINE ONLINE racnode2 ora....de2.ons application ONLINE ONLINE racnode2 ora....de2.vip application ONLINE ONLINE racnode2 [oracle@racnode1 [oracle@racnode1 [oracle@racnode1 [oracle@racnode1 ~]$ ~]$ ~]$ ~]$ srvctl srvctl srvctl srvctl start start start start asm -n racnode1 asm -n racnode2 database -d racdb service -d racdb

[oracle@racnode1 ~]$ cluvfy stage -post crsinst -n racnode1,racnode2 Performing post-checks for cluster services setup Checking node reachability... Node reachability check passed from node "racnode1". Checking user equivalence... User equivalence check passed for user "oracle". Checking Cluster manager integrity... Checking CSS daemon... Daemon status check passed for "CSS daemon". Cluster manager integrity check passed. Checking cluster integrity... Cluster integrity check passed Checking OCR integrity... Checking the absence of a non-clustered configuration... All nodes free of non-clustered, local-only configurations. Uniqueness check for OCR device passed. Checking the version of OCR... OCR of correct Version "2" exists. Checking data integrity of OCR... Data integrity check for OCR passed. OCR integrity check passed. Checking CRS integrity... Checking daemon liveness... Liveness check passed for "CRS daemon". Checking daemon liveness... Liveness check passed for "CSS daemon". Checking daemon liveness... Liveness check passed for "EVM daemon". Checking CRS health... CRS health check passed. CRS integrity check passed.

Checking node application existence... Checking existence of VIP node application (required) Check passed. Checking existence of ONS node application (optional) Check passed. Checking existence of GSD node application (optional) Check passed. Post-check for cluster services setup was successful.

About the Author Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX, Linux, and Windows server environment. Jeff's other interests include mathematical encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 18 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science.

OCR File and Voting Disk Administration by Example - (Oracle 10g)

by Jeff Hunter, Sr. Database Administrator Contents
1. 2. 3.

Overview Example Configuration Administering the OCR File View OCR Configuration Information Add an OCR File

Relocate an OCR File Repair an OCR File on a Local Node Remove an OCR File 4. Backup the OCR File Automatic OCR Backups Manual OCR Exports 5. Recover the OCR File Recover OCR from Valid OCR Mirror Recover OCR from Automatically Generated Physical Backup Recover OCR from an OCR Export File 6. Administering the Voting Disk View Voting Disk Configuration Information Add a Voting Disk Remove a Voting Disk Relocate a Voting Disk 7. Backup the Voting Disk 8. Recover the Voting Disk 9. Move the Voting Disk and OCR from OCFS to RAW Devices Move the OCR Move the Voting Disk 10. About the Author

Overview Oracle Clusterware 10g, formerly known as Cluster Ready Services (CRS) is software that when installed on servers running the same operating system, enables the servers to be bound together to operate and function as a single server or cluster. This infrastructure simplifies the requirement for an Oracle Real Application Clusters (RAC) database by providing cluster software that is tightly integrated with the Oracle Database. The Oracle Clusterware requires two critical clusterware components: a voting disk to record node membership information and the Oracle Cluster Registry (OCR) to record cluster configuration information: Voting Disk

The voting disk is a shared partition that Oracle Clusterware uses to verify cluster node membership and status. Oracle Clusterware uses the voting disk to determine which instances are members of a cluster by way of a health check and arbitrates cluster ownership among the instances in case of network failures. The primary function of the voting disk is to manage node membership and prevent what is known as Split Brain Syndrome in which two or more instances attempt to control the RAC database. This can occur in cases where there is a break in communication between nodes through the interconnect. The voting disk must reside on a shared disk(s) that is accessible by all of the nodes in the cluster. For high availability, Oracle recommends that you have multiple voting disks. Oracle Clusterware can be configured to maintain multiple voting disks (multiplexing) but you must have an odd number of voting disks, such as three, five, and so on. Oracle Clusterware supports a maximum of 32 voting disks. If you define a single voting disk, then you should use external mirroring to provide redundancy. A node must be able to access more than half of the voting disks at any time. For example, if you have five voting disks configured, then a node must be able to access at least three of the voting disks at any time. If a node cannot access the minimum required number of voting disks it is evicted, or removed, from the cluster. After the cause of the failure has been corrected and access to the voting disks has been restored, you can instruct Oracle Clusterware to recover the failed node and restore it to the cluster. Oracle Cluster Registry (OCR) Maintains cluster configuration information as well as configuration information about any cluster database within the cluster. OCR is the repository of configuration information for the cluster that manages information about like the cluster node list and instance-to-node mapping information. This configuration information is used by many of the processes that make up the CRS as well as other cluster-aware applications which use this repository to share information amoung them. Some of the main components included in the OCR are:

Node membership information Database instance, node, and other mapping information ASM (if configured) Application resource profiles such as VIP addresses, services, etc. Service characteristics Information about processes that Oracle Clusterware controls

Information about any third-party applications controlled by CRS (10g R2 and later)

The OCR stores configuration information in a series of key-value pairs within a directory tree structure. To view the contents of the OCR in a human-readable format, run the ocrdump command. This will dump the contents of the OCR into an ASCII text file in the current directory named OCRDUMPFILE. The OCR must reside on a shared disk(s) that is accessible by all of the nodes in the cluster. Oracle Clusterware 10g Release 2 allows you to multiplex the OCR and Oracle recommends that you use this feature to ensure cluster high availability. Oracle Clusterware allows for a maximum of two OCR locations; one is the primary and the second is an OCR mirror. If you define a single OCR, then you should use external mirroring to provide redundancy. You can replace a failed OCR online, and you can update the OCR through supported APIs such as Enterprise Manager, the Server Control Utility (SRVCTL), or the Database Configuration Assistant (DBCA). This article provides a detailed look at how to administer the two critical Oracle Clusterware components the voting disk and the Oracle Cluster Registry (OCR). The examples described in this guide were tested with Oracle RAC 10g Release 2 (10.2.0.4) on the Linux x86 platform.
It is highly recommended to take a backup of the voting disk and OCR file before making any changes! Instruction are included in this guide on how to perform backups of the voting disk andOCR file. CRS_home

The Oracle Clusterware binaries included in this article (i.e. crs_stat, ocrcheck, crsctl, etc.) are being executed from the Oracle Clusterware home directory which for the purpose of this article is /u01/app/crs. The environment variable $ORA_CRS_HOME is set for both the oracleand root user accounts to this directory and is also included in the $PATH:
[root@racnode1 ~]# echo $ORA_CRS_HOME /u01/app/crs [root@racnode1 ~]# which ocrcheck /u01/app/crs/bin/ocrcheck

Example Configuration

The example configuration used in this article consists of a two-node RAC with a clustered database named racdb.idevelopment.info running Oracle RAC 10g Release 2 on the Linux x86 platform. The two node names are racnode1 and racnode2, each hosting a single Oracle instance named racdb1 and racdb2 respectively. For a detailed guide on building the example clustered database environment, please see: Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 5.3 / iSCSI) The example Oracle Clusterware environment is configured with a single voting disk and a single OCR file on an OCFS2 clustered file system. Note that the voting disk is owned by the oracle user in the oinstall group with 0644 permissions while the OCR file is owned by root in the oinstallgroup with 0640 permissions:
[oracle@racnode1 ~]$ ls -l /u02/oradata/racdb total 16608 -rw-r--r-- 1 oracle oinstall 10240000 Aug 26 22:43 CSSFile drwxr-xr-x 2 oracle oinstall 3896 Aug 26 23:45 dbs/ -rw-r----- 1 root oinstall 6836224 Sep 3 23:47 OCRFile

Check Current OCR File

[oracle@racnode1 ~]$ ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name is as follows : : 2 : 262120 : 4660 : 257460 : 1331197 : /u02/oradata/racdb/OCRFile Device/File integrity check succeeded Device/File not configured Cluster registry integrity check succeeded

Check Current Voting Disk

[oracle@racnode1 ~]$ crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile located 1 votedisk(s).

Preparation To prepare for the examples used in this guide, five new iSCSI volumes were created from the SAN and will be bound to RAW devices on all nodes in the RAC cluster. These five new volumes will be used to demonstrate how to move the current voting disk and OCR file from an OCFS2 file system to RAW devices:

Five New iSCSI Volumes and their Local Device Name Mappings iSCSI Target Name
iqn.2006-01.com.openfiler:racdb.ocr1 iqn.2006-01.com.openfiler:racdb.ocr2 iqn.2006-01.com.openfiler:racdb.voting1 iqn.2006-01.com.openfiler:racdb.voting2 iqn.2006-01.com.openfiler:racdb.voting3

Local Device Name

/dev/iscsi/ocr1/part /dev/iscsi/ocr2/part /dev/iscsi/voting1/part /dev/iscsi/voting2/part /dev/iscsi/voting3/part

Disk Size
512 MB 512 MB 32 MB 32 MB 32 MB

After creating the new iSCSI volumes from the SAN, they now need to be configured for access and bound to RAW devices by all Oracle RAC nodes in the database cluster.
1.

From all Oracle RAC nodes in the cluster as root, discover the five new iSCSI volumes from the SAN which will be used to store the voting disks and OCR files.
[root@racnode1 ~]# iscsiadm -m discovery -t sendtargets -p openfiler1-san 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm1 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm2 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.crs 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.ocr1 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.ocr2 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.voting1 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.voting2 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.voting3 [root@racnode2 ~]# iscsiadm -m discovery -t sendtargets -p openfiler1-san 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm1 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm2 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.crs 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.ocr1 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.ocr2 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.voting1 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.voting2 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.voting3

2. Manually login to the new iSCSI targets from all Oracle RAC nodes in the cluster.
[root@racnode1 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.ocr1 -p

192.168.2.195 -l [root@racnode1 ~]# iscsiadm 192.168.2.195 -l [root@racnode1 ~]# iscsiadm -p 192.168.2.195 -l [root@racnode1 ~]# iscsiadm -p 192.168.2.195 -l [root@racnode1 ~]# iscsiadm -p 192.168.2.195 -l [root@racnode2 ~]# iscsiadm 192.168.2.195 -l [root@racnode2 ~]# iscsiadm 192.168.2.195 -l [root@racnode2 ~]# iscsiadm -p 192.168.2.195 -l [root@racnode2 ~]# iscsiadm -p 192.168.2.195 -l [root@racnode2 ~]# iscsiadm -p 192.168.2.195 -l

-m node -T iqn.2006-01.com.openfiler:racdb.ocr2 -p -m node -T iqn.2006-01.com.openfiler:racdb.voting1 -m node -T iqn.2006-01.com.openfiler:racdb.voting2 -m node -T iqn.2006-01.com.openfiler:racdb.voting3 -m node -T iqn.2006-01.com.openfiler:racdb.ocr1 -p -m node -T iqn.2006-01.com.openfiler:racdb.ocr2 -p -m node -T iqn.2006-01.com.openfiler:racdb.voting1 -m node -T iqn.2006-01.com.openfiler:racdb.voting2 -m node -T iqn.2006-01.com.openfiler:racdb.voting3

3. Create a single primary partition on each of the five new iSCSI volumes that span the entire disk. Perform this from only one of the Oracle RAC nodes in the cluster:
[root@racnode1 [root@racnode1 [root@racnode1 [root@racnode1 [root@racnode1 ~]# ~]# ~]# ~]# ~]# fdisk fdisk fdisk fdisk fdisk /dev/iscsi/ocr1/part /dev/iscsi/ocr2/part /dev/iscsi/voting1/part /dev/iscsi/voting2/part /dev/iscsi/voting3/part

4. Re-scan the SCSI bus from all Oracle RAC nodes in the cluster:
[root@racnode2 ~]# partprobe

5.

Create a shell script (/usr/local/bin/setup_raw_devices.sh) on all Oracle RAC nodes in the cluster to bind the five Oracle Clusterware component devices to RAW devices as follows:
# +---------------------------------------------------------+ # | FILE: /usr/local/bin/setup_raw_devices.sh | # +---------------------------------------------------------+ # +---------------------------------------------------------+ # | Bind OCR files to RAW device files. | # +---------------------------------------------------------+ /bin/raw /dev/raw/raw1 /dev/iscsi/ocr1/part1 /bin/raw /dev/raw/raw2 /dev/iscsi/ocr2/part1 sleep 3 /bin/chown root:oinstall /dev/raw/raw1 /bin/chown root:oinstall /dev/raw/raw2 /bin/chmod 0640 /dev/raw/raw1 /bin/chmod 0640 /dev/raw/raw2

# +---------------------------------------------------------+ # | Bind voting disks to RAW device files. | # +---------------------------------------------------------+ /bin/raw /dev/raw/raw3 /dev/iscsi/voting1/part1 /bin/raw /dev/raw/raw4 /dev/iscsi/voting2/part1 /bin/raw /dev/raw/raw5 /dev/iscsi/voting3/part1 sleep 3 /bin/chown oracle:oinstall /dev/raw/raw3 /bin/chown oracle:oinstall /dev/raw/raw4 /bin/chown oracle:oinstall /dev/raw/raw5 /bin/chmod 0644 /dev/raw/raw3 /bin/chmod 0644 /dev/raw/raw4 /bin/chmod 0644 /dev/raw/raw5

6. From all Oracle RAC nodes in the cluster, change the permissions of the new shell script to execute:
[root@racnode1 ~]# chmod 755 /usr/local/bin/setup_raw_devices.sh [root@racnode2 ~]# chmod 755 /usr/local/bin/setup_raw_devices.sh

7. Manually execute the new shell script from all Oracle RAC nodes in the cluster to bind the voting disks to RAW devices:
[root@racnode1 ~]# /usr/local/bin/setup_raw_devices.sh /dev/raw/raw1: bound to major 8, minor 97 /dev/raw/raw2: bound to major 8, minor 17 /dev/raw/raw3: bound to major 8, minor 1 /dev/raw/raw4: bound to major 8, minor 49 /dev/raw/raw5: bound to major 8, minor 33 [root@racnode2 ~]# /usr/local/bin/setup_raw_devices.sh /dev/raw/raw1: bound to major 8, minor 65 /dev/raw/raw2: bound to major 8, minor 49 /dev/raw/raw3: bound to major 8, minor 33 /dev/raw/raw4: bound to major 8, minor 1 /dev/raw/raw5: bound to major 8, minor 17

8. Check that the character (RAW) devices were created from all Oracle RAC nodes in the cluster:
[root@racnode1 ~]# ls -l /dev/raw total 0 crw-r----- 1 root oinstall 162, crw-r----- 1 root oinstall 162, crw-r--r-- 1 oracle oinstall 162, crw-r--r-- 1 oracle oinstall 162, crw-r--r-- 1 oracle oinstall 162,

1 2 3 4 5

Sep Sep Sep Sep Sep

24 24 24 24 24

00:48 00:48 00:48 00:48 00:48

raw1 raw2 raw3 raw4 raw5

[root@racnode2 ~]# ls -l /dev/raw total 0 crw-r----- 1 root oinstall 162, 1 Sep 24 00:48 raw1 crw-r----- 1 root oinstall 162, 2 Sep 24 00:48 raw2

crw-r--r-- 1 oracle oinstall 162, 3 Sep 24 00:48 raw3 crw-r--r-- 1 oracle oinstall 162, 4 Sep 24 00:48 raw4 crw-r--r-- 1 oracle oinstall 162, 5 Sep 24 00:48 raw5 [root@racnode1 ~]# raw -qa /dev/raw/raw1: bound to major /dev/raw/raw2: bound to major /dev/raw/raw3: bound to major /dev/raw/raw4: bound to major /dev/raw/raw5: bound to major [root@racnode2 ~]# raw -qa /dev/raw/raw1: bound to major /dev/raw/raw2: bound to major /dev/raw/raw3: bound to major /dev/raw/raw4: bound to major /dev/raw/raw5: bound to major 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, minor minor minor minor minor minor minor minor minor minor 97 17 1 49 33 65 49 33 1 17

9. Include the new shell script in /etc/rc.local to run on each boot from all

Oracle RAC nodes in the cluster:

[root@racnode1 ~]# echo "/usr/local/bin/setup_raw_devices.sh" >> /etc/rc.local [root@racnode2 ~]# echo "/usr/local/bin/setup_raw_devices.sh" >> /etc/rc.local

10. Once

the raw devices are created, use the dd command to zero out the device and make sure no data is written to the raw devices. Only perform this action from one of the Oracle RAC nodes in the cluster:

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw1 dd: writing to '/dev/raw/raw1': No space left on device 1048516+0 records in 1048515+0 records out 536839680 bytes (537 MB) copied, 773.145 seconds, 694 kB/s [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw2 dd: writing to '/dev/raw/raw2': No space left on device 1048516+0 records in 1048515+0 records out 536839680 bytes (537 MB) copied, 769.974 seconds, 697 kB/s [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw3 dd: writing to '/dev/raw/raw3': No space left on device 65505+0 records in 65504+0 records out 33538048 bytes (34 MB) copied, 47.9176 seconds, 700 kB/s [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw4 dd: writing to '/dev/raw/raw4': No space left on device 65505+0 records in 65504+0 records out 33538048 bytes (34 MB) copied, 47.9915 seconds, 699 kB/s

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw5 dd: writing to '/dev/raw/raw5': No space left on device 65505+0 records in 65504+0 records out 33538048 bytes (34 MB) copied, 48.2684 seconds, 695 kB/s

Administering the OCR File View OCR Configuration Information Two methods exist to verify how many OCR files are configured for the cluster as well as their location. If the cluster is up and running, use the ocrcheck utility as either the oracle or root user account:
[oracle@racnode1 ~]$ ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name (primary) is as follows : : 2 : 262120 : 4660 : 257460 : 1331197 : /u02/oradata/racdb/OCRFile

<-- OCR

Device/File integrity check succeeded Device/File not configured <-- OCR Mirror

(not configured) Cluster registry integrity check succeeded

If CRS is down, you can still determine the location and number of OCR files by viewing the file ocr.loc, whose location is somewhat platform dependent. For example, on the Linux platform it is located in /etc/oracle/ocr.loc while on Sun Solaris it is located at/var/opt/oracle/ocr.loc:
[root@racnode1 ~]# cat /etc/oracle/ocr.loc ocrconfig_loc=/u02/oradata/racdb/OCRFile local_only=FALSE

To view the actual contents of the OCR in a human-readable format, run the ocrdump command. This command requires the CRS stack to be running. Running the ocrdump command will dump the contents of the OCR into an ASCII text file in the current directory namedOCRDUMPFILE:
[root@racnode1 ~]# ocrdump [root@racnode1 ~]# ls -l OCRDUMPFILE -rw-r--r-- 1 root root 250304 Oct 2 22:46 OCRDUMPFILE

The ocrdump utility also allows for different output options:

# # Write OCR contents to specified file name. # [root@racnode1 ~]# ocrdump /tmp/'hostname'_ocrdump_'date +%m%d%y:%H%M' # # Print OCR contents to the screen. # [root@racnode1 ~]# ocrdump -stdout -keyname SYSTEM.css # # Write OCR contents out to XML format. # [root@racnode1 ~]# ocrdump -stdout -keyname SYSTEM.css -xml > ocrdump.xml

Add an OCR File Starting with Oracle Clusterware 10g Release 2 (10.2), users now have the ability to multiplex (mirror) the OCR. Oracle Clusterware allows for a maximum of two OCR locations; one is the primary and the second is an OCR mirror. To avoid simultaneous loss of multiple OCR files, each copy of the OCR should be placed on a shared storage device that does not share any components (controller, interconnect, and so on) with the storage devices used for the other OCR file. Before attempting to add a mirrored OCR, determine how many OCR files are currently configured for the cluster as well as their location. If the cluster is up and running, use the ocrcheck utility as either the oracle or root user account:
[oracle@racnode1 ~]$ ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name (primary) is as follows : : 2 : 262120 : 4660 : 257460 : 1331197 : /u02/oradata/racdb/OCRFile

<-- OCR

Device/File integrity check succeeded (not configured yet) Device/File not configured <-- OCR Mirror

Cluster registry integrity check succeeded

If CRS is down, you can still determine the location and number of OCR files by viewing the file ocr.loc, whose location is somewhat platform dependent. For

example, on the Linux platform it is located in /etc/oracle/ocr.loc while on Sun Solaris it is located at/var/opt/oracle/ocr.loc:
[root@racnode1 ~]# cat /etc/oracle/ocr.loc ocrconfig_loc=/u02/oradata/racdb/OCRFile local_only=FALSE

The results above indicate I have only one OCR file and that it is located on an OCFS2 file system. Since we are allowed a maximum of two OCR locations, I intend to create an OCR mirror and locate it on the same OCFS2 file system in the same directory as the primary OCR. Please note that I am doing this for the sake brevity. The OCR mirror should always be placed on a separate device than the primary OCR file to guard against a single point of failure. Note that the Oracle Clusterware stack should be online and running on all nodes in the cluster while adding, replacing, or removing the OCR location and hence does not require any system downtime.
The operations performed in this section affect the OCR for the entire cluster. However, theocrconfig command cannot modify OCR configuration information for nodes that are shut down or for nodes on which Oracle Clusterware is not running. So, you should avoid shutting down nodes while modifying the OCR using the ocrconfig command. If for any reason, any of the nodes in the cluster are shut down while modifying the OCR using the ocrconfig command, you will need to perform a repair on the stopped node before it can brought online to join the cluster. Please see the section "Repair an OCR File on a Local Node" for instructions on repairing the OCR file on the affected node.

You can add an OCR mirror after an upgrade or after completing the Oracle Clusterware installation. The Oracle Universal Installer (OUI) allows you to configure either one or two OCR locations during the installation of Oracle Clusterware. If you already mirror the OCR, then you do not need to add a new OCR location; Oracle Clusterware automatically manages two OCRs when you configure normal redundancy for the OCR. As previously mentioned, Oracle RAC environments do not support more than two OCR locations; a primary OCR and a secondary (mirrored) OCR. Run the following command to add or relocate an OCR mirror using either destination_file or disk to designate the target location of the additional OCR:
ocrconfig -replace ocrmirror <destination_file> ocrconfig -replace ocrmirror <disk>

You must be logged in as the root user to run the ocrconfig command. Please note that ocrconfig -replace is the only way to add/relocate OCR files/mirrors. Attempting to copy the existing OCR file to a new location and then manually adding/changing the file pointer in the ocr.loc file is not supported and will actually fail to work.

For example:
# # Verify CRS is running on node 1. # [root@racnode1 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy # # Verify CRS is running on node 2. # [root@racnode2 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy # # Configure the shared OCR destination_file/disk before # attempting to create the new ocrmirror on it. This example # creates a destination_file on an OCFS2 file system. # Failure to pre-configure the new destination_file/disk # before attempting to run ocrconfig will result in the # following error: # # PROT-21: Invalid parameter # [root@racnode1 ~]# cp /dev/null /u02/oradata/racdb/OCRFile_mirror [root@racnode1 ~]# chown root /u02/oradata/racdb/OCRFile_mirror [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/OCRFile_mirror [root@racnode1 ~]# chmod 640 /u02/oradata/racdb/OCRFile_mirror # # Add new OCR mirror. # [root@racnode1 ~]# ocrconfig -replace ocrmirror /u02/oradata/racdb/OCRFile_mirror

After adding the new OCR mirror, check that it can be seen from all nodes in the cluster:

# # Verify new OCR mirror from node # [root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name Device/File Name OCR Mirror

1. is as follows : : 2 : 262120 : 4668 : 257452 : 1331197 : /u02/oradata/racdb/OCRFile Device/File integrity check succeeded : /u02/oradata/racdb/OCRFile_mirror <-- New Device/File integrity check succeeded

Cluster registry integrity check succeeded [root@racnode1 ~]# cat /etc/oracle/ocr.loc #Device/file getting replaced by device /u02/oradata/racdb/OCRFile_mirror ocrconfig_loc=/u02/oradata/racdb/OCRFile ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror # # Verify new OCR mirror from node # [root@racnode2 ~]# ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name Device/File Name OCR Mirror

2. is as follows : : 2 : 262120 : 4668 : 257452 : 1331197 : /u02/oradata/racdb/OCRFile Device/File integrity check succeeded : /u02/oradata/racdb/OCRFile_mirror <-- New Device/File integrity check succeeded

Cluster registry integrity check succeeded [root@racnode2 ~]# cat /etc/oracle/ocr.loc #Device/file getting replaced by device /u02/oradata/racdb/OCRFile_mirror ocrconfig_loc=/u02/oradata/racdb/OCRFile ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror

As mentioned earlier, you can have at most two OCR files in the cluster; the primary OCR and a single OCR mirror. Attempting to add an extra mirror will actually relocate the current OCR mirror to the new location specified in the command:

[root@racnode1 ~]# cp /dev/null /u02/oradata/racdb/OCRFile_mirror2 [root@racnode1 ~]# chown root /u02/oradata/racdb/OCRFile_mirror2 [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/OCRFile_mirror2 [root@racnode1 ~]# chmod 640 /u02/oradata/racdb/OCRFile_mirror2 [root@racnode1 ~]# ocrconfig -replace ocrmirror /u02/oradata/racdb/OCRFile_mirror2 [root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 4668 Available space (kbytes) : 257452 ID : 1331197 Device/File Name : /u02/oradata/racdb/OCRFile Device/File integrity check succeeded Device/File Name : /u02/oradata/racdb/OCRFile_mirror2 <-Mirror was Relocated! Device/File integrity check succeeded Cluster registry integrity check succeeded

Relocate an OCR File Just as we were able to add a new ocrmirror while the CRS stack was online, the same holds true when relocating an OCR file or OCR mirror and therefore does not require any system downtime.
You can relocate OCR only when the OCR is mirrored. A mirror copy of the OCR file is required to move the OCR online. If there is no mirror copy of the OCR, first create the mirror using the instructions in the previous section. Attempting to relocate OCR when an OCR mirror does not exist will produce the following error:
ocrconfig -replace ocr /u02/oradata/racdb/OCRFile PROT-16: Internal Error

If the OCR mirror is not required in the cluster after relocating the OCR, it can be safely removed.

Run the following command as the root account to relocate the current OCR file to a new location using either destination_file or disk to designate the new target location for the OCR:
ocrconfig -replace ocr <destination_file> ocrconfig -replace ocr <disk>

Run the following command as the root account to relocate the current OCR mirror to a new location using either destination_file ordisk to designate the new target location for the OCR mirror:
ocrconfig -replace ocrmirror <destination_file> ocrconfig -replace ocrmirror <disk>

The following example assumes the OCR is mirrored and demonstrates how to relocate the current OCR file (/u02/oradata/racdb/OCRFile) from the OCFS2 file system to a new raw device (/dev/raw/raw1):
# # Verify CRS is running on node 1. # [root@racnode1 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy # # Verify CRS is running on node 2. # [root@racnode2 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy # # Verify current OCR configuration. # [root@racnode2 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 4668 Available space (kbytes) : 257452 ID : 1331197 Device/File Name : /u02/oradata/racdb/OCRFile <-- Current OCR to Relocate Device/File integrity check succeeded Device/File Name : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check succeeded Cluster registry integrity check succeeded # # Verify new raw storage device exists, is configured with # the correct permissions, and can be seen from all nodes # in the cluster. # [root@racnode1 ~]# ls -l /dev/raw/raw1 crw-r----- 1 root oinstall 162, 1 Oct 2 19:54 /dev/raw/raw1

[root@racnode2 ~]# ls -l /dev/raw/raw1 crw-r----- 1 root oinstall 162, 1 Oct 2 19:54 /dev/raw/raw1 # # Clear out the contents from the new raw device. # [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw1 # # Relocate primary OCR file to new raw device. Note that # there is no deletion of the old OCR file but simply a # replacement. # [root@racnode1 ~]# ocrconfig -replace ocr /dev/raw/raw1

After relocating the OCR file, check that the change can be seen from all nodes in the cluster:
# # Verify new OCR file from node 1. # [root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 4668 Available space (kbytes) : 257452 ID : 1331197 Device/File Name : /dev/raw/raw1 <-- Relocated OCR Device/File integrity check succeeded Device/File Name : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check succeeded Cluster registry integrity check succeeded [root@racnode1 ~]# cat /etc/oracle/ocr.loc #Device/file /u02/oradata/racdb/OCRFile getting replaced by device /dev/raw/raw1 ocrconfig_loc=/dev/raw/raw1 ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror # # Verify new OCR file from node 2. # [root@racnode2 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 4668 Available space (kbytes) : 257452 ID : 1331197 Device/File Name : /dev/raw/raw1 <-- Relocated OCR Device/File integrity check succeeded Device/File Name : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check succeeded

Cluster registry integrity check succeeded [root@racnode2 ~]# cat /etc/oracle/ocr.loc #Device/file /u02/oradata/racdb/OCRFile getting replaced by device /dev/raw/raw1 ocrconfig_loc=/dev/raw/raw1 ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror

After verifying the relocation was successful, remove the old OCR file at the OS level:
[root@racnode1 ~]# rm -v /u02/oradata/racdb/OCRFile removed '/u02/oradata/racdb/OCRFile'

Repair an OCR File on a Local Node It was mentioned in the previous section that the ocrconfig command cannot modify OCR configuration information for nodes that are shut down or for nodes on which Oracle Clusterware is not running. You may need to repair an OCR configuration on a particular node if your OCR configuration changes while that node is stopped. For example, you may need to repair the OCR on a node that was shut down while you were adding, replacing, or removing an OCR. To repair an OCR configuration, run the following command as root from the node on which you have stopped the Oracle Clusterware daemon:
ocrconfig repair ocr device_name

To repair an OCR mirror configuration, run the following command as root from the node on which you have stopped the Oracle Clusterware daemon:
ocrconfig repair ocrmirror device_name

You cannot perform this operation on a node on which the Oracle Clusterware daemon is running. The CRS stack must be shutdown before attempting to repair the OCR configuration on the local node.

The ocrconfig repair command changes the OCR configuration only on the node from which you run this command. For example, if the OCR mirror was relocated to a disk named /dev/raw/raw2 from racnode1 while the node racnode2 was down, then use the command ocrconfig -repair ocrmirror /dev/raw/raw2 on racnode2 while the CRS stack is down on that node to repair its OCR configuration:
# # Shutdown CRS stack on node 2 and verify the CRS stack is not up. #

[root@racnode2 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode2 ~]# ps -ef | grep d.bin | grep -v grep # # Relocate OCR mirror to new raw device from node 1. Note # that node 2 is down (actually CRS down on node 2) while # we relocate the OCR mirror. # [root@racnode1 ~]# ocrconfig -replace ocrmirror /dev/raw/raw2 # # Verify relocated OCR mirror from node 1. # [root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 4668 Available space (kbytes) : 257452 ID : 1331197 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/raw/raw2 <-- Relocated OCR Mirror Device/File integrity check succeeded Cluster registry integrity check succeeded [root@racnode1 ~]# cat /etc/oracle/ocr.loc #Device/file /u02/oradata/racdb/OCRFile_mirror getting replaced by device /dev/raw/raw2 ocrconfig_loc=/dev/raw/raw1 ocrmirrorconfig_loc=/dev/raw/raw2 # # Node 2 does not know about the OCR mirror relocation. # [root@racnode2 ~]# cat /etc/oracle/ocr.loc #Device/file /u02/oradata/racdb/OCRFile getting replaced by device /dev/raw/raw1 ocrconfig_loc=/dev/raw/raw1 ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror # # While the CRS stack is down on node 2, perform a local OCR # repair operation to inform the node of the relocated OCR # mirror. The ocrconfig -repair option will only update the # OCR configuration information on node 2. If there were # other nodes shutdown during the relocation, they too will # need repaired. # [root@racnode2 ~]# ocrconfig -repair ocrmirror /dev/raw/raw2

# # Verify the repair updated the OCR configuration on node 2. # [root@racnode2 ~]# cat /etc/oracle/ocr.loc #Device/file /u02/oradata/racdb/OCRFile_mirror getting replaced by device /dev/raw/raw2 ocrconfig_loc=/dev/raw/raw1 ocrmirrorconfig_loc=/dev/raw/raw2 # # Bring up the CRS stack on node 2. # [root@racnode2 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly # # Verify node 2 is back online. # [root@racnode2 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application ONLINE ONLINE racnode1 ora....b1.inst application ONLINE ONLINE racnode1 ora....b2.inst application ONLINE ONLINE racnode2 ora....srvc.cs application ONLINE ONLINE racnode1 ora....db1.srv application ONLINE ONLINE racnode1 ora....db2.srv application ONLINE ONLINE racnode2 ora....SM1.asm application ONLINE ONLINE racnode1 ora....E1.lsnr application ONLINE ONLINE racnode1 ora....de1.gsd application ONLINE ONLINE racnode1 ora....de1.ons application ONLINE ONLINE racnode1 ora....de1.vip application ONLINE ONLINE racnode1 ora....SM2.asm application ONLINE ONLINE racnode2 ora....E2.lsnr application ONLINE ONLINE racnode2 ora....de2.gsd application ONLINE ONLINE racnode2 ora....de2.ons application ONLINE ONLINE racnode2 ora....de2.vip application ONLINE ONLINE racnode2

Remove an OCR File To remove an OCR, you need to have at least one OCR online. You may need to perform this to reduce overhead or for other storage reasons, such as stopping a mirror to move it to SAN, RAID etc. Carry out the following steps:

Check if at least one OCR is online Verify the CRS stack is online preferably on all nodes Remove the OCR or OCR mirror If using a clustered file system, remove the deleted file at the OS level

Run the following command as the root account to delete the current OCR or the current OCR mirror:
ocrconfig -replace ocr or ocrconfig -replace ocrmirror

For example:
# # Verify CRS is running on node 1. # [root@racnode1 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy # # Verify CRS is running on node 2. # [root@racnode2 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy # # Get the existing OCR file information by running ocrcheck # utility. # [root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 4668 Available space (kbytes) : 257452 ID : 1331197 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /dev/raw/raw2 <-- OCR Mirror to be Removed Device/File integrity check succeeded Cluster registry integrity check succeeded # # Delete OCR mirror from the cluster configuration. # [root@racnode1 ~]# ocrconfig -replace ocrmirror

After removing the new OCR mirror, check that the change is seen from all nodes in the cluster:
# # Verify OCR mirror was removed from node 1.

# [root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name

is as follows : : 2 : 262120 : 4668 : 257452 : 1331197 : /dev/raw/raw1 Device/File integrity check succeeded Device/File not configured <-- OCR Mirror

Removed Cluster registry integrity check succeeded # # Verify OCR mirror was removed from node 2. # [root@racnode2 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 4668 Available space (kbytes) : 257452 ID : 1331197 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File not configured Removed Cluster registry integrity check succeeded <-- OCR Mirror

Removing the OCR or OCR mirror from the cluster configuration does not remove the physical file at the OS level when using a clustered file system.

Backup the OCR File There are two methods for backing up the contents of the OCR and each backup method can be used for different recovery purposes. This section discusses how to ensure the stability of the cluster by implementing a robust backup strategy. The first type of backup relies on automatically generated OCR file copies which are sometimes referred to as physical backups. These physical OCR file copies are automatically generated by the CRSD process on the master node and are primarily used to recover the OCR from a lost or corrupt OCR file. Your backup strategy should include procedures to copy these automatically generated OCR file copies to a secure

location which is accessible from all nodes in the cluster in the event the OCR needs to be restored. The second type of backup uses manual procedures to create OCR export files; also known as logical backups. Creating a manual OCR export file should be performed both before and after making significant configuration changes to the cluster, such as adding or deleting nodes from your environment, modifying Oracle Clusterware resources, or creating a database. If in the event a configuration change is made to the OCR that causes errors, the OCR can be restored to a previous state by performing an import of the logical backup taken before the configuration change. Please note that an OCR logical export can also be used to restore the OCR from a lost or corrupt OCR file.
Unlike the methods used to backup the voting disk, attempting to backup the OCR by copying the OCR file directly at the OS level is not a valid backup and will result in errors after the restore!

Because of the importance of OCR information, Oracle recommends that you make copies of the automatically created backup files and an OCR export at least once a day. The following is a working UNIX script that can be scheduled in CRON to backup the OCR File(s) and the Voting Disk(s) on a regular basis: crs_components_backup_10g.ksh Automatic OCR Backups The Oracle Clusterware automatically creates OCR physical backups every four hours. At any one time, Oracle always retains the last 3 backup copies of the OCR that are 4 hours old. The CRSD process that creates these backups also creates and retains an OCR backup for each full day and at the end of each week. You cannot customize the backup frequencies or the number of OCR physical backup files that Oracle retains. The default location for generating physical backups on UNIX-based systems is CRS_home/cdata/cluster_name where cluster_nameis the name of your cluster (i.e. crs). Use the ocrconfig -showbackup command to view all current OCR physical backups that were taken from the master node:
[oracle@racnode1 ~]$ ocrconfig -showbackup racnode1 2009/09/29 13:05:22 /u01/app/crs/cdata/crs

racnode1 racnode1 racnode1 racnode1

2009/09/29 09:05:22 2009/09/29 05:05:22 2009/09/28 05:05:21 2009/09/22 05:05:13

/u01/app/crs/cdata/crs /u01/app/crs/cdata/crs /u01/app/crs/cdata/crs /u01/app/crs/cdata/crs

[oracle@racnode1 ~]$ ls -l $ORA_CRS_HOME/cdata/crs total 59276 -rw-r--r-- 1 root root 8654848 Sep 29 13:05 backup00.ocr physical backup -rw-r--r-- 1 root root 8654848 Sep 29 09:05 backup01.ocr -rw-r--r-- 1 root root 8654848 Sep 29 05:05 backup02.ocr -rw-r--r-- 1 root root 8654848 Sep 29 05:05 day_.ocr -rw-r--r-- 1 root root 8654848 Sep 28 05:05 day.ocr -rw-r--r-- 1 root root 8654848 Sep 29 05:05 week_.ocr -rw-r--r-- 1 root root 8654848 Sep 22 05:05 week.ocr

<-- Most recent

<-- One day old <-- One week old

You can change the location where the CRSD process writes the physical OCR copies to using:
ocrconfig -backuploc <new_dirname>

Restoring the OCR from an automatic physical backup is accomplished using the ocrconfig -restore command. Note that the CRS stack needs to be shutdown on all nodes in the cluster prior to running the restore operation:
ocrconfig -restore <backup_file_name>

You cannot restore the OCR from a physical backup using the -import option. The only method to restore the OCR from a physical backup is to use the -restore option. The Master Node As documented in Doc ID: 357262.1 on the My Oracle Support web site, the CRSD process only creates automatic OCR physical backups on one node in the cluster, which is the OCR master node. It does not create automatic backup copies on the other nodes; only from the OCR master node. If the master node fails, the OCR backups will be created from the new master node. You can determine which node in the cluster is the master node by examining the $ORA_CRS_HOME/log/<node_name>/cssd/ocssd.log file on any node in the cluster. In this log file, check for reconfiguration information (reconfiguration successful) after which you will see which node is the master and how many nodes are active in the cluster:
Node 1 - (racnode1) [ CSSD]CLSS-3000: reconfiguration successful, incarnation 1 with 2 nodes

CSSD]CLSS-3001: local node number 1, master node number 1

Node 2 - (racnode2) [ CSSD]CLSS-3000: reconfiguration successful, incarnation 1 with 2 nodes [ CSSD]CLSS-3001: local node number 2, master node number 1

Another quick approach is to use either of the following methods:

Node 1 - (racnode1) # grep -i "master node" $ORA_CRS_HOME/log/racnode?/cssd/ocssd.log | tail -1 [ CSSD]CLSS-3001: local node number 1, master node number 1 Node 2 - (racnode2) # grep -i "master node" $ORA_CRS_HOME/log/racnode?/cssd/ocssd.log | tail -1 [ CSSD]CLSS-3001: local node number 2, master node number 1 # If not found in the ocssd.log, then look through all # of the ocssd archives: Node 1 - (racnode1) # for x in 'ls -tr $ORA_CRS_HOME/log/racnode?/cssd/ocssd.*' do grep -i "master node" $x; done | tail -1 [ CSSD]CLSS-3001: local node number 1, master node number 1 Node 2 - (racnode2) # for x in 'ls -tr $ORA_CRS_HOME/log/racnode?/cssd/ocssd.*' do grep -i "master node" $x; done | tail -1 [ CSSD]CLSS-3001: local node number 2, master node number 1 # The master node information is confirmed by the # ocrconfig -showbackup command: # ocrconfig -showbackup racnode1 racnode1 racnode1 racnode1 racnode1 2009/09/29 13:05:22 2009/09/29 09:05:22 2009/09/29 05:05:22 2009/09/28 05:05:21 2009/09/22 05:05:13 /u01/app/crs/cdata/crs /u01/app/crs/cdata/crs /u01/app/crs/cdata/crs /u01/app/crs/cdata/crs /u01/app/crs/cdata/crs

Because of the importance of OCR information, Oracle recommends that you make copies of the automatically created backup files at least once a day from the master node to a different device from where the primary OCR resides. You can use any backup software to copy the automatically generated physical backup files to a stable backup location:

[root@racnode1 ~]# cp -p -v -f -R /u01/app/crs/cdata /u02/crs_backup/ocrbackup/RACNODE1 '/u01/app/crs/cdata/crs/day_.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE1/cdata/crs/day_.ocr' '/u01/app/crs/cdata/crs/backup02.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE1/cdata/crs/backup02.ocr' '/u01/app/crs/cdata/crs/backup01.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE1/cdata/crs/backup01.ocr' '/u01/app/crs/cdata/crs/week_.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE1/cdata/crs/week_.ocr' '/u01/app/crs/cdata/crs/day.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE1/cdata/crs/day.ocr' '/u01/app/crs/cdata/crs/backup00.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE1/cdata/crs/backup00.ocr' '/u01/app/crs/cdata/crs/week.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE1/cdata/crs/week.ocr' [root@racnode2 ~]# cp -p -v -f -R /u01/app/crs/cdata /u02/crs_backup/ocrbackup/RACNODE2 '/u01/app/crs/cdata/crs/day_.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE2/cdata/crs/day_.ocr' '/u01/app/crs/cdata/crs/backup02.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE2/cdata/crs/backup02.ocr' '/u01/app/crs/cdata/crs/backup01.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE2/cdata/crs/backup01.ocr' '/u01/app/crs/cdata/crs/week_.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE2/cdata/crs/week_.ocr' '/u01/app/crs/cdata/crs/day.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE2/cdata/crs/day.ocr' '/u01/app/crs/cdata/crs/backup00.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE2/cdata/crs/backup00.ocr' '/u01/app/crs/cdata/crs/week.ocr' -> '/u02/crs_backup/ocrbackup/RACNODE2/cdata/crs/week.ocr'

It is possible and recommended that shared storage be used for the backup location(s). Keep in mind that if the master node goes down and cannot be rebooted, it is possible to loose all OCR physical backups if they were all on that node. The OCR backup process, however, will start on the new master node within four hours for all new backups. It is highly recommended that you integrate OCR backups with your normal database backup strategy. If possible, use a backup location that is shared by all nodes in the cluster. Manual OCR Exports Performing a manual export of the OCR should be done before and after making significant configuration changes to the cluster, such as adding or deleting nodes from your environment, modifying Oracle Clusterware resources, or creating a database. This type of backup is often referred to as a logical backup. If in the event a configuration change is made to the OCR that causes errors, the OCR can be restored to its previous state by performing an import of the logical backup taken before the

configuration change. For example, if you have unresolvable configuration problems, or if you are unable to restart your cluster database after such changes, then you can restore your configuration by importing the saved OCR content from a valid configuration. Please note that an OCR logical export can also be used to restore the OCR from a lost or corrupt OCR file. To export the contents of the OCR to a dump file, use the following command, where backup_file_name is the name of the OCR logical backup file you want to create:
ocrconfig -export <backup_file_name>

For example:
[root@racnode1 ~]# ocrconfig -export /u02/crs_backup/ocrbackup/RACNODE1/exports/OCRFileBackup.dmp # A second export is not strictly required, however, there is no such thing as too many backups! [root@racnode2 ~]# ocrconfig -export /u02/crs_backup/ocrbackup/RACNODE2/exports/OCRFileBackup.dmp

To restore the OCR from an export/logical backup, use the ocrconfig import command. Note that the CRS stack needs to be shutdown on all nodes in the cluster prior to running the restore operation. In addition, the total space required for the restored OCR location (typically 280MB) has to be pre-allocated. This is especially important when the OCR is located on a clustered file system like OCFS2.
ocrconfig import <export_file_name>

You cannot restore the OCR from a logical backup using the -restore option. The only method to restore the OCR from a logical export is to use the -import option.
You must be logged in as the root user to run the ocrconfig command.

Recover the OCR File

If an application fails, then before attempting to restore the OCR, restart the application. As a definitive verification that the OCR failed, run theocrcheck command:
[root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name OCR (primary) Valid Device/File Name OCR (mirror) Valid is as follows : : 2 : 262120 : 4668 : 257452 : 1331197 : /u02/oradata/racdb/OCRFile Device/File integrity check succeeded : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check succeeded

<-<--

Cluster registry integrity check succeeded

The example above indicates that both the primary OCR and OCR mirror checks were successful and that no problems exist with the OCR configuration. If the ocrcheck command does not display the message 'Device/File integrity check succeeded' for at least one copy of the OCR, then both the primary OCR and the OCR mirror have failed. In this case, the CRS stack must be brought down on all nodes in the cluster to restore the OCR from a previous physical backup copy or an OCR export. If there is at least one copy of the OCR available (either the primary OCR or the OCR mirror), you can use that valid copy to restore the contents of the other copy of the OCR. The restore in this case can be accomplished using the ocrconfig -replace command and does not require the applications or CRS stack to be down. This section describes a number of possible OCR recovery scenarios using the OCR configuration described by the output of the ocrcheck command above. Both the primary OCR and the OCR mirror are located on an OCFS2 file system in the same directory. The recovery scenarios demonstrated in this section will make use of both types of OCR backups automatically generated OCR file copies and manually created OCR export files.
Although it should go without saying, DO NOT perform these recovery scenarios on a critical system like production!

Recover OCR from Valid OCR Mirror

This section demonstrates how to restore the OCR when only one of the OCR copies is missing or corrupt. The restore process will use the good OCR copy (whether its the primary OCR or the OCR mirror) to restore the missing/corrupt copy. Remember that if there is at least one copy of the OCR available, you can use that valid copy to restore the contents of the other copy of the OCR. The best part about this type of recovery is that it doesn't require any downtime! Oracle Clusterware and the applications can remain online during the recovery process. For the purpose of this example, let's corrupt the primary OCR file:
[root@racnode1 ~]# dd if=/dev/zero of=/u02/oradata/racdb/OCRFile bs=4k count=100 100+0 records in 100+0 records out 409600 bytes (410 kB) copied, 0.00756842 seconds, 54.1 MB/s

Running ocrcheck picks up the now corrupted primary OCR file:

[root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name the other device Device/File Name is as follows : : 2 : 262120 : 4668 : 257452 : 1331197 : /u02/oradata/racdb/OCRFile <-- Corrupt OCR Device/File needs to be synchronized with : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check succeeded

Cluster registry integrity check succeeded

Note that after loosing the one OCR copy (in this case, the primary OCR file), Oracle Clusterware and the applications remain online:
[root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application ONLINE ONLINE racnode1 ora....b1.inst application ONLINE ONLINE racnode1 ora....b2.inst application ONLINE ONLINE racnode2 ora....srvc.cs application ONLINE ONLINE racnode2 ora....db1.srv application ONLINE ONLINE racnode1 ora....db2.srv application ONLINE ONLINE racnode2 ora....SM1.asm application ONLINE ONLINE racnode1 ora....E1.lsnr application ONLINE ONLINE racnode1 ora....de1.gsd application ONLINE ONLINE racnode1 ora....de1.ons application ONLINE ONLINE racnode1 ora....de1.vip application ONLINE ONLINE racnode1 ora....SM2.asm application ONLINE ONLINE racnode2

ora....E2.lsnr ora....de2.gsd ora....de2.ons ora....de2.vip

application application application application

ONLINE ONLINE ONLINE ONLINE

ONLINE ONLINE ONLINE ONLINE

racnode2 racnode2 racnode2 racnode2

While the applications and CRS remain online, perform the following steps to recover the primary OCR using the contents of the OCR mirror. 1. When using a clustered file system, remove the corrupt OCR file and re-initialize it:
2. 3. 4. 5. [root@racnode1 [root@racnode1 [root@racnode1 [root@racnode1 [root@racnode1 ~]# ~]# ~]# ~]# ~]# rm /u02/oradata/racdb/OCRFile cp /dev/null /u02/oradata/racdb/OCRFile chown root /u02/oradata/racdb/OCRFile chgrp oinstall /u02/oradata/racdb/OCRFile chmod 640 /u02/oradata/racdb/OCRFile

NOTE: If the target OCR is located on a raw device, verify the permissions are applied correctly for an OCR file (owned by root:oinstall with 0640 permissions), that the device is being shared by all nodes in the cluster, and finally use the dd command from only one node in the cluster to zero out the device and make sure no data is written to the raw device.
[root@racnode1 ~]# ls -l /dev/raw/raw1 crw-r----- 1 root oinstall 162, 1 Oct 6 11:05 /dev/raw/raw1 [root@racnode2 ~]# ls -l /dev/raw/raw1 crw-r----- 1 root oinstall 162, 1 Oct 6 11:04 /dev/raw/raw1 [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw1

6.

Restore the primary OCR using the contents of the OCR mirror. Note that this operation is the same process used when adding a new OCR location:
[root@racnode1 ~]# ocrconfig -replace ocr /u02/oradata/racdb/OCRFile

NOTE: If the target OCR is located on a raw device, substitute the path name above with that of the shared device name: (i.e. /dev/raw/raw1)

7. Verify the restore was successful by viewing the Clusterware alert log file.
8. [root@racnode1 ~]# tail $ORA_CRS_HOME/log/racnode1/alertracnode1.log 9. ... 10. 2009-10-06 17:46:51.118 [crsd(11054)]CRS-1007:The OCR/OCR mirror location was replaced by /u02/oradata/racdb/OCRFile.

11. Verify

the OCR configuration by running the ocrcheck 12. [root@racnode1 ~]# ocrcheck 13. Status of Oracle Cluster Registry is as follows : 14. Version : 2 15. Total space (kbytes) : 262120

command:

16. 17. 18. 19. 20. 21. 22. 23.

Used space (kbytes) Available space (kbytes) ID Device/File Name succeeded succeeded

<-- Primary OCR Restored Device/File Name : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check Cluster registry integrity check succeeded

: 4668 : 257452 : 1331197 : /u02/oradata/racdb/OCRFile Device/File integrity check

24. As

the oracle user account with user equivalence enabled on all the nodes, run the cluvfy command to validate the OCR configuration:
[oracle@racnode1 ~]$ ssh racnode1 "hostname; date" racnode1 Tue Oct 6 17:52:52 EDT 2009 [oracle@racnode1 ~]$ ssh racnode2 "hostname; date" racnode2 Tue Oct 6 17:51:50 EDT 2009 [oracle@racnode1 ~]$ cluvfy comp ocr -n all Verifying OCR integrity Checking OCR integrity... Checking the absence of a non-clustered configuration... All nodes free of non-clustered, local-only configurations. Uniqueness check for OCR device passed. Checking the version of OCR... OCR of correct Version "2" exists. Checking data integrity of OCR... Data integrity check for OCR passed. OCR integrity check passed. Verification of OCR integrity was successful.

25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51.

Recover OCR from Automatically Generated Physical Backup This section demonstrates how to recover the Oracle Cluster Registry from a lost or corrupt OCR file. This example assumes that both the primary OCR and the OCR mirror are lost from an accidental delete by a user and that the latest automatic OCR backup copy on the master node is accessible.

At this time, the second node in the cluster (racnode2) is the master node and currently available. We will be restoring the OCR using the latest OCR backup copy from racnode2 which is located at /u01/app/crs/cdata/crs/backup00.ocr. Let's now corrupt the OCR by removing both the primary OCR and the OCR mirror:
[root@racnode1 ~]# rm /u02/oradata/racdb/OCRFile [root@racnode1 ~]# rm /u02/oradata/racdb/OCRFile_mirror

Running ocrcheck fails to provide any useful information given that both OCR files are lost
[root@racnode1 ~]# ocrcheck PROT-602: Failed to retrieve data from the cluster registry

Note that after loosing both OCR files, Oracle Clusterware and the applications remain online. Before restoring the OCR, the applications and CRS will need to be shutdown as described in the steps below. Perform the following steps to recover the OCR from the latest automatically generated physical backup:
1.

With CRS still online, identify the master node (which in this example is racnode2) and all OCR backups using the ocrconfig -showbackup command:

2. [root@racnode1 ~]# ocrconfig -showbackup 3. 4. racnode2 2009/10/07 12:05:18 /u01/app/crs/cdata/crs 5. 6. racnode2 2009/10/07 08:05:17 /u01/app/crs/cdata/crs 7. 8. racnode2 2009/10/07 04:05:17 /u01/app/crs/cdata/crs 9. 10. racnode2 2009/10/07 00:05:16 /u01/app/crs/cdata/crs 11. racnode1 2009/09/24 08:49:19 /u01/app/crs/cdata/crs

Note that ocrconfig -showbackup may result in a segmentation fault or simply not show any results if CRS is shutdown. 12. For documentation purposes, identify the number and location of all configured OCR files that will be recovered in this example.
13. [root@racnode2 ~]# cat /etc/oracle/ocr.loc 14. #Device/file /u02/oradata/racdb/OCRFile getting replaced by device /u02/oradata/racdb/OCRFile 15. ocrconfig_loc=/u02/oradata/racdb/OCRFile

ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror

16. Although

all OCR files have been lost or corrupted, the Oracle Clusterware daemons as well as the clustered database remain running. In this scenario, Oracle Clusterware and all managed resources need to be shut down in order to recover the OCR. Attempting to stop CRS using crsctl stop crs will fail given it cannot write to the now lost/corrupt OCR file:

17. [root@racnode1 ~]# crsctl stop crs OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

With the environment in this unstable state, shutdown all database instances from all nodes in the cluster and then reboot each node:
[oracle@racnode1 ~]$ sqlplus / as sysdba SQL> shutdown immediate [root@racnode1 ~]# reboot -----------------------------------------------[oracle@racnode2 ~]$ sqlplus / as sysdba SQL> shutdown immediate [root@racnode2 ~]# reboot

18. When the Oracle RAC nodes come back up, note that Oracle Clusterware will fail to start as a result of the lost/corrupt OCR file:
19. [root@racnode1 ~]# crs_stat -t 20. CRS-0184: Cannot communicate with the CRS daemon. 21. 22. [root@racnode2 ~]# crs_stat -t CRS-0184: Cannot communicate with the CRS daemon.

23. When

using a clustered file system, re-initialize both the primary OCR and the OCR mirror target locations identified earlier in the/etc/oracle/ocr.loc file:
[root@racnode1 [root@racnode1 [root@racnode1 [root@racnode1 [root@racnode1 ~]# ~]# ~]# ~]# ~]# rm -f /u02/oradata/racdb/OCRFile cp /dev/null /u02/oradata/racdb/OCRFile chown root /u02/oradata/racdb/OCRFile chgrp oinstall /u02/oradata/racdb/OCRFile chmod 640 /u02/oradata/racdb/OCRFile

24. 25. 26. 27. 28. 29. 30. 31.

[root@racnode1 ~]# rm -f /u02/oradata/racdb/OCRFile_mirror [root@racnode1 ~]# cp /dev/null /u02/oradata/racdb/OCRFile_mirror

32. [root@racnode1 ~]# chown root /u02/oradata/racdb/OCRFile_mirror 33. [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/OCRFile_mirror [root@racnode1 ~]# chmod 640 /u02/oradata/racdb/OCRFile_mirror

NOTE: If the target OCR is located on a raw device(s), verify the permissions are applied correctly for an OCR file (owned by root:oinstall with 0640 permissions), that the device is being shared by all nodes in the cluster, and finally use the dd command from only one node in the cluster to zero out the device(s) and make sure no data is written to the raw device(s).
[root@racnode1 ~]# ls -l /dev/raw/raw[12] crw-r----- 1 root oinstall 162, 1 Oct 7 15:00 /dev/raw/raw1 crw-r----- 1 root oinstall 162, 2 Oct 7 15:00 /dev/raw/raw2 [root@racnode2 ~]# ls -l /dev/raw/raw[12] crw-r----- 1 root oinstall 162, 1 Oct 7 14:59 /dev/raw/raw1 crw-r----- 1 root oinstall 162, 2 Oct 7 14:59 /dev/raw/raw2 [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw1 <-- OCR (primary) [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw2 <-- OCR (mirror)

34. Before

restoring the OCR, dump the contents of the physical backup you intend to recover from the master node (racnode2) to validate its availability as well as the accuracy of its contents:

35. [root@racnode2 ~]# ocrdump -backupfile /u01/app/crs/cdata/crs/backup00.ocr [root@racnode2 ~]# less OCRDUMPFILE

36. With

CRS down, perform the restore operation from the master node (racnode2) by applying the latest automatically generated physical backup:
[root@racnode2 ~]# ocrconfig -restore /u01/app/crs/cdata/crs/backup00.ocr

37. Restart

Oracle Clusterware on all of the nodes in the cluster by rebooting each node or by running the crsctl start crscommand:
[root@racnode1 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly [root@racnode2 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly

38. 39. 40. 41. 42. 43.

44. Verify

the OCR configuration by running the ocrcheck 45. [root@racnode1 ~]# ocrcheck 46. Status of Oracle Cluster Registry is as follows : 47. Version : 2 48. Total space (kbytes) : 262120

command:

49. 50. 51. 52. 53. 54. 55. 56.

Used space (kbytes) Available space (kbytes) ID Device/File Name

succeeded <-- Primary OCR Restored Device/File Name : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check succeeded <-- Mirror OCR Restored Cluster registry integrity check succeeded

: 4668 : 257452 : 1331197 : /u02/oradata/racdb/OCRFile Device/File integrity check

57. As

the oracle user account with user equivalence enabled on all the nodes, run the cluvfy command to validate the OCR configuration:
[oracle@racnode1 ~]$ ssh racnode1 "hostname; date" racnode1 Wed Oct 7 16:29:49 EDT 2009 [oracle@racnode1 ~]$ ssh racnode2 "hostname; date" racnode2 Wed Oct 7 16:29:06 EDT 2009 [oracle@racnode1 ~]$ cluvfy comp ocr -n all Verifying OCR integrity Checking OCR integrity... Checking the absence of a non-clustered configuration... All nodes free of non-clustered, local-only configurations. Uniqueness check for OCR device passed. Checking the version of OCR... OCR of correct Version "2" exists. Checking data integrity of OCR... Data integrity check for OCR passed. OCR integrity check passed. Verification of OCR integrity was successful.

58. 59. 60. 61. 62. 63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74. 75. 76. 77. 78. 79. 80. 81. 82. 83. 84.

85.
86. 87. 88. 89. 90. 91. 92. 93. 94. 95.

Finally, verify the applications are running:

[root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application ONLINE ONLINE racnode1 ora....b1.inst application ONLINE ONLINE racnode1 ora....b2.inst application ONLINE ONLINE racnode2 ora....srvc.cs application ONLINE ONLINE racnode2 ora....db1.srv application ONLINE ONLINE racnode1 ora....db2.srv application ONLINE ONLINE racnode2 ora....SM1.asm application ONLINE ONLINE racnode1

Recover OCR from an OCR Export File

96. ora....E1.lsnr application 97. ora....de1.gsd application 98. ora....de1.ons application 99. ora....de1.vip application 100. ora....SM2.asm application 101. ora....E2.lsnr application 102. ora....de2.gsd application 103. ora....de2.ons application ora....de2.vip application

ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

racnode1 racnode1 racnode1 racnode1 racnode2 racnode2 racnode2 racnode2 racnode2

This section demonstrates how to restore the Oracle Cluster Registry to a valid state after an OCR configuration change causes unresolvable errors and renders the cluster as unusable. This example can also be used to recover the OCR from a lost / corrupt OCR file. It is assumed a manual OCR export was taken prior to making the OCR configuration change that caused problems with the cluster registry:
[root@racnode1 ~]# ocrconfig -export /u02/crs_backup/ocrbackup/RACNODE1/exports/OCRFileBackup.dmp

Perform the following steps to restore the previous configuration stored in the OCR from an OCR export file: 1. For documentation purposes, identify the number and location of all configured OCR files that will be recovered in this example.
2. [root@racnode2 ~]# cat /etc/oracle/ocr.loc 3. #Device/file /u02/oradata/racdb/OCRFile getting replaced by device /u02/oradata/racdb/OCRFile 4. ocrconfig_loc=/u02/oradata/racdb/OCRFile ocrmirrorconfig_loc=/u02/oradata/racdb/OCRFile_mirror

5.

Place the OCR export file that you created previously using the ocrconfig -export command on a local disk for the node that will be performing the import:

6. [root@racnode1 ~]# mkdir -p /u03/crs_backup/ocrbackup/exports 7. [root@racnode1 ~]# cd /u02/crs_backup/ocrbackup/RACNODE1/exports 8. [root@racnode1 ~]# cp -p OCRFileBackup.dmp /u03/crs_backup/ocrbackup/exports 9. [root@racnode1 ~]# ls -l /u03/crs_backup/ocrbackup/exports 10. total 112 -rw-r--r-- 1 root root 110233 Oct 8 09:38 OCRFileBackup.dmp

NOTE: The ocrconfig -import process is unable to read an OCR export file from an OCFS2 file system. Attempting to import an OCR export file that is located on an OCFS2 file system will fail with the following error:
[root@racnode1 ~]# ocrconfig -import /u02/crs_backup/ocrbackup/RACNODE1/exports/OCRFileBackup.dmp PROT-8: Failed to import data from specified file to the cluster registry

Investigating the $ORA_CRS_HOME/log/<hostname>/client/ocrconfig_pid.log will reveal the error:

... [ OCRCONF][3012240]Error[112] encountered when reading from import file ...

The solution is to copy the OCR dump file to be imported from the OCFS2 file system to a file system on the local disk. 11. As
12. 13. 14. 15. 16. 17.

the root user, stop Oracle Clusterware on all the nodes in the cluster by executing the following command:
[root@racnode1 ~]# crsctl stop crs Stopping resources. This could take several minutes. Error while stopping resources. Possible cause: CRSD is down. [root@racnode2 ~]# crsctl stop crs Stopping resources. This could take several minutes. Error while stopping resources. Possible cause: CRSD is down.

18. When

using a clustered file system, re-initialize / pre-allocate the space (typically 280MB) for both the primary OCR and the OCR mirror target locations identified earlier in the /etc/oracle/ocr.loc file:

19. [root@racnode1 ~]# rm -f /u02/oradata/racdb/OCRFile 20. [root@racnode1 ~]# dd if=/dev/zero of=/u02/oradata/racdb/OCRFile bs=4096 count=65587 21. [root@racnode1 ~]# chown root /u02/oradata/racdb/OCRFile 22. [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/OCRFile 23. [root@racnode1 ~]# chmod 640 /u02/oradata/racdb/OCRFile 24. 25. [root@racnode1 ~]# rm -f /u02/oradata/racdb/OCRFile_mirror 26. [root@racnode1 ~]# dd if=/dev/zero of=/u02/oradata/racdb/OCRFile_mirror bs=4096 count=65587 27. [root@racnode1 ~]# chown root /u02/oradata/racdb/OCRFile_mirror 28. [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/OCRFile_mirror [root@racnode1 ~]# chmod 640 /u02/oradata/racdb/OCRFile_mirror

NOTE: If the target OCR is located on a raw device(s), verify the permissions are applied correctly for an OCR file (owned by root:oinstall with 0640 permissions), that the device is being shared by all nodes in the cluster, and finally use the dd command from only one node in the cluster to zero out the device(s) and make sure no data is written to the raw device(s).
[root@racnode1 ~]# ls -l /dev/raw/raw[12] crw-r----- 1 root oinstall 162, 1 Oct 8 09:43 /dev/raw/raw1 crw-r----- 1 root oinstall 162, 2 Oct 8 09:43 /dev/raw/raw2 [root@racnode2 ~]# ls -l /dev/raw/raw[12] crw-r----- 1 root oinstall 162, 1 Oct 8 09:42 /dev/raw/raw1 crw-r----- 1 root oinstall 162, 2 Oct 8 09:42 /dev/raw/raw2 [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw1 <-- OCR (primary)

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw2 <-- OCR (mirror)

29. With

CRS down as the root user, restore the OCR data by importing the contents of the OCR export file using the following command:
[root@racnode1 ~]# ocrconfig -import /u03/crs_backup/ocrbackup/exports/OCRFileBackup.dmp

30. Restart

Oracle Clusterware on all of the nodes in the cluster by rebooting each node or by running the crsctl start crscommand:
[root@racnode1 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly [root@racnode2 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly

31. 32. 33. 34. 35. 36.

37. Verify

the OCR configuration by running the

ocrcheck

command:

38. [root@racnode1 ~]# ocrcheck 39. Status of Oracle Cluster Registry 40. Version 41. Total space (kbytes) 42. Used space (kbytes) 43. Available space (kbytes) 44. ID 45. Device/File Name 46. succeeded <-- Primary OCR Restored 47. Device/File Name 48. succeeded <-- Mirror OCR Restored 49. Cluster registry integrity

is as follows : : 2 : 262120 : 4668 : 257452 : 1331197 : /u02/oradata/racdb/OCRFile Device/File integrity check : /u02/oradata/racdb/OCRFile_mirror Device/File integrity check check succeeded

50. As

the oracle user account with user equivalence enabled on all the nodes, run the cluvfy command to validate the OCR configuration:
[oracle@racnode1 ~]$ ssh racnode1 "hostname; date" racnode1 Thu Oct 8 11:34:15 EDT 2009 [oracle@racnode1 ~]$ ssh racnode2 "hostname; date" racnode2 Thu Oct 8 11:33:33 EDT 2009 [oracle@racnode1 ~]$ cluvfy comp ocr -n all Verifying OCR integrity

51. 52. 53. 54. 55. 56. 57. 58. 59. 60. 61.

62. 63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74. 75. 76. 77.

Checking OCR integrity... Checking the absence of a non-clustered configuration... All nodes free of non-clustered, local-only configurations. Uniqueness check for OCR device passed. Checking the version of OCR... OCR of correct Version "2" exists. Checking data integrity of OCR... Data integrity check for OCR passed. OCR integrity check passed. Verification of OCR integrity was successful.

78.
79. 80. 81. 82. 83. 84. 85. 86. 87. 88. 89. 90. 91. 92. 93. 94. 95. 96.

Finally, verify the applications are running:

[root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application ONLINE ONLINE racnode1 ora....b1.inst application ONLINE ONLINE racnode1 ora....b2.inst application ONLINE ONLINE racnode2 ora....srvc.cs application ONLINE ONLINE racnode2 ora....db1.srv application ONLINE ONLINE racnode1 ora....db2.srv application ONLINE ONLINE racnode2 ora....SM1.asm application ONLINE ONLINE racnode1 ora....E1.lsnr application ONLINE ONLINE racnode1 ora....de1.gsd application ONLINE ONLINE racnode1 ora....de1.ons application ONLINE ONLINE racnode1 ora....de1.vip application ONLINE ONLINE racnode1 ora....SM2.asm application ONLINE ONLINE racnode2 ora....E2.lsnr application ONLINE ONLINE racnode2 ora....de2.gsd application ONLINE ONLINE racnode2 ora....de2.ons application ONLINE ONLINE racnode2 ora....de2.vip application ONLINE ONLINE racnode2

Administering the Voting Disk View Voting Disk Configuration Information Use the crsctl utility to verify how many voting disks are configured for the cluster as well as their location. The the crsctl command can be run as either the oracle or root user account:
[oracle@racnode1 ~]$ crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile located 1 votedisk(s).

Add a Voting Disk Adding or removing a voting disk from the cluster is a fairly straightforward process. Oracle Clusterware 10g Release 1 (10.1) only allowed for one voting disk while Oracle Oracle Clusterware 10g Release 2 (10.2) lifted this restriction to allow for 32 voting disks. Having multiple voting disks available to the cluster removes the voting disk as a single point of failure and eliminates the need to mirror them outside of Oracle Clusterware (i.e. RAID). The Oracle Universal Installer (OUI) allows you to configure either one or three voting disks during the installation of Oracle Clusterware. Having three voting disks available allows Oracle Clusterware (CRS) to continue operating uninterrupted when any one of the voting disks fail.
When deciding how many voting disks is appropriate for your environment, consider that for the cluster to survive failure of x number of voting disks, you need to configure (2x + 1) voting disks. For example, to allow for the failure of 2 voting disks, you would need to configure 5 voting disks.

When allocating shared raw storage devices for the voting disk(s), keep in mind that each voting disk requires 20MB of raw storage. OCR Corruption after Adding/Removing Voting Disk when CRS Stack is Running In addition to allowing for more than one voting disk in the cluster, the Oracle10g R2 documentation also indicates that adding and removing voting disks can be performed while CRS is online and does not require any cluster-wide downtime. After reading of this new capability, I immediately tried adding a new voting while CRS was running only to be greeted with the following error:
[root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw3 Cluster is not in a ready state for online disk addition

After some research, it appears this is a known issue on at least the Linux and Sun Solaris platform with the 10.2.0.1.0 release and is fully documented in Oracle Bug 4898020: ADDING VOTING DISK ONLINE CRASH THE CRS. Some have reported that this issue was to be fixed with the 10.2.0.4 patch set; however that is the release I am currently using and the bug still exists. In order to workaround this bug, you must first shut down CRS and then use the force flag when running the crsctlcommand. Do not attempt to add or remove a voting disk to the cluster using the -force flag while CRS is online. Oracle

Clusterware should be shut down on all nodes in the cluster before adding or removing voting disks.
Using the -force flag to add or remove a voting disk while the Oracle Clusterware stack is active on any node in the cluster may corrupt your cluster configuration. Bring down CRS on all nodes in the cluster prior to modifying the voting disk configuration using the -force flag to avoid interacting with active Oracle Clusterware daemons. If the Oracle Clusterware stack is online while attempting to use the -force flag, all nodes in the cluster will reboot due to the css shutdown and corruption of your cluster configuration is very likely. For a detailed discussion on this issue, please see Oracle Doc ID: 390880.1 "OCR Corruption after Adding/Removing voting disk to a cluster when CRS stack is running) on the My Oracle Support web site.

To add a new voting disk to the cluster, use the following command where path is the fully qualified path for the additional voting disk. Run the following command as the root user to add a voting disk:
crsctl add css votedisk <path>

You must be logged in as the root user to run the crsctl command to add/remove voting disks.

The following example demonstrates how to add two new voting disks to the current cluster. The new voting disks will reside on the same OCFS2 file system in the same directory as the current voting disk. Please note that I am doing this for the sake brevity. Multiplexed voting disks should always be placed on a separate device than the current voting disk to guard against a single point of failure. Stop all application processes, shut down CRS on all nodes, and Oracle10g R2 users should use the -force flag to the crsctlcommand when adding the new voting disk(s). For example:
# # Query current voting disk configuration. # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile located 1 votedisk(s). #

# Stop all application processes. # [root@racnode1 ~]# srvctl stop database -d racdb [root@racnode1 ~]# srvctl stop asm -n racnode1 [root@racnode1 ~]# srvctl stop asm -n racnode2 [root@racnode1 ~]# srvctl stop nodeapps -n racnode1 [root@racnode1 ~]# srvctl stop nodeapps -n racnode2 # # Verify all application processes are OFFLINE. # [root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....srvc.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....E1.lsnr application OFFLINE OFFLINE ora....de1.gsd application OFFLINE OFFLINE ora....de1.ons application OFFLINE OFFLINE ora....de1.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....E2.lsnr application OFFLINE OFFLINE ora....de2.gsd application OFFLINE OFFLINE ora....de2.ons application OFFLINE OFFLINE ora....de2.vip application OFFLINE OFFLINE # # Shut down CRS on node 1 and verify the CRS stack is not up. # [root@racnode1 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode1 ~]# ps -ef | grep d.bin | grep -v grep # # Shut down CRS on node 2 and verify the CRS stack is not up. # [root@racnode2 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode2 ~]# ps -ef | grep d.bin | grep -v grep #

# Take a backup of the current voting disk. # [oracle@racnode1 ~]$ dd if=/u02/oradata/racdb/CSSFile of=/home/oracle/VotingDiskBackup.dmp bs=4k 2500+0 records in 2500+0 records out 10240000 bytes (10 MB) copied, 0.272872 seconds, 37.5 MB/s # # Add two new voting disks. # [root@racnode1 ~]# crsctl add css votedisk /u02/oradata/racdb/CSSFile_mirror1 -force Now formatting voting disk: /u02/oradata/racdb/CSSFile_mirror1 successful addition of votedisk /u02/oradata/racdb/CSSFile_mirror1. [root@racnode1 ~]# crsctl add css votedisk /u02/oradata/racdb/CSSFile_mirror2 -force Now formatting voting disk: /u02/oradata/racdb/CSSFile_mirror2 successful addition of votedisk /u02/oradata/racdb/CSSFile_mirror2. # # Set the appropriate permissions on the new voting disks. # [root@racnode1 ~]# chown oracle /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chmod 644 /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chown oracle /u02/oradata/racdb/CSSFile_mirror2 [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/CSSFile_mirror2 [root@racnode1 ~]# chmod 644 /u02/oradata/racdb/CSSFile_mirror2

If the new voting disks will be created on raw devices # # Clear out the contents from the new raw devices. # [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw3 [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw4 # # Add two new voting disks. # [root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw3 -force Now formatting voting disk: /dev/raw/raw3 successful addition of votedisk /dev/raw/raw3. [root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw4 -force Now formatting voting disk: /dev/raw/raw4 successful addition of votedisk /dev/raw/raw4.

After adding the new voting disk(s), check that they can be seen from all nodes in the cluster:

# # Verify new voting disk access from node 1. # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile 1. 0 /u02/oradata/racdb/CSSFile_mirror1 2. 0 /u02/oradata/racdb/CSSFile_mirror2 located 3 votedisk(s). # # Verify new voting disk access from node 2. # [root@racnode2 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile 1. 0 /u02/oradata/racdb/CSSFile_mirror1 2. 0 /u02/oradata/racdb/CSSFile_mirror2 located 3 votedisk(s).

After verifying the new voting disk(s) can be seen from all nodes in the cluster, restart CRS and the application processes:
# # Restart CRS and application processes from node 1. # [root@racnode1 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly # # Restart CRS and application processes from node 2. # [root@racnode2 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly

Remove a Voting Disk As discussed in the previous section, Oracle Clusterware must be shut down on all nodes in the cluster before adding or removing voting disks. Just as we were required to add the -force flag when adding a voting disk, the same holds true for Oracle10g R2 users attempting to remove a voting disk:
[root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror1 Cluster is not in a ready state for online disk removal [root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror2 Cluster is not in a ready state for online disk removal

The CRS stack must be shutdown on all nodes in the the cluster before attempting to use the -force flag. Failure to do so may result in OCR corruption.

Use the following command as the root user to remove a voting disk where path is the fully qualified path for the voting disk to be removed:
crsctl delete css votedisk <path>

You must be logged in as the root user to run the crsctl command to add/remove voting disks.

The "crsctl delete css votedisk" command deletes an existing voting disk from the cluster. This command does not, however, remove the physical file at the OS level if using a clustered file system nor does it clear the data from a raw storage device. The following example demonstrates how to delete two voting disks from the current cluster. Stop all application processes, shut down CRS on all nodes, and Oracle10g R2 users should use the -force flag to the crsctl command when removing a voting disk(s). For example:
# # Query current voting disk configuration. # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile 1. 0 /u02/oradata/racdb/CSSFile_mirror1 2. 0 /u02/oradata/racdb/CSSFile_mirror2 located 3 votedisk(s). # # Stop all application processes. # [root@racnode1 ~]# srvctl stop database -d racdb [root@racnode1 ~]# srvctl stop asm -n racnode1 [root@racnode1 ~]# srvctl stop asm -n racnode2 [root@racnode1 ~]# srvctl stop nodeapps -n racnode1 [root@racnode1 ~]# srvctl stop nodeapps -n racnode2 # # Verify all application processes are OFFLINE. # [root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....srvc.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....E1.lsnr application OFFLINE OFFLINE ora....de1.gsd application OFFLINE OFFLINE

ora....de1.ons ora....de1.vip ora....SM2.asm ora....E2.lsnr ora....de2.gsd ora....de2.ons ora....de2.vip

application application application application application application application

OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE

OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE

# # Shut down CRS on node 1 and verify the CRS stack is not up. # [root@racnode1 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode1 ~]# ps -ef | grep d.bin | grep -v grep # # Shut down CRS on node 2 and verify the CRS stack is not up. # [root@racnode2 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode2 ~]# ps -ef | grep d.bin | grep -v grep # # Remove two voting disks. # [root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror1 -force successful deletion of votedisk /u02/oradata/racdb/CSSFile_mirror1. [root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror2 -force successful deletion of votedisk /u02/oradata/racdb/CSSFile_mirror2. # # Remove voting disk files at the OS level. # [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror2

If the voting disks are on raw devices # # Remove two voting disks. #

[root@racnode1 ~]# crsctl delete css votedisk /dev/raw/raw3 -force successful deletion of votedisk /dev/raw/raw3. [root@racnode1 ~]# crsctl delete css votedisk /dev/raw/raw4 -force successful deletion of votedisk /dev/raw/raw4. # # (Optional) # Clear out the old contents (voting disk data) from the raw devices. # [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw3 [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw4

After removing the voting disk(s), check that the voting disk(s) were removed from the cluster and the new voting disk configuration is seen from all nodes in the cluster:
# # Verify voting disk(s) deleted from node 1. # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile located 1 votedisk(s). # # Verify voting disk(s) deleted from node 2. # [root@racnode2 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile located 1 votedisk(s).

After verifying the voting disk(s) have been removed, restart CRS and the application processes on all nodes in the cluster:
# # Restart CRS and application processes from node 1. # [root@racnode1 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly # # Restart CRS and application processes from node 2. # [root@racnode2 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly

Relocate a Voting Disk The process of moving a voting disk consists simply of removing the old voting disk and adding a new voting disk to the destination location:

crsctl delete css votedisk <old_path> -force crsctl add css votedisk <new_path> -force

As discussed earlier in this section, Oracle Clusterware must be shut down on all nodes in the cluster before adding or removing voting disks. Oracle10g R2 users are required to add the -force flag when removing/adding a voting disk. The CRS stack must be shutdown on all nodes in the the cluster before attempting to use the force flag. Failure to do so may result in OCR corruption.
# # Determine the current location and number of voting disks. # If there is only one voting disk location then first add # at least one new location before attempting to move the # current voting disk. The following will show that I have # only one voting disk location and will need to add at # least one additional voting disk in order to perform the # move. After the move, this temporary voting disk can be # removed from the cluster. The remainder of this example # will provide the instructions required to move the current # voting disk from its current location on an OCFS2 file # system to a new shared raw device (/dev/raw/raw3). # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile located 1 votedisk(s). # # Stop all application processes. # [root@racnode1 ~]# srvctl stop database -d racdb [root@racnode1 ~]# srvctl stop asm -n racnode1 [root@racnode1 ~]# srvctl stop asm -n racnode2 [root@racnode1 ~]# srvctl stop nodeapps -n racnode1 [root@racnode1 ~]# srvctl stop nodeapps -n racnode2 # # Verify all application processes are OFFLINE. # [root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....srvc.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....E1.lsnr application OFFLINE OFFLINE ora....de1.gsd application OFFLINE OFFLINE ora....de1.ons application OFFLINE OFFLINE ora....de1.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE

ora....E2.lsnr ora....de2.gsd ora....de2.ons ora....de2.vip

application application application application

OFFLINE OFFLINE OFFLINE OFFLINE

OFFLINE OFFLINE OFFLINE OFFLINE

# # Shut down CRS on node 1 and verify the CRS stack is not up. # [root@racnode1 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode1 ~]# ps -ef | grep d.bin | grep -v grep # # Shut down CRS on node 2 and verify the CRS stack is not up. # [root@racnode2 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode2 ~]# ps -ef | grep d.bin | grep -v grep # # Before moving the current voting disk # (/u02/oradata/racdb/CSSFile) to a new location, we first # need to add at least one new voting disks to the cluster. # [root@racnode1 ~]# cp /dev/null /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chown oracle /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chmod 644 /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# crsctl add css votedisk /u02/oradata/racdb/CSSFile_mirror1 -force Now formatting voting disk: /u02/oradata/racdb/CSSFile_mirror1 successful addition of votedisk /u02/oradata/racdb/CSSFile_mirror1. # # Use the dd command to zero out the device and make sure # no data is written to the raw device. # [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw3 # # Delete the old voting disk (the voting disk that is to be # moved). # [root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile -force successful deletion of votedisk /u02/oradata/racdb/CSSFile.

# # Add the new voting disk to the new location. # [root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw3 -force Now formatting voting disk: /dev/raw/raw3 successful addition of votedisk /dev/raw/raw3. # # (Optional) # Remove the temporary voting disk. # [root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror1 -force successful deletion of votedisk /u02/oradata/racdb/CSSFile_mirror1. # # Remove all deleted voting disk files from the OCFS2 file system. # [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror1 # # Verify voting disk(s) relocation from node 1. # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /dev/raw/raw3 located 1 votedisk(s). # # Verify voting disk(s) relocation from node 2. # [root@racnode2 ~]# crsctl query css votedisk 0. 0 /dev/raw/raw3 located 1 votedisk(s). # # After verifying the voting disk(s) have been moved, restart # CRS and the application processes on all nodes in the # cluster. # [root@racnode1 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly [root@racnode2 ~]# crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly

Backup the Voting Disk

Backing up the voting disk(s) is often performed on a regular basis by the DBA to guard the cluster against a single point of failure as the result of hardware failure or user error. Because the node membership information does not usually change, it is not a strict requirement that you back up the voting disk every day. At a minimum, however, your backup strategy should include procedures to back up all voting disks at the following times and make certain that the backups are stored in a secure location that is accessible from all nodes in the cluster in the event the voting disk(s) need to be restored:

After installing Oracle Clusterware After adding nodes to or deleting nodes from the cluster After performing voting disk add or delete operations

Oracle Clusterware 10g Release 1 (10.1) only allowed for one voting disk while Oracle Clusterware 10g Release 2 (10.2) lifted this restriction to allow for 32 voting disks. For high availability, Oracle recommends that Oracle Clusterware 10g R2 users configure multiple voting disks while keeping in mind that you must have an odd number of voting disks, such as three, five, and so on. To avoid simultaneous loss of multiple voting disks, each voting disk should be placed on a shared storage device that does not share any components (controller, interconnect, and so on) with the storage devices used for the other voting disks. If you define a single voting disk, then you should use external mirroring to provide redundancy. To make a backup copy of the voting disk on UNIX/Linux, use the dd command:
dd if=<voting_disk_name> of=<backup_file_name> bs=<block_size>

Perform this operation on every voting disk where voting_disk_name is the name of the active voting disk (input file), backup_file_name is the name of the file to which you want to back up the voting disk contents (output file), and block_size is the value to set both the input and output block sizes. As a general rule on most platforms, including Linux and Sun, the block size for the dd command should be 4k to ensure that the backup of the voting disk gets complete blocks. If your voting disk is stored on a raw device, use the device name in place of voting_disk_name. For example:
dd if=/dev/raw/raw3 of=/u03/crs_backup/votebackup/VotingDiskBackup.dmp bs=4k

When you use the dd command to make backups of the voting disk, the backup can be performed while the Cluster Ready Services (CRS) process is active; you do not need

to stop the CRS daemons (namely, the crsd.bin process) before taking a backup of the voting disk. The following is a working UNIX script that can be scheduled in CRON to backup the OCR File and the Voting Disk on a regular basis: crs_components_backup_10g.ksh For the purpose of this example, the current Oracle Clusterware environment is configured with three voting disks on an OCFS2 clustered file system that will be backed up to a local file system on one of the nodes in the cluster. For example:
# # Query the location and number of voting disks. # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile 1. 0 /u02/oradata/racdb/CSSFile_mirror1 2. 0 /u02/oradata/racdb/CSSFile_mirror2 # # Backup all three voting disks. # [root@racnode1 ~]# dd if=/u02/oradata/racdb/CSSFile of=/u03/crs_backup/votebackup/CSSFile.bak bs=4k 2500+0 records in 2500+0 records out 10240000 bytes (10 MB) copied, 0.259862 seconds, 39.4 MB/s [root@racnode1 ~]# dd if=/u02/oradata/racdb/CSSFile_mirror1 of=/u03/crs_backup/votebackup/CSSFile_mirror1.bak bs=4k 2500+0 records in 2500+0 records out 10240000 bytes (10 MB) copied, 0.295964 seconds, 34.6 MB/s [root@racnode1 ~]# dd if=/u02/oradata/racdb/CSSFile_mirror2 of=/u03/crs_backup/votebackup/CSSFile_mirror2.bak bs=4k 2500+0 records in 2500+0 records out 10240000 bytes (10 MB) copied, 0.249039 seconds, 41.1 MB/s

Recover the Voting Disk The recommended way to recover from a lost or corrupt voting disk is to restore it from a previous good backup that was taken with the dd command. There are actually very few steps required to restore the voting disks:

1. Shutdown CRS on all nodes in the cluster. 2. List the current location of the voting disks. 3. Restore each of the voting disks using the dd command from a previous good backup of the voting disks that was taken using the same ddcommand. 4. Re-start CRS on all nodes in the cluster. For example:
[root@racnode1 ~]# crsctl stop crs [root@racnode2 ~]# crsctl stop crs [root@racnode1 ~]# crsctl query css votedisk [root@racnode1 ~]# # Do this for all voting disks... [root@racnode1 ~]# dd if=<backup_voting_disk> of=<voting_disk_name> bs=4k [root@racnode1 ~]# crsctl start crs [root@racnode2 ~]# crsctl start crs

The following is an example of what occurs on all RAC nodes when a voting disk is destroyed. This example will manually corrupt all voting disks in the cluster. After the Oracle RAC nodes reboot from the crash, we will follow up with the steps required to restore the lost/corrupt voting disk which will make use of the voting disk backups that were created in the previous section.
Although it should go without saying, DO NOT perform this recovery scenario on a critical system like production!

First, let's check the status of the cluster and all RAC components, list the current location of the voting disk(s), and finally list the voting disk backup that will be used to recover from:
[root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application ONLINE ONLINE racnode2 ora....b1.inst application ONLINE ONLINE racnode1 ora....b2.inst application ONLINE ONLINE racnode2 ora....srvc.cs application ONLINE ONLINE racnode2 ora....db1.srv application ONLINE ONLINE racnode1 ora....db2.srv application ONLINE ONLINE racnode2 ora....SM1.asm application ONLINE ONLINE racnode1 ora....E1.lsnr application ONLINE ONLINE racnode1 ora....de1.gsd application ONLINE ONLINE racnode1 ora....de1.ons application ONLINE ONLINE racnode1 ora....de1.vip application ONLINE ONLINE racnode1

ora....SM2.asm ora....E2.lsnr ora....de2.gsd ora....de2.ons ora....de2.vip

application application application application application

ONLINE ONLINE ONLINE ONLINE ONLINE

ONLINE ONLINE ONLINE ONLINE ONLINE

racnode2 racnode2 racnode2 racnode2 racnode2

[root@racnode1 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile 1. 0 /u02/oradata/racdb/CSSFile_mirror1 2. 0 /u02/oradata/racdb/CSSFile_mirror2 located 3 votedisk(s). [root@racnode1 ~]# ls -l /u03/crs_backup/votebackup total 30048 -rw-r--r-- 1 root root 10240000 Oct 8 21:24 CSSFile.bak -rw-r--r-- 1 root root 10240000 Oct 8 21:24 CSSFile_mirror1.bak -rw-r--r-- 1 root root 10240000 Oct 8 21:25 CSSFile_mirror2.bak

The next step is to simulate the corruption or loss of the voting disk(s). Oracle RAC 10g R1 / R2 (not patched with 10.2.0.4) If you are using Oracle RAC 10g R1 or Oracle RAC 10g R2 (not patched with 10.2.0.4), simply write zero's to one of the voting disk:
[root@racnode1 ~]# dd if=/dev/zero of=/u02/oradata/racdb/CSSFile Both RAC servers are now stuck and will be rebooted by CRS...

Oracle RAC 11g or higher (including Oracle RAC 10g R2 patched with 10.2.0.4) Starting with Oracle RAC 11g R1 (including Oracle RAC 10g R2 patched with 10.2.0.4), attempting to corrupt a voting disk using dd will result in all nodes being rebooted, however, Oracle Clusterware will re-construct the corrupt voting disk and successfully bring up the RAC components. Because the voting disks do not contain persistent data, CSSD is able to fully reconstruct the voting disks so long as the cluster is running. This feature was introduced with Oracle Clusterware 11.1 and is also available with Oracle Clusterware 10.2 patched with 10.2.0.4. This makes it a bit more difficult to corrupt a voting disk by simply writing zero's to it. You would need to find a way to dd the voting disks and stop the cluster before any of the voting disks could be automatically recovered by CSSD. Good luck with that! To simulate the corruption (actually the loss) of the voting disk and have both nodes

crash, I'm simply going to delete all of the voting disks and then manually reboot the nodes:
Delete the voting disk... [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror2 Reboot both nodes to simulate the crash... [root@racnode1 ~]# reboot [root@racnode2 ~]# reboot

After the reboot, CRS will not come up and all RAC components will be down:
[root@racnode1 ~]# crs_stat -t CRS-0184: Cannot communicate with the CRS daemon. [root@racnode2 ~]# crs_stat -t CRS-0184: Cannot communicate with the CRS daemon.

Ok, let's start the recovery process.

# # Locate the voting disk backups that were taken in the # previous section. # [root@racnode1 ~]# cd /u03/crs_backup/votebackup [root@racnode1 votebackup]# ls -l *.bak -rw-r--r-- 1 root root 10240000 Oct 8 21:24 CSSFile.bak -rw-r--r-- 1 root root 10240000 Oct 8 21:24 CSSFile_mirror1.bak -rw-r--r-- 1 root root 10240000 Oct 8 21:25 CSSFile_mirror2.bak # # Recover the voting disk (or voting disks) using the same # dd command that was used to back it up, but with the input # file and output file in reverse. # [root@racnode1 ~]# dd if=/u03/crs_backup/votebackup/CSSFile.bak of=/u02/oradata/racdb/CSSFile bs=4k 2500+0 records in 2500+0 records out 10240000 bytes (10 MB) copied, 0.252425 seconds, 40.6 MB/s [root@racnode1 ~]# dd if=/u03/crs_backup/votebackup/CSSFile_mirror1.bak of=/u02/oradata/racdb/CSSFile_mirror1 bs=4k 2500+0 records in 2500+0 records out 10240000 bytes (10 MB) copied, 0.217645 seconds, 47.0 MB/s [root@racnode1 ~]# dd if=/u03/crs_backup/votebackup/CSSFile_mirror2.bak of=/u02/oradata/racdb/CSSFile_mirror2 bs=4k 2500+0 records in 2500+0 records out

10240000 bytes (10 MB) copied, 0.220051 seconds, 46.5 MB/s # # Verify the permissions # set appropriately. # [root@racnode1 ~]# chown [root@racnode1 ~]# chgrp [root@racnode1 ~]# chmod on the recovered voting disk(s) are oracle /u02/oradata/racdb/CSSFile oinstall /u02/oradata/racdb/CSSFile 644 /u02/oradata/racdb/CSSFile

[root@racnode1 ~]# chown oracle /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chmod 644 /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# chown oracle /u02/oradata/racdb/CSSFile_mirror2 [root@racnode1 ~]# chgrp oinstall /u02/oradata/racdb/CSSFile_mirror2 [root@racnode1 ~]# chmod 644 /u02/oradata/racdb/CSSFile_mirror2 # # With the recovered voting disk(s) in place, restart CRS # on all Oracle RAC nodes. # [root@racnode1 ~]# crsctl start crs [root@racnode2 ~]# crsctl start crs

If you have multiple voting disks, then you can remove the voting disks and add them back into your environment using the crsctl delete css votedisk path and crsctl add css votedisk path commands respectively, where path is the complete path of the location on which the voting disk resides.

After recovering the voting disk, run through several tests to verify that Oracle Clusterware is functioning correctly:
[root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application ONLINE ONLINE racnode1 ora....b1.inst application ONLINE ONLINE racnode1 ora....b2.inst application ONLINE ONLINE racnode2 ora....srvc.cs application ONLINE ONLINE racnode2 ora....db1.srv application ONLINE ONLINE racnode1 ora....db2.srv application ONLINE ONLINE racnode2 ora....SM1.asm application ONLINE ONLINE racnode1 ora....E1.lsnr application ONLINE ONLINE racnode1 ora....de1.gsd application ONLINE ONLINE racnode1 ora....de1.ons application ONLINE ONLINE racnode1 ora....de1.vip application ONLINE ONLINE racnode1 ora....SM2.asm application ONLINE ONLINE racnode2 ora....E2.lsnr application ONLINE ONLINE racnode2 ora....de2.gsd application ONLINE ONLINE racnode2 ora....de2.ons application ONLINE ONLINE racnode2 ora....de2.vip application ONLINE ONLINE racnode2

[root@racnode1 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy

Move the Voting Disk and OCR from OCFS to RAW Devices This section provides instructions on how to move the OCR and all voting disks used throughout this article from an OCFS2 file system to raw storage devices.
OCR / Voting Disk Mappings from OCFS2 to Raw Storage OCR Component OCR File (primary) Current Location on OCFS2
/u02/oradata/racdb/OCRFile

New Location
/dev/raw/raw1

OCR File (mirror) /u02/oradata/racdb/OCRFile_mirror Vote Disk 1 Vote Disk 2 Vote Disk 3
/u02/oradata/racdb/CSSFile /u02/oradata/racdb/CSSFile_mirror1 /u02/oradata/racdb/CSSFile_mirror2

/dev/raw/raw2 /dev/raw/raw3 /dev/raw/raw4 /dev/raw/raw5

Move the OCR

# # The new raw storage devices for OCR should be owned by the # root user, must be in the oinstall group, and must have # permissions set to 640. Provide at least 280MB of disk # space for each OCR file and verify the raw storage devices # can be seen from all nodes in the cluster. # [root@racnode1 ~]# ls -l /dev/raw/raw[12] crw-r----- 1 root oinstall 162, 1 Oct 8 21:55 /dev/raw/raw1 crw-r----- 1 root oinstall 162, 2 Oct 8 21:55 /dev/raw/raw2 [root@racnode2 ~]# ls -l /dev/raw/raw[12] crw-r----- 1 root oinstall 162, 1 Oct 8 21:54 /dev/raw/raw1 crw-r----- 1 root oinstall 162, 2 Oct 8 21:54 /dev/raw/raw2 # # Use the dd command to zero out the devices and make sure # no data is written to the raw devices. # [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw1

[root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw2 # # Verify CRS is running on node 1. # [root@racnode1 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy # # Verify CRS is running on node 2. # [root@racnode2 ~]# crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy # # Query the current location and number of OCR files on # the OCFS2 file system. # [root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 4676 Available space (kbytes) : 257444 ID : 1513888898 Device/File Name : /u02/oradata/racdb/OCRFile <-- OCR (primary) Device/File integrity check succeeded Device/File Name : /u02/oradata/racdb/OCRFile_mirror <-- OCR (mirror) Device/File integrity check succeeded Cluster registry integrity check succeeded # # Move OCR and OCR mirror to new storage location. # [root@racnode1 ~]# ocrconfig -replace ocr /dev/raw/raw1 [root@racnode1 ~]# ocrconfig -replace ocrmirror /dev/raw/raw2 # # Verify OCR relocation from node # [root@racnode1 ~]# ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name 1. is as follows : : 2 : 262120 : 4676 : 257444 : 1513888898 : /dev/raw/raw1 Device/File integrity check succeeded

Device/File Name

: /dev/raw/raw2 Device/File integrity check succeeded

Cluster registry integrity check succeeded # # Verify OCR relocation from node # [root@racnode2 ~]# ocrcheck Status of Oracle Cluster Registry Version Total space (kbytes) Used space (kbytes) Available space (kbytes) ID Device/File Name Device/File Name 2. is as follows : : 2 : 262120 : 4676 : 257444 : 1513888898 : /dev/raw/raw1 Device/File integrity check succeeded : /dev/raw/raw2 Device/File integrity check succeeded

Cluster registry integrity check succeeded # # Remove all deleted OCR files from the OCFS2 file system. # [root@racnode1 ~]# rm /u02/oradata/racdb/OCRFile [root@racnode1 ~]# rm /u02/oradata/racdb/OCRFile_mirror

Move the Voting Disk

# # The new raw storage devices for the voting disks should be # owned by the oracle user, must be in the oinstall group, # and and must have permissions set to 644. Provide at least # 20MB of disk space for each voting disk and verify the raw # storage devices can be seen from all nodes in the cluster. # [root@racnode1 ~]# ls -l /dev/raw/raw[345] crw-r--r-- 1 oracle oinstall 162, 3 Oct 8 22:44 /dev/raw/raw3 crw-r--r-- 1 oracle oinstall 162, 4 Oct 8 22:45 /dev/raw/raw4 crw-r--r-- 1 oracle oinstall 162, 5 Oct 9 00:22 /dev/raw/raw5 [root@racnode2 ~]# ls -l /dev/raw/raw[345] crw-r--r-- 1 oracle oinstall 162, 3 Oct 8 22:53 /dev/raw/raw3 crw-r--r-- 1 oracle oinstall 162, 4 Oct 8 22:54 /dev/raw/raw4 crw-r--r-- 1 oracle oinstall 162, 5 Oct 9 00:23 /dev/raw/raw5 # # Use the dd command to zero out the devices and make sure # no data is written to the raw devices. # [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw3 [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw4 [root@racnode1 ~]# dd if=/dev/zero of=/dev/raw/raw5 # # Query the current location and number of voting disks on

# the OCFS2 file system. There needs to be at least two # voting disks configured before attempting to perform the # move. # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /u02/oradata/racdb/CSSFile 1. 0 /u02/oradata/racdb/CSSFile_mirror1 2. 0 /u02/oradata/racdb/CSSFile_mirror2 located 3 votedisk(s). # # Stop all application processes. # [root@racnode1 ~]# srvctl stop database -d racdb [root@racnode1 ~]# srvctl stop asm -n racnode1 [root@racnode1 ~]# srvctl stop asm -n racnode2 [root@racnode1 ~]# srvctl stop nodeapps -n racnode1 [root@racnode1 ~]# srvctl stop nodeapps -n racnode2 # # Verify all application processes are OFFLINE. # [root@racnode1 ~]# crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora....srvc.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....E1.lsnr application OFFLINE OFFLINE ora....de1.gsd application OFFLINE OFFLINE ora....de1.ons application OFFLINE OFFLINE ora....de1.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....E2.lsnr application OFFLINE OFFLINE ora....de2.gsd application OFFLINE OFFLINE ora....de2.ons application OFFLINE OFFLINE ora....de2.vip application OFFLINE OFFLINE # # Shut down CRS on node 1 and verify the CRS stack is not up. # [root@racnode1 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode1 ~]# ps -ef | grep d.bin | grep -v grep #

# Shut down CRS on node 2 and verify the CRS stack is not up. # [root@racnode2 ~]# crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@racnode2 ~]# ps -ef | grep d.bin | grep -v grep # # Move all three voting disks to new storage location. # [root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile -force successful deletion of votedisk /u02/oradata/racdb/CSSFile. [root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw3 -force Now formatting voting disk: /dev/raw/raw3 successful addition of votedisk /dev/raw/raw3. [root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror1 -force successful deletion of votedisk /u02/oradata/racdb/CSSFile_mirror1. [root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw4 -force Now formatting voting disk: /dev/raw/raw4 successful addition of votedisk /dev/raw/raw4. [root@racnode1 ~]# crsctl delete css votedisk /u02/oradata/racdb/CSSFile_mirror2 -force successful deletion of votedisk /u02/oradata/racdb/CSSFile_mirror2. [root@racnode1 ~]# crsctl add css votedisk /dev/raw/raw5 -force Now formatting voting disk: /dev/raw/raw5 successful addition of votedisk /dev/raw/raw5. # # Verify voting disk(s) relocation from node 1. # [root@racnode1 ~]# crsctl query css votedisk 0. 0 /dev/raw/raw3 1. 0 /dev/raw/raw4 2. 0 /dev/raw/raw5 located 3 votedisk(s). # # Verify voting disk(s) relocation from node 2. # [root@racnode2 ~]# crsctl query css votedisk 0. 0 /dev/raw/raw3 1. 0 /dev/raw/raw4 2. 0 /dev/raw/raw5 located 3 votedisk(s).

# # Remove all deleted voting disk files from the OCFS2 file system. # [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror1 [root@racnode1 ~]# rm /u02/oradata/racdb/CSSFile_mirror2 # # With all voting disks now located on raw storage devices, # restart CRS on all Oracle RAC nodes. # [root@racnode1 ~]# crsctl start crs [root@racnode2 ~]# crsctl start crs

About the Author Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX, Linux, and Windows server environment. Jeff's other interests include mathematical encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 18 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science.

Enabling Archive Logs in a RAC Environment

by Jeff Hunter, Sr. Database Administrator

Whether a single instance or clustered database, Oracle tracks (logs) all changes to database blocks in online redolog files. In an Oracle RAC environment, each instance will have its own set of online redolog files known as a thread. Each Oracle instance will use its set (group) of online redologs in a circular manner. Once an online redolog fills, Oracle moves to the next one. If the database is in "Archive Log Mode", Oracle will make a copy of the online redo log before it gets reused. A thread must contain at least two online redologs (or online redolog groups). The same holds true for a single instance configuration. The single instance must contain at least two online redologs (or online redolog groups). The size of an online redolog file is completely independent of another intances' redolog size. Although in most configurations the size is the same, it may be different depending on the workload and backup /

recovery considerations for each node. It is also worth mentioning that each instance has exclusive write access to its own online redolog files. In a correctly configured RAC environment, however, each instance can read another instance's current online redolog file to perform instance recovery if that instance was terminated abnormally. It is therefore a requirement that online redo logs be located on a shared storage device (just like the database files). As already mentioned, Oracle writes to its online redolog files in a circular manner. When the current online redolog fills, Oracle will switch to the next one. To facilitate media recovery, Oracle allows the DBA to put the database into "Archive Log Mode" which makes a copy of the online redolog after it fills (and before it gets reused). This is a process known as archiving. The Database Creation Assistant (DBCA) allows users to configure a new database to be in archive log mode, however most DBA's opt to bypass this option. In cases like this where the database is in no archive log mode, it is a simple task to put the database into archive log mode. Note however that this will require a short database outage. From one of the nodes in the Oracle RAC 10g configuration, use the following tasks to put a RAC enabled database into archive log mode. For the purpose of this article, I will use the node racnode1 which runs the racdb1 instance:

1. Login to one of the nodes (i.e. racnode1) and disable the cluster instance parameter by
setting cluster_database to FALSE from the current instance:

$ sqlplus "/ as sysdba" SQL> alter system set cluster_database=false scope=spfile sid='racdb1';

2. Shutdown all instances accessing the clustered database:

$ srvctl stop database -d racdb

3. Using the local instance, MOUNT the database:

$ sqlplus "/ as sysdba" SQL> startup mount 4. Enable archiving: SQL> alter database archivelog;

5. Re-enable support for clustering by modifying the instance

parameter cluster_database to TRUE from the current instance: SQL> alter system set cluster_database=true scope=spfile sid='racdb1'; 6. Shutdown the local instance: SQL> shutdown immediate

7. Bring all instance back up using srvctl:

$ srvctl start database -d racdb

8. (Optional) Bring any services (i.e. TAF) back up using srvctl:

$ srvctl start service -d racdb 9. Login to the local instance and verify Archive Log Mode is enabled:

$ sqlplus "/ as sysdba" SQL> archive log list Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence

Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 83 84 84

After enabling Archive Log Mode, each instance in the RAC configuration can automatically archive redologs!

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX, Linux, and Windows server environment. Jeff's other interests include mathematical encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 18 years and maintains his own website site at:http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science

Controlling the Oracle Clusterware Services (CRS) 10g in UNIX / Linux

by Jeff Hunter, Sr. Database Administrator Contents
1. 2. 3.

Overview Oracle10g R2 Oracle10g R1

Overview This article was motivated by Metalink Doc ID: 309542.1. In this article, I will explain how to manage (start, stop, enable, disable) the Oracle10g CRS / Clusterware services on the Unix / Linux platform. It applies to both Oracle10g R1 (10.1.0) and Oracle10g R2 (10.2.0). Whenever a node managed by Oracle10g CRS / Clusterware comes up, the Clusterware processes (CRS stack) start up automatically. It is possible, however, to manually control these services (i.e. start / stop) using scripts installed by Oracle. In Oracle10g R1, users can invoke the init.crs script. Starting with Oracle10g R2, users can also use the more powerful and user-friendly script named crsctl.

Oracle10g R2 Starting with Oracle10g R2, you can use the $ORA_CRS_HOME/bin/crsctl script to control CRS / Clusterware services. Run the crsctl script with no arguments to display usage parameters. Stopping CRS Stack
crsctl stop crs

This is often used when applying patches or for planned outages. Disabling CRS Stack
crsctl disable crs

This disables CRS / Clusterware services from being started in subsequent reboots. It will not stop the currently running CRS / Clusterware - it only ensures that the services do not come up after a reboot. Enabling CRS Stack
crsctl enable crs

Use this to enable CRS / Clusterware to be started in subsequent reboots. Starting CRS Stack
crsctl start crs

Used to start CRS / Clusterware services.

Oracle10g R1 In Oracle10g R1 (and in R2) you can use the CRS init script init.crs to control CRS services. Run the init.crs script with no arguments to display usage parameters. The location of the init.crs script is operating system dependant:

Solaris: the scripts are in /etc/init.d Linux: the scripts are in /etc/init.d HP-UX: the scripts are in /sbin/init.d AIX: the scripts are in /etc

Stopping CRS Stack

init.crs stop

This is often used when applying patches or for planned outages. Disabling CRS Stack
init.crs disable

This disables CRS / Clusterware services from being started in subsequent reboots. It will not stop the currently running CRS / Clusterware - it only ensures that the services do not come up after a reboot. Enabling CRS Stack
init.crs enable

Use this to enable CRS / Clusterware to be started in a subsequent reboots.

Starting CRS Stack

init.crs start

Using this method to start CRS / Clusterware services is only supported starting from 10.1.0.4. In previous versions (10.1.0.3 and 10.1.0.2), the node would need to be rebooted in order to start the CRS / Clusterware services. The only exception to this is when applying patchsets. The patchset will run scripts to sucessfully start the CRS / Clusterware services.
Notice that init.crs is not the only script or component of CRS / Clusterware. For example:
[root@linux1 init.d]# ls -l /etc/init.d/init.* -r-xr-xr-x 1 root root 1951 Jul 18 12:49 /etc/init.d/init.crs -r-xr-xr-x 1 root root 4714 Jul 18 12:49 /etc/init.d/init.crsd -r-xr-xr-x 1 root root 35394 Jul 18 12:49 /etc/init.d/init.cssd -r-xr-xr-x 1 root root 3190 Jul 18 12:49 /etc/init.d/init.evmd

Never attempt to manually use the individual components of CRS / Clusterware likeinit.crsd, init.cssd, and init.evmd.

Automatically Restarting ONS - (Oracle10g R1)

by Jeff Hunter, Sr. Database Administrator Contents
1. 2.

Overview Solution

Overview For Oracle Database 10g release 1 (both 10.1.0.2 and 10.1.0.3) installations, the Oracle Notification Service (ONS) AUTO_START parameter is set to 0 on each node of the cluster. This bug seems to exist for all UNIX platforms (Solaris, Linux, etc.) For this reason, CRS does not automatically start the

ONS component when the node is restarted. This issue is documented and being tracked with Oracle bug 4011834. You can verify this using the following command in UNIX:
$ $ORA_CRS_HOME/bin/crs_stat ... NAME=ora.vmlinux1.ons TYPE=application TARGET=OFFLINE STATE=OFFLINE ... NAME=ora.vmlinux2.ons TYPE=application TARGET=OFFLINE STATE=OFFLINE

Solution To work around this issue, perform the following steps as the CRS owner (the user account that installed CRS and in UNIX is usually the oracle user account) for each ONS resource. For the purpose of this article, I will also be giving the commands I ran on one of the nodes in my cluster, vmlinux1:
1.

Change directory to the

cd $ORA_CRS_HOME/crs/public

crs_home/crs/public

directory.

2. Use the following command to create a text file containing the profile resources:
crs_home/bin/crs_stat -p ora.nodename.ons > ora.nodename.ons.cap

For example,
$ORA_CRS_HOME/bin/crs_stat -p ora.vmlinux1.ons > ora.vmlinux1.ons.cap

3.

Enter the following command to change the AUTO_START parameter value to 1 in the

ora.nodename.ons.cap

file:

crs_home/bin/crs_profile -update ora.nodename.ons -o as=1

For example,
$ORA_CRS_HOME/bin/crs_profile -update ora.vmlinux1.ons -o as=1

4. Enter the following command to update the resource profile:

crs_home/bin/crs_register -u ora.nodename.ons

For example,
$ORA_CRS_HOME/bin/crs_register -u ora.vmlinux1.ons

Remove a Node from an Existing Oracle RAC 10g R2 Cluster on Linux - (RHEL 5.3)
by Jeff Hunter, Sr. Database Administrator Contents
1. 2. 3. 4. 5. 6. 7. 8. 9.

Overview Remove the Oracle Instance on the Node to be Deleted ASM Instance Clean-Up Procedures Remove the Listener From the Node to be Deleted Remove the Oracle Database Software From the Node to be Deleted Remove the Node to be Deleted from Oracle Clusterware OCFS2 Clean-Up Procedures Remove Access Permissions on Openfiler for the Node to be Deleted About the Author

Overview Although not as exciting as building an Oracle RAC or adding a new node and instance to a clustered database; removing a node from a clustered environment is just as important to understand for a DBA managing Oracle RAC 10g. While it is true that most of the attention in a clustered database environment is focused on extending the

database tier to support increased demand, the exact opposite is just as likely to be encountered where the DBA needs to remove a node from an existing RAC environment. It may be that a server is being underutilized in the database cluster and could be better served in another business unit. Another scenario is a node failure. In this case, a node can be removed from the cluster while the remaining nodes continue to service ongoing requests. This document is an extension to two of my articles: "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 5.3 / iSCSI)" and "Add a Node to an Existing Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 5.3 / iSCSI)". Contained in this new article are the steps required to remove a single node from an already running and configured three-node Oracle RAC 10g Release 2 environment on the CentOS 5 platform. The node being removed is the third node I added in the second article. Although this article was written and tested on CentOS 5.3, it should work unchanged with Red Hat Enterprise Linux 5 Update 3. When removing the Oracle Cluster and Oracle Database software, you must first remove the Oracle Database software and then remove the Oracle Clusterware from the node that you are deleting. In other words, you remove the software components from the node you are deleting in the reverse order that you originally installed the clusterware and Oracle database software components. It is important that you perform each step contained this article in the order provided. This article assumes the following:
The reader has already built and configured a three-node Oracle RAC 10g Release 2 environment using the articles "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 5.3 / iSCSI)" and "Add a Node to an Existing Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 5.3 / iSCSI)". These two articles provide comprehensive instructions for building a two-node RAC cluster and then adding a third node to the cluster named linux3. All Oracle RAC nodes are single processor machines running CentOS 5.3 x86, Oracle RAC 10g Release 2 for Linux x86, OCFS2, and ASMLib 2.0. The current three-node RAC environment actually consists of four machines three named linux1, linux2, and linux3 which each run an Oracle10g instance and a forth node to run the network storage server named openfiler1.

Note: The current three-node Oracle RAC environment has been upgraded from its base release (10.2.0.1.0) to Oracle RAC 10g Release 2 (10.2.0.4) Patch Set 3 for Linux x86 by applying the 6810189 patchset (p6810189_10204_Linux-x86.zip). The patchset was applied to Oracle Clusterware and the Oracle Database software. The procedures for installing patchsets are not included in any of the parent article(s).

The third node in the current Oracle RAC environment named linux3 (running the racdb3 instance) will be removed from the cluster making it a two-node cluster. Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and Oracle Database software installed on their local disks. The current three-node Oracle RAC environment does not use shared Oracle homes for the Clusterware or Database software. The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. Instructions for removing the OCFS2 software from the deleted node against a "live" OCFS2 file system will be included. Automatic Storage Management (ASM) is being used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area. In addition to removing the ASM instance from the deleted node, we will also be removing ASMLib from this node. During the creation of the existing two-node cluster and the addition of the third node, the installation of Oracle Clusterware and the Oracle Database software were only performed from one node in the RAC cluster namely from linux1 as the oracle user account. The Oracle Universal Installer (OUI) on that particular node would then use the ssh and scp commands to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. The oracle user account on the node running the OUI (runInstaller) had to be trusted by all other nodes in the RAC cluster. This meant that theoracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI (linux1) against all other Linux servers in the cluster without being prompted for a password. The same security requirements hold true for this article. User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securely removed from the node to be deleted (linux3). This task will need to be performed from a node that is to remain a part of the RAC cluster database environment (linux1 in this example) using ssh and scp without being prompted for a password. All shared disk storage for the existing Oracle RAC is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.3 (Final) x86_64. While removing node linux3 from the Oracle RAC cluster, we will also be removing access permissions to the Openfiler server from linux3.

The following is a conceptual look at what the environment will look like after removing the third Oracle RAC node (linux3) from the cluster. Click on the graphic below to enlarge the image:

Figure 1: Removing linux3 from the current Oracle RAC 10g Release 2 Environment

While this article provides comprehensive instructions for successfully removing a node fro existing Oracle RAC 10g system, it is by no means a substitute for the official Oracle documentation. In addition to this article, users should also consult the following Oracle documents to gain a full understanding of alternative configuration options, installation, an administration with Oracle RAC 10g. Oracle's official documentation site is docs.oracle.com

Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) Linux Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 1 Release 2 (10.2) 2 Day + Real Application Clusters Guide - 10g Release 2 (10.2)

Remove the Oracle Instance on the Node to be Deleted

Remove the Oracle instance running on the node to be deleted using DBCA!

Overview Before removing any of the Oracle software from the node to be deleted (linux3), the first step is to remove the Oracle instance (racdb3) from the RAC database. This task can be performed using either Enterprise Manager or the Database Configuration Assistant (DBCA). For the purpose of this article, I am opting to use the DBCA. The DBCA will need to be run from a node other than the node that hosts the instance you want to delete. The suggested node to run the DBCA from is the same node used to install the original Oracle RAC environment from which for this example is linux1. Verifying Terminal Shell Environment Before starting the DBCA, you should first verify you are logged onto the server you will be running the DBCA from (i.e. linux1) then run the xhost command as root from the console to allow X Server connections. Next, login as the oracle user account. If you are using a remote client to connect to the node performing the installation (SSH / Telnet to linux1 from a workstation configured with an X Server), you will need to set the DISPLAY variable to point to your local workstation. Finally, verify remote access / user equivalence to all nodes in the cluster: Verify Server and Enable X Server Access
[root@linux1 ~]# hostname

linux1 [root@linux1 ~]# xhost + access control disabled, clients can connect from any host

Login as the oracle User Account and Set DISPLAY (if necessary)
[root@linux1 ~]# su - oracle [oracle@linux1 [oracle@linux1 [oracle@linux1 [oracle@linux1 ~]$ ~]$ ~]$ ~]$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the DBCA from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method,user equivalence will need to be enabled on any new terminal shell session before attempting to run the DBCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
[oracle@linux1 ~]$ exec /usr/bin/ssh-agent $SHELL [oracle@linux1 ~]$ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) [oracle@linux1 ~]$ ssh linux1 "date;hostname" Fri Sep 4 10:23:48 EDT 2009 linux1 [oracle@linux1 ~]$ ssh linux2 "date;hostname" Fri Sep 4 10:24:13 EDT 2009 linux2 [oracle@linux1 ~]$ ssh linux3 "date;hostname" Fri Sep 4 10:24:13 EDT 2009 linux3

Remove Database Instance To start the database instance removal process, run the following from linux1:
[oracle@linux1 ~]$ dbca &

Screen Name Welcome Screen Operations Instance Management

Response Select Oracle Real Application Clusters database. Select Instance Management. Select Delete an instance. Provides a list of clustered databases running on the node. For the purpose of this example, the clustered database running on node linux1 is racdb. Select this clustered database.

List of cluster databases At the bottom of this screen, the DBCA requests you to "Specify a user with SYSDBA system privileges": Username: sys Password: <sys_password> Click Next to continue. List of cluster database instances This screen provides a list of all instances currently available on the cluster, their status, and which node they reside on. Select the node:instance to delete (linux3:racdb3) and Click Next to continue. If the current clustered database has any database services defined, the next screen allows the DBA to re-configure those database services to account for the instance being removed. In this example, the existing clustered database has one service defined named racdb_srvc. With the "racdb_srvc" database service selected, change the details to Not Used for the instance to be removed (racdb3) and leave the "TAF Policy" set to Basic. Click Finish to remove the instance. Database Configuration Assistant: Summary After verifying the instance removal options in the summary dialog, Click OK to begin the instance management process. A second dialog box is presented which includes the warning: Database Configuration Assistant "The Database Configuration Assistance will delete the Oracle instance and its associated OFA directory structure. All information about this instance will be deleted. Do you want to proceed?" Click OK to acknowledge this dialog. Database Configuration Assistant Progress Screen A progress bar is display while the instance is being deleted. Once the instance management process is complete, the DBCA prompts the user with a dialog and the message "Do you want to perform

Database Services

another operation?" Click No to end and exit the DBCA utility.

When the Oracle Database Configuration Assistant has completed, the racdb3 instance (and associated services) will have been successfully removed from linux3! Verify Database Instance and Associated Objects Were Removed Check Cluster Services
[oracle@linux1 ~]$ $ORA_CRS_HOME/bin/crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application ONLINE ONLINE linux1 ora....b1.inst application ONLINE ONLINE linux1 ora....b2.inst application ONLINE ONLINE linux2 ora....srvc.cs application ONLINE ONLINE linux1 ora....db1.srv application ONLINE ONLINE linux1 ora....db2.srv application ONLINE ONLINE linux2 ora....SM1.asm application ONLINE ONLINE linux1 ora....E1.lsnr application ONLINE ONLINE linux1 ora....de1.gsd application ONLINE ONLINE linux1 ora....de1.ons application ONLINE ONLINE linux1 ora....de1.vip application ONLINE ONLINE linux1 ora....SM2.asm application ONLINE ONLINE linux2 ora....E2.lsnr application ONLINE ONLINE linux2 ora....de2.gsd application ONLINE ONLINE linux2 ora....de2.ons application ONLINE ONLINE linux2 ora....de2.vip application ONLINE ONLINE linux2 ora....SM3.asm application ONLINE ONLINE linux3 ora....E3.lsnr application ONLINE ONLINE linux3 ora....de3.gsd application ONLINE ONLINE linux3 ora....de3.ons application ONLINE ONLINE linux3 ora....de3.vip application ONLINE ONLINE linux3 - or [oracle@linux1 ~]$ rac_crs_stat HA Resource ----------ora.racdb.db ora.racdb.racdb1.inst ora.racdb.racdb2.inst ora.racdb.racdb_srvc.cs ora.racdb.racdb_srvc.racdb1.srv ora.racdb.racdb_srvc.racdb2.srv ora.linux1.ASM1.asm ora.linux1.LISTENER_linux1.lsnr ora.linux1.gsd ora.linux1.ons ora.linux1.vip Target -----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE State ----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on on on on on on

linux1 linux1 linux2 linux1 linux1 linux2 linux1 linux1 linux1 linux1 linux1

ora.linux2.ASM2.asm ora.linux2.LISTENER_linux2.lsnr ora.linux2.gsd ora.linux2.ons ora.linux2.vip ora.linux3.ASM3.asm ora.linux3.LISTENER_linux3.lsnr ora.linux3.gsd ora.linux3.ons ora.linux3.vip

ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on on on on on

linux2 linux2 linux2 linux2 linux2 linux3 linux3 linux3 linux3 linux3

Verify Instance we Removed Login to one of the instances and query the
gv$instance

view:

SQL> select inst_id, instance_name, status, to_char(startup_time, 'DD-MONYYYY HH24:MI:SS') 2 from gv$instance order by inst_id; INST_ID ---------1 2 INSTANCE_NAME ---------------racdb1 racdb2 STATUS -----------OPEN OPEN TO_CHAR(STARTUP_TIME -------------------04-SEP-2009 00:01:58 04-SEP-2009 00:02:07

Verify Database Files / Tablespaces have been Removed Check that the UNDO tablespace that was defined for has been removed:
SQL> select tablespace_name from dba_tablespaces order by 1; TABLESPACE_NAME -----------------EXAMPLE SOE SOEINDEX SYSAUX SYSTEM TEMP UNDOTBS1 UNDOTBS2 USERS 9 rows selected. racdb3 (UNDOTBS3)

Check that the online redo log files that were defined for been removed:

racdb3

have

SQL> select inst_id, group#, member from gv$logfile order by 1,2,3; INST_ID GROUP# MEMBER

------- ------ ---------------------------------------------------------1 1 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_1.260.696523713 1 1 +RACDB_DATA1/racdb/onlinelog/group_1.268.696523711 1 2 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_2.259.696523721 1 2 +RACDB_DATA1/racdb/onlinelog/group_2.262.696523717 1 3 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_3.258.696526639 1 3 +RACDB_DATA1/racdb/onlinelog/group_3.264.696526635 1 4 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_4.257.696526645 1 4 +RACDB_DATA1/racdb/onlinelog/group_4.263.696526643 2 1 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_1.260.696523713 2 1 +RACDB_DATA1/racdb/onlinelog/group_1.268.696523711 2 2 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_2.259.696523721 2 2 +RACDB_DATA1/racdb/onlinelog/group_2.262.696523717 2 3 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_3.258.696526639 2 3 +RACDB_DATA1/racdb/onlinelog/group_3.264.696526635 2 4 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_4.257.696526645 2 4 +RACDB_DATA1/racdb/onlinelog/group_4.263.696526643 16 rows selected.

Update TNSNAMES Login to all machines that were accessing the instance removed and update the tnsnames.ora file (if necessary).

ASM Instance Clean-Up Procedures

Remove ASM instance and ASMLib from the node to be deleted!

Overview When the clustered database is running on ASM (as is the case in this example), the steps in this section describe how to remove the ASM instance (and ASMLib) from the node to be deleted. If the Oracle RAC database is not running on ASM, this section can be skipped.

The tasks in this section will need to be run from a node other than the node that hosts the ASM instance you want to delete. The suggested node to run these steps from is the same node used to install the original Oracle RAC environment from which for this example is linux1. Remove the ASM Instance from the Oracle Cluster Registry File Run the following from
linux1

as the

oracle

user account:

[oracle@linux1 ~]$ srvctl stop asm -n linux3 [oracle@linux1 ~]$ srvctl remove asm -n linux3

Remove ASM Directories from the Node to be Deleted Remove all ASM directories from the node to be deleted (linux3) as the oracle user account:
[oracle@linux3 ~]$ cd $ORACLE_BASE/admin [oracle@linux3 ~]$ rm -rf +ASM [oracle@linux3 ~]$ cd $ORACLE_HOME/dbs [oracle@linux3 ~]$ rm -f *ASM*

You may also have to edit the /etc/oratab file on the node to be deleted and remove entries beginning with +ASM to delete the references of the ASM instance.

Remove ASMLib from the Node to be Deleted The Linux nodes in the current Oracle RAC database make use of ASMLib so that the raw block devices managed by ASM use ASMLib calls as opposed to standard Linux I/O system calls. In this section we will remove the ASMLib kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account:
[root@linux3 ~]# /etc/init.d/oracleasm stop

Dropping Oracle ASMLib disks: [ OK ] Shutting down the Oracle ASMLib driver: [ [root@linux3 ~]# rpm -qa | grep oracleasm oracleasm-2.6.18-128.el5-2.0.5-1.el5 oracleasmlib-2.0.4-1.el5 oracleasm-support-2.1.3-1.el5

OK

[root@linux3 ~]# rpm -ev oracleasm-2.6.18-128.el5-2.0.5-1.el5 oracleasmlib2.0.4-1.el5 oracleasm-support-2.1.3-1.el5 warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 ~]# ~]# ~]# ~]# ~]# ~]# ~]# ~]# ~]# ~]# rm rm rm rm rm rm rm rm rm rm -f -f -f -f -f -f -f -f -f -f /etc/sysconfig/oracleasm.rpmsave /etc/sysconfig/oracleasm-_dev_oracleasm /etc/rc.d/init.d/oracleasm /etc/rc0.d/*oracleasm* /etc/rc1.d/*oracleasm* /etc/rc2.d/*oracleasm* /etc/rc3.d/*oracleasm* /etc/rc4.d/*oracleasm* /etc/rc5.d/*oracleasm* /etc/rc6.d/*oracleasm*

Remove the Listener From the Node to be Deleted

Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Network Configuration Assistant (NETCA) will remove the node-specific listener from the clustered configuration for the node to be deleted.

Overview Each of the nodes in the current Oracle RAC environment use a node-specific listener named LISTENER_nodename which runs out of the local Oracle home. In this section, you will use the Network Configuration Assistant (NETCA) to delete the TNS listener for the node to be deleted from the existing clustered configuration. The NETCA program will be run from linux1 with user equivalence enabled to all

nodes in the cluster. Although it is possible to re-create this listener in another Oracle home (on a different node), we will simply be removing it from the cluster. Verifying Terminal Shell Environment As was performed earlier, the terminal shell environment needs to be configured for remote access and user equivalence to all Oracle RAC nodes before running the NETCA. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable: Login as the oracle User Account and Set DISPLAY (if necessary)
[root@linux1 ~]# su - oracle [oracle@linux1 [oracle@linux1 [oracle@linux1 [oracle@linux1 ~]$ ~]$ ~]$ ~]$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the NETCA from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the NETCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
[oracle@linux1 ~]$ exec /usr/bin/ssh-agent $SHELL [oracle@linux1 ~]$ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) [oracle@linux1 ~]$ ssh linux1 "date;hostname" Fri Sep 4 10:23:48 EDT 2009 linux1

[oracle@linux1 ~]$ ssh linux2 "date;hostname" Fri Sep 4 10:24:13 EDT 2009 linux2 [oracle@linux1 ~]$ ssh linux3 "date;hostname" Fri Sep 4 10:24:13 EDT 2009 linux3

Run the Network Configuration Assistant To start the NETCA, run the following from
[oracle@linux1 ~]$ netca & linux1:

The following table walks you through the process of reconfiguring the clustered TNS listeners to remove the node-specific listener from the node to be deleted.
Screen Name Select the Type of Oracle Net Services Configuration Select the nodes to configure Type of Configuration Response

Select Cluster configuration

Only select the Oracle RAC node to remove the listener from: linux3.

Select Listener configuration. The following screens are now like any other normal listener configuration when removing a listener: What do you want to do: Delete Select a listener you want to delete: LISTENER Acknowledge warning dialog: Yes Listener deleted [ Next ] Listener configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen. Click Finish to exit the NETCA.

Listener Configuration Next 4 Screens

Type of Configuration

Verify TNS Listener Configuration

The Oracle node-specific TNS listener processes should now only be running on two nodes in the RAC cluster (linux1 and linux2):
[oracle@linux1 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX1 [oracle@linux1 ~]$ $ORA_CRS_HOME/bin/crs_stat ora.linux1.LISTENER_LINUX1.lsnr NAME=ora.linux1.LISTENER_LINUX1.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux1 ===================== [oracle@linux2 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX2 [oracle@linux2 ~]$ $ORA_CRS_HOME/bin/crs_stat ora.linux2.LISTENER_LINUX2.lsnr NAME=ora.linux2.LISTENER_LINUX2.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux2 ===================== [oracle@linux3 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' [oracle@linux3 ~]$ $ORA_CRS_HOME/bin/crs_stat ora.linux3.LISTENER_LINUX3.lsnr CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.

Remove the Oracle Database Software From the Node to be Deleted

Remove the Oracle Database software and update the Node List for the node to be deleted!

Overview In this section, you will be removing the node to be deleted (linux3) from the Node List for its Oracle inventory. Next will be to remove

the Oracle Database software from the node to be deleted using the Oracle Universal Installer (OUI). Finally, update the Oracle inventory on all remaining Oracle RAC nodes to reflect the nodes which will remain in the cluster. Update Node List for Oracle Database Software - (Remove linux3) From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Database software:
[oracle@linux3 ~]$ DISPLAY=<your local workstation>:0.0; export DISPLAY [oracle@linux3 ~]$ cd $ORACLE_HOME/oui/bin [oracle@linux3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES="" -local Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.

Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!

De-install Oracle Database Software Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Database software. Make certain that you choose the home to be removed and not just the products under that home. From linux3 as the oracle user account, run the following:
[oracle@linux3 ~]$ cd $ORACLE_HOME/oui/bin [oracle@linux3 bin]$ ./runInstaller Screen Name Welcome Screen Inventory: Response Click the Installed Products button. Check the Oracle home to deleted (OraDb10g_home1) and click

Contents Tab Confirmation

the Remove button. Acknowledge the warning dialog by clicking Yes. A progress bar is displayed while the Oracle Database software is being removed. Once this process has completed, you are returned to the "Inventory: Contents Tab" dialog. After confirming the Oracle Database software (Oracle home) was successfully removed, Click Close to exit this dialog. Click Cancel to exit the OUI.

Deinstallation Process

Welcome Screen

Update Node List for Remaining Nodes in the Cluster Finally, from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster:
[oracle@linux1 ~]$ DISPLAY=<your local workstation>:0.0; export DISPLAY [oracle@linux1 ~]$ cd $ORACLE_HOME/oui/bin [oracle@linux1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={linux1,linux2}" Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.

Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!

Remove the Node to be Deleted from Oracle Clusterware

Remove the node to be deleted from Oracle Clusterware and update the Node List for all remaining nodes!

Overview After successfully removing the Oracle Database software from the node to be deleted, the next major step is to remove the node to be deleted from Oracle Clusterware. Remove Node-Specific Interface Configuration Run the following commands to remove node-specific interface configurations from the node to be deleted. For this example, run these commands from linux3 as the oracle user account:
[oracle@linux3 ~]$ $ORA_CRS_HOME/bin/racgons remove_config linux3:6200

In the above command, replace 6200 with the port number you get in the remoteport section of the ons.config file found in the $ORA_CRS_HOME/opmn/conf directory:
[oracle@linux3 ~]$ grep '^remoteport' $ORA_CRS_HOME/opmn/conf/ons.config remoteport=6200 [oracle@linux3 ~]$ $ORA_CRS_HOME/bin/oifcfg delif -node linux3 PROC-4: The cluster registry key to be operated on does not exist. PRIF-11: cluster registry error

From the research that I have performed, I believe the above error can be safely ignored. The "oifcfg delif" command attempts to delete a node-specific interface defined for node linux3 in the Oracle Cluster Registry file (OCR). This means it is looking for the presence of a key in the OCR file named "SYSTEM.css.interfaces.nodelinux3". If you were to dump the contents of the OCR file, you would quickly see that no key is present with that name:
[oracle@linux3 ~]$ $ORA_CRS_HOME/bin/ocrdump [oracle@linux3 ~]$ grep 'SYSTEM.css.interfaces.nodelinux3' OCRDUMPFILE

After examining the contents of the OCRDUMPFILE, interfaces are indeed defined, however they are identified as global:
[oracle@linux3 ~]$ grep 'SYSTEM.css.interfaces.global' OCRDUMPFILE [SYSTEM.css.interfaces.global] [SYSTEM.css.interfaces.global.eth0] [SYSTEM.css.interfaces.global.eth0.192|d168|d1|d0] [SYSTEM.css.interfaces.global.eth0.192|d168|d1|d0.1] [SYSTEM.css.interfaces.global.eth2] [SYSTEM.css.interfaces.global.eth2.192|d168|d3|d0]

[SYSTEM.css.interfaces.global.eth2.192|d168|d3|d0.1]

Given there are no node-specific interface entries defined and we obviously don't want to delete the interface keys configured globally, the oifcfg delif command and its errors can be safely ignored.

Disable Oracle Clusterware Applications From the node you are deleting from the cluster (linux3), run the script $ORA_CRS_HOME/install/rootdelete.sh to disable the Oracle Clusterware applications that are on the node. This script should only be run once. Given the Clusterware software install is on local disk (non-shared), make certain to use the nosharedhome argument. The default for this script is sharedhome which prevents you from updating the permissions of local files such that they can be removed by the oracle user account. Running this script will stop the CRS stack and delete the ocr.loc file on the node to be removed. The nosharedvar option assumes the ocr.loc file is not on a shared file system. While logged into linux3 as the root user account, run the following:
[oracle@linux3 ~]$ su [root@linux3 oracle]# cd $ORA_CRS_HOME/install [root@linux3 oracle]# ./rootdelete.sh local nosharedvar nosharedhome CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'. Shutting down Oracle Cluster Ready Services (CRS): Sep 04 11:57:30.100 | INF | daemon shutting down Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. Shutdown has begun. The daemons should exit soon. Checking to see if Oracle CRS stack is down... Oracle CRS stack is not running. Oracle CRS stack is down now. Removing script for Oracle Cluster Ready services Updating ocr file for downgrade Cleaning up SCR settings in '/etc/oracle/scls_scr'

Delete Node from Cluster and Update OCR

Upon successful completion of the rootdelete.sh script, run the rootdeletenode.sh script to delete the node (linux3) from the Oracle cluster and to update the Oracle Cluster Registry (OCR). This script should be run from a pre-existing / available node in the cluster (linux1) as the root user account: Before executing rootdeletenode.sh, we need to know the node number associated with the node name to be deleted from the cluster. To determine the node number, run the following command as the oracle user account from linux1:
[oracle@linux1 ~]$ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2 linux3 3

From the listing above, the node number for linux3 is 3. While logged into linux1 as the root user account, run the following using the name linux3 and the node number 3:
[oracle@linux1 ~]$ su [root@linux1 oracle]# cd $ORA_CRS_HOME/install [root@linux1 install]# ./rootdeletenode.sh linux3,3 CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'. CRS-0210: Could not find resource 'ora.linux3.ons'. CRS-0210: Could not find resource 'ora.linux3.vip'. CRS-0210: Could not find resource 'ora.linux3.gsd'. CRS-0210: Could not find resource ora.linux3.vip. CRS nodeapps are deleted successfully clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully deleted 14 values from OCR. Key SYSTEM.css.interfaces.nodelinux3 marked for deletion is not there. Ignoring. Successfully deleted 5 keys from OCR. Node deletion operation successful. 'linux3,3' deleted successfully

To verify that the node was successfully removed, use the following as either the oracle or root user:
[oracle@linux1 ~]$ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2

Update Node List for Oracle Clusterware Software - (Remove linux3)

From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Clusterware software:
[oracle@linux3 ~]$ DISPLAY=<your local workstation>:0.0; export DISPLAY [oracle@linux3 ~]$ cd $ORA_CRS_HOME/oui/bin [oracle@linux3 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES="" -local CRS=true Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.

Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!

De-install Oracle Clusterware Software Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Clusterware software. Make certain that you choose the home to be removed and not just the products under that home. From linux3 as the oracle user account, run the following:
[oracle@linux3 ~]$ cd $ORA_CRS_HOME/oui/bin [oracle@linux3 bin]$ ./runInstaller Screen Name Welcome Screen Inventory: Contents Tab Confirmation Deinstallation Process Response Click the Installed Products button. Check the Oracle home to deleted (OraCrs10g_home) and click the Remove button. Acknowledge the warning dialog by clicking Yes to remove the Oracle Clusterware software and to remove the /u01/app/crs directory. A progress bar is displayed while the Oracle Clusterware software is being removed. Once this process has completed, you are returned to the "Inventory: Contents Tab" dialog. After confirming the Oracle Clusterware software (Clusterware home) was successfully removed,Click Close to

exit this dialog. Welcome Screen Click Cancel to exit the OUI.

Update Node List for Remaining Nodes in the Cluster Finally, from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster:
[oracle@linux1 ~]$ DISPLAY=<your local workstation>:0.0; export DISPLAY [oracle@linux1 ~]$ cd $ORA_CRS_HOME/oui/bin [oracle@linux1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME "CLUSTER_NODES={linux1,linux2}" CRS=true Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.

Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!

Verify Node to be Deleted is no Longer a Member of the Cluster Run the following commands to verify that the node to be deleted from the Oracle RAC cluster is no longer a member of the cluster and to verify that the Oracle Clusterware components have been successfully removed from that node. Run the following commands from linux1 as the oracle user account:
[oracle@linux1 ~]$ srvctl status nodeapps -n linux3 PRKC-1056 : Failed to get the hostname for node linux3 PRKH-1001 : HASContext Internal Error [OCR Error(Native: getHostName:[21])]

The error above indicates that cluster.

linux3

is no longer a member of the

[oracle@linux1 ~]$ $ORA_CRS_HOME/bin/crs_stat | grep -i linux3

You should not see any output from the above command
[oracle@linux1 ~]$ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2

You should see the present node list without the deleted node (that is linux1 and linux2 only). Remove/Rename any Remaining Oracle Files from Node to be Deleted From the node to be deleted (linux3), remove/rename any remaining Oracle files while logged in as the root user account:
[root@linux3 ~]# mv -f /etc/inittab.no_crs /etc/inittab [root@linux3 ~]# rm -f /etc/inittab.orig [root@linux3 ~]# rm -f /etc/inittab.crs [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 ~]# ~]# ~]# ~]# ~]# ~]# ~]# ~]# ~]# rm rm rm rm rm rm rm rm rm -rf /etc/oracle -f /etc/oratab -f /etc/oraInst.loc -rf /etc/ORCLcluster -rf /u01/app/oracle -rf /u01/app/crs -f /usr/local/bin/coraenv -f /usr/local/bin/dbhome -f /usr/local/bin/oraenv

Finally, remove the Oracle user account and all associated UNIX groups from linux3:
[root@linux3 [root@linux3 [root@linux3 [root@linux3 ~]# ~]# ~]# ~]# userdel -r oracle groupdel oinstall groupdel dba groupdel oper

OCFS2 Clean-Up Procedures

Remove OCFS2 from the node to be deleted!

Overview The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. It is also used to store the shared SPFILE for ASM. This section includes instructions for removing the OCFS2 software from the deleted node against a "live" OCFS2 file system. Remove OCFS2 from the Node to be Deleted In this section we will remove the OCFS2 kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account:
[root@linux3 ~]# umount /u02 [root@linux3 ~]# /etc/init.d/o2cb offline ocfs2 [root@linux3 ~]# /etc/init.d/o2cb unload [root@linux3 ~]# rpm -qa | grep ocfs2 ocfs2-2.6.18-128.el5-1.4.2-1.el5 ocfs2console-1.4.2-1.el5 ocfs2-tools-1.4.2-1.el5 [root@linux3 ~]# rpm -ev ocfs2-2.6.18-128.el5-1.4.2-1.el5 ocfs2console-1.4.21.el5 ocfs2-tools-1.4.2-1.el5 warning: /etc/sysconfig/o2cb saved as /etc/sysconfig/o2cb.rpmsave [root@linux3 ~]# rm -f /etc/sysconfig/o2cb.rpmsave [root@linux3 ~]# rm -rf /etc/ocfs2

Remove Node to be Deleted from the OCFS2 Cluster The next step is to remove the Oracle RAC node being deleted (linux3) from the "live" OCFS2 cluster. This entails running theo2cb_ctl command-line utility from the remaining two RAC nodes linux1 and linux2. As root, run the following from linux1 and then linux2:
[root@linux1 ~]# o2cb_ctl -D -u -n linux3 o2cb_ctl: Not yet supported

[root@linux2 ~]# o2cb_ctl -D -u -n linux3 o2cb_ctl: Not yet supported

o2cb_ctl parameters:
-D -u : Delete an object from the existing OCFS2 Cluster Configuration. : Valid only with -D. When deleting something (node or cluster), it will also remove it from the live cluster (/config). If the parameter is not specified, then only update the /etc/ocfs2/cluster.conf. : Object name which is usually the node name or cluster name.

-n

At the time of this writing, the latest version of OCFS2 (namely the o2cb cluster stack)only supports "adding" nodes dynamically. Deleting a node requires the OCFS2 cluster to be down which means a database outage. This is a serious shortcoming with OCFS2 that hasn't been addressed for some time. Currently, an enhancement request (oss.oracle.com Bugzilla Bug 694) has been submitted to Oracle, however no target date or version has been set for this enhancement.

Removing linux3 from the OCFS2 cluster will have to be performed manually during the next scheduled database outage. After bringing the clustered database and all Oracle Clusterware services down on both linux1 and linux2, remove the following stanza from the /etc/ocfs2/cluster.conf file on the remaining Oracle RAC nodes (linux1 and linux2):
node: ip_port = 7777 ip_address = 192.168.2.107 number = 2 name = linux3 cluster = ocfs2 Before closing out the /etc/ocfs2/cluster.conf file, change the value

ofnode_count to 2 on the remaining Oracle RAC nodes (linux1 and linux2):

node_count = 2

After modifying the /etc/ocfs2/cluster.conf file on the remaining Oracle RAC nodes (linux1 and linux2), un-mount the OCFS2 file system and restart the o2cb cluster stack:
# umount /u02 # /etc/init.d/o2cb offline ocfs2 # /etc/init.d/o2cb unload # /etc/init.d/o2cb load # /etc/init.d/o2cb online ocfs2

# mount /u02

After mounting the OCFS2 file system on the remaining Oracle RAC nodes, verify thatlinux3 does not exist in the live OCFS2 cluster only linux1 and linux2 should be present:
[root@linux1 ~]# o2cb_ctl -I -n linux1 node: name = linux1 cluster = ocfs2 number = 0 ip_address = 192.168.2.100 ip_port = 7777 status = configured [root@linux1 ~]# o2cb_ctl -I -n linux2 node: name = linux2 cluster = ocfs2 number = 1 ip_address = 192.168.2.101 ip_port = 7777 status = configured [root@linux1 ~]# o2cb_ctl -I -n linux3

Remove OCFS2 Mount Options from /etc/fstab Finally, remove the following entry from the
LABEL=oracrsfiles /u02 ocfs2 /etc/fstab

from
0 0

linux3:

_netdev,datavolume,nointr

Remove Access Permissions on Openfiler for the Node to be Deleted

Remove network access permissions in Openfiler for linux3 and disable auto-discovery / auto-logon of iSCSI volumes from linux3!

Overview

At this point, linux3 has been successfully removed from the Oracle RAC configuration. Remember that in the existing Oracle RAC configuration, all shared disk storage is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.3 (Final) x86_64. When we added linux3 to the current Oracle RAC configuration in the article "Add a Node to an Existing Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 5.3 / iSCSI)", we needed to allow network access in Openfiler for linux3. Next, we granted permission to the shared iSCSI volumes for linux3 by updating the network ACL for each of the iSCSI targets. Finally, we configured the iSCSI initiator service on linux3 to automatically discover and login to all available iSCSI volumes on each boot. In this final section of the article, we will be disabling automatic discovery and automatic login to iSCSI volumes from Openfiler as well as revoking network access and permissions to the iSCSI targets for the node linux3. Disable iSCSI Initiator Service on linux3 Modify the iSCSI initiator service on linux3 so it will not automatically start and therefore will not attempt to discover iSCSI volumes from the Openfiler server. Manually Logout of iSCSI Target(s)
[root@linux3 ~]# iscsiadm 192.168.2.195 --logout [root@linux3 ~]# iscsiadm 192.168.2.195 --logout [root@linux3 ~]# iscsiadm 192.168.2.195 --logout [root@linux3 ~]# iscsiadm 192.168.2.195 --logout [root@linux3 ~]# iscsiadm 192.168.2.195 --logout -m node -T iqn.2006-01.com.openfiler:racdb.asm4 -p -m node -T iqn.2006-01.com.openfiler:racdb.asm3 -p -m node -T iqn.2006-01.com.openfiler:racdb.asm2 -p -m node -T iqn.2006-01.com.openfiler:racdb.asm1 -p -m node -T iqn.2006-01.com.openfiler:racdb.crs -p

Verify we are logged out of the iSCSI target by looking at the /dev/disk/bypath directory. If no other iSCSI targets exist on the client node, then after logging out from the iSCSI target, the mappings for all targets should be gone and the following command should not find any files or directories:

[root@linux3 ~]# (cd /dev/disk/by-path; ls -l *openfiler* | awk '{FS=" "; print $9 " " $10 " " $11}') ls: *openfiler*: No such file or directory

Delete Target and Disable Automatic Login Update the record entry on the client node to disable automatic logins to the iSCSI targets:
[root@linux3 ~]# iscsiadm 192.168.2.195 --op update [root@linux3 ~]# iscsiadm 192.168.2.195 --op update [root@linux3 ~]# iscsiadm 192.168.2.195 --op update [root@linux3 ~]# iscsiadm 192.168.2.195 --op update [root@linux3 ~]# iscsiadm 192.168.2.195 --op update -m -n -m -n -m -n -m -n -m -n node -T iqn.2006-01.com.openfiler:racdb.asm4 -p node.startup -v manual node -T iqn.2006-01.com.openfiler:racdb.asm3 -p node.startup -v manual node -T iqn.2006-01.com.openfiler:racdb.asm2 -p node.startup -v manual node -T iqn.2006-01.com.openfiler:racdb.asm1 -p node.startup -v manual node -T iqn.2006-01.com.openfiler:racdb.crs -p node.startup -v manual

Delete the iSCSI target:

[root@linux3 ~]# iscsiadm -m 01.com.openfiler:racdb.asm4 [root@linux3 ~]# iscsiadm -m 01.com.openfiler:racdb.asm3 [root@linux3 ~]# iscsiadm -m 01.com.openfiler:racdb.asm2 [root@linux3 ~]# iscsiadm -m 01.com.openfiler:racdb.asm1 [root@linux3 ~]# iscsiadm -m 01.com.openfiler:racdb.crs node --op delete --targetname iqn.2006node --op delete --targetname iqn.2006node --op delete --targetname iqn.2006node --op delete --targetname iqn.2006node --op delete --targetname iqn.2006-

Remove udev Rules Files If the iSCSI targets being removed is the only remaining target and you don't plan on adding any further iSCSI targets in the future, then it is safe to remove the iSCSI rules file and its call-out script:
[root@linux3 ~]# rm /etc/udev/rules.d/55-openiscsi.rules [root@linux3 ~]# rm /etc/udev/scripts/iscsidev.sh

Disable the iSCSI (Initiator) Service If the iSCSI target being removed is the only remaining target and you don't plan on adding any further iSCSI targets in the future, then it is safe to disable the iSCSI Initiator Service:
[root@linux3 ~]# service iscsid stop [root@linux3 ~]# chkconfig iscsid off

[root@linux3 ~]# chkconfig iscsi off

Revoke Network Access and Permissions from linux3 for each iSCSI Volume Revoking Openfiler network access and permissions from linux3 can be done using the Openfiler Storage Control Center a browser based tool over an https connection on port 446. For example:
https://openfiler1.idevelopment.info:446/

From the Openfiler Storage Control Center home page, login as an administrator. The default administration login credentials for Openfiler are:

Username: openfiler Password: password

The first page the administrator sees is the [Status] / [System Information] screen. To start the process of removing access to the current iSCSI logical volumes from linux3, navigate to [Volumes] / [iSCSI Targets]. Then click on the grey sub-tab named "Target Configuration". On this page is a section named "Select iSCSI Target" where users can select which of the iSCSI targets to display and/or edit:

Figure 2: Select iSCSI Target Select the first iSCSI target (which in this example is iqn.200601.com.openfiler:racdb.crs) and click the [Change] button. Next, click the grey subtab named "Network ACL" (next to "LUN Mapping" sub-tab). For the current iSCSI target, change the "Access" forlinux3 from 'Allow' to 'Deny' and click the 'Update' button:

Figure 3: Update Network ACL After updating the Network ACL for the first iSCSI Target, click on the grey sub-tab named "Target Configuration" and select the next iSCSI target in the "Select iSCSI Target" section. Click the [Change] button and continue to update the Network ACL for this iSCSI target (changing the "Access" for linux3 from 'Allow' to 'Deny' under the "Network ACL" grey sub-tab). Continue this process until access has been revoked from linux3 to all five iSCSI targets. Remove linux3 from Openfiler Local Network Configuration The final step is to revoke network access to Openfiler from linux3. This task can be completed using the Openfiler Storage Control Center by navigating to [System] / [Network Setup]. The Local Networks screen allows an administrator to setup (add/remove) networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we want to remove linux3 from the Openfiler local network configuration. Simply, check the linux3-priv / 192.168.2.107 row and click the [Update] button at the bottom of the screen to remove linux3from the Openfiler local network configuration.

The following image shows the step required to remove linux3 and update the local network configuration within Openfiler:

Figure 4: Configure Openfiler Network Access for deleted Oracle RAC Node

About the Author Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX, Linux, and Windows server environment. Jeff's other interests include mathematical encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 18 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science.

DBA Tips Archive for Oracle

Add a Node to an Existing Oracle RAC 10g R2 Cluster on Linux - (RHEL 5.3)
by Jeff Hunter, Sr. Database Administrator Contents Overview Hardware and Costs Install the Linux Operating System Install Required Linux Packages for Oracle RAC Network Configuration Configure Network Security on the Openfiler Storage Server 7. Configure the iSCSI Initiator 8. Create "oracle" User and Directories 9. Configure the Linux Server for Oracle 10. Configure the "hangcheck-timer" Kernel Module 11. Configure RAC Nodes for Remote Access using SSH 12. All Startup Commands for New Oracle RAC Node 13. Install and Configure Oracle Cluster File System (OCFS2) 14. Install and Configure Automatic Storage Management (ASMLib 2.0) 15. Pre-Installation Tasks for Oracle10g Release 2 16. Extend Oracle Clusterware Software to the New Node 17. Extend Oracle Database Software to the New Node 18. Add Listener to New Node 19. Add Database Instance to the New Node 20. About the Author
1. 2. 3. 4. 5. 6.

Overview As your organization grows, so too does your need for more application and database resources to support the company's IT systems. Oracle RAC 10g provides a scalable framework which allows DBA's to effortlessly extend the database tier to support this increased demand. As the number of users and transactions increase, additional Oracle instances can be added to the Oracle database cluster to distribute the extra load. This document is an extension to my article "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 5.3 / iSCSI)". Contained in this new article are the steps required to add a single node to an already running and configured two-node Oracle RAC 10g Release 2 environment on the CentOS 5 (x86) platform. Although this article was written and tested on CentOS 5.3 Linux, it should work unchanged with Red Hat Enterprise Linux 5 Update 3. This article assumes the following:
The reader has already built and configured a two-node Oracle RAC 10g Release 2 environment using the article "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 5.3 / iSCSI)". The article provides comprehensive instructions for building a two-node RAC cluster, each with a single processor running CentOS 5.3 (x86), Oracle RAC 10g Release 2 for Linux x86, OCFS2, and ASMLib 2.0. The current two-node RAC environment actually consists of three machines two named linux1 and linux2which each run an Oracle10g instance and a third node to run the network storage server namedopenfiler1.

Note: The current two-node Oracle RAC environment has been upgraded from its base release (10.2.0.1.0) to Oracle RAC 10g Release 2 (10.2.0.4) Patch Set 3 for Linux x86 by applying the 6810189 patchset (p6810189_10204_Linux-x86.zip). The patchset was applied to Oracle Clusterware and the Oracle Database software. The procedures for installing patch sets are not included in any of the parent article(s).
To maintain the current naming convention, the new Oracle RAC node to be added to the existing cluster will be named linux3 (running a new instance named racdb3) making it a three-node cluster. The new Oracle RAC node should have the same operating system version and installed patches as the current two-node cluster. Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and

Oracle Database software installed on their local disks. The current two-node Oracle RAC environment does not use shared Oracle homes for the Clusterware or Database software. The software owner for the Oracle Clusterware and Oracle Database installs will be "oracle". It is important that the UID and GID of the oracle user account on the new node be identical to that of the existing RAC nodes. For the purpose of this example, the oracle user account will be defined as follows:
[oracle@linux1 ~]$ id oracle uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper)

The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. Instructions for installing and adding the new Oracle RAC node to the "live" OCFS2 file system will be included. Automatic Storage Management (ASM) is being used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area. In addition to ASM, we will also be configuring ASMLib on the new Oracle RAC node. To add instances to an existing RAC database, Oracle Corporation recommends using the Oracle cloning procedures which is described in the Oracle Universal Installer and OPatch User's Guide. This article, however, uses manual procedures to add nodes and instances to the existing Oracle RAC cluster. The manual procedures method described in this article involve extending the RAC database by first extending the Oracle Clusterware home to the new Oracle RAC node and then extending the Oracle Database home. In other words, you extend the software onto the new node in the same order as you installed the clusterware and Oracle database software components on the existing two-node RAC. During the creation of the existing two-node cluster, the installation of Oracle Clusterware and the Oracle Database software were only performed from one node in the RAC cluster namely from linux1 as theoracle user account. The Oracle Universal Installer (OUI) on that particular node would then use the sshand scp commands to run remote commands on and copy the Oracle software to all other nodes within the RAC cluster. The oracle user account on the node running the OUI (runInstaller) had to be trusted by all other nodes in the RAC cluster. This meant that the oracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI (linux1) against all other Linux servers in the cluster without being prompted for a password. The same security requirements hold true for this article.User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securely copied from linux1 to the new Oracle RAC node (linux3)

using ssh and scp without being prompted for a password. All shared disk storage for the existing Oracle RAC is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.3 (Final) x86_64.

Powered by rPath Linux, Openfiler is a free browser-based network storage management utility that delivers file-based Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single framework. The entire software stack interfaces with open source applications such as Apache, Samba, LVM2, ext3, Linux NFS and iSCSI Enterprise Target. Openfiler combines these ubiquitous technologies into a small, easy to manage solution fronted by a powerful web-based management interface.
These articles provide a low cost alternative for those who want to become familiar with Oracle RAC 10gusing commercial off the shelf components and downloadable software. Bear in mind that these articles are provided for educational purposes only so the setup is kept simple to demonstrate ideas and concepts. For example, the disk mirroring configured in this article will be setup on one physical disk only, while in practice that should be performed on multiple physical drives. In addition, each Linux node will only be configured with two network cards one for the public network (eth0) and one for the private cluster interconnect "and" network storage server for shared iSCSI access (eth1). For a production RAC implementation, the private interconnect should be at least Gigabit (or more) and "only" be used by Oracle to transfer Cluster Manager and Cache Fusion related data. A third dedicated network interface (i.e. eth2) should be configured on another Gigabit network for access to the network storage server (Openfiler).

The following is a conceptual look at what the environment will look like after adding the third Oracle RAC node (linux3) to the cluster. Click on the graphic below to enlarge the image:

Figure 1: Adding linux3 to the current Oracle RAC 10g Release 2 Environment

While this article provides comprehensive instructions for successfully adding a node to an existing Oracle RAC 10gsystem, it is by no means a substitute for the official Oracle documentation. In addition to this article, users should also consult the following Oracle documents to gain a full understanding of alternative configuration options, installation, an administration with Oracle RAC 10g. Oracle's official documentation site is docs.oracle.com

Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) Linux Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 1 Release 2 (10.2) 2 Day + Real Application Clusters Guide - 10g Release 2 (10.2)

Hardware and Costs The hardware used in this article to build the third node (linux3) consists of a Linux workstation and components which can be purchased at many local computer stores or over the Internet (i.e. Stallard Technologies, Inc.).
Oracle RAC Node 3 - (linux3)
Dell Dimension 3000 Series - Intel(R) Pentium(R) 4 Processor at 2.80GHz - 2GB DDR SDRAM (at 333MHz) - 60GB 7200 RPM Internal Hard Drive - Integrated Intel 3D AGP Graphics - Integrated 10/100 Ethernet - (Broadcom BCM4401) - CDROM (48X Max Variable) - 3.5" Floppy - No Keyboard, Monitor, or Mouse - (Connected to KVM Switch) 1 - Ethernet LAN Card Each Linux server for Oracle RAC should contain two NIC adapters. The Dell Dimension includes an integrated 10/100 Ethernet adapter that will be used to connect to the public network. The second NIC adapter will be used for the private network (RAC interconnect and Openfiler networked storage). Select the appropriate NIC adapter that is compatible with the maximum data transmission speed of the network switch to be used for the private network. For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network. Used for RAC interconnect to linux1, linux2 and Openfiler networked storage. Gigabit Ethernet Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT) 2 - Network Cables Category 6 patch cable - (Connect linux3 to public network) Category 6 patch cable - (Connect linux3 to interconnect Ethernet switch) US$10 US$10 US$35

US$300

Total

US$355

We are about to start the installation process. As we start to go into the details of the installation, it should be noted that most of the tasks within this document will need to be performed on the new Oracle RAC node (linux3). I will indicate at the beginning of each section whether or not the task(s) should be performed on the new Oracle RAC node, the current Oracle RAC node(s), or on the network storage server (openfiler1).

Install the Linux Operating System

Perform the following installation on the new Oracle RAC node!

After procuring the required hardware, it is time to start the configuration process. The first task we need to perform is to install the Linux operating system. As already mentioned, this article will use CentOS 5.3 (x86) and follows Oracle's suggestion of performing a "default RPMs" installation type to ensure all expected Linux O/S packages are present for a successful Oracle RDBMS installation. Downloading CentOS CentOS.org Download and burn the following ISO images to CD/DVD for CentOS Release 5 Update 3 for either x86 or x86_64 depending on your hardware architecture. 32-bit (x86) Installations

CentOS-5.3-i386-bin-1of6.iso CentOS-5.3-i386-bin-2of6.iso CentOS-5.3-i386-bin-3of6.iso CentOS-5.3-i386-bin-4of6.iso CentOS-5.3-i386-bin-5of6.iso CentOS-5.3-i386-bin-6of6.iso

(624 (636 (634 (632 (638 (625

MB) MB) MB) MB) MB) MB)

If the Linux RAC nodes have a DVD installed, you may find it more convenient to make use of the single DVD image:

CentOS-5.3-i386-bin-DVD.iso (3.7 GB)

64-bit (x86_64) Installations

CentOS-5.3-x86_64-bin-1of7.iso CentOS-5.3-x86_64-bin-2of7.iso CentOS-5.3-x86_64-bin-3of7.iso CentOS-5.3-x86_64-bin-4of7.iso CentOS-5.3-x86_64-bin-5of7.iso CentOS-5.3-x86_64-bin-6of7.iso CentOS-5.3-x86_64-bin-7of7.iso

(619 (554 (637 (629 (635 (637 (638

MB) MB) MB) MB) MB) MB) MB)

If the Linux RAC nodes have a DVD installed, you may find it more convenient to make use of the single DVD image:

CentOS-5.3-x86_64-bin-DVD.iso (4.2 GB)

If you are downloading the above ISO files to a MS Windows machine, there are many options for burning these images (ISO files) to a CD. You may already be familiar with and have the proper software to burn images to CD. If you are not familiar with this process and do not have the required software to burn images to CD, here are just two (of many) software packages that can be used:

UltraISO Magic ISO Maker

Installing CentOS This section provides a summary of the screens used to install CentOS. For more detailed installation instructions, it is possible to use the manuals from Red Hat Linux http://www.redhat.com/docs/manuals/. I would suggest, however, that the instructions I have provided below be used for this Oracle RAC 10g configuration.

Before installing the Linux operating system on the new Oracle RAC node, you should have the two NIC interfaces (cards) installed.

After downloading and burning the CentOS images (ISO files) to CD/DVD, insert CentOS Disk #1 into the new Oracle RAC server (linux3 in this example), power it on, and answer the installation screen prompts as noted below. Boot Screen The first screen is the CentOS boot screen. At the boot: prompt, hit [Enter] to start the installation process. Media Test When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any errors, the media burning software would have warned us. After several seconds, the installer should then detect the video card, monitor, and mouse. The installer then goes into GUI mode. Welcome to CentOS At the welcome screen, click [Next] to continue. Language / Keyboard Selection The next two screens prompt you for the Language and Keyboard settings. In almost all cases, you can accept the defaults. Make the appropriate selection for your configuration and click [Next] to continue. Disk Partitioning Setup Keep the default selection to [Remove linux partitions on selected drives and create default layout] and check the option to [Review and modify partitioning layout]. Click [Next] to continue. You will then be prompted with a dialog window asking if you really want to remove all Linux partitions. Click [Yes] to acknowledge this warning. Partitioning

The installer will then allow you to view (and modify if needed) the disk partitions it automatically selected. For most automatic layouts, the installer will choose 100MB for /boot, double the amount of RAM (systems with <= 2,048MB RAM) or an amount equal to RAM (systems with > 2,048MB RAM) for swap, and the rest going to the root (/) partition. Starting with RHEL 4, the installer will create the same disk configuration as just noted but will create them using the Logical Volume Manager (LVM). For example, it will partition the first hard drive (/dev/hda for my configuration) into two partitions one for the /boot partition (/dev/hda1) and the remainder of the disk dedicate to a LVM named VolGroup00 (/dev/hda2). The LVM Volume Group (VolGroup00) is then partitioned into two LVM partitions - one for the root filesystem (/) and another for swap. The main concern during the partitioning phase is to ensure enough swap space is allocated as required by Oracle (which is a multiple of the available RAM). The following is Oracle's requirement for swap space:
Available RAM Between 1,024MB and 2,048MB Between 2,049MB and 8,192MB More than 8,192MB Swap Space Required 1.5 times the size of RAM Equal to the size of RAM .75 times the size of RAM

For the purpose of this install, I will accept all automatically preferred sizes. (Including 4,096MB for swap since I have 2,048MB of RAM installed. If for any reason, the automatic layout does not configure an adequate amount of swap space, you can easily change that from this screen. To increase the size of the swap partition, [Edit] the volume group VolGroup00. This will bring up the "Edit LVM Volume Group: VolGroup00" dialog. First, [Edit] and decrease the size of the root file system (/) by the amount you want to add to the swap partition. For example, to add another 512MB to swap, you would decrease the size of the root file system by 512MB (i.e. 36,032MB - 512MB = 35,520MB). Now add the space you decreased from the root file system (512MB) to the swap partition. When completed, click [OK] on the "Edit LVM Volume Group: VolGroup00" dialog.

Once you are satisfied with the disk layout, click [Next] to continue. Boot Loader Configuration The installer will use the GRUB boot loader by default. To use the GRUB boot loader, accept all default values and click [Next] to continue. Network Configuration I made sure to install both NIC interfaces (cards) in the new Linux machine before starting the operating system installation. This screen should have successfully detected each of the network devices. Since we will be using this machine to host an Oracle instance, there will be several changes that need to be made to the network configuration. The settings you make here will, of course, depend on your network configuration. The key point to make is that the machine should never be configured with DHCP since it will be used to host an Oracle instance. You will need to configure the machine with static IP addresses. You will also need to configure the server with a real host name. First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to not activate eth1 by default. Second, [Edit] both eth0 and eth1 as follows. Verify that the option "Enable IPv4 support" is selected. Click off the option for "Use dynamic IP configuration (DHCP)" and configure a static IP address and Netmask for your environment. Click off the option to "Enable IPv6 support". You may choose to use different IP addresses for both eth0 and eth1 that I have documented in this guide and that is OK. Put eth1 (the interconnect) on a different subnet than eth0 (the public network): eth0: - Check ON the option to [Enable IPv4 support] - Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration) IPv4 Address: 192.168.1.107 Prefix (Netmask): 255.255.255.0 - Check OFF the option to [Enable IPv6 support] eth1: - Check ON the option to [Enable IPv4 support] - Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration) IPv4 Address: 192.168.2.107

Prefix (Netmask): 255.255.255.0 - Check OFF the option to [Enable IPv6 support] Continue by setting your hostname manually. I used "linux3" for this new Oracle RAC node. Finish this dialog off by supplying your gateway and DNS servers. Time Zone Selection Select the appropriate time zone for your environment and click [Next] to continue. Set Root Password Select a root password and click [Next] to continue. Package Installation Defaults By default, CentOS Linux installs most of the software required for a typical server. There are several other packages (RPMs), however, that are required to successfully install the Oracle Database software. The installer includes a "Customize software" selection that allows the addition of RPM groupings such as "Development Libraries" or "Legacy Library Support". The ADDITION of such RPM groupings is NOT an issue. De-selecting any "default RPM" groupings or individual RPMs, however, can result in failed Oracle Clusterware and Oracle Database installation attempts. For the purpose of this article, select the radio button [Customize now] and click [Next] to continue. This is where you pick the packages to install. Most of the packages required for the Oracle software are grouped into "Package Groups" (i.e. Application -> Editors). Since these nodes will be hosting the Oracle Clusterware and Oracle RAC software, verify that at least the following package groups are selected for install. For many of the Linux package groups, not all of the packages associated with that group get selected for installation. (Note the "Optional packages" button after selecting a package group.) So although the package group gets selected for install, some of the packages required by Oracle do not get installed. In fact, there are some packages that are required by Oracle that do not belong to any of the available package groups (i.e. libaio-devel). Not to worry. A complete list of required packages for Oracle Clusterware 10g and Oracle RAC 10g for CentOS 5 will be provided at the end of this section. These packages will need to be manually installed from the CentOS CDs after the operating system install. For now, install the following package groups:

Desktop Environments

GNOME Desktop Environment Applications o Editors o Graphical Internet o Text-based Internet Development o Development Libraries o Development Tools o Legacy Software Development Servers o Server Configuration Tools Base System o Administration Tools o Base o Java o Legacy Software Support o System Tools o X Window System
o

In addition to the above packages, select any additional packages you wish to install for this node keeping in mind to NOT de-select any of the "default" RPM packages. After selecting the packages to install click [Next] to continue. About to Install This screen is basically a confirmation screen. Click [Next] to start the installation. If you are installing CentOS using CDs, you will be asked to switch CDs during the installation process depending on which packages you selected. Congratulations And that's it. You have successfully installed CentOS on the new Oracle RAC server (linux3). The installer will eject the CD/DVD from the CD-ROM drive. Take out the CD/DVD and click [Reboot] to reboot the system. Post Installation Wizard Welcome Screen When the system boots into CentOS Linux for the first time, it will prompt you with another Welcome screen for the "Post Installation Wizard". The post installation wizard allows you to make final O/S configuration settings. On the "Welcome" screen, click [Forward] to continue.

Firewall On this screen, make sure to select the [Disabled] option and click [Forward] to continue. You will be prompted with a warning dialog about not setting the firewall. When this occurs, click [Yes] to continue. SELinux On the SELinux screen, choose the [Disabled] option and click [Forward] to continue. You will be prompted with a warning dialog warning that changing the SELinux setting will require rebooting the system so the entire file system can be relabeled. When this occurs, click [Yes] to acknowledge a reboot of the system will occur after firstboot (Post Installation Wizard) is completed. Kdump Accept the default setting on the Kdump screen (disabled) and click [Forward] to continue. Date and Time Settings Adjust the date and time settings if necessary and click [Forward] to continue. Create User Create any additional (non-oracle) operating system user accounts if desired and click [Forward] to continue. For the purpose of this article, I will not be creating any additional operating system accounts. I will be creating the "oracle" user account during the Oracle database installation later in this guide. If you chose not to define any additional operating system user accounts, click [Continue] to acknowledge the warning dialog. Sound Card This screen will only appear if the wizard detects a sound card. On the sound card screen click [Forward] to continue.

Additional CDs On the "Additional CDs" screen click [Finish] to continue. Reboot System Given we changed the SELinux option (to disabled), we are prompted to reboot the system. Click [OK] to reboot the system for normal use. Login Screen After rebooting the machine, you are presented with the login screen. Login using the "root" user account and the password you provided during the installation.

Install Required Linux Packages for Oracle RAC

Install the following required Linux packages on the new Oracle RAC node!

After installing CentOS Linux, the next step is to verify and install all packages (RPMs) required by both Oracle Clusterware and Oracle RAC. Although many of the required packages for Oracle were installed in the section Install the Linux Operating System, several will be missing either because they were considered optional within the package group or simply didn't exist in any package group! The packages listed in this section (or later versions) are required for Oracle Clusterware 10g Release 2 and Oracle RAC 10g Release 2 running on the CentOS 5 or Red Hat Enterprise Linux 5 platform. 32-bit (x86) Installations

binutils-2.17.50.0.6-2.el5 compat-libstdc++-296-2.96-138 compat-libstdc++-33-3.2.3-61

elfutils-libelf-0.125-3.el5 elfutils-libelf-devel-0.125 gcc-4.1.1-52 gcc-c++-4.1.1-52 glibc-2.5-12 glibc-common-2.5-12 glibc-devel-2.5-12 glibc-headers-2.5-12 libaio-0.3.106 libaio-devel-0.3.106 libgcc-4.1.1-52 libstdc++-4.1.1 libstdc++-devel-4.1.1-52.e15 libXp-1.0.0-8 make-3.81-1.1 openmotif-2.2.3 sysstat-7.0.0 unixODBC-2.2.11 unixODBC-devel-2.2.11

Note that the openmotif RPM packages are required to install Oracle demos. This article does not cover the installation of Oracle demos. Each of the packages listed above can be found on CD #1, CD #2, and CD #3 on the CentOS 5 - (x86) CDs. While it is possible to query each individual package to determine which ones are missing and need to be installed, an easier method is to run the rpm -Uvh PackageName command from the five CDs as follows. For packages that already exist and are up to date, the RPM command will simply ignore the install and print a warning message to the console that the package is already installed.
# From CentOS 5.3 (x86)- [CD #1] mkdir -p /media/cdrom mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh binutils-2.* rpm -Uvh elfutils-libelf-0.* rpm -Uvh glibc-2.* rpm -Uvh glibc-common-2.* rpm -Uvh libaio-0.* rpm -Uvh libgcc-4.* rpm -Uvh libstdc++-4.* rpm -Uvh make-3.* cd / eject

# From CentOS 5.3 (x86) - [CD #2] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh elfutils-libelf-devel-0.* rpm -Uvh glibc-devel-2.* rpm -Uvh glibc-headers-2.* rpm -Uvh gcc-4.* rpm -Uvh gcc-c++-4.* rpm -Uvh libstdc++-devel-4.* rpm -Uvh unixODBC-2.* cd / eject # From CentOS 5.3 (x86) - [CD #3] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh compat-libstdc++-296* rpm -Uvh compat-libstdc++-33* rpm -Uvh libaio-devel-0.* rpm -Uvh libXp-1.* rpm -Uvh openmotif-2.* rpm -Uvh unixODBC-devel-2.* cd / eject # From CentOS 5.3 (x86) - [CD #4] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh sysstat-7.* cd / eject

64-bit (x86_64) Installations

binutils-2.17.50.0.6-2.el5 compat-libstdc++-296-2.96-138 compat-libstdc++-33-3.2.3-61 compat-libstdc++-33-3.2.3-61 (32 bit) elfutils-libelf-0.125-3.el5 elfutils-libelf-0.125-3.el5 (32 bit) elfutils-libelf-devel-0.125 gcc-4.1.1 gcc-c++-4.1.1 glibc-2.5-12 glibc-2.5-12 (32 bit) glibc-common-2.5

glibc-devel-2.5-12 glibc-devel-2.5-12 (32 bit) glibc-headers-2.5-12 libaio-0.3.106 libaio-0.3.106 (32 bit) libaio-devel-0.3.106 libgcc-4.1.1 libgcc-4.1.1 (32 bit) libstdc++-4.1.1 libstdc++-4.1.1 (32 bit) libstdc++-devel-4.1.1-52.e15 libXp-1.0.0-8 libXp-1.0.0-8 (32 bit) make-3.81-1.1 openmotif-2.2.3 sysstat-7.0.0 unixODBC-2.2.11 unixODBC-2.2.11 (32 bit) unixODBC-devel-2.2.11

Note that the openmotif RPM packages are required to install Oracle demos. This article does not cover the installation of Oracle demos. Each of the packages listed above can be found on CD #1, CD #2, CD #3, and CD #4 on the CentOS 5 - (x86_64) CDs. While it is possible to query each individual package to determine which ones are missing and need to be installed, an easier method is to run the rpm -Uvh PackageNamecommand from the five CDs as follows. For packages that already exist and are up to date, the RPM command will simply ignore the install and print a warning message to the console that the package is already installed.
# From CentOS 5.3 (x86_64)- [CD #1] mkdir -p /media/cdrom mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh binutils-2.* rpm -Uvh elfutils-libelf-0.* rpm -Uvh glibc-2.* rpm -Uvh glibc-common-2.* rpm -Uvh libaio-0.* rpm -Uvh libgcc-4.* rpm -Uvh libstdc++-4.* rpm -Uvh make-3.* cd / eject

# From CentOS 5.3 (x86_64) - [CD #2] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh elfutils-libelf-devel-0.* rpm -Uvh glibc-devel-2.* rpm -Uvh glibc-headers-2.* rpm -Uvh gcc-4.* rpm -Uvh gcc-c++-4.* rpm -Uvh libstdc++-devel-4.* rpm -Uvh unixODBC-2.* cd / eject # From CentOS 5.3 (x86_64) - [CD #3] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh compat-libstdc++-296* rpm -Uvh compat-libstdc++-33* rpm -Uvh libaio-devel-0.* rpm -Uvh libXp-1.* rpm -Uvh openmotif-2.* rpm -Uvh unixODBC-devel-2.* cd / eject # From CentOS 5.3 (x86_64) - [CD #4] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh sysstat-7.* cd / eject

Network Configuration

Perform the following network configuration tasks on the new Oracle RAC node!

Introduction to Network Settings Although we configured several of the network settings during the installation of CentOS, it is important to not skip this section as it

contains critical steps that are required for a successful RAC environment. During the Linux O/S install we already configured the IP address and host name for the new Oracle RAC node. We now need to configure the /etc/hosts file as well as adjusting several of the network settings for the interconnect. All nodes in the RAC cluster should have one static IP address for the public network and one static IP address for the private cluster interconnect. Do not use DHCP naming for the public IP address or the interconnects; you need static IP addresses! The private interconnect should only be used by Oracle to transfer Cluster Manager and Cache Fusion related data along with data for the network storage server (Openfiler). Note that Oracle does not support using the public network interface for the interconnect. You must have one network interface for the public network and another network interface for the private interconnect. For a production RAC implementation, the interconnect should be at least Gigabit (or more) and only be used by Oracle as well as having the network storage server (Openfiler) on a separate Gigabit network. Configuring Public and Private Network With the new Oracle RAC node, we need to configure the network for access to the public network as well as the private interconnect. The easiest way to configure network settings in Red Hat Linux is with the program Network Configuration. This application can be started from the commandline as the "root" user account as follows:
[root@linux3 ~]# /usr/bin/system-config-network &

Do not use DHCP naming for the public IP address or the interconnects we need static IP addresses!

Using the Network Configuration application, you need to configure both NIC devices as well as the /etc/hosts file on all nodes in the RAC cluster. Both of these tasks can be completed using the Network Configuration GUI. Notice that the /etc/hosts settings should the same for all nodes and that I removed any entry that has to do with IPv6 (for example, ::1 localhost6.localdomain6 localhost6).

Please note that for the purpose of this example configuration the /etc/hosts entries will be the same for all three Oracle RAC nodes (linux1, linux2, and linux3) as well as the network storage server (openfiler1): Our example configuration will use the following settings for all nodes:
Oracle RAC Node 3 - (linux3)
Devi IP Address Subnet ce eth0 Gateway Purpose

192.168.1.1 255.255.25 192.168. Connects linux3 to the public network 07 5.0 1.1 192.168.2.1 255.255.25 07 5.0 Connects linux3 (interconnect) to linux1/linux2 (linux1-priv/linux2-priv)

eth1

/etc/hosts 127.0.0.1 # Public Network 192.168.1.100 192.168.1.101 192.168.1.107 localhost.localdomain localhost - (eth0) linux1 linux2 linux3

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv 192.168.2.107 linux3-priv # Public Virtual 192.168.1.200 192.168.1.201 192.168.1.207 IP (VIP) addresses - (eth0:1) linux1-vip linux2-vip linux3-vip

# Private Storage Network for Openfiler - (eth1) 192.168.1.195 openfiler1 192.168.2.195 openfiler1-priv

Oracle RAC Node 2 - (linux2)

Devi IP Address Subnet ce eth0 Gateway Purpose

192.168.1.1 255.255.25 192.168. Connects linux2 to the public network 01 5.0 1.1 192.168.2.1 255.255.25 01 5.0 Connects linux2 (interconnect) to linux1/linux3 (linux1-priv/linux3-priv)

eth1

/etc/hosts 127.0.0.1 # Public Network 192.168.1.100 192.168.1.101 192.168.1.107 localhost.localdomain localhost - (eth0) linux1 linux2 linux3

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv 192.168.2.107 linux3-priv # Public Virtual 192.168.1.200 192.168.1.201 192.168.1.207 IP (VIP) addresses - (eth0:1) linux1-vip linux2-vip linux3-vip

# Private Storage Network for Openfiler - (eth1) 192.168.1.195 openfiler1 192.168.2.195 openfiler1-priv

Oracle RAC Node 1 - (linux1)

Devi IP Address Subnet ce eth0 Gateway Purpose

192.168.1.1 255.255.25 192.168. Connects linux1 to the public network 00 5.0 1.1 192.168.2.1 255.255.25 00 5.0 Connects linux1 (interconnect) to linux2/linux3 (linux2-priv/linux3-priv)

eth1

/etc/hosts 127.0.0.1 # Public Network 192.168.1.100 192.168.1.101 192.168.1.107 localhost.localdomain localhost - (eth0) linux1 linux2 linux3

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv 192.168.2.107 linux3-priv # Public Virtual 192.168.1.200 192.168.1.201 192.168.1.207 IP (VIP) addresses - (eth0:1) linux1-vip linux2-vip linux3-vip

# Private Storage Network for Openfiler - (eth1)

192.168.1.195 192.168.2.195

openfiler1 openfiler1-priv

In the screen shots below, only the new Oracle RAC node (linux3) is shown. Ensure that the /etc/hosts file is updated on all participating nodes to access the new Oracle RAC node!

Figure 2: Network Configuration Screen - Node 3 (linux3)

Figure 3: Ethernet Device Screen - eth0 (linux3)

Figure 4: Ethernet Device Screen - eth1 (linux3)

Figure 5: Network Configuration Screen - /etc/hosts (linux3) Once the network is configured, you can use the ifconfig command to verify everything is working. The following example is from the new Oracle RAC node linux3:
[root@linux3 ~]# /sbin/ifconfig -a eth0 Link encap:Ethernet HWaddr 00:1E:2A:37:6B:9E inet addr:192.168.1.107 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::21e:2aff:fe37:6b9e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1167677 errors:0 dropped:0 overruns:0 frame:0 TX packets:1842517 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:576629131 (549.9 MiB) TX bytes:2143836310 (1.9 GiB) Interrupt:209 Base address:0xef00 eth1 Link encap:Ethernet HWaddr 00:0E:0C:C0:78:64 inet addr:192.168.2.107 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20e:cff:fec0:7864/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:48 errors:0 dropped:0 overruns:0 frame:0 TX packets:59 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4782 (4.6 KiB) TX bytes:5564 (5.4 KiB) Base address:0xdd80 Memory:fe9c0000-fe9e0000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2034 errors:0 dropped:0 overruns:0 frame:0 TX packets:2034 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2386686 (2.2 MiB) TX bytes:2386686 (2.2 MiB) Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

sit0

Verify Network Access to All Nodes Verify that the new Oracle RAC node has access to the public and private network for all current nodes. From linux3:
[root@linux3 ~]# ping -c 1 linux1 | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms [root@linux3 ~]# ping -c 1 linux1-priv | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms [root@linux3 ~]# ping -c 1 linux2 | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms [root@linux3 ~]# ping -c 1 linux2-priv | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms [root@linux3 ~]# ping -c 1 openfiler1 | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms [root@linux3 ~]# ping -c 1 openfiler1-priv | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms

Confirm the RAC Node Name is Not Listed in Loopback Address Ensure that the new Oracle RAC node (linux3) is not included for the loopback address in the /etc/hosts file. If the machine name is listed in the in the loopback address entry as below:

it will need to be removed as shown below:

127.0.0.1

127.0.0.1

linux3 localhost.localdomain localhost

localhost.localdomain localhost

If the RAC node name is listed for the loopback address, you will receive the following error during the RAC installation:
ORA-00603: ORACLE server session terminated by fatal error

or

ORA-29702: error occurred in Cluster Group Service operation

Confirm localhost is defined in the /etc/hosts file for the loopback address Ensure that the entry for localhost.localdomain and localhost are included for the loopback address in the /etc/hosts file on the new Oracle RAC node:
127.0.0.1 localhost.localdomain localhost

If an entry does not exist for localhost in the /etc/hosts file, Oracle Clusterware will be unable to start the application resources notably the ONS process. The error would indicate "Failed to get IP for localhost" and will be written to the log file for ONS. For example:
CRS-0215 could not start resource 'ora.linux3.ons'. Check log file "/u01/app/crs/log/linux3/racg/ora.linux3.ons.log" for more details.

The ONS log file will contain lines similar to the following:
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved. 2007-04-14 13:10:02.729: [ RACG][3086871296][13316][3086871296] [ora.linux3.ons]: Failed to get IP for localhost (1) Failed to get IP for localhost (1) Failed to get IP for localhost (1) onsctl: ons failed to start ...

Adjusting Network Settings With Oracle 9.2.0.1 and later, Oracle makes use of UDP as the default protocol on Linux for inter-process communication (IPC), such as Cache Fusion and Cluster Manager buffer transfers between instances within the RAC cluster.

Oracle strongly suggests to adjust the default and maximum receive buffer size (SO_RCVBUF socket option) to 1024KB and the default and maximum send buffer size (SO_SNDBUF socket option) to 256KB. The receive buffers are used by TCP and UDP to hold received data until it is read by the application. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This means that datagrams will be discarded if they don't fit in the socket receive buffer, potentially causing the sender to overwhelm the receiver.
The default and maximum window size can be changed in the /proc file system without reboot:
[root@linux3 ~]# sysctl -w net.core.rmem_default=1048576 net.core.rmem_default = 1048576 [root@linux3 ~]# sysctl -w net.core.rmem_max=1048576 net.core.rmem_max = 1048576 [root@linux3 ~]# sysctl -w net.core.wmem_default=262144 net.core.wmem_default = 262144 [root@linux3 ~]# sysctl -w net.core.wmem_max=262144 net.core.wmem_max = 262144

The above commands made the changes to the already running OS. You should now make the above changes permanent (for each reboot) by adding the following lines to the /etc/sysctl.conf file on the new Oracle RAC node:
# # # # # # # # # # # # # # # # # # # # +---------------------------------------------------------+ | ADJUSTING NETWORK SETTINGS | +---------------------------------------------------------+ | With Oracle 9.2.0.1 and onwards, Oracle now makes use | | of UDP as the default protocol on Linux for | | inter-process communication (IPC), such as Cache Fusion | | and Cluster Manager buffer transfers between instances | | within the RAC cluster. Oracle strongly suggests to | | adjust the default and maximum receive buffer size | | (SO_RCVBUF socket option) to 1024KB, and the default | | and maximum send buffer size (SO_SNDBUF socket option) | | to 256KB. The receive buffers are used by TCP and UDP | | to hold received data until it is read by the | | application. The receive buffer cannot overflow because | | the peer is not allowed to send data beyond the buffer | | size window. This means that datagrams will be | | discarded if they don't fit in the socket receive | | buffer. This could cause the sender to overwhelm the | | receiver. | +---------------------------------------------------------+

# +---------------------------------------------------------+ # | Default setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. | # +---------------------------------------------------------+ net.core.rmem_default=1048576 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. | # +---------------------------------------------------------+ net.core.rmem_max=1048576 # +---------------------------------------------------------+ # | Default setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.core.wmem_default=262144 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.core.wmem_max=262144

Check and turn off UDP ICMP rejections: During the Linux installation process, I indicated to not configure the firewall option. By default the option to configure a firewall is selected by the installer. This has burned me several times so I like to do a double-check that the firewall option is not configured and to ensure udp ICMP filtering is turned off. If UDP ICMP is blocked or rejected by the firewall, the Oracle Clusterware software will crash after several minutes of running. When the Oracle Clusterware process fails, you will have something similar to the following in the <machine_name>_evmocr.log file:
08/29/2005 22:17:19 oac_init:2: Could not connect to server, clsc retcode = 9 08/29/2005 22:17:19 a_init:12!: Client init unsuccessful : [32] ibctx:1:ERROR: INVALID FORMAT proprinit:problem reading the bootblock or superbloc 22

When experiencing this type of error, the solution is to remove the udp ICMP (iptables) rejection rule - or to simply have the firewall option turned off. The Oracle Clusterware software will then start to operate normally and not crash. The following commands should be executed as the root user account:

1. Check to ensure that the firewall option is turned off. If the firewall option is stopped (like it is in my example below) you do not have to proceed with the following steps.
2. [root@linux3 ~]# /etc/rc.d/init.d/iptables status Firewall is stopped.

3. If the firewall option is operating you will need to first manually disable UDP ICMP rejections:
4. [root@linux3 ~]# /etc/rc.d/init.d/iptables stop 5. 6. Flushing firewall rules: [ OK ] 7. Setting chains to policy ACCEPT: filter [ OK ] Unloading iptables modules: [ OK ]

8. Then, to turn UDP ICMP rejections off for next server reboot (which should always be turned off):
[root@linux3 ~]# chkconfig iptables off

Configure Network Security on the Openfiler Storage Server

Perform the following configuration tasks on the network storage server (openfiler1)!

With the network now setup, the next step is to configure network access in Openfiler so that the new Oracle RAC node (linux3) has permissions to the shared iSCSI volumes used in the current Oracle RAC 10g environment. For the purpose of this example, all iSCSI traffic will use the private network interface eth1 which in this article is on the 192.168.2.0 network. Openfiler administration is performed using the Openfiler Storage Control Center a browser based tool over an https connection on port 446. For example:
https://openfiler1.idevelopment.info:446/

From the Openfiler Storage Control Center home page, login as an administrator. The default administration login credentials for Openfiler are:

Username:

openfiler

Password:

password

The first page the administrator sees is the [Status] / [System Information] screen. Services This article assumes that the current Oracle RAC 10g environment is operational and therefore the iSCSI services should already be enabled within Openfiler. To verify the iSCSI services are running, use the Openfiler Storage Control Center and navigate to [Services] / [Manage Services]:

Figure 6: Verify iSCSI Services are Enabled Another method is to SSH into the Openfiler server and verify the iscsi-target service is running:
[root@openfiler1 ~]# service iscsi-target status ietd (pid 3961) is running...

Network Access Configuration The next step is to configure network access in Openfiler to identify the new Oracle RAC node (linux3) which will need to access the iSCSI volumes used in the current

Oracle RAC 10g environment. Note that this step does not actually grant the appropriate permissions to the iSCSI volumes required by the new Oracle RAC node. That will be accomplished later in this section by updating the ACL for each of the current iSCSI logical volumes. As in the previous section, configuring network access is accomplished using the Openfiler Storage Control Center by navigating to [System] / [Network Setup]. The "Network Access Configuration" section (at the bottom of the page) allows an administrator to setup networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we will want to add the new Oracle RAC node individually rather than allowing the entire 192.168.2.0 network have access to Openfiler resources. When entering the new Oracle RAC node, note that the 'Name' field is just a logical name used for reference only. As a convention when entering nodes, I simply use the node name defined for that IP address. Next, when entering the actual node in the 'Network/Host' field, always use its IP address even though its host name may already be defined in your /etc/hosts file or DNS. Lastly, when entering actual hosts in our Class C network, use a subnet mask of 255.255.255.255. It is important to remember that you will be entering the IP address of the private network (eth1) for the new Oracle RAC node. The following image shows the results of adding the new Oracle RAC node linux3 to the local network configuration:

Figure 7: Configure Openfiler Network Access for new Oracle RAC Node Current Logical iSCSI Volumes The current Openfiler configuration contains five logical iSCSI volumes in a single volume group named rac1.
iSCSI / Logical Volumes Volume Name racdb-crs Volume Description racdb - Oracle Clusterware racdb - ASM Volume 1 racdb - ASM Volume 2 racdb - ASM Volume 3 Required Space Filesystem (MB) Type 2,048 iSCSI

racdb-asm1

16,984

iSCSI

racdb-asm2

16,984

iSCSI

racdb-asm3

16,984

iSCSI

racdb-asm4

racdb - ASM Volume 4

16,984

iSCSI

To view the available iSCSI volumes from within the Openfiler Storage Control Center, navigate to [Volumes] / [Manage Volumes]. There we will see all five logical volumes within the volume group rac1:

Figure 8: Current Logical (iSCSI) Volumes Update Network ACL

Before the new Oracle RAC node can have access to the current iSCSI targets, it needs to be granted the appropriate permissions. Awhile back, we configured network access in Openfiler for the new Oracle RAC node. The new Oracle RAC node will need to access the current iSCSI targets through the storage (private) network. In this section, we grant the new Oracle RAC node access to the current iSCSI targets. The current Openfiler configuration contains an iSCSI name (the Target IQN) which each iSCSI logical volume listed in the previous section is mapped to:
iSCSI Target / Logical Volume Mappings Target IQN iSCSI Volume Name
racdb-crs

Volume Description racdb - Oracle Clusterware racdb - ASM Volume 1 racdb - ASM Volume 2 racdb - ASM Volume 3 racdb - ASM Volume 4

iqn.2006-01.com.openfiler:racdb.crs

iqn.2006-01.com.openfiler:racdb.asm1 racdb-asm1 iqn.2006-01.com.openfiler:racdb.asm2 racdb-asm2 iqn.2006-01.com.openfiler:racdb.asm3 racdb-asm3 iqn.2006-01.com.openfiler:racdb.asm4 racdb-asm4

To start the process of allowing access to the current iSCSI logical volumes from the new Oracle RAC node, navigate to [Volumes] / [iSCSI Targets]. Then click on the grey sub-tab named "Target Configuration". On this page is a section named "Select iSCSI Target" where users can select which of the iSCSI targets to display and/or edit:

Figure 9: Select iSCSI Target Select the first iSCSI target (which in this example is iqn.200601.com.openfiler:racdb.crs) and click the [Change] button. Next, click the grey subtab named "Network ACL" (next to "LUN Mapping" sub-tab). As you can see, all nodes in the cluster will have their "Access" set to 'Allow' with the exception of the new Oracle RAC node. For the current iSCSI target, change the "Access" for the new Oracle RAC node from 'Deny' to 'Allow' and click the 'Update' button:

Figure 10: Update Network ACL

After updating the Network ACL for the first iSCSI Target, click on the grey sub-tab named "Target Configuration" and select the next iSCSI target in the "Select iSCSI Target" section. Click the [Change] button and continue to update the Network ACL for this iSCSI target (changing the "Access" for the new Oracle RAC node from 'Deny' to 'Allow' under the "Network ACL" grey sub-tab). Continue this process until access has been granted to the new Oracle RAC node to all five iSCSI targets.

Configure the iSCSI Initiator

Configure the iSCSI initiator on the new Oracle RAC node!

An iSCSI client can be any system (Linux, Unix, MS Windows, Apple Mac, etc.) for which iSCSI support (a driver) is available. In our case, the clients are the three Oracle RAC nodes, (linux1, linux2, and linux3), running CentOS 5. In this section we will be configuring the iSCSI software initiator on the new Oracle RAC node linux3. CentOS 5.3 includes the Open-iSCSI iSCSI software initiator which can be found in the iscsi-initiator-utils RPM. This is a change from previous versions of CentOS (4.x) which included the Linux iscsi-sfnet software driver developed as part of the Linux-iSCSI Project. All iSCSI management tasks like discovery and logins will use the command-line interface iscsiadm which is included with Open-iSCSI. The iSCSI software initiator will be configured to automatically login to the network storage server (openfiler1) and discover the iSCSI volumes listed in the previous section. We will then go through the steps of creating persistent local SCSI device names (i.e. /dev/iscsi/asm1) for each of the iSCSI target names discovered using udev. Having a consistent local SCSI device name and which iSCSI target it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM. Before we can do any of this, however, we must first install the iSCSI initiator software on the new Oracle RAC node. Installing the iSCSI (initiator) service

With CentOS 5.3, the Open-iSCSI iSCSI software initiator does not get installed by default. The software is included in the iscsi-initiator-utils package which can be found on CD #1. To determine if this package is installed (which in most cases, it will not be), perform the following on the new Oracle RAC node:
[root@linux3 ~]# rpm -qa | grep iscsi-initiator-utils

If the iscsi-initiator-utils package is not installed, load CD #1 into the new Oracle RAC node and perform the following:
[root@linux3 [root@linux3 [root@linux3 [root@linux3 [root@linux3 ~]# ~]# ~]# ~]# ~]# mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh iscsi-initiator-utils-6.2.0.868-0.18.el5.i386.rpm cd / eject

Configure the iSCSI (initiator) service After verifying that the iscsi-initiator-utils package is installed on the new Oracle RAC node, start the iscsid service and enable it to automatically start when the system boots. We will also configure the iscsi service to automatically start which logs into iSCSI targets needed at system startup.
[root@linux3 ~]# service iscsid start Turning off network shutdown. Starting iSCSI daemon: [ [ OK ] [root@linux3 ~]# chkconfig iscsid on [root@linux3 ~]# chkconfig iscsi on OK ]

Now that the iSCSI service is started, use the iscsiadm command-line interface to discover all available targets on the network storage server:
[root@linux3 ~]# iscsiadm -m discovery -t sendtargets -p openfiler1-priv 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm1 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm2 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm3 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm4 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.crs

Manually Login to iSCSI Targets At this point the iSCSI initiator service has been started and the new Oracle RAC node was able to discover the available targets from the network storage server. The next step is to manually login to each of the available targets which can be done using

the iscsiadmcommand-line interface. Note that I had to specify the IP address and not the host name of the network storage server (openfiler1-priv) - I believe this is required given the discovery (above) shows the targets using the IP address.
[root@linux3 ~]# 192.168.2.195 -l [root@linux3 ~]# 192.168.2.195 -l [root@linux3 ~]# 192.168.2.195 -l [root@linux3 ~]# 192.168.2.195 -l [root@linux3 ~]# 192.168.2.195 -l iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm1 -p iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm2 -p iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm3 -p iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm4 -p iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p

Configure Automatic Login The next step is to ensure the client will automatically login to each of the targets listed above when the machine is booted (or the iSCSI initiator service is started/restarted):
[root@linux3 ~]# iscsiadm 192.168.2.195 --op update [root@linux3 ~]# iscsiadm 192.168.2.195 --op update [root@linux3 ~]# iscsiadm 192.168.2.195 --op update [root@linux3 ~]# iscsiadm 192.168.2.195 --op update [root@linux3 ~]# iscsiadm 192.168.2.195 --op update -m -n -m -n -m -n -m -n -m -n node -T iqn.2006-01.com.openfiler:racdb.asm1 -p node.startup -v automatic node -T iqn.2006-01.com.openfiler:racdb.asm2 -p node.startup -v automatic node -T iqn.2006-01.com.openfiler:racdb.asm3 -p node.startup -v automatic node -T iqn.2006-01.com.openfiler:racdb.asm4 -p node.startup -v automatic node -T iqn.2006-01.com.openfiler:racdb.crs -p node.startup -v automatic

Create Persistent Local SCSI Device Names In this section, we will go through the steps to create persistent local SCSI device names for each of the iSCSI target names. This will be done using udev. Having a consistent local SCSI device name and which iSCSI target it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM. When any of the Oracle RAC nodes boot and the iSCSI initiator service is started, it will automatically login to each of the targets configured in a random fashion and map them to the next available local SCSI device name. For example, the target iqn.200601.com.openfiler:racdb.asm1 may get mapped to /dev/sda. I can actually determine the current mappings for all targets by looking at the /dev/disk/by-path directory:

[root@linux3 ~]# (cd /dev/disk/by-path; ls -l *openfiler* | awk '{FS=" "; print $9 " " $10 " " $11}') ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm1-lun-0 -> ../../sda ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm1-lun-0-part1 -> ../../sda1 ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm2-lun-0 -> ../../sdb ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm2-lun-0-part1 -> ../../sdb1 ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm3-lun-0 -> ../../sdc ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm3-lun-0-part1 -> ../../sdc1 ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm4-lun-0 -> ../../sdd ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm4-lun-0-part1 -> ../../sdd1 ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.crs-lun-0 -> ../../sde ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.crs-lun-0-part1 -> ../../sde1

Using the output from the above listing, we can establish the following current mappings:
Current iSCSI Target Name to local SCSI Device Name Mappings iSCSI Target Name iqn.2006-01.com.openfiler:racdb.asm1 iqn.2006-01.com.openfiler:racdb.asm2 iqn.2006-01.com.openfiler:racdb.asm3 iqn.2006-01.com.openfiler:racdb.asm4 iqn.2006-01.com.openfiler:racdb.crs SCSI Device Name
/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde

This mapping, however, may change every time the Oracle RAC node is rebooted. For example, after a reboot it may be determined that the iSCSI target iqn.200601.com.openfiler:racdb.asm1 gets mapped to the local SCSI device /dev/sdd. It is therefore impractical to rely on using the local SCSI device name given there is no way to predict the iSCSI target mappings after a reboot.

What we need is a consistent device name we can reference (i.e. /dev/iscsi/asm1) that will always point to the appropriate iSCSI target through reboots. This is where the Dynamic Device Management tool named udev comes in. udev provides a dynamic device directory using symbolic links that point to the actual device using a configurable set of rules. When udev receives a device event (for example, the client logging in to an iSCSI target), it matches its configured rules against the available device attributes provided in sysfs to identify the device. Rules that match may provide additional device information or specify a device node name and multiple symlink names and instruct udev to run additional programs (a SHELL script for example) as part of the device event handling process. The first step is to create a new rules file. The file will be named /etc/udev/rules.d/55-openiscsi.rules and contain only a single line of name=value pairs used to receive events we are interested in. It will also define a callout SHELL script (/etc/udev/scripts/iscsidev.sh) to handle the event. Create the following rules file /etc/udev/rules.d/55-openiscsi.rules on the new Oracle RAC node:
/etc/udev/rules.d/55-openiscsi.rules # /etc/udev/rules.d/55-openiscsi.rules KERNEL=="sd*", BUS=="scsi", PROGRAM="/etc/udev/scripts/iscsidev.sh %b",SYMLINK+="iscsi/%c/part%n"

We now need to create the UNIX SHELL script that will be called when this event is received. Let's first create a separate directory on the new Oracle RAC node where udev scripts can be stored:
[root@linux3 ~]# mkdir -p /etc/udev/scripts

Next, create the UNIX shell script /etc/udev/scripts/iscsidev.sh on the new Oracle RAC node:
/etc/udev/scripts/iscsidev.sh #!/bin/sh # FILE: /etc/udev/scripts/iscsidev.sh BUS=${1} HOST=${BUS%%:*} [ -e /sys/class/iscsi_host ] || exit 1

file="/sys/class/iscsi_host/host$ {HOST}/device/session*/iscsi_session*/targetname" target_name=$(cat ${file}) # This is not an open-scsi drive if [ -z "${target_name}" ]; then exit 1 fi # Check if QNAP drive check_qnap_target_name=${target_name%%:*} if [ $check_qnap_target_name = "iqn.2004-04.com.qnap" ]; then target_name=`echo "${target_name%.*}"` fi echo "${target_name##*.}"

After creating the UNIX SHELL script, change it to executable:

[root@linux3 ~]# chmod 755 /etc/udev/scripts/iscsidev.sh

Now that udev is configured, restart the iSCSI service on the new Oracle RAC node:
[root@linux3 ~]# service iscsi stop Logging out of session [sid: 1, target: iqn.2006-01.com.openfiler:racdb.asm1, portal: 192.168.2.195,3260] Logging out of session [sid: 2, target: iqn.2006-01.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260] Logging out of session [sid: 3, target: iqn.2006-01.com.openfiler:racdb.asm3, portal: 192.168.2.195,3260] Logging out of session [sid: 4, target: iqn.2006-01.com.openfiler:racdb.asm4, portal: 192.168.2.195,3260] Logging out of session [sid: 5, target: iqn.2006-01.com.openfiler:racdb.crs, portal: 192.168.2.195,3260] Logout of [sid: 1, target: iqn.2006-01.com.openfiler:racdb.asm1, portal: 192.168.2.195,3260]: successful Logout of [sid: 2, target: iqn.2006-01.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260]: successful Logout of [sid: 3, target: iqn.2006-01.com.openfiler:racdb.asm3, portal: 192.168.2.195,3260]: successful Logout of [sid: 4, target: iqn.2006-01.com.openfiler:racdb.asm4, portal: 192.168.2.195,3260]: successful Logout of [sid: 5, target: iqn.2006-01.com.openfiler:racdb.crs, portal: 192.168.2.195,3260]: successful Stopping iSCSI daemon: [root@linux3 ~]# service iscsi start iscsid dead but pid file exists Turning off network shutdown. Starting iSCSI daemon: [ OK ] [ OK ] Setting up iSCSI targets: Logging in to [iface: default, target: iqn.200601.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm1, portal: 192.168.2.195,3260] Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.crs, portal: 192.168.2.195,3260] Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm4, portal: 192.168.2.195,3260] Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm3, portal: 192.168.2.195,3260] Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260]: successful Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm1, portal: 192.168.2.195,3260]: successful Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.crs, portal: 192.168.2.195,3260]: successful Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm4, portal: 192.168.2.195,3260]: successful Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm3, portal: 192.168.2.195,3260]: successful [ OK ]

Let's see if our hard work paid off:

[root@linux3 ~]# ls -l /dev/iscsi/* /dev/iscsi/asm1: total 0 lrwxrwxrwx 1 root root 9 Sep 2 22:29 part -> ../../sda lrwxrwxrwx 1 root root 10 Sep 2 22:29 part1 -> ../../sda1 /dev/iscsi/asm2: total 0 lrwxrwxrwx 1 root root 9 Sep lrwxrwxrwx 1 root root 10 Sep /dev/iscsi/asm3: total 0 lrwxrwxrwx 1 root root 9 Sep lrwxrwxrwx 1 root root 10 Sep /dev/iscsi/asm4: total 0 lrwxrwxrwx 1 root root 9 Sep lrwxrwxrwx 1 root root 10 Sep /dev/iscsi/crs: total 0 lrwxrwxrwx 1 root root 9 Sep lrwxrwxrwx 1 root root 10 Sep

2 22:29 part -> ../../sdc 2 22:29 part1 -> ../../sdc1

2 22:29 part -> ../../sdb 2 22:29 part1 -> ../../sdb1

2 22:29 part -> ../../sde 2 22:29 part1 -> ../../sde1

2 22:29 part -> ../../sdd 2 22:29 part1 -> ../../sdd1

The listing above shows that udev did the job it was suppose to do! We now have a consistent set of local device names that can be used to reference the iSCSI targets. For example, we can safely assume that the device name /dev/iscsi/asm1/part will always reference the iSCSI target iqn.2006-01.com.openfiler:racdb.asm1. We now

have a consistent iSCSI target name to local device name mapping which is described in the following table:
iSCSI Target Name to Local Device Name Mappings iSCSI Target Name iqn.2006-01.com.openfiler:racdb.asm1 iqn.2006-01.com.openfiler:racdb.asm2 iqn.2006-01.com.openfiler:racdb.asm3 iqn.2006-01.com.openfiler:racdb.asm4 iqn.2006-01.com.openfiler:racdb.crs Local Device Name
/dev/iscsi/asm1/part /dev/iscsi/asm2/part /dev/iscsi/asm3/part /dev/iscsi/asm4/part /dev/iscsi/crs/part

Create "oracle" User and Directories

Perform the following tasks on the new Oracle RAC node!

In this section we will create the oracle UNIX user account, recommended O/S groups, and all required directories. The following O/S groups will be created:
Description Oracle Inventory and Software Owner Database Administrator Database Operator SYSDBA SYSOPER OSDBA OSOPER Oracle Privilege Oracle Group Name UNIX Group name oinstall dba oper

We will be using the Oracle Cluster File System, Release 2 (OCFS2) to store the files required to be shared for the Oracle Clusterware software. When using OCFS2, the UID of the UNIX user "oracle" and GID of the UNIX group "oinstall" must be the same on all Oracle RAC nodes in the cluster. If either the UID or GID are different, the files on the OCFS2 file system will show up as "unowned" or may even be owned by a different user. For this example, I will use 501 for the "oracle" UID and 501 for the "oinstall" GID. Note that members of the UNIX group oinstall are considered the "owners" of the Oracle software. Members of the dba group can administer Oracle databases, for example starting up and shutting down databases. Members of the optional group oper have a limited set of database administrative privileges such as managing and running backups. The default name for this group is oper. To use this group, choose the "Custom" installation type to install the Oracle database software. In this article, we are creating the oracle user account to have all responsibilities!
This guide adheres to the Optimal Flexible Architecture (OFA) for naming conventions used in creating the directory structure.

Create Group and User for Oracle Lets start this section by creating the UNIX oinstall, dba, and oper group and oracle user account:
[root@linux3 [root@linux3 [root@linux3 [root@linux3 /bin/bash -c ~]# groupadd -g 501 oinstall ~]# groupadd -g 502 dba ~]# groupadd -g 503 oper ~]# useradd -m -u 501 -g oinstall -G dba,oper -d /home/oracle -s "Oracle Software Owner" oracle

[root@linux3 ~]# id oracle uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper)

Set the password for the oracle account:

[root@linux3 ~]# passwd oracle Changing password for user oracle. New UNIX password: xxxxxxxxxxx Retype new UNIX password: xxxxxxxxxxx passwd: all authentication tokens updated successfully.

Verify That the User nobody Exists

Before installing the Oracle software, complete the following procedure to verify that the user nobody exists on the system: 1. To determine if the user exists, enter the following command:
2. [root@linux3 ~]# id nobody uid=99(nobody) gid=99(nobody) groups=99(nobody)

If this command displays information about the nobody user, then you do not have to create that user.
3.

If the user nobody does not exist, then enter the following command to create it:
[root@linux3 ~]# /usr/sbin/useradd nobody

Create the Oracle Base Directory The next step is to create a new directory that will be used to store the Oracle Database software. When configuring the oracle user's environment (later in this section) we will be assigning the location of this directory to the $ORACLE_BASE environment variable. The following assumes that the directories are being created in the root file system. Please note that this is being done for the sake of simplicity and is not recommended as a general practice. Normally, these directories would be created on a separate file system. After the directory is created, you must then specify the correct owner, group, and permissions for it. Perform the following on the new Oracle RAC node:
[root@linux3 ~]# mkdir -p /u01/app/oracle [root@linux3 ~]# chown -R oracle:oinstall /u01/app/oracle [root@linux3 ~]# chmod -R 775 /u01/app/oracle

At the end of this procedure, you will have the following:

owned by root. /u01/app owned by root. /u01/app/oracle owned by oracle:oinstall with 775 permissions. This ownership and permissions enables the OUI to create the oraInventory directory, in the path /u01/app/oracle/oraInventory.
/u01

Create the Oracle Clusterware Home Directory Next, create a new directory that will be used to store the Oracle Clusterware software. When configuring the oracle user's environment (later in this section) we will be assigning the location of this directory to the $ORA_CRS_HOME environment variable. As noted in the previous section, the following assumes that the directories are being created in the root file system. This is being done for the sake of simplicity and is not recommended as a general practice. Normally, these directories would be created on a separate file system. After the directory is created, you must then specify the correct owner, group, and permissions for it. Perform the following on the new Oracle RAC node:
[root@linux3 ~]# mkdir -p /u01/app/crs [root@linux3 ~]# chown -R oracle:oinstall /u01/app/crs [root@linux3 ~]# chmod -R 775 /u01/app/crs

At the end of this procedure, you will have the following:

owned by root. /u01/app owned by root. /u01/app/crs owned by oracle:oinstall with 775 permissions. These permissions are required for Oracle Clusterware installation and are changed during the installation process.
/u01

Create Mount Point for OCFS2 / Clusterware Let's now create the mount point for the Oracle Cluster File System, Release 2 (OCFS2) that will be used to store the two Oracle Clusterware shared files. Perform the following on the new Oracle RAC node:
[root@linux3 ~]# mkdir -p /u02 [root@linux3 ~]# chown -R oracle:oinstall /u02 [root@linux3 ~]# chmod -R 775 /u02

Create Login Script for oracle User Account

To ensure that the environment is setup correctly for the "oracle" UNIX userid on the new Oracle RAC node, use the following.bash_profile:
When you are setting the Oracle environment variables for each Oracle RAC node, ensure to assign each RAC node a unique Oracle SID! For this example, I used:

For this example, I used:

linux1 : ORACLE_SID=racdb1 linux2 : ORACLE_SID=racdb2 linux3 : ORACLE_SID=racdb3

Login to the new Oracle RAC node as the oracle user account:
[root@linux3 ~]# su - oracle .bash_profile for Oracle User # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi alias ls="ls -FA" alias s="screen -DRRS iPad -t iPad" export JAVA_HOME=/usr/local/java # User export export export export export specific environment and startup programs ORACLE_BASE=/u01/app/oracle ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1 ORA_CRS_HOME=/u01/app/crs ORACLE_PATH=$ORACLE_BASE/dba_scripts/sql:.:$ORACLE_HOME/rdbms/admin CV_JDKHOME=/usr/local/java

# Each RAC node must have a unique ORACLE_SID. (i.e. racdb1, racdb2, racdb3...) export ORACLE_SID=racdb3 export PATH=.:${JAVA_HOME}/bin:$JAVA_HOME/db/bin:${PATH}:$HOME/bin: $ORACLE_HOME/bin export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin export PATH=${PATH}:$ORACLE_BASE/dba_scripts/bin export ORACLE_TERM=xterm export TNS_ADMIN=$ORACLE_HOME/network/admin export ORA_NLS10=$ORACLE_HOME/nls/data export NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"

export export export export export export export export export export export export export

LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib CLASSPATH=$ORACLE_HOME/JRE CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/oc4j/ant/lib/ant.jar CLASSPATH=${CLASSPATH}:$ORACLE_HOME/oc4j/ant/lib/ant-launcher.jar CLASSPATH=${CLASSPATH}:$JAVA_HOME/db/lib/derby.jar CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib THREADS_FLAG=native TEMP=/tmp TMPDIR=/tmp

Configure the Linux Server for Oracle

Perform the following tasks on the new Oracle RAC node!

The kernel parameters and shell limits discussed in this section will need to be defined on the new Oracle RAC node every time the machine is booted. This section provides information about setting those kernel parameters required for Oracle. Instructions for placing them in a startup script (/etc/sysctl.conf) are included in the section "All Startup Commands for New Oracle RAC Node".

Overview This section focuses on configuring the new Oracle RAC Linux server - getting it prepared for the Oracle RAC 10g installation. This includes verifying enough swap space, setting shared memory and semaphores, setting the maximum number of file handles, setting the IP local port range, setting shell limits for the oracle user, activating all kernel parameters for the system, and finally how to verify the correct date and time for all nodes in the cluster. There are several different ways to configure (set) these parameters. For the purpose of this article, I will be making all changes permanent (through reboots) by placing all commands in the /etc/sysctl.conf file.

Swap Space Considerations Installing Oracle Database 10g Release 2 on RHEL/OL 5 requires a minimum of 1024MB of memory. (Note: An inadequate amount of swap during the installation will cause the Oracle Universal Installer to either "hang" or "die") To check the amount of memory you have, type:
[root@linux3 ~]# cat /proc/meminfo | grep MemTotal MemTotal: 2074068 kB

To check the amount of swap you have allocated, type:

[root@linux3 ~]# cat /proc/meminfo | grep SwapTotal SwapTotal: 4128760 kB

If you have less than 2048MB of memory (between your RAM and SWAP), you can add temporary swap space by creating a temporary swap file. This way you do not have to use a raw device or even more drastic, rebuild your system. As root, make a file that will act as additional swap space, let's say about 500MB:
[root@linux3 ~]# dd if=/dev/zero of=tempswap bs=1k count=500000

Now we should change the file permissions:

[root@linux3 ~]# chmod 600 tempswap

Finally we format the "partition" as swap and add it to the swap space:
[root@linux3 ~]# mke2fs tempswap [root@linux3 ~]# mkswap tempswap [root@linux3 ~]# swapon tempswap

Configuring Kernel Parameters and Shell Limits The kernel parameters and shell limits presented in this section are recommended values only as documented by Oracle. For production database systems, Oracle recommends that you tune these values to optimize the performance of the system. On the new Oracle RAC node, verify that the kernel parameters described in this section are set to values greater than or equal to the recommended values. Also note that when setting the four semaphore values that all four values need to be entered on one line.

Configuring Kernel Parameters Oracle Database 10g Release 2 on RHEL/OL 5 requires the kernel parameter settings shown below. The values given are minimums, so if your system uses a larger value, do not change it:
kernel.shmmax = 4294967295 kernel.shmall = 268435456 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000 net.core.rmem_default=1048576 net.core.rmem_max=1048576 net.core.wmem_default=262144 net.core.wmem_max=262144

RHEL/OL 5 already comes configured with default values defined for the following kernel parameters:
kernel.shmall kernel.shmmax

Use the default values if they are the same or larger than the required values. This article assumes a fresh new install of CentOS 5 and as such, many of the required kernel parameters are already set (see above). This being the case, you can simply copy / paste the following to the new Oracle RAC node while logged in as root:
[root@linux3 ~]# cat >> /etc/sysctl.conf <<EOF kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000 EOF

The above command persisted the required kernel parameters through reboots by inserting them in the /etc/sysctl.conf startup file. Linux allows modification of these kernel parameters to the current system while it is up and running, so there's no need to reboot the system after making kernel parameter changes. To activate the new kernel parameter values for the currently running system, run the following as root on the new Oracle RAC node:
[root@linux3 ~]# sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 4294967295 kernel.shmall = 268435456 net.core.rmem_default = 1048576 net.core.rmem_max = 1048576 net.core.wmem_default = 262144 net.core.wmem_max = 262144 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000

Verify the new kernel parameter values by running the following on the new Oracle RAC node:
[root@linux3 ~]# /sbin/sysctl -a | grep shm vm.hugetlb_shm_group = 0 kernel.shmmni = 4096 kernel.shmall = 268435456 kernel.shmmax = 4294967295 [root@linux3 ~]# /sbin/sysctl -a | grep sem kernel.sem = 250 32000 100 128 [root@linux3 ~]# /sbin/sysctl -a | grep file-max fs.file-max = 65536 [root@linux3 ~]# /sbin/sysctl -a | grep ip_local_port_range net.ipv4.ip_local_port_range = 1024 65000 [root@linux3 ~]# /sbin/sysctl -a | grep 'core\.[rw]mem' net.core.rmem_default = 1048576 net.core.wmem_default = 262144 net.core.rmem_max = 1048576 net.core.wmem_max = 262144

Setting Shell Limits for the oracle User To improve the performance of the software on Linux systems, Oracle recommends you increase the following shell limits for the oracle user:
Shell Limit Maximum number of open file descriptors Item in limits.conf nofile Hard Limit 65536

Maximum number of processes available to a nproc single user

16384

To make these changes, run the following as root:

[root@linux3 ~]# cat >> /etc/security/limits.conf <<EOF oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 EOF [root@linux3 ~]# cat >> /etc/pam.d/login <<EOF session required /lib/security/pam_limits.so EOF

Update the default shell startup file for the "oracle" UNIX account.

For the Bourne, Bash, or Korn shell, add the following lines to the /etc/profile file by running the following command:
[root@linux3 ~]# cat >> /etc/profile <<EOF if [ \$USER = "oracle" ]; then if [ \$SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi EOF

For the C shell (csh or tcsh), add the following lines to the /etc/csh.login file by running the following command:
[root@linux3 ~]# cat >> /etc/csh.login <<EOF if ( \$USER == "oracle" ) then limit maxproc 16384 limit descriptors 65536 endif EOF

Setting the Correct Date and Time on the new Oracle RAC Node When adding the new Oracle RAC node to the cluster, the Oracle Universal Installer (OUI) copies the Oracle Clusterware and Oracle Database software from the source

RAC node (linux1 in this article) to the new node in the cluster (linux3). During the remote copy process, the OUI will execute the UNIX "tar" command on the remote node (linux3) to extract the files that were archived and copied over. If the date and time on the node performing the install is greater than that of the node it is copying to, the OUI will throw an error from the "tar" command indicating it is attempting to extract files stamped with a time in the future:
Error while copying directory /u01/app/crs with exclude file list 'null' to nodes 'linux3'. [PRKC-1002 : All the submitted commands did not execute successfully] --------------------------------------------linux3: /bin/tar: ./bin/lsnodes: time stamp 2009-09-02 23:07:04 is 735 s in the future /bin/tar: ./bin/olsnodes: time stamp 2009-09-02 23:07:04 is 735 s in the future ...(more errors on this node)

Please note that although this would seem like a severe error from the OUI, it can safely be disregarded as a warning. The "tar" command DOES actually extract the files; however, when you perform a listing of the files (using ls -l) on the remote node (the new Oracle RAC node), they will be missing the time field until the time on the remote server is greater than the timestamp of the file. Before attempting to add the new node, ensure that all nodes in the cluster are set as closely as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server. Accessing a Network Time Protocol server, however, may not always be an option. In this case, when manually setting the date and time for the nodes in the cluster, ensure that the date and time of the node you are performing the software installations from (linux1) is less than the new node being added to the cluster (linux3). I generally use a 20 second difference as shown in the following example: Show the date and time from linux1:
[root@linux3 ~]# date Wed Sep 2 23:09:00 EDT 2009

Setting the date and time on the new Oracle RAC node linux3:
[root@linux3 ~]# date -s "9/2/2009 23:09:20"

The RAC configuration described in this article does not make use of a Network Time Protocol server.

Configure the "hangcheck-timer" Kernel Module

Perform the following tasks on the new Oracle RAC node!

Oracle 9.0.1 and 9.2.0.1 used a userspace watchdog daemon called watchdogd to monitor the health of the cluster and to restart a RAC node in case of a failure. Starting with Oracle 9.2.0.2 (and still available in Oracle10g Release 2), the watchdog daemon has been deprecated by a Linux kernel module named hangcheck-timer which addresses availability and reliability problems much better. The hang-check timer is loaded into the Linux kernel and checks if the system hangs. It will set a timer and check the timer after a certain amount of time. There is a configurable threshold to hang-check that, if exceeded will reboot the machine. Although the hangchecktimer module is not required for Oracle Clusterware (Cluster Manager) operation, it is highly recommended by Oracle. The hangcheck-timer.ko Module The hangcheck-timer module uses a kernel-based timer that periodically checks the system task scheduler to catch delays in order to determine the health of the system. If the system hangs or pauses, the timer resets the node. The hangcheck-timer module uses theTime Stamp Counter (TSC) CPU register which is a counter that is incremented at each clock signal. The TCS offers much more accurate time measurements since this register is updated by the hardware automatically. Much more information about the
hangcheck-timer

project can be found here.

Installing the hangcheck-timer.ko Module The hangcheck-timer was normally shipped only by Oracle, however, this module is now included with Red Hat Linux AS starting with kernel versions 2.4.9-e.12 and

higher. The hangcheck-timer should already be included. Use the following to ensure that you have the module included:
[root@linux3 ~]# find /lib/modules -name "hangcheck-timer.ko" /lib/modules/2.6.18-128.el5/kernel/drivers/char/hangcheck-timer.ko

In the above output, we care about the hangcheck timer object (hangcheck-timer.ko) in the /lib/modules/2.6.18-128.el5/kernel/drivers/char directory. Configuring and Loading the hangcheck-timer Module There are two key parameters to the hangcheck-timer module:
hangcheck-tick:

This parameter defines the period of time between checks of system health. The default value is 60 seconds; Oracle recommends setting it to 30 seconds. hangcheck-margin: This parameter defines the maximum hang delay that should be tolerated before hangcheck-timer resets the RAC node. It defines the margin of error in seconds. The default value is 180 seconds; Oracle recommends setting it to 180 seconds.
The two hangcheck-timer module parameters indicate how long a RAC node must hang before it will reset the system. A node reset will occur when the following is true:
system hang time > (hangcheck_tick + hangcheck_margin)

Configuring Hangcheck Kernel Module Parameters Each time the hangcheck-timer kernel module is loaded (manually or by Oracle) it needs to know what value to use for each of the two parameters we just discussed: (hangcheck-tick and hangcheck-margin). These values need to be available after each reboot of the Linux server. To do this, make an entry with the correct values to the /etc/modprobe.conf file as follows:
[root@linux3 ~]# echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf

Each time the hangcheck-timer kernel module gets loaded, it will use the values defined by the entry I made in the/etc/modprobe.conf file. Manually Loading the Hangcheck Kernel Module for Testing Oracle is responsible for loading the hangcheck-timer kernel module when required. It is for this reason that it is not required to perform a modprobe or insmod of the hangcheck-timer kernel module in any of the startup files (i.e. /etc/rc.local). It is only out of pure habit that I continue to include a modprobe of the hangchecktimer kernel module in the/etc/rc.local file. Someday I will get over it, but realize that it does not hurt to include a modprobe of the hangcheck-timer kernel module during startup. So to keep myself sane and able to sleep at night, I always configure the loading of the hangcheck-timer kernel module on each startup as follows:
[root@linux3 ~]# echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local

You don't have to manually load the hangcheck-timer kernel module using modprobe orinsmod after each reboot. The hangcheck-timer module will be loaded by Oracle (automatically) when needed.

Now, to test the hangcheck-timer kernel module to verify it is picking up the correct parameters we defined in the/etc/modprobe.conf file, use the modprobe command. Although you could load the hangcheck-timer kernel module by passing it the appropriate parameters (e.g. insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180), we want to verify that it is picking up the options we set in the /etc/modprobe.conf file. To manually load the hangcheck-timer kernel module and verify it is using the correct values defined in the/etc/modprobe.conf file, run the following command:
[root@linux3 ~]# modprobe hangcheck-timer [root@linux3 ~]# grep Hangcheck /var/log/messages | tail -2 Sep 2 23:32:00 linux3 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds). Sep 2 23:32:00 linux3 kernel: Hangcheck: Using get_cycles().

Configure RAC Nodes for Remote Access using SSH

Perform the following configuration procedures on linux1 and the new Oracle RAC node!

During the creation of the existing two-node cluster, the installation of Oracle Clusterware and the Oracle Database software were only performed from one node in the RAC cluster namely from linux1 as the oracle user account. The Oracle Universal Installer (OUI) on that particular node would then use the ssh and scp commands to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. Theoracle user account on the node running the OUI (runInstaller) had to be trusted by all other nodes in the RAC cluster. This meant that theoracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI against all other Linux servers in the cluster without being prompted for a password. The same security requirements hold true for this article. User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securely copied from linux1 to the new Oracle RAC node (linux3) using ssh and scpwithout being prompted for a password. As was the case when configuring the existing two-node cluster, this article assumes the Oracle software installation to the new Oracle RAC node will be performed from linux1. This section provides the methods required for configuring SSH1, an RSA key, and user equivalence for the new Oracle RAC node. Configuring the Secure Shell To determine if SSH is installed and running on the new Oracle RAC node, enter the following command:
[root@linux3 ~]# pgrep sshd 3695

If SSH is running, then the response to this command is a list of process ID number(s). Creating the RSA Keys on the new Oracle RAC Node

The first step in configuring SSH is to create an RSA public/private key pair on the new Oracle RAC node. An RSA public/private key should already exist on both of the two nodes in the current two-node cluster. The command to do this will create a public and private key for RSA (for a total of two keys per node). The content of the RSA public keys will then be copied into an authorized key file onlinux1 which is then distributed to all other Oracle RAC nodes in the cluster. Use the following steps to create the RSA key pair from the new Oracle RAC node (linux3); 1. Log on as the "oracle" UNIX user account.
[root@linux3 ~]# su - oracle

2.

If necessary, create the .ssh directory in the "oracle" user's home directory and set the correct permissions on it:

3. [oracle@linux3 ~]$ mkdir -p ~/.ssh [oracle@linux3 ~]$ chmod 700 ~/.ssh

4. Enter the following command to generate an RSA key pair (public and private key) for the SSH protocol:
[oracle@linux3 ~]$ /usr/bin/ssh-keygen -t rsa

At the prompts:
o o

Accept the default location for the key files. Enter and confirm a pass phrase. This should be different from the "oracle" UNIX user account password however it is not a requirement.

This command will write the public key to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file. Note that you should never distribute the private key to anyone! Updating and Distributing the "authorized key file" from linux1 Now that the new Oracle RAC node contains a public and private key for RSA, you will need to update the authorized key file onlinux1 to add (append) the new RSA public key from linux3. An authorized key file is nothing more than a single file that contains a copy of everyone's (every node's) RSA public key. Once the authorized key

file contains all of the public keys, it is then distributed to all other nodes in the cluster. Complete the following steps on linux1 to update and then distribute the authorized key file to all nodes in the Oracle RAC cluster.
1.

As the "oracle" UNIX user account, verify that an authorized key file currently exists on the node linux1(~/.ssh/authorized_keys). The authorized key file should already exist from the initial installation of Oracle RAC.
2 01:18 id_rsa.pub

2. [oracle@linux1 ~]$ cd ~/.ssh 3. [oracle@linux1 ~]$ ls -l *.pub -rw-r--r-- 1 oracle oinstall 223 Sep

4.

In this step, use SCP (Secure Copy) or SFTP (Secure FTP) to copy the content of the ~/.ssh/id_rsa.pub public key from the new Oracle RAC node to the authorized key file on linux1. You will be prompted for the oracle UNIX user account password for the new Oracle RAC node. Again, this task will be performed from linux1.
[oracle@linux1 ~]$ ssh linux3 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys The authenticity of host 'linux3 (192.168.1.107)' can't be established. RSA key fingerprint is f5:38:37:e8:84:4e:bd:6d:6b:25:f7:94:58:e8:b2:7a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'linux3,192.168.1.107' (RSA) to the list of known hosts. oracle@linux3's password: xxxxx

The first time you use SSH to connect to a node from a particular system, you will see a message similar to the following:
The authenticity of host 'linux3 (192.168.1.107)' can't be established. RSA key fingerprint is f5:38:37:e8:84:4e:bd:6d:6b:25:f7:94:58:e8:b2:7a. Are you sure you want to continue connecting (yes/no)? yes Enter yes at the prompt to continue. You should not see this message

again when you connect from this system to the same node.

5.

At this point, we have the RSA public key from every node in the cluster (including the new Oracle RAC node) in the authorized key file on linux1. We now need to copy it to all other nodes in the cluster. Use the scp command from linux1 to

copy the authorized key file to all remaining nodes in the RAC cluster:
6. [oracle@linux1 ~]$ scp ~/.ssh/authorized_keys linux2:.ssh/authorized_keys 7. Enter passphrase for key '/home/oracle/.ssh/id_rsa': xxxxx 8. authorized_keys 100% 1191 1.2KB/s 00:00 9. 10. [oracle@linux1 ~]$ scp ~/.ssh/authorized_keys linux3:.ssh/authorized_keys 11. oracle@linux3's password: xxxxx authorized_keys 100% 1191 1.2KB/s 00:00

12. Change

the permission of the authorized key file on all Oracle RAC nodes in the cluster by logging into the node and running the following:

13. [oracle@linux1 ~]$ chmod 600 ~/.ssh/authorized_keys 14. [oracle@linux2 ~]$ chmod 600 ~/.ssh/authorized_keys [oracle@linux3 ~]$ chmod 600 ~/.ssh/authorized_keys

15. At

this point, if you use ssh to log in to or run a command on the new Oracle RAC node, you are prompted for the pass phrase that you specified when you created the RSA key. For example, test the following from linux1:

16. [oracle@linux1 ~]$ ssh linux3 hostname 17. Enter passphrase for key '/home/oracle/.ssh/id_rsa': xxxxx linux3

If you see any other messages or text, apart from the host name, then the Oracle installation can fail. Make any changes required to ensure that only the host name is displayed when you enter these commands. You should ensure that any part of a login script(s) that generate any output, or ask any questions, are modified so that they act only when the shell is an interactive shell.

Enabling SSH User Equivalency for the Current Shell Session When running the addNode.sh script from linux1 (which runs the OUI), it will need to run the secure shell tool commands (ssh andscp) on the new Oracle RAC node without being prompted for a pass phrase. Even though SSH is now configured on all Oracle RAC nodes in the cluster, using the secure shell tool commands will still prompt for a pass phrase. Before running the addNode.sh script, you need to enable user equivalence for the terminal session you plan to run the script from. For the purpose of this article, the addNode.shscript will be run from linux1.

User equivalence will need to be enabled on any new terminal shell session on linux1 before attempting to run the addNode.sh script. If you log out and log back in to the node you will be performing the Oracle installation from, you must enable user equivalence for the terminal shell session as this is not done by default. To enable user equivalence for the current terminal shell session, perform the following steps:
1.

Log on to the node where you want to run the addNode.sh script from (linux1) as the "oracle" UNIX user account.
[root@linux1 ~]# su - oracle

2. Enter the following commands:

3. [oracle@linux1 ~]$ exec /usr/bin/ssh-agent $SHELL 4. [oracle@linux1 ~]$ /usr/bin/ssh-add 5. Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

At the prompt, enter the pass phrase for each key that you generated.
6.

If SSH is configured correctly, you will be able to use the ssh and scp commands without being prompted for a password or pass phrase from this terminal session:

7. [oracle@linux1 ~]$ ssh linux1 "date;hostname" 8. Thu Sep 3 00:02:34 EDT 2009 9. linux1 10. 11. [oracle@linux1 ~]$ ssh linux2 "date;hostname" 12. Thu Sep 3 00:03:04 EDT 2009 13. linux2 14. 15. [oracle@linux1 ~]$ ssh linux3 "date;hostname" 16. Thu Sep 3 00:03:25 EDT 2009 linux3

The commands above should display the date set on each Oracle RAC node along with its hostname. If any of the nodes prompt for a password or pass phrase then verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys.

Also, if you see any other messages or text, apart from the date and hostname, then the Oracle installation can fail. Make any changes required to ensure that only the date is displayed when you enter these commands. You should ensure that any part of a login script(s) that generate any output, or ask any questions, are modified so that they act only when the shell is an interactive shell.

17. The

Oracle Universal Installer is a GUI interface and requires the use of an X Server. From the terminal session enabled for user equivalence (the node you will be running the addNode.sh script from), set the environment variable DISPLAY to a valid X Windows display: Bourne, Korn, and Bash shells:
[oracle@linux1 ~]$ DISPLAY=<Any X-Windows Host>:0 [oracle@linux1 ~]$ export DISPLAY

C shell:
[oracle@linux1 ~]$ setenv DISPLAY <Any X-Windows Host>:0

After setting the DISPLAY variable to a valid X Windows display, you should perform another test of the current terminal session to ensure that X11 forwarding is not enabled:
[oracle@linux1 ~]$ ssh linux1 hostname linux1 [oracle@linux1 ~]$ ssh linux2 hostname linux2 [oracle@linux1 ~]$ ssh linux3 hostname linux3

18. You

must run the addNode.sh script from this terminal session or remember to repeat the steps to enable user equivalence (steps 2, 3, and 4 from this section) before you start the Oracle Universal Installer from a different terminal session.

For more information on configuring SSH and user equivalence in an Oracle RAC 10g environment, see the section "Configure RAC Nodes for Remote Access using SSH" in the parent article.

All Startup Commands for New Oracle RAC Node

Verify that the following startup commands are included on the new Oracle RAC node!

Up to this point, we have talked in great detail about the parameters and resources that need to be configured on the new Oracle RAC node for the Oracle RAC 10g configuration. This section will review those parameters, commands, and entries (in previous sections of this document) that need to occur on the new Oracle RAC node when it is booted. In this section, I provide all of the commands, parameters, and entries that have been discussed so far that will need to be included in the startup scripts for the new Oracle RAC node. For each of the startup files below, I indicate in blue the entries that should be included in each of the startup files in order to provide a successful RAC node.
/etc/modprobe.conf

All parameters and values to be used by kernel modules.

/etc/modprobe.conf alias eth0 r8169 alias eth1 e1000 alias scsi_hostadapter ata_piix alias snd-card-0 snd-intel8x0 options snd-card-0 index=0 options snd-intel8x0 index=0 remove snd-intel8x0 { /usr/sbin/alsactl store 0 >/dev/null 2>&1 || : ; }; /sbin/modprobe -r --ignore-remove snd-intel8x0 options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

/etc/sysctl.conf

We wanted to adjust the default and maximum send buffer size as well as the default and maximum receive buffer size for the interconnect. This file also contains those parameters responsible for configuring shared memory, semaphores, file handles, and local IP range for use by the Oracle instance.
/etc/sysctl.conf

# Kernel sysctl configuration file for Red Hat Linux # # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and # sysctl.conf(5) for more details. # Controls IP packet forwarding net.ipv4.ip_forward = 0 # Controls source route verification net.ipv4.conf.default.rp_filter = 1 # Do not accept source routing net.ipv4.conf.default.accept_source_route = 0 # Controls the System Request debugging functionality of the kernel kernel.sysrq = 0 # Controls whether core dumps will append the PID to the core filename # Useful for debugging multi-threaded applications kernel.core_uses_pid = 1 # Controls the use of TCP syncookies net.ipv4.tcp_syncookies = 1 # Controls the maximum size of a message, in bytes kernel.msgmnb = 65536 # Controls the default maxmimum size of a mesage queue kernel.msgmax = 65536 # # # # # # # # # # # # # # # # # # # # +---------------------------------------------------------+ | ADJUSTING NETWORK SETTINGS | +---------------------------------------------------------+ | With Oracle 9.2.0.1 and onwards, Oracle now makes use | | of UDP as the default protocol on Linux for | | inter-process communication (IPC), such as Cache Fusion | | and Cluster Manager buffer transfers between instances | | within the RAC cluster. Oracle strongly suggests to | | adjust the default and maximum receive buffer size | | (SO_RCVBUF socket option) to 1024MB, and the default | | and maximum send buffer size (SO_SNDBUF socket option) | | to 256KB. The receive buffers are used by TCP and UDP | | to hold received data until it is read by the | | application. The receive buffer cannot overflow because | | the peer is not allowed to send data beyond the buffer | | size window. This means that datagrams will be | | discarded if they don't fit in the socket receive | | buffer. This could cause the sender to overwhelm the | | receiver. | +---------------------------------------------------------+

# +---------------------------------------------------------+ # | Default setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. | # +---------------------------------------------------------+ net.core.rmem_default=1048576

# +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. | # +---------------------------------------------------------+ net.core.rmem_max=1048576 # +---------------------------------------------------------+ # | Default setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.core.wmem_default=262144 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.core.wmem_max=262144 # # # # # # # # +---------------------------------------------------------+ | ADJUSTING ADDITIONAL KERNEL PARAMETERS FOR ORACLE | +---------------------------------------------------------+ | Configure the kernel parameters for all Oracle Linux | | servers by setting shared memory and semaphores, | | setting the maximum amount of file handles, and setting | | the IP local port range. | +---------------------------------------------------------+

# +---------------------------------------------------------+ # | SHARED MEMORY | # +---------------------------------------------------------+ # Controls the maximum shared segment size, in bytes kernel.shmmax = 4294967295 # Controls the maximum number of shared memory segments, in pages kernel.shmall = 268435456 # Controls the maximum number of shared memory segments system wide kernel.shmmni = 4096 # +---------------------------------------------------------+ # | SEMAPHORES | # | ---------| # | | # | SEMMSL_value SEMMNS_value SEMOPM_value SEMMNI_value | # | | # +---------------------------------------------------------+ kernel.sem=250 32000 100 128 # +---------------------------------------------------------+ # | FILE HANDLES | # ----------------------------------------------------------+ fs.file-max=65536 # +---------------------------------------------------------+ # | LOCAL IP RANGE |

# ----------------------------------------------------------+ net.ipv4.ip_local_port_range=1024 65000

Verify that each of the required kernel parameters (above) are configured in the/etc/sysctl.conf file. Then, ensure that each of these parameters are truly in effect by running the following command on the new Oracle RAC node:
[root@linux3 ~]# sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 net.core.rmem_default = 1048576 net.core.rmem_max = 1048576 net.core.wmem_default = 262144 net.core.wmem_max = 262144 kernel.shmmax = 4294967295 kernel.shmall = 268435456 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000

/etc/hosts

All machine/IP entries for nodes in the RAC cluster.

/etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 # Public Network 192.168.1.100 192.168.1.101 192.168.1.107 localhost.localdomain - (eth0) linux1 linux2 linux3 localhost

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv 192.168.2.107 linux3-priv # Public Virtual IP (VIP) addresses - (eth0:1) 192.168.1.200 linux1-vip

192.168.1.201 192.168.1.207

linux2-vip linux3-vip

# Private Storage Network for Openfiler - (eth1) 192.168.1.195 openfiler1 192.168.2.195 openfiler1-priv # Miscellaneous Nodes 192.168.1.1 router 192.168.1.102 alex 192.168.1.103 nascar 192.168.1.105 packmule 192.168.1.106 melody 192.168.1.120 cartman 192.168.1.121 domo 192.168.1.122 switch1 192.168.1.190 george 192.168.1.245 accesspoint

/etc/udev/rules.d/55-openiscsi.rules /etc/udev/rules.d/55-openiscsi.rules # /etc/udev/rules.d/55-openiscsi.rules KERNEL=="sd*", BUS=="scsi", PROGRAM="/etc/udev/scripts/iscsidev.sh %b",SYMLINK+="iscsi/%c/part%n"

/etc/udev/scripts/iscsidev.sh /etc/udev/scripts/iscsidev.sh #!/bin/sh # FILE: /etc/udev/scripts/iscsidev.sh BUS=${1} HOST=${BUS%%:*} [ -e /sys/class/iscsi_host ] || exit 1 file="/sys/class/iscsi_host/host$ {HOST}/device/session*/iscsi_session*/targetname" target_name=$(cat ${file}) # This is not an open-scsi drive if [ -z "${target_name}" ]; then exit 1 fi

echo "${target_name##*.}"

/etc/rc.local

Loading the hangcheck-timer kernel module.

/etc/rc.local #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local # # # # +---------------------------------------------------------+ | HANGCHECK TIMER | | (I do not believe this is required, but doesn't hurt) | +---------------------------------------------------------+

/sbin/modprobe hangcheck-timer

Install and Configure Oracle Cluster File System (OCFS2)

Perform the following tasks on the new Oracle RAC node!

Overview The current two-node Oracle RAC database makes use of the Oracle Cluster File System, Release 2 (OCFS2) to store the two files that are required to be shared by the Oracle Clusterware software. Note that for each of the two shared Oracle Clusterware shared files, a mirrored copy was created making for five files in total:

Oracle Cluster Registry (OCR) o File 1 : /u02/oradata/racdb/OCRFile

File 2 : /u02/oradata/racdb/OCRFile_mirror CRS Voting Disk o File 1 : /u02/oradata/racdb/CSSFile o File 2 : /u02/oradata/racdb/CSSFile_mirror1 o File 3 : /u02/oradata/racdb/CSSFile_mirror2
o

Along with these two groups of files (the OCR and Voting disk), we also used this space to store the shared ASM SPFILE for all Oracle RAC instances. In this section, we will download and install the release of OCFS2 used for the current two-node cluster on the new Oracle RAC node. See the following page for more information on OCFS2 (including Installation Notes) for Linux: OCFS2 Project Documentation Download OCFS2 First, let's download the same OCFS2 distribution used for the current two-node RAC. The OCFS2 distribution comprises of two sets of RPMs; namely, the kernel module and the tools. The latest kernel module is available for download fromhttp://oss.oracle.com/projects/ocfs2/files/ and the tools from http://oss.oracle.com/projects/ocfs2-tools/files/. Download the appropriate RPMs starting with the same OCFS2 kernel module (the driver) used for the current two-node RAC. With CentOS 5.3, I am using kernel release 2.6.18-128.el5. The appropriate OCFS2 kernel module was found in the latest release of OCFS2 at the time of this writing (OCFS2 Release 1.4.2-1 at the time of this writing). The available OCFS2 kernel modules for Linux kernel 2.6.18-128.el5 are listed below. Always download the OCFS2 kernel module that matches the distribution, platform, kernel version and the kernel flavor (smp, hugemem, psmp, etc). 32-bit (x86) Installations ocfs2-2.6.18-128.el5-1.4.2-1.el5.i686.rpm - (Package for default kernel) ocfs2-2.6.18-128.el5PAE-1.4.2-1.el5.i686.rpm - (Package for PAE

kernel) ocfs2-2.6.18-128.el5xen-1.4.2-1.el5.i686.rpm - (Package for xen kernel) Next, download both the OCFS2 tools and the OCFS2 console applications: ocfs2-tools-1.4.2-1.el5.i386.rpm - (OCFS2 tools) ocfs2console-1.4.2-1.el5.i386.rpm - (OCFS2 console) 64-bit (x86_64) Installations ocfs2-2.6.18-128.el5-1.4.2-1.el5.x86_64.rpm - (Package for default kernel) ocfs2-2.6.18-128.el5xen-1.4.2-1.el5.x86_64.rpm - (Package for xen kernel) Next, download both the OCFS2 tools and the OCFS2 console applications: ocfs2-tools-1.4.2-1.el5.x86_64.rpm - (OCFS2 tools) ocfs2console-1.4.2-1.el5.x86_64.rpm - (OCFS2 console)
The OCFS2 Console is optional but highly recommended. The ocfs2consoleapplication requires e2fsprogs, glib2 2.2.3 or later, vte 0.11.10 or later, pygtk2 (EL4) or python-gtk (SLES9) 1.99.16 or later, python 2.3 or later and ocfs2-tools. If you were curious as to which OCFS2 driver release you need, use the OCFS2 release that matches your kernel version. To determine your kernel release:
[root@linux3 ~]# uname -a Linux linux3 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux

Install OCFS2 I will be installing the OCFS2 files onto a single processor / x86 machines. The installation process is simply a matter of running the following command on the new Oracle RAC node as the root user account:
[root@linux3 ~]# rpm -Uvh ocfs2-2.6.18-128.el5-1.4.2-1.el5.i686.rpm \

ocfs2console-1.4.2-1.el5.i386.rpm \ ocfs2-tools-1.4.2-1.el5.i386.rpm Preparing... ########################################### 1:ocfs2-tools ########################################### 2:ocfs2-2.6.18-128.el5 ########################################### 3:ocfs2console ###########################################

[100%] [ 33%] [ 67%] [100%]

Disable SELinux (RHEL4 U2 and higher) Users of RHEL4 U2 and higher (CentOS 5.3 is based on RHEL5 U3) are advised that OCFS2 currently does not work with SELinux enabled. If you are using RHEL4 U2 or higher (which includes us since we are using CentOS 5.3) you will need to disable SELinux (using tool system-config-securitylevel) to get the O2CB service to execute. During the installation of CentOS, we Disabled SELinux on the SELinux screen. If, however, you did not disable SELinux during the installation phase, you can use the tool system-config-securitylevel to disable SELinux. If you did not follow the instructions to disable the SELinux option during the installation of CentOS (or if you simply want to verify it is truly disable), run the "Security Level Configuration" GUI utility:
[root@linux3 ~]# /usr/bin/system-config-securitylevel &

This will bring up the following screen:

Figure 11: Security Level Configuration Opening Screen / Firewall Disabled Now, click the SELinux tab and select the "Disabled" option. After clicking [OK], you will be presented with a warning dialog. Simply acknowledge this warning by clicking "Yes". Your screen should now look like the following after disabling the SELinux option:

Figure 12: SELinux Disabled If you needed to disable SELinux in this section on the new Oracle RAC node, it will need to be rebooted to implement the change. SELinux must be disabled before you can continue with configuring OCFS2! Configure OCFS2 The next step is to generate and configure the /etc/ocfs2/cluster.conf file on the new Oracle RAC node. The easiest way to accomplish this is to run the GUI tool ocfs2console. The /etc/ocfs2/cluster.conf file will contain hostnames and IP addresses for "all" nodes in the cluster. After creating the /etc/ocfs2/cluster.conf on the new Oracle RAC node, these changes will then be distributed to the other two current RAC nodes using the o2cb_ctl command-line utility. In this section, we will not only create and configure the /etc/ocfs2/cluster.conf file using ocfs2console, but will also create and start the cluster stack O2CB. When the /etc/ocfs2/cluster.conf file is not present, (as will be the case in our example), theocfs2console tool will create this file along with a new cluster stack service (O2CB) with a default cluster name of ocfs2.

OCFS2 will be configured to use the private network (192.168.2.0) for all of its network traffic as recommended by Oracle. While OCFS2 does not take much bandwidth, it does require the nodes to be alive on the network and sends regular keepalive packets to ensure that they are. To avoid a network delay being interpreted as a node disappearing on the net which could lead to a node-selffencing, a private interconnect is recommended. It is safe to use the same private interconnect for both Oracle RAC and OCFS2. A popular question then is what node name should be used when adding nodes to an OCFS2 cluster and should it be related to the IP address? When adding nodes to an OCFS2 cluster, the node name being entered must match the hostname of the machine or the OCFS2 console will throw an error. The IP address, however, need not be the one associated with that hostname. In other words, any valid IP address on that node can be used. OCFS2 will not attempt to match the node name (hostname) with the specified IP address.
[root@linux3 ~]# ocfs2console &

This will bring up the GUI as shown below:

Figure 13: ocfs2console Screen Using the ocfs2console GUI tool, perform the following steps: Select [Cluster] -> [Configure Nodes...]. This will start the OCFS2 Cluster Stack (Figure 14). Acknowledge this Information dialog box by clicking [Close]. You will then be presented with the "Node Configuration" dialog. 2. On the "Node Configuration" dialog, click the [Add] button. o This will bring up the "Add Node" dialog. o In the "Add Node" dialog, enter the Host name and IP address for the first node in the cluster. Leave the IP Port set to its default value of 7777. In my example, I added all three nodes using linux1 / 192.168.2.100 for the first node, linux2 /192.168.2.101 for the second node and linux3 / 192.168.2.107 for the third node. Note: The node name you enter "must" match the
1.

hostname of the machine and the IP addresses will use the privateinterconnect. o Click [Apply] on the "Node Configuration" dialog - All nodes should now be "Active" as shown in Figure 15. o Click [Close] on the "Node Configuration" dialog. 3. After verifying all values are correct, exit the application using [File] -> [Quit]. These steps need only be performed on the new Oracle RAC node.

Figure 14: Starting the OCFS2 Cluster Stack The following dialog shows the OCFS2 settings I used when configuring the new Oracle RAC node:

Figure 15: Configuring Nodes for OCFS2

After exiting the ocfs2console, you will have a /etc/ocfs2/cluster.conf similar to the following. In the next section, this file (along with other changes) will be distributed to the current two RAC nodes:
/etc/ocfs2/cluster.conf node:

ip_port = 7777 ip_address = 192.168.2.100 number = 0 name = linux1 cluster = ocfs2 ip_port = 7777 ip_address = 192.168.2.101 number = 1 name = linux2 cluster = ocfs2

node:

node:

ip_port = 7777 ip_address = 192.168.2.107 number = 2 name = linux3 cluster = ocfs2 node_count = 3 name = ocfs2

cluster:

Add New Oracle RAC Node to the OCFS2 Cluster The next step is to add the new Oracle RAC node (linux3) to the current "live" OCFS2 cluster. This entails running the o2cb_ctlcommand-line utility from the current two RAC nodes linux1 and linux2. As root, run the following from linux1 and then linux2:
[root@linux1 ~]# o2cb_ctl -C -i -n linux3 -t node -a number=2 -a ip_address=192.168.2.107 -a ip_port=7777 -a cluster=ocfs2 Node linux3 created [root@linux2 ~]# o2cb_ctl -C -i -n linux3 -t node -a number=2 -a ip_address=192.168.2.107 -a ip_port=7777 -a cluster=ocfs2 Node linux3 created

o2cb_ctl parameters:
-C -i : Create an object in the OCFS2 Cluster Configuration. : Valid only with -C. When creating something (node or cluster), it will also install it in the live cluster (/config). If the parameter is not specified, then only update the /etc/ocfs2/cluster.conf. : Object name which is usually the node name or cluster name. : Type can be cluster, node or heartbeat. : Attribute in the format "parameter=value" which will be set in the file /etc/ocfs2/cluster.conf file. Since nodes are numbered starting with zero, the third node in the OCFS2 cluster will be "number=2". Set the IP address which in this example will be the private interconnect "ip_address=192.168.2.107". The port number used in the current two-node cluster is "ip_port=7777". Finally, identify which OCFS2 cluster to use which in our case is named "cluster=ocfs2".

-n -t -a

Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold Next, configure the on-boot properties of the OC2B driver on the new Oracle RAC node so that the cluster stack services will start on each boot. You will also be adjusting the OCFS2 Heartbeat Threshold from its default setting of 31 to 61. Set the on-boot properties as follows:
[root@linux3 ~]# /etc/init.d/o2cb offline ocfs2 [root@linux3 ~]# /etc/init.d/o2cb unload [root@linux3 ~]# /etc/init.d/o2cb configure Configuring the O2CB driver. This will configure the on-boot properties of the O2CB driver. The following questions will determine whether the driver is loaded on boot. The current values will be shown in brackets ('[]'). Hitting without typing an answer will keep that current value. Ctrl-C will abort. Load O2CB driver on boot (y/n) [n]: y Cluster stack backing O2CB [o2cb]: o2cb Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2 Specify heartbeat dead threshold (>=7) [31]: 61 Specify network idle timeout in ms (>=5000) [30000]: 30000 Specify network keepalive delay in ms (>=1000) [2000]: 2000 Specify network reconnect delay in ms (>=2000) [2000]: 2000 Writing O2CB configuration: OK Loading filesystem "configfs": OK Mounting configfs filesystem at /sys/kernel/config: OK Loading filesystem "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting O2CB cluster ocfs2: OK

Mount the OCFS2 File System Since the clustered file system already exists, the next step is to simply mount it on the new Oracle RAC node. Let's first do it using the command-line, then I'll show how to include it in the /etc/fstab to have it mount on each boot. The current OCFS2 file system was created with the label oracrsfiles which will be used when mounting. First, here is how to manually mount the OCFS2 file system from the command-line. Remember that this needs to be performed as theroot user account on the new Oracle RAC node:
[root@linux3 ~]# mount -t ocfs2 -o datavolume,nointr -L "oracrsfiles" /u02

If the mount was successful, you will simply get your prompt back. We should, however, run the following checks to ensure the file system is mounted correctly. Let's use the mount command to ensure that the clustered file system is really mounted:
[root@linux3 ~]# mount /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/hda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) domo-san:Public on /domo type nfs (rw,addr=192.168.2.121) configfs on /sys/kernel/config type configfs (rw) ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) /dev/sdc1 on /u02 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)

Please take note of the datavolume option I am using to mount the new file system. Oracle database users must mount any volume that will contain the Voting Disk file, Cluster Registry (OCR), Data files, Redo logs, Archive logs and Control files with thedatavolume mount option so as to ensure that the Oracle processes open the files with the o_direct flag. The nointr option ensures that the I/O's are not interrupted by signals. Any other type of volume, including an Oracle home (which I will not be using for this article), should not be mounted with this mount option.
Why does it take so much time to mount the volume? It takes around 5 seconds for a volume to mount. It does so as to let the heartbeat thread stabilize. In a later release, Oracle plans to add support for a global heartbeat, which will make most mounts instant.

Configure OCFS2 to Mount Automatically at Startup This section provides the steps necessary to mount the OCFS2 file system each time the new Oracle RAC node is booted using its label. We start by adding the following line to the /etc/fstab file on the new Oracle RAC node:
LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0

Notice the "_netdev" option for mounting this file system. The _netdev mount option is a must for OCFS2 volumes. This mount option indicates that the volume is to be mounted after the network is started and dismounted before the network is shutdown.

Now, let's make sure that the ocfs2.ko kernel module is being loaded and that the file system will be mounted during the boot process. If you have been following along with the examples in this article, the actions to load the kernel module and mount the OCFS2 file system should already be enabled. However, we should still check those options by running the following as the root user account on the new Oracle RAC node:
[root@linux3 ~]# chkconfig --list o2cb o2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off

The flags that I have marked in bold should be set to "on". Check Permissions on OCFS2 File System From the new Oracle RAC node, use the ls command to check ownership. The permissions should be set to 0775 with owner "oracle" and group "oinstall".
[root@linux3 ~]# ls -ld /u02 drwxrwxr-x 6 oracle oinstall 3896 Aug 26 23:41 /u02

Verify Access to the Shared Clusterware Files From the new Oracle RAC node as the oracle user account, use the ls command to verify access to the Oracle Clusterware shared files (OCR file and Voting Disk):

[oracle@linux3 ~]$ ls -l /u02/oradata/racdb total 14820 -rw-r--r-- 1 oracle oinstall 10240000 Aug 26 -rw-r--r-- 1 oracle oinstall 10240000 Aug 26 -rw-r--r-- 1 oracle oinstall 10240000 Aug 26 drwxr-x--- 2 oracle oinstall 3896 Aug 26 -rw-r----- 1 root oinstall 5074944 Sep 2 -rw-r----- 1 root oinstall 5074944 Sep 2

22:43 22:43 22:43 23:45 14:18 14:18

CSSFile CSSFile_mirror1 CSSFile_mirror2 dbs/ OCRFile OCRFile_mirror

Install and Configure Automatic Storage Management (ASMLib 2.0)

Perform the following tasks on the new Oracle RAC node!

Introduction The current two-node Oracle RAC database makes use of Automatic Storage Management (ASM) to be used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area. In this section, we will download, install, and configure ASMLib on the new Oracle RAC node. Using this method, Oracle database files will be stored on raw block devices managed by ASM using ASMLib calls. RAW devices are not required with this method as ASMLib works with block devices. Download the ASMLib 2.0 Packages The ASMLib distribution comprises of two sets of RPMs; namely, the kernel module and the ASMLib tools. We start this section by downloading the same ASMLib distribution used for the current two-node RAC - (2.0.5-1). Like the Oracle Cluster File System, we need to download the appropriate version of the ASMLib driver for the Linux kernel which in my case is kernel 2.6.18-128.el5 running on two single processor / x86 machines:
[root@linux3 ~]# uname -a

Linux linux1 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux

If you do not currently have an account with Oracle OTN, you will need to create one. This is a FREE account!

Oracle ASMLib Downloads for Red Hat Enterprise Linux Server 5 32-bit (x86) Installations oracleasm-2.6.18-128.el5-2.0.5-1.el5.i686.rpm - (Package for default kernel) oracleasm-2.6.18-128.el5PAE-2.0.5-1.el5.i686.rpm - (Package for PAE kernel) oracleasm-2.6.18-128.el5xen-2.0.5-1.el5.i686.rpm - (Package for xen kernel) Next, download the following ASMLib tools: oracleasm-support-2.1.3-1.el5.i386.rpm - (Driver support files) oracleasmlib-2.0.4-1.el5.i386.rpm - (Userspace library) 64-bit (x86_64) Installations oracleasm-2.6.18-128.el5-2.0.5-1.el5.x86_64.rpm - (Package for default kernel) oracleasm-2.6.18-128.el5xen-2.0.5-1.el5.x86_64.rpm - (Package for xen kernel) Next, download the following ASMLib tools: oracleasm-support-2.1.3-1.el5.x86_64.rpm - (Driver support files) oracleasmlib-2.0.4-1.el5.x86_64.rpm - (Userspace library) Install ASMLib 2.0 Packages

I will be installing the ASMLib files onto the new Oracle RAC node (linux3) which is a single processor machine. The installation process is simply a matter of running the following command on the new Oracle RAC node as the root user account:
[root@linux3 ~]# rpm -Uvh oracleasm-2.6.18-128.el5-2.0.5-1.el5.i686.rpm oracleasmlib-2.0.4-1.el5.i386.rpm \ oracleasm-support-2.1.3-1.el5.i386.rpm Preparing... ########################################### 1:oracleasm-support ########################################### 2:oracleasm-2.6.18-128.el########################################### 3:oracleasmlib ########################################### \ [100%] [ 33%] [ 67%] [100%]

Configure and Loading the ASMLib 2.0 Packages After downloading and installing the ASMLib 2.0 Packages for Linux, we now need to configure and load the ASM kernel module. Run the following as root on the new Oracle RAC node:
[root@linux3 ~]# /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracle Default group to own the driver interface []: oinstall Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ]

Scan for ASM Disks From the new Oracle RAC node, you can now perform a scandisk to recognize the current volumes. Even though the above configuration automatically ran the scandisk utility, I still like to manually perform this step!
[root@linux3 ~]# /etc/init.d/oracleasm scandisks Scanning system for ASM disks [ OK ]

We can now test that the ASM disks were successfully identified using the following command as the root user account:
[root@linux3 ~]# /etc/init.d/oracleasm listdisks VOL1 VOL2 VOL3 VOL4

Pre-Installation Tasks for Oracle10g Release 2

Perform the following checks on the new Oracle RAC node and run the Oracle Cluster Verification Utility (CVU) from linux1!

Before propagating the Oracle Clusterware and Oracle RAC software to the new Oracle RAC node, it is highly recommended to run the Cluster Verification Utility (CVU) against all Oracle RAC nodes (which will include the new Oracle RAC node) to verify the hardware and software configuration. CVU is a command-line utility provided on the Oracle Clusterware installation media. It is responsible for performing various system checks to assist you with confirming the Oracle RAC nodes are properly configured for Oracle Clusterware and Oracle Real Application Clusters installation. The CVU only needs to be run from the node you will be performing the Oracle installations from (linux1 in this article). Prerequisites for Using Cluster Verification Utility Install cvuqdisk RPM (RHEL Users Only) The first pre-requisite for running the CVU pertains to users running Oracle Linux, Red Hat Linux, CentOS, and SuSE. If you are using any of the above listed operating systems, then you must download and install the package cvuqdisk to the new Oracle RAC node (linux3). Without cvuqdisk, CVU will be unable to discover shared disks and you will receive the error message "Package cvuqdisk not installed" when you run CVU. The cvuqdisk RPM can be found on the Oracle Clusterware installation media in the rpm directory. For the purpose of this article, the Oracle Clusterware media was

extracted to the /home/oracle/orainstall/clusterware directory onlinux1. Note that before installing the cvuqdisk RPM, we need to set an environment variable named CVUQDISK_GRP to point to the group that will own the cvuqdisk utility. The default group is oinstall which is the group we are using for theoracle UNIX user account in this article. Locate and copy the cvuqdisk RPM from linux1 to linux2 as the "oracle" user account:
[oracle@linux1 ~]$ ssh linux3 "mkdir -p /home/oracle/orainstall/clusterware/rpm" [oracle@linux1 ~]$ scp /home/oracle/orainstall/clusterware/rpm/cvuqdisk1.0.1-1.rpm linux3:/home/oracle/orainstall/clusterware/rpm

Perform the following steps as the "root" user account on the new Oracle RAC node to install the cvuqdisk RPM:
[root@linux3 ~]# cd /home/oracle/orainstall/clusterware/rpm [root@linux3 ~]# CVUQDISK_GRP=oinstall; export CVUQDISK_GRP [root@linux3 ~]# rpm -iv cvuqdisk-1.0.1-1.rpm Preparing packages for installation... cvuqdisk-1.0.1-1 [root@linux3 ~]# ls -l /usr/sbin/cvuqdisk -rwsr-x--- 1 root oinstall 4168 Jun 2 2005 /usr/sbin/cvuqdisk

Verify Remote Access / User Equivalence The CVU should be run from linux1 the node we will be extending the Oracle software from. Before running CVU, login as the oracle user account and verify remote access / user equivalence is configured to all nodes in the cluster. When using the secure shell method, user equivalence will need to be enabled for the terminal shell session before attempting to run the CVU. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:
[oracle@linux1 ~]$ exec /usr/bin/ssh-agent $SHELL [oracle@linux1 ~]$ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

Checking Pre-Installation Tasks for CRS with CVU Once all prerequisites for running the CVU utility have been met, we can now check that all pre-installation tasks for Oracle Clusterware are completed by executing the

following command as the "oracle" UNIX user account (with user equivalence enabled) from linux1:
[oracle@linux1 ~]$ cd /home/oracle/orainstall/clusterware/cluvfy [oracle@linux1 ~]$ mkdir -p jdk14 [oracle@linux1 ~]$ unzip jrepack.zip -d jdk14 [oracle@linux1 ~]$ CV_HOME=/home/oracle/orainstall/clusterware/cluvfy; export CV_HOME [oracle@linux1 ~]$ CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14; export CV_JDKHOME [oracle@linux1 ~]$ ./runcluvfy.sh stage -pre crsinst -n linux1,linux2,linux3 -verbose

Review the CVU report. Note that there are several errors you may ignore in this report. If your system only has 1GB of RAM memory, you may receive an error during the "Total memory" check:
Check: Total memory Node Name Available ------------ -----------------------linux3 1009.65MB (1033880KB) linux2 1009.65MB (1033880KB) linux1 1009.65MB (1033880KB) Result: Total memory check failed. Required -----------------------1GB (1048576KB) 1GB (1048576KB) 1GB (1048576KB) Comment ---------failed failed failed

As you can see from the output above, the requirement is for 1GB of memory (1048576 KB). Although your system may have 1GB of memory installed in each of the Oracle RAC nodes, the Linux kernel is calculating it to be 1033880 KB which comes out to be 14696 KB short. This can be considered close enough and safe to continue with the installation. As I mentioned earlier in this article, I highly recommend all Oracle RAC nodes have 2GB of RAM memory or higher for performance reasons. The first error is with regards to finding a suitable set of interfaces for VIPs which can be safely ignored. This is a bug documented in Metalink Note 338924.1:
Suitable interfaces for the private interconnect on subnet "192.168.2.0": linux3 eth1:192.168.2.107 linux2 eth1:192.168.2.101 linux1 eth1:192.168.2.100 ERROR: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check failed.

As documented in the note, this error can be safely ignored.

The last set of errors that can be ignored deal with specific RPM package versions that are not required with CentOS 5. For example:

compat-db-4.0.14-5 compat-gcc-7.3-2.96.128 compat-gcc-c++-7.3-2.96.128 compat-libstdc++-7.3-2.96.128 compat-libstdc++-devel-7.3-2.96.128 compat-libstdc++-devel-7.3-2.96.128

While these specific packages are listed as missing in the CVU report, please ensure that the correct versions of the compat-* packages are installed on the new Oracle RAC node. For example, in CentOS 5 Update 3, these would be:

compat-gcc-34-3.4.6-4 compat-gcc-34-c++-3.4.6-4 compat-libstdc++-33-3.2.3-61

Checking the Hardware and Operating System Setup with CVU The next CVU check to run will verify the hardware and operating system setup. Again, run the following as the "oracle" UNIX user account from linux1:
[oracle@linux1 ~]$ cd /home/oracle/orainstall/clusterware/cluvfy [oracle@linux1 ~]$ ./runcluvfy.sh stage -post hwos -n linux1,linux2,linux3 -verbose

Review the CVU report. As with the previous check (pre-installation tasks for CRS), the check for finding a suitable set of interfaces for VIPs will fail and can be safely ignored Also note you may receive warnings in the "Checking shared storage accessibility..." portion of the report:
Checking shared storage accessibility... WARNING: Unable to determine the sharedness of /dev/sde on nodes: linux3,linux3,linux3,linux3,linux3,linux2,linux2,linux2,linux2,linux2 ,linux1,linux1,linux1,linux1,linux1 Shared storage check failed on nodes "linux3,linux2,linux1".

If this occurs, this too can be safely ignored. While we know the disks are visible and shared from all of our Oracle RAC nodes in the cluster, the check itself may fail. Several reasons for this have been documented. The first came from Metalink indicating that cluvfycurrently does not work with devices other than SCSI devices. This would include devices like EMC PowerPath and volume groups like those from Openfiler. At the time of this writing, no workaround exists other than to use manual methods for detecting shared devices. Another reason for this error was documented by Bane Radulovic at Oracle Corporation. His research shows that CVU calls smartclton Linux, and the problem is that smartclt does not return the serial number from our iSCSI devices. For example, a check against/dev/sde shows:
[root@linux3 ~]# /usr/sbin/smartctl -i /dev/sde smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen Home page is http://smartmontools.sourceforge.net/ Device: Openfile Virtual disk Version: 0 Serial number: Device type: disk Local Time is: Mon Sep 3 02:02:53 2007 EDT Device supports SMART and is Disabled Temperature Warning Disabled or Not Supported

At the time of this writing, it is unknown if the Openfiler developers have plans to fix this.

Extend Oracle Clusterware Software to the New Node

Extend the Oracle Clusterware software to the new Oracle RAC node from linux1!

Overview In this section, we will extend the current Oracle RAC database by adding the new Oracle RAC node linux3. The new node will need to be added to the cluster at the clusterware layer so that the other nodes in the RAC cluster consider it to be part of the cluster.

When you extend an Oracle RAC database, you must first extend the Oracle Clusterware home to the new node and then extend the Oracle Database home. In other words, you extend the software onto the new node in the same order as you installed the clusterware and Oracle database components on the existing nodes. Oracle Clusterware is already installed on the cluster. The task in this section is to add the new Oracle RAC node to the clustered configuration. This is done by executing the Oracle provided utility addNode.sh from one of the existing nodes in the cluster; namelylinux1. This script is located in the Oracle Clusterware's home oui/bin directory (/u01/app/crs/oui/bin). During the add nodeprocess, the shared Oracle Cluster Registry file and Voting Disk will be updated with the information regarding the new node. Verifying Terminal Shell Environment Before starting the Oracle Universal Installer, you should first verify you are logged onto the server you will be running the installer from (i.e. linux1) then run the xhost command as root from the console to allow X Server connections. Next, login as the oracle user account. If you are using a remote client to connect to the node performing the installation (SSH / Telnet to linux1 from a workstation configured with an X Server), you will need to set the DISPLAY variable to point to your local workstation. Finally, verify remote access / user equivalence to all nodes in the cluster: Verify Server and Enable X Server Access
[root@linux1 ~]# hostname linux1 [root@linux1 ~]# xhost + access control disabled, clients can connect from any host

Login as the oracle User Account and Set DISPLAY (if necessary)
[root@linux1 ~]# su - oracle [oracle@linux1 [oracle@linux1 [oracle@linux1 [oracle@linux1 ~]$ ~]$ ~]$ ~]$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer from against the new Oracle RAC node without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
[oracle@linux1 ~]$ exec /usr/bin/ssh-agent $SHELL [oracle@linux1 ~]$ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) [oracle@linux1 ~]$ ssh linux1 "date;hostname" Thu Sep 3 13:08:19 EDT 2009 linux1 [oracle@linux1 ~]$ ssh linux3 "date;hostname" Thu Sep 3 13:08:42 EDT 2009 linux3

Configure Oracle Clusterware on the New Node The next step is to configure Oracle Clusterware on the new Oracle RAC node linux3. As previously mentioned, this is performed by executing the new addNode.sh utility located in the Oracle Clusterware's home oui/bin directory (/u01/app/crs/oui/bin) fromlinux1:
[oracle@linux1 ~]$ hostname linux1 [oracle@linux1 ~]$ id -a uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper) [oracle@linux1 ~]$ cd /u01/app/crs/oui/bin [oracle@linux1 ~]$ ./addNode.sh Screen Name Welcome Screen Specify Cluster Nodes to Add to Installation Response Click Next

In this screen, the OUI lists all existing nodes in the top portion labeled "Existing Nodes". the bottom half of the screen labeled "Specify New Nodes", enter the information for the new node in the appropriate fields: Public Node Name Private Node Name Virtual Node Name

linux3

linux3-priv

linux3-vip

Click Next to continue. Cluster Node Additional Summary Execute Configuration Scripts

Verify the new Oracle RAC node is listed under the "New Nodes" drill-down. Click Install t start the installation!

Once all of the required Oracle Clusterware components have been copied from linux1 to linux3, the OUI prompts to execute three files as described in the following sections.

From linux3

Navigate to the /u01/app/oracle/oraInventory directory on linux3 and run orainstRoot.sh as the "root" user acco

From linux1

Important: As documented in Metalink Note 392415.1, the rootaddnode.sh script (which is run in this section) error out at the end with "Connection refused" (PRKC-1044) when trying to trying to add a new node to the clu The reason this error occurs is because the "oracle" user account on the node running the rootaddnode.sh script setup with SSH for remote access to the new node and has a non-empty SSH passphrase. Note that for obvious security reasons, the "oracle" user account is typically setup with a non-empty pass phrase for SSH keys and wo thus succumb to this error. The rootaddnode.sh script uses SSH to check remote node connectivity from linux1 linux3. If it gives any prompt, it will consider ssh is not configured properly. The script will then use rsh instead rsh is not configured, then it will error out with "Connection refused". If you are using SSH for user equivalenc I am in this article), you will need to temporarily define an empty rsa passphrase for the "oracle" user account o linux1 as follows: [oracle@linux1 ~]$ ssh-keygen -p Enter file in which the key is (/home/oracle/.ssh/id_rsa): Enter old passphrase: [OLD PASSPHRASE] Key has comment '/home/oracle/.ssh/id_rsa' Enter new passphrase (empty for no passphrase): [JUST HIT ENTER WITHOUT ENTERING A PASSPHRASE] Enter same passphrase again: [JUST HIT ENTER WITHOUT ENTERING A PASSPHRASE] Your identification has been saved with the new passphrase.

After temporarily defining an empty rsa passphrase for the "oracle" user account, navigate to the/u01/app/crs/install directory on linux1 and run rootaddnode.sh as the "root" user account. The rootaddnode. script will add the new node information to the Oracle Cluster Registry (OCR) file using the srvctl utility.

After running the rootaddnode.sh script from linux1, you can set your passphrase back to the old passphrase usi the same "ssh-keygen -p" command.

From linux3

Finally, navigate to the /u01/app/crs directory on linux3 and run root.sh as the "root" user account.

If the Oracle Clusterware home directory is a subdirectory of the ORACLE_BASE directory (which should nev be!), you will receive several warnings regarding permissions while running the root.sh script on the new node. These warnings can be safely ignored.

The root.sh may take awhile to run. With Oracle Clusterware version 10.2.0.4, the root.sh script should comple successfully. With Oracle version 10.2.0.1, when running the root.sh on linux3, you may receive critical errors. Use the troubleshooting methods described in the parent to this article: Running root.sh on the Last Node will Fail! If the vipca failed to run (10.2.0.1 users), re-run vipca (GUI) manually as root from linux3 (the node where the occurred). Please keep in mind that vipca is a GUI and will need to set your DISPLAY variable accordingly to X server: [root@linux3 ~]# $ORA_CRS_HOME/bin/vipca When the "VIP Configuration Assistant" appears, this is how I answered the screen prompts: Welcome: Click Next Network interfaces: Select only the public interface - eth0 Virtual IPs for cluster nodes: Node Name: linux1 IP Alias Name: linux1-vip IP Address: 192.168.1.200 Subnet Mask: 255.255.255.0 Node Name: linux2 IP Alias Name: linux2-vip IP Address: 192.168.1.201 Subnet Mask: 255.255.255.0 Node Name: linux3 IP Alias Name: linux3-vip IP Address: 192.168.1.207 Subnet Mask: 255.255.255.0 Summary: Click Finish Configuration Assistant Progress Dialog: Click OK after configuration is complete. Configuration Results: Click Exit Go back to the OUI and acknowledge the "Execute Configuration scripts" dialog window. End of installation At the end of the installation, exit from the OUI.

Verify Oracle Clusterware Installation After extending Oracle Clusterware to the new node, we can run through several tests to verify the install was successful. Run the following commands on the new Oracle RAC node (linux3) as the "oracle" user account: Check Cluster Nodes
[oracle@linux3 ~]$ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2 linux3 3

Confirm Oracle Clusterware Function

[oracle@linux3 ~]$ $ORA_CRS_HOME/bin/crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------ora.racdb.db application 0/0 0/1 ONLINE ONLINE linux2 ora....b1.inst application 0/5 0/0 ONLINE ONLINE linux1 ora....b2.inst application 0/5 0/0 ONLINE ONLINE linux2 ora....srvc.cs application 0/0 0/1 ONLINE ONLINE linux1 ora....db1.srv application 0/0 0/0 ONLINE ONLINE linux1 ora....db2.srv application 0/0 0/0 ONLINE ONLINE linux2 ora....SM1.asm application 0/5 0/0 ONLINE ONLINE linux1 ora....E1.lsnr application 0/5 0/0 ONLINE ONLINE linux1 ora....de1.gsd application 0/5 0/0 ONLINE ONLINE linux1 ora....de1.ons application 0/3 0/0 ONLINE ONLINE linux1 ora....de1.vip application 0/0 0/0 ONLINE ONLINE linux1 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE linux2 ora....E2.lsnr application 0/5 0/0 ONLINE ONLINE linux2 ora....de2.gsd application 0/5 0/0 ONLINE ONLINE linux2 ora....de2.ons application 0/3 0/0 ONLINE ONLINE linux2 ora....de2.vip application 0/0 0/0 ONLINE ONLINE linux2 ora....de3.gsd application 0/5 0/0 ONLINE ONLINE linux3 ora....de3.ons application 0/3 0/0 ONLINE ONLINE linux3 ora....de3.vip application 0/0 0/0 ONLINE ONLINE linux3

Check CRS Status

[oracle@linux3 ~]$ $ORA_CRS_HOME/bin/crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy

Check Oracle Clusterware Auto-Start Scripts on New Node (linux3)

[oracle@linux3 ~]$ ls -l /etc/init.d/init.* -rwxr-xr-x 1 root root 2236 Sep 3 13:19 /etc/init.d/init.crs* -rwxr-xr-x 1 root root 4926 Sep 3 13:19 /etc/init.d/init.crsd*

-rwxr-xr-x 1 root root 53446 Sep -rwxr-xr-x 1 root root 3208 Sep

3 13:19 /etc/init.d/init.cssd* 3 13:19 /etc/init.d/init.evmd*

Extend Oracle Database Software to the New Node

Extend the Oracle Database software to the new Oracle RAC node from linux1!

Overview After copying and configuring the Oracle Clusterware software to the new node, we now need to copy the Oracle Database software from one of the existing nodes to linux3. This is done by executing the Oracle provided utility addNode.sh from one of the existing nodes in the cluster; namely linux1. This script is located in the $ORACLE_HOME/oui/bin directory (/u01/app/oracle/product/10.2.0/db_1/oui/bin). Verifying Terminal Shell Environment As discussed in the previous section, the terminal shell environment needs to be configured for remote access and user equivalence to the new Oracle RAC node before running the Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable: Login as the oracle User Account and Set DISPLAY (if necessary)
[root@linux1 ~]# su - oracle [oracle@linux1 [oracle@linux1 [oracle@linux1 [oracle@linux1 ~]$ ~]$ ~]$ ~]$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer from against the new Oracle RAC node without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
[oracle@linux1 ~]$ exec /usr/bin/ssh-agent $SHELL [oracle@linux1 ~]$ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) [oracle@linux1 ~]$ ssh linux1 "date;hostname" Thu Sep 3 14:19:17 EDT 2009 linux1 [oracle@linux1 ~]$ ssh linux3 "date;hostname" Thu Sep 3 14:19:43 EDT 2009 linux3

Install Oracle Database Software on the New Node Copy the Oracle Database software to the new Oracle RAC node linux3. As previously mentioned, this is performed by executing the new addNode.sh utility located in the $ORACLE_HOME/oui/bin directory from linux1:
[oracle@linux1 ~]$ hostname linux1 [oracle@linux1 ~]$ id -a uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper) [oracle@linux1 ~]$ cd /u01/app/oracle/product/10.2.0/db_1/oui/bin [oracle@linux1 ~]$ ./addNode.sh Screen Name Welcome Screen Specify Cluster Nodes to Add to Response Click Next

In this screen, the OUI lists all of the nodes already part of the installation in the top port labeled "Existing Nodes". On the bottom half of the screen labeled "Specify New Nodes" list of new nodes which can be added. By default linux3 is selected. Verify linux3 is selec

Installation Cluster Node Additional Summary

(checked) and Click Next to continue.

Verify the new Oracle RAC node is listed under the "New Nodes" drill-down. Click Install t start the installation!

Execute Configuration Scripts

Once all of the required Oracle Database components have been copied from linux1 to linux3, the OUI prompts to execute the root.sh on the new Oracle RAC node. Navigate to the /u01/app/oracle/product/10.2.0/db_1 directory on linux3 and run root.sh as the "root" user account.

After running the root.sh script on the new Oracle RAC node, go back to the OUI and acknowledge the "Execu Configuration scripts" dialog window. End of installation At the end of the installation, exit from the OUI.

Add Listener to New Node

Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Network Configuration Assistant (NETCA) will setup the TNS listener in a clustered configuration to include the new node in the cluster.

Overview In this section, you will use the Network Configuration Assistant (NETCA) to setup the TNS listener in a clustered configuration to include the new Oracle RAC node. The NETCA program will be run from linux1 with user equivalence enabled to all nodes in the cluster. Verifying Terminal Shell Environment As discussed in the previous section, the terminal shell environment needs to be configured for remote access and user equivalence to

the new Oracle RAC node before running the NETCA. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and theDISPLAY variable: Login as the oracle User Account and Set DISPLAY (if necessary)
[root@linux1 ~]# su - oracle [oracle@linux1 [oracle@linux1 [oracle@linux1 [oracle@linux1 ~]$ ~]$ ~]$ ~]$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the NETCA from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the NETCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
[oracle@linux1 ~]$ exec /usr/bin/ssh-agent $SHELL [oracle@linux1 ~]$ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) [oracle@linux1 ~]$ ssh linux1 "date;hostname" Thu Sep 3 14:19:17 EDT 2009 linux1 [oracle@linux1 ~]$ ssh linux3 "date;hostname" Thu Sep 3 14:19:43 EDT 2009 linux3

Run the Network Configuration Assistant To start the NETCA, run the following from
linux1:

[oracle@linux1 ~]$ netca &

The following table walks you through the process of reconfiguring the TNS listeners in a clustered configuration to include the new node.
Screen Name Select the Type of Oracle Net Services Configuration Select the nodes to configure Type of Configuration Response

Select Cluster configuration

Only select the new Oracle RAC node: linux3. Select Listener configuration. The following screens are now like any other normal listener configuration. You can simply accept the default parameters for the next six screens: What do you want to do: Add Listener name: LISTENER Selected protocols: TCP Port number: 1521 Configure another listener: No Listener configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen. Select Naming Methods configuration. The following screens are: Selected Naming Methods: Local Naming Naming Methods configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen. Click Finish to exit the NETCA.

Listener Configuration Next 6 Screens

Type of Configuration

Naming Methods Configuration

Type of Configuration

Verify TNS Listener Configuration The Oracle TNS listener process should now be running on all three nodes in the RAC cluster:
[oracle@linux1 ~]$ hostname linux1

[oracle@linux1 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX1 [oracle@linux1 ~]$ $ORA_CRS_HOME/bin/crs_stat ora.linux1.LISTENER_LINUX1.lsnr NAME=ora.linux1.LISTENER_LINUX1.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux1 ===================== [oracle@linux2 ~]$ hostname linux2 [oracle@linux2 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX2 [oracle@linux2 ~]$ $ORA_CRS_HOME/bin/crs_stat ora.linux2.LISTENER_LINUX2.lsnr NAME=ora.linux2.LISTENER_LINUX2.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux2 ===================== [oracle@linux3 ~]$ hostname linux3 [oracle@linux3 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX3 [oracle@linux3 ~]$ $ORA_CRS_HOME/bin/crs_stat ora.linux3.LISTENER_LINUX3.lsnr NAME=ora.linux3.LISTENER_LINUX3.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux3

Add Database Instance to the New Node

Add the new Oracle instance to the new Oracle RAC node using DBCA!

Overview The final step in extending the Oracle RAC database is to add a new database instance to the new Oracle RAC node. The database instance will be named racdb3 and hosted on the new node linux3. This process can be performed using either Enterprise Manager or the Database Configuration Assistant (DBCA). For the purpose of this article, I am opting to use the DBCA. Before executing the DBCA, make certain that $ORACLE_HOME and $PATH are set appropriately for the$ORACLE_BASE/product/10.2.0/db_1 environment. You should also verify that all services we have installed up to this point (Oracle TNS listener, Oracle Clusterware processes, etc.) are running before attempting to start the clustered database creation process. The DBCA program will be run from linux1 with user equivalence enabled to all nodes in the cluster. Verifying Terminal Shell Environment As discussed in the previous section, the terminal shell environment needs to be configured for remote access and user equivalence to the new Oracle RAC node before running the DBCA. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and theDISPLAY variable: Login as the oracle User Account and Set DISPLAY (if necessary)
[root@linux1 ~]# su - oracle [oracle@linux1 [oracle@linux1 [oracle@linux1 [oracle@linux1 ~]$ ~]$ ~]$ ~]$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence

Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the DBCA from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method,user equivalence will need to be enabled on any new terminal shell session before attempting to run the DBCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
[oracle@linux1 ~]$ exec /usr/bin/ssh-agent $SHELL [oracle@linux1 ~]$ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) [oracle@linux1 ~]$ ssh linux1 "date;hostname" Thu Sep 3 14:19:17 EDT 2009 linux1 [oracle@linux1 ~]$ ssh linux3 "date;hostname" Thu Sep 3 14:19:43 EDT 2009 linux3

Add Database Instance to New Node To start the database instance creation process for the new Oracle RAC node, run the following from linux1:
[oracle@linux1 ~]$ dbca & Screen Name Welcome Screen Operations Instance Management Response Select Oracle Real Application Clusters database. Select Instance Management. Select Add an instance.

List of cluster databases Provides a list of clustered databases running on the node. For the purpose of this example, the clustered database running on node linux1 is racdb. Select this clustered database. At the bottom of this screen, the DBCA requests you to "Specify a user with SYSDBA system privileges":

Username: sys Password: <sys_password> Click Next to continue. List of cluster database instances This screen provides a list of all instances currently available on the cluster, their status, and which node they reside on. Verify this list is correct and Click Next to continue. This screen lists the next instance name in the series and requests the node on which to add the instance to. In this example, the next instance name is racdb3 and the node name to create it on is linux3. For this example, the default values are correct (instance name "racdb3" to be added to node "linux3"). After verifying these values, Click Next to continue. After clicking Next, there will be a small pause before the next screen appears as the DBCA determines the current state of the new node and what services (if any) are configured on the existing nodes. If the current clustered database has any database services defined, the next screen allows the DBA to configure those database services for the new instance. In this example, the existing clustered database has one service defined namedracdb_srvc. With the "racdb_srvc" database service selected, change the details to Preferred for the new instance (racdb3) and the "TAF Policy" set to Basic. By default, the DBCA does a good job of determining the instance specific files such as an UNDO tablespace (UNDOTBS3), database files for this tablespace, and two redo log groups. Verify the storage options and Click Finish to add the instance. After verifying the instance creation options in the summary dialog, Click OK to begin the instance management process. During the add instance step, the DBCA verifies the new node and then checks to determine if ASM is present on the existing cluster (which in this example, ASM is configured). The DBCA presents a dialog box indicating that "ASM is present on the cluster but needs to be extended to the following nodes: [linux3]. Do you want ASM to be extended?" Click on Yes to add the ASM instance to the new node. NOTE: In the previous section (Add Listener to New Node), I provided instructions to setup the TNS listener in a clustered configuration to include the new Oracle RAC node using NETCA. If the listener is not yet configured on the

Instance naming and node selection

Database Services

Instance Storage

Database Configuration Assistant: Summary Extend ASM

new Oracle RAC node, the DBCA will prompt the user with a dialog asking to configure a new listener using port 1521 and listener name "LISTENER_LINUX3". The TNS listener must be present and started on the new Oracle RAC node in order to create and start the ASM instance on the new node. Database Configuration Assistant Progress Screen A progress bar is display while the new instance is being configured. Once the instance management process is complete, the DBCA prompts the user with a dialog and the message "Do you want to perform another operation?" Click No to end and exit the DBCA utility. The DBCA will automatically start the new instance (racdb3) on the node linux3. If any services were configured during the instance management process, however, they are left in an offline state. For the purpose of this example, I had to manually start the "racdb_srvc" service for the database: $ srvctl start service -s racdb_srvc -d racdb -i racdb3

Start New Database Services

When the Oracle Database Configuration Assistant has completed, you will have successfully extended the current Oracle RAC database! Verify New Database Environment Check Cluster Services
[oracle@linux1 ~]$ $ORA_CRS_HOME/bin/crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.racdb.db application ONLINE ONLINE linux2 ora....b1.inst application ONLINE ONLINE linux1 ora....b2.inst application ONLINE ONLINE linux2 ora....b3.inst application ONLINE ONLINE linux3 ora....srvc.cs application ONLINE ONLINE linux1 ora....db1.srv application ONLINE ONLINE linux1 ora....db2.srv application ONLINE ONLINE linux2 ora....db3.srv application ONLINE ONLINE linux3 ora....SM1.asm application ONLINE ONLINE linux1 ora....E1.lsnr application ONLINE ONLINE linux1 ora....de1.gsd application ONLINE ONLINE linux1 ora....de1.ons application ONLINE ONLINE linux1 ora....de1.vip application ONLINE ONLINE linux1 ora....SM2.asm application ONLINE ONLINE linux2 ora....E2.lsnr application ONLINE ONLINE linux2 ora....de2.gsd application ONLINE ONLINE linux2 ora....de2.ons application ONLINE ONLINE linux2 ora....de2.vip application ONLINE ONLINE linux2 ora....SM3.asm application ONLINE ONLINE linux3 ora....E3.lsnr application ONLINE ONLINE linux3

ora....de3.gsd application ora....de3.ons application ora....de3.vip application - or -

ONLINE ONLINE ONLINE

ONLINE ONLINE ONLINE

linux3 linux3 linux3

[oracle@linux1 ~]$ rac_crs_stat HA Resource ----------ora.racdb.db ora.racdb.racdb1.inst ora.racdb.racdb2.inst ora.racdb.racdb3.inst ora.racdb.racdb_srvc.cs ora.racdb.racdb_srvc.racdb1.srv ora.racdb.racdb_srvc.racdb2.srv ora.racdb.racdb_srvc.racdb3.srv ora.linux1.ASM1.asm ora.linux1.LISTENER_linux1.lsnr ora.linux1.gsd ora.linux1.ons ora.linux1.vip ora.linux2.ASM2.asm ora.linux2.LISTENER_linux2.lsnr ora.linux2.gsd ora.linux2.ons ora.linux2.vip ora.linux3.ASM3.asm ora.linux3.LISTENER_linux3.lsnr ora.linux3.gsd ora.linux3.ons ora.linux3.vip

Target -----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

State ----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on on on on on on on on on on on on on on on on on on

linux2 linux1 linux2 linux3 linux1 linux1 linux2 linux3 linux1 linux1 linux1 linux1 linux1 linux2 linux2 linux2 linux2 linux2 linux3 linux3 linux3 linux3 linux3

Verify New Instance Login to one of the instances and query the
gv$instance

view:

SQL> select inst_id, instance_name, status, to_char(startup_time, 'DD-MONYYYY HH24:MI:SS') 2 from gv$instance order by inst_id; INST_ID ---------1 2 3 INSTANCE_NAME ---------------racdb1 racdb2 racdb3 STATUS -----------OPEN OPEN OPEN TO_CHAR(STARTUP_TIME -------------------02-SEP-2009 17:27:52 02-SEP-2009 17:28:57 03-SEP-2009 14:47:57

Update TNSNAMES Login to all machines that will be accessing the new instance and update the tnsnames.ora file (if necessary).

Verify Enterprise Manager - Database Control The DBCA should have updated and added the new node(s) to EM Database Control. Bring up a web browser and navigate to:
http://linux3:1158/em

About the Author Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX, Linux, and Windows server environment. Jeff's other interests include mathematical encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 18 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science.

Copyright (c) 1998-2012 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info. I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.

Last modified on Wednesday, 16-May-2012 10:38:05 EDT Page Count: 10631

DBA Tips Archive for Oracle

Remove a Node from an Existing Oracle RAC 10g R2 Cluster on Linux - (RHEL 4.5)
by Jeff Hunter, Sr. Database Administrator Contents
1. 2. 3. 4. 5. 6. 7. 8. 9.

Overview Remove the Oracle Instance on the Node to be Deleted ASM Instance Clean-Up Procedures Remove the Listener From the Node to be Deleted Remove the Oracle Database Software From the Node to be Deleted Remove the Node to be Deleted from Oracle Clusterware OCFS2 Clean-Up Procedures Remove Access Permissions on Openfiler for the Node to be Deleted About the Author

Overview Although not as exciting as building an Oracle RAC or adding a new node and instance to a clustered database; removing a node from a clustered environment is just as important to understand for a DBA

managing Oracle RAC 10g. While it is true that most of the attention in a clustered database environment is focused on extending the database tier to support increased demand, the exact opposite is just as likely to be encountered where the DBA needs to remove a node from an existing RAC environment. It may be that a server is being underutilized in the database cluster and could be better served in another business unit. Another scenario is a node failure. In this case, a node can be removed from the cluster while the remaining nodes continue to service ongoing requests. This document is an extension to two of my articles: "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)" and "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)". Contained in this new article are the steps required to remove a single node from an already running and configured three-node Oracle RAC 10g Release 2 environment on the CentOS 32bit (x86) platform. The node being removed is the third node I added in the second article. Although this article was written and tested on CentOS 4.5 Linux, it should work unchanged with Red Hat Enterprise Linux 4 Update 5. When removing the Oracle Cluster and Oracle Database software, you must first remove the Oracle Database software and then remove the Oracle Clusterware from the node that you are deleting. In other words, you remove the software components from the node you are deleting in the reverse order that you originally installed the clusterware and Oracle database software components. It is important that you perform each step contained this article in the order provided. This article assumes the following:
The reader has already built and configured a three-node Oracle RAC 10g Release 2 environment using the articles "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)" and "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)". These two articles provide comprehensive instructions for building a two-node RAC cluster and then adding a third node to the cluster named linux3. All Oracle RAC nodes are single processor machines running CentOS 4.5, Oracle RAC 10g Release 2, OCFS2, and ASMLib 2.0. The current three-node RAC environment actually consists of four machines three named linux1, linux2, and linux3 which each run an Oracle10g instance and a forth node to run the network storage server named openfiler1.

Note: The current three-node Oracle RAC environment has been upgraded from its base release (10.2.0.1.0) to version 10.2.0.3.0 by applying the 5337014 patchset (p5337014_10203_LINUX.zip).

The patchset was applied to Oracle Clusterware and the Oracle Database software. I also applied the one-off patchset - "BUNDLE Patch for Base Bug 6000740" (MLR7 ON TOP OF 10.2.0.3) to the Oracle Clusterware and Oracle Database software. The procedures for installing both patchsets are not included in any of the parent article(s).
The third node in the current Oracle RAC environment named linux3 (running the orcl3 instance) will be removed from the cluster making it a two-node cluster. Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and Oracle Database software installed on their local disks. The current three-node Oracle RAC environment does not use shared Oracle homes for the Clusterware or Database software. The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. Instructions for removing the OCFS2 software from the deleted node against a "live" OCFS2 file system will be included.

Automatic Storage Management (ASM) is being used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area. In addition to removing the ASM instance from the deleted node, we will also be removing ASMLib from this node. During the creation of the existing two-node cluster and the addition of the third node, the installation of Oracle Clusterware and the Oracle Database software were only performed from one node in the RAC cluster namely from linux1 as the oracle user account. The Oracle Universal Installer (OUI) on that particular node would then use the ssh and scp commands to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. The oracle user account on the node running the OUI (runInstaller) had to be trusted by all other nodes in the RAC cluster. This meant that theoracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI (linux1) against all other Linux servers in the cluster without being prompted for a password. The same security requirements hold true for this article. User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securely removed from the node to be deleted (linux3). This task will need to be performed from a node that is to remain a part of the RAC cluster database environment (linux1 in this example) using ssh and scp without being prompted for a password. All shared disk storage for the existing Oracle RAC is based on iSCSI using a Network

Storage Server; namely Openfiler Release 2.2 (respin 2). While removing the node linux3 from the Oracle RAC cluster, we will also be removing access permissions to the Openfiler server from linux3.

The following is a conceptual look at what the environment will look like after removing the third Oracle RAC node (linux3) from the cluster. Click on the graphic below to enlarge the image:

Figure 1: Removing linux3 from the current Oracle RAC 10g Release 2 Environment

While this article provides comprehensive instructions for successfully removing a node fro existing Oracle RAC 10g system, it is by no means a substitute for the official Oracle documentation. In addition to this article, users should also consult the following Oracle

documents to gain a full understanding of alternative configuration options, installation, an administration with Oracle RAC 10g. Oracle's official documentation site is docs.oracle.com

Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) Linux Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 1 Release 2 (10.2) 2 Day + Real Application Clusters Guide - 10g Release 2 (10.2)

Remove the Oracle Instance on the Node to be Deleted

Remove the Oracle instance running on the node to be deleted using DBCA!

Overview Before removing any of the Oracle software from the node to be deleted (linux3), the first step is to remove the Oracle instance (orcl3) from the RAC database. This task can be performed using either Enterprise Manager or the Database Configuration Assistant (DBCA). For the purpose of this article, I am opting to use the DBCA. The DBCA will need to be run from a node other than the node that hosts the instance you want to delete. The suggested node to run the DBCA from is the same node used to install the original Oracle RAC environment from which for this example is linux1. Verifying Terminal Shell Environment Before starting the DBCA, you should first verify you are logged onto the server you will be running the DBCA from (i.e. linux1) then run the xhost command as root from the console to allow X Server connections. Next, login as the oracle user account. If you are using a remote client to connect to the node performing the installation

(SSH / Telnet to linux1 from a workstation configured with an X Server), you will need to set the DISPLAY variable to point to your local workstation. Finally, verify remote access / user equivalence to all nodes in the cluster: Verify Server and Enable X Server Access
# hostname linux1 # xhost + access control disabled, clients can connect from any host

Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the secure shell commands (ssh or scp) on the Linux server you will be running the DBCA from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the DBCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) $ ssh linux1 "date;hostname" Fri Feb 29 00:08:32 EST 2008 linux1 $ ssh linux2 "date;hostname" Fri Feb 29 00:10:10 EST 2008 linux2 $ ssh linux3 "date;hostname" Fri Feb 29 00:07:25 EST 2008

linux3

Remove Database Instance To start the database instance removal process, run the following from linux1:
$ dbca & Screen Name Welcome Screen Operations Instance Management Response Select Oracle Real Application Clusters database. Select Instance Management. Select Delete an instance. Provides a list of clustered databases running on the node. For the purpose of this example, the clustered database running on node linux1 is orcl. Select this clustered database. List of cluster databases At the bottom of this screen, the DBCA requests you to "Specify a user with SYSDBA system privileges": Username: sys Password: <sys_password> Click Next to continue. List of cluster database instances This screen provides a list of all instances currently available on the cluster, their status, and which node they reside on. Select the node:instance to delete (linux3:orcl3) and Click Next to continue. If the current clustered database has any database services defined, the next screen allows the DBA to re-configure those database services to account for the instance being removed. In this example, the existing clustered database has one service defined named orcl_taf. With the "orcl_taf" database service selected, change the details to Not used for the instance to be removed (orcl3) and leave the "TAF Policy" set to Basic. Click Finish to remove the instance. Database Configuration Assistant: Summary Database Configuration After verifying the instance removal options in the summary dialog, Click OK to begin the instance management process. A second dialog box is presented which includes the warning:

Database Services

Assistant

"The Database Configuration Assistance will delete the Oracle instance and its associated OFA directory structure. All information about this instance will be deleted. Do you want to proceed?" Click OK to acknowledge this dialog.

Database Configuration Assistant Progress Screen

A progress bar is display while the instance is being deleted. Once the instance management process is complete, the DBCA prompts the user with a dialog and the message "Do you want to perform another operation?" Click No to end and exit the DBCA utility.

When the Oracle Database Configuration Assistant has completed, the orcl3 instance (and associated services) will have been successfully removed from linux3! Verify Database Instance and Associated Objects Were Removed Check Cluster Services
$ $ORA_CRS_HOME/bin/crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....SM1.asm application ONLINE ONLINE linux1 ora....X1.lsnr application ONLINE ONLINE linux1 ora.linux1.gsd application ONLINE ONLINE linux1 ora.linux1.ons application ONLINE ONLINE linux1 ora.linux1.vip application ONLINE ONLINE linux1 ora....SM2.asm application ONLINE ONLINE linux2 ora....X2.lsnr application ONLINE ONLINE linux2 ora.linux2.gsd application ONLINE ONLINE linux2 ora.linux2.ons application ONLINE ONLINE linux2 ora.linux2.vip application ONLINE ONLINE linux2 ora....SM3.asm application ONLINE ONLINE linux3 ora....X3.lsnr application ONLINE ONLINE linux3 ora.linux3.gsd application ONLINE ONLINE linux3 ora.linux3.ons application ONLINE ONLINE linux3 ora.linux3.vip application ONLINE ONLINE linux3 ora.orcl.db application ONLINE ONLINE linux1 ora....l1.inst application ONLINE ONLINE linux1 ora....l2.inst application ONLINE ONLINE linux2 ora...._taf.cs application ONLINE ONLINE linux2 ora....cl1.srv application ONLINE ONLINE linux1 ora....cl2.srv application ONLINE ONLINE linux2 - or $ rac_crs_stat HA Resource Target State

----------ora.linux1.ASM1.asm ora.linux1.LISTENER_LINUX1.lsnr ora.linux1.gsd ora.linux1.ons ora.linux1.vip ora.linux2.ASM2.asm ora.linux2.LISTENER_LINUX2.lsnr ora.linux2.gsd ora.linux2.ons ora.linux2.vip ora.linux3.ASM3.asm ora.linux3.LISTENER_LINUX3.lsnr ora.linux3.gsd ora.linux3.ons ora.linux3.vip ora.orcl.db ora.orcl.orcl1.inst ora.orcl.orcl2.inst ora.orcl.orcl_taf.cs ora.orcl.orcl_taf.orcl1.srv ora.orcl.orcl_taf.orcl2.srv

-----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on on on on on on on on on on on on on on on on

linux1 linux1 linux1 linux1 linux1 linux2 linux2 linux2 linux2 linux2 linux3 linux3 linux3 linux3 linux3 linux1 linux1 linux2 linux2 linux1 linux2

Verify Instance we Removed Login to one of the instances and query the
gv$instance

view:

SQL> select inst_id, instance_name, status, to_char(startup_time, 'DD-MONYYYY HH24:MI:SS') 2 from gv$instance order by inst_id; INST_ID ---------1 2 INSTANCE_NAME ---------------orcl1 orcl2 STATUS -----------OPEN OPEN TO_CHAR(STARTUP_TIME -------------------27-FEB-2008 01:26:52 27-FEB-2008 01:27:22

Verify Database Files / Tablespaces have been Removed Check that the UNDO tablespace that was defined for has been removed:
SQL> select tablespace_name from dba_tablespaces order by 1; TABLESPACE_NAME -----------------EXAMPLE SOE SOEINDEX SYSAUX SYSTEM TEMP orcl3 (UNDOTBS3)

UNDOTBS1 UNDOTBS2 USERS

Check that the online redo log files that were defined for been removed:

orcl3

have

SQL> select inst_id, group#, member from gv$logfile order by 1,2,3; INST_ID GROUP# MEMBER ------- ------ --------------------------------------------------------1 1 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.641829669 1 1 +ORCL_DATA1/orcl/onlinelog/group_1.257.641829667 1 2 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.258.641829675 1 2 +ORCL_DATA1/orcl/onlinelog/group_2.258.641829671 1 3 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.641832297 1 3 +ORCL_DATA1/orcl/onlinelog/group_3.266.641832295 1 4 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.641832303 1 4 +ORCL_DATA1/orcl/onlinelog/group_4.267.641832299 2 1 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.641829669 2 1 +ORCL_DATA1/orcl/onlinelog/group_1.257.641829667 2 2 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_2.258.641829675 2 2 +ORCL_DATA1/orcl/onlinelog/group_2.258.641829671 2 3 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_3.259.641832297 2 3 +ORCL_DATA1/orcl/onlinelog/group_3.266.641832295 2 4 +FLASH_RECOVERY_AREA/orcl/onlinelog/group_4.260.641832303 2 4 +ORCL_DATA1/orcl/onlinelog/group_4.267.641832299

Update TNSNAMES Login to all machines that were accessing the instance removed and update the tnsnames.ora file (if necessary).

ASM Instance Clean-Up Procedures

Remove ASM instance and ASMLib from the node to be deleted!

Overview

When the clustered database is running on ASM (as is the case in this example), the steps in this section describe how to remove the ASM instance (and ASMLib) from the node to be deleted. If the Oracle RAC database is not running on ASM, this section can be skipped. The tasks in this section will need to be run from a node other than the node that hosts the ASM instance you want to delete. The suggested node to run these steps from is the same node used to install the original Oracle RAC environment from which for this example is linux1. Remove the ASM Instance from the Oracle Cluster Registry File Run the following from
linux1

as the

oracle

user account:

$ srvctl stop asm -n linux3 $ srvctl remove asm -n linux3

Remove ASM Directories from the Node to be Deleted Remove all ASM directories from the node to be deleted (linux3) as the oracle user account:
$ cd $ORACLE_BASE/admin $ rm -rf +ASM $ cd $ORACLE_HOME/dbs $ rm -f *ASM*

You may also have to edit the /etc/oratab file on the node to be deleted and remove entries beginning with +ASM to delete the references of the ASM instance.

Remove ASMLib from the Node to be Deleted

The Linux nodes in the current Oracle RAC database make use of ASMLib so that the raw block devices managed by ASM use ASMLib calls as opposed to standard Linux I/O system calls. In this section we will remove the ASMLib kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account:
# /etc/init.d/oracleasm stop Unmounting ASMlib driver filesystem: [ Unloading module "oracleasm": [ OK ] # rpm -qa | grep oracleasm oracleasm-support-2.0.3-1 oracleasm-2.6.9-55.EL-2.0.3-1 oracleasmlib-2.0.2-1 # rpm -ev oracleasm-support-2.0.3-1 oracleasm-2.6.9-55.EL-2.0.3-1 oracleasmlib-2.0.2-1 warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave # # # # # # # # # rm rm rm rm rm rm rm rm rm -f -f -f -f -f -f -f -f -f /etc/sysconfig/oracleasm.rpmsave /etc/rc.d/init.d/oracleasm /etc/rc0.d/*oracleasm* /etc/rc1.d/*oracleasm* /etc/rc2.d/*oracleasm* /etc/rc3.d/*oracleasm* /etc/rc4.d/*oracleasm* /etc/rc5.d/*oracleasm* /etc/rc6.d/*oracleasm* OK ]

Remove the Listener From the Node to be Deleted

Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Network Configuration Assistant (NETCA) will remove the node-specific listener from the clustered configuration for the node to be deleted.

Overview

Each of the nodes in the current Oracle RAC environment use a node-specific listener named LISTENER_nodename which runs out of the local Oracle home. In this section, you will use the Network Configuration Assistant (NETCA) to delete the TNS listener for the node to be deleted from the existing clustered configuration. The NETCA program will be run from linux1 with user equivalence enabled to all nodes in the cluster. Although it is possible to re-create this listener in another Oracle home (on a different node), we will simply be removing it from the cluster. Verifying Terminal Shell Environment As was performed earlier, the terminal shell environment needs to be configured for remote access and user equivalence to all Oracle RAC nodes before running the NETCA. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable: Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the secure shell commands (ssh or scp) on the Linux server you will be running the NETCA from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the NETCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:

$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) $ ssh linux1 "date;hostname" Fri Feb 29 00:08:32 EST 2008 linux1 $ ssh linux2 "date;hostname" Fri Feb 29 00:10:10 EST 2008 linux2 $ ssh linux3 "date;hostname" Fri Feb 29 00:07:25 EST 2008 linux3

Run the Network Configuration Assistant To start the NETCA, run the following from
$ netca & linux1:

The following table walks you through the process of reconfiguring the clustered TNS listeners to remove the node-specific listener from the node to be deleted.
Screen Name Select the Type of Oracle Net Services Configuration Select the nodes to configure Type of Configuration Response

Select Cluster configuration

Only select the Oracle RAC node to remove the listener from: linux3.

Select Listener configuration. The following screens are now like any other normal listener configuration when removing a listener: What do you want to do: Delete Select a listener you want to delete: LISTENER Acknowledge warning dialog: Yes Listener deleted [ Next ] Listener configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen.

Listener Configuration Next 4 Screens

Type of Configuration

Click Finish to exit the NETCA.

Verify TNS Listener Configuration The Oracle node-specific TNS listener processes should now only be running on two nodes in the RAC cluster (linux1 and linux2):
$ hostname linux1 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX1 $ $ORA_CRS_HOME/bin/crs_stat ora.linux1.LISTENER_LINUX1.lsnr NAME=ora.linux1.LISTENER_LINUX1.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux1 ===================== $ hostname linux2 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX2 $ $ORA_CRS_HOME/bin/crs_stat ora.linux2.LISTENER_LINUX2.lsnr NAME=ora.linux2.LISTENER_LINUX2.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux2 ===================== $ hostname linux3 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' $ $ORA_CRS_HOME/bin/crs_stat ora.linux3.LISTENER_LINUX3.lsnr CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'.

Remove the Oracle Database Software From the Node to be Deleted

Remove the Oracle Database software and update the Node List for the node to be deleted!

Overview In this section, you will be removing the node to be deleted (linux3) from the Node List for its Oracle inventory. Next will be to remove the Oracle Database software from the node to be deleted using the Oracle Universal Installer (OUI). Finally, update the Oracle inventory on all remaining Oracle RAC nodes to reflect the nodes which will remain in the cluster. Update Node List for Oracle Database Software - (Remove linux3) From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Database software:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY $ cd $ORACLE_HOME/oui/bin $ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME CLUSTER_NODES="" -local Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.

Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!

De-install Oracle Database Software

Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Database software. Make certain that you choose the home to be removed and not just the products under that home. From linux3 as the oracle user account, run the following:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY $ cd $ORACLE_HOME/oui/bin $ ./runInstaller Screen Name Welcome Screen Inventory: Contents Tab Confirmation Response Click the Installed Products button. Check the Oracle home to deleted (OraDb10g_home1) and click the Remove button. Acknowledge the warning dialog by clicking Yes. A progress bar is displayed while the Oracle Database software is being removed. Once this process has completed, you are returned to the "Inventory: Contents Tab" dialog. After confirming the Oracle Database software (Oracle home) was successfully removed, Click Close to exit this dialog. Click Cancel to exit the OUI.

Deinstallation Process

Welcome Screen

Update Node List for Remaining Nodes in the Cluster Finally, from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY $ cd $ORACLE_HOME/oui/bin $ ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={linux1,linux2}" Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.

Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!

Remove the Node to be Deleted from Oracle Clusterware

Remove the node to be deleted from Oracle Clusterware and update the Node List for all remaining nodes!

Overview After successfully removing the Oracle Database software from the node to be deleted, the next major step is to remove the node to be deleted from Oracle Clusterware. Remove Node-Specific Interface Configuration Run the following commands to remove node-specific interface configurations from the node to be deleted. For this example, run these commands from linux3 as the oracle user account:
$ $ORA_CRS_HOME/bin/racgons remove_config linux3:6200

In the above command, replace 6200 with the port number you get in the remoteportsection of the ons.config file found in the $ORA_CRS_HOME/opmn/conf directory:
$ grep '^remoteport' $ORA_CRS_HOME/opmn/conf/ons.config remoteport=6200 $ $ORA_CRS_HOME/bin/oifcfg delif -node linux3 PROC-4: The cluster registry key to be operated on does not exist. PRIF-11: cluster registry error

From the research that I have performed, I believe the above error can be safely ignored. The "oifcfg delif" command attempts to delete a node-specific interface defined for node linux3 in the Oracle Cluster Registry file (OCR). This means it is looking for the presence of a key in

the OCR file named "SYSTEM.css.interfaces.nodelinux3". If you were to dump the contents of the OCR file, you would quickly see that no key is present with that name:
$ $ORA_CRS_HOME/bin/ocrdump $ grep 'SYSTEM.css.interfaces.nodelinux3' OCRDUMPFILE

After examining the contents of the OCRDUMPFILE, interfaces are indeed defined, however they are identified as global:
$ grep 'SYSTEM.css.interfaces.global' OCRDUMPFILE [SYSTEM.css.interfaces.global] [SYSTEM.css.interfaces.global.eth0] [SYSTEM.css.interfaces.global.eth0.192|d168|d1|d0] [SYSTEM.css.interfaces.global.eth0.192|d168|d1|d0.1] [SYSTEM.css.interfaces.global.eth2] [SYSTEM.css.interfaces.global.eth2.192|d168|d3|d0] [SYSTEM.css.interfaces.global.eth2.192|d168|d3|d0.1]

Given there are no node-specific interface entries defined and we obviously don't want to delete the interface keys configured globally, the oifcfg delif command and its errors can be safely ignored!

Disable Oracle Clusterware Applications From the node you are deleting from the cluster (linux3), run the script $ORA_CRS_HOME/install/rootdelete.sh to disable the Oracle Clusterware applications that are on the node. This script should only be run once. Given the Clusterware software install is on local disk (non-shared), make certain to use the nosharedhome argument. The default for this script is sharedhome which prevents you from updating the permissions of local files such that they can be removed by the oracle user account. Running this script will stop the CRS stack and delete the ocr.loc file on the node to be removed. The nosharedvar option assumes the ocr.loc file is not on a shared file sytem. While logged into linux3 as the root user account, run the following:
$ su # cd $ORA_CRS_HOME/install # ./rootdelete.sh local nosharedvar nosharedhome CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'. Shutting down Oracle Cluster Ready Services (CRS): Stopping resources. This could take several minutes.

Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. Shutdown has begun. The daemons should exit soon. Checking to see if Oracle CRS stack is down... Oracle CRS stack is not running. Oracle CRS stack is down now. Removing script for Oracle Cluster Ready services Updating ocr file for downgrade Cleaning up SCR settings in '/etc/oracle/scls_scr'

Delete Node from Cluster and Update OCR Upon successful completion of the rootdelete.sh script, run the rootdeletenode.sh script to delete the node (linux3) from the Oracle cluster and to update the Oracle Cluster Registry (OCR). This script should be run from a pre-existing / available node in the cluster (linux1) as the root user account: Before executing rootdeletenode.sh, we need to know the node number associated with the node name to be deleted from the cluster. To determine the node number, run the following command as the oracle user account from linux1:
$ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2 linux3 3

From the listing above, the node number for linux3 is 3. While logged into linux1 as the root user account, run the following using the name linux3 and the node number 3:
$ su # cd $ORA_CRS_HOME/install # ./rootdeletenode.sh linux3,3 CRS-0210: Could not find resource 'ora.linux3.LISTENER_LINUX3.lsnr'. CRS-0210: Could not find resource 'ora.linux3.ons'. CRS-0210: Could not find resource 'ora.linux3.vip'. CRS-0210: Could not find resource 'ora.linux3.gsd'. CRS-0210: Could not find resource ora.linux3.vip. CRS nodeapps are deleted successfully clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully deleted 14 values from OCR. Key SYSTEM.css.interfaces.nodelinux3 marked for deletion is not there. Ignoring.

Successfully deleted 5 keys from OCR. Node deletion operation successful. 'linux3,3' deleted successfully

To verify that the node was successfully removed, use the following as either the oracle or root user:
$ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2

Update Node List for Oracle Clusterware Software - (Remove linux3) From the node to be deleted (linux3), run the OUI as the oracle user account to update the inventory node list for the Oracle Clusterware software:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY $ cd $ORA_CRS_HOME/oui/bin $ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME CLUSTER_NODES="" -local CRS=true Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.

Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!

De-install Oracle Clusterware Software Next, run the OUI from the node to be deleted (linux3) to de-install the Oracle Clusterware software. Make certain that you choose the home to be removed and not just the products under that home. From linux3 as the oracle user account, run the following:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY $ cd $ORA_CRS_HOME/oui/bin $ ./runInstaller

Screen Name Welcome Screen Inventory: Contents Tab Confirmation

Response Click the Installed Products button. Check the Oracle home to deleted (OraCrs10g_home) and click the Remove button. Acknowledge the warning dialog by clicking Yes to remove the Oracle Clusterware software and to remove the /u01/app/crs directory. A progress bar is displayed while the Oracle Clusterware software is being removed. Once this process has completed, you are returned to the "Inventory: Contents Tab" dialog. After confirming the Oracle Clusterware software (Clusterware home) was successfully removed,Click Close to exit this dialog. Click Cancel to exit the OUI.

Deinstallation Process

Welcome Screen

Update Node List for Remaining Nodes in the Cluster Finally, from linux1 logged in as the oracle user account (and user equivalence enabled), update the Oracle Clusterware inventory node list for all nodes that will remain in the cluster:
$ DISPLAY=<your local workstation>:0.0; export DISPLAY $ cd $ORA_CRS_HOME/oui/bin $ ./runInstaller -updateNodeList ORACLE_HOME=$ORA_CRS_HOME "CLUSTER_NODES={linux1,linux2}" CRS=true Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed. The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oracle/oraInventory 'UpdateNodeList' was successful.

Although the OUI does not launch an installer GUI, the DISPLAY environment variable still needs to be set!

Verify Node to be Deleted is no Longer a Member of the Cluster

Run the following commands to verify that the node to be deleted from the Oracle RAC cluster is no longer a member of the cluster and to verify that the Oracle Clusterware components have been successfully removed from that node. Run the following commands from linux1 as the oracle user account:
$ srvctl status nodeapps -n linux3 PRKC-1056 : Failed to get the hostname for node linux3 PRKH-1001 : HASContext Internal Error [OCR Error(Native: getHostName:[21])]

The error above indicates that cluster.

linux3

is no longer a member of the

$ $ORA_CRS_HOME/bin/crs_stat | grep -i linux3

You should not see any output from the above command
$ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2

You should see the present node list without the deleted node (that is linux1 and linux2 only). Remove/Rename any Remaining Oracle Files from Node to be Deleted From the node to be deleted (linux3), remove/rename any remaining Oracle files while logged in as the root user account:
# mv -f /etc/inittab.no_crs /etc/inittab # rm -f /etc/inittab.orig # rm -f /etc/inittab.crs # # # # # # # # # rm rm rm rm rm rm rm rm rm -rf /etc/oracle -f /etc/oratab -f /etc/oraInst.loc -rf /etc/ORCLcluster -rf /u01/app/oracle -rf /u01/app/crs -f /usr/local/bin/coraenv -f /usr/local/bin/dbhome -f /usr/local/bin/oraenv

Finally, remove the Oracle user account and all associated UNIX groups from linux3:

# userdel -r oracle # groupdel oinstall # groupdel dba

OCFS2 Clean-Up Procedures

Remove OCFS2 from the node to be deleted!

Overview The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. It is also used to store the shared SPFILE for ASM. This section includes instructions for removing the OCFS2 software from the deleted node against a "live" OCFS2 file system. Remove OCFS2 from the Node to be Deleted In this section we will remove the OCFS2 kernel driver, supporting software, and associated directories. Run the following from the node to be deleted (linux3) as the root user account:
# umount /u02 # /etc/init.d/o2cb offline ocfs2 # /etc/init.d/o2cb unload # rpm -qa | grep ocfs2 ocfs2-tools-1.2.4-1 ocfs2-2.6.9-55.EL-1.2.5-6 ocfs2console-1.2.4-1 # rpm -ev ocfs2-tools-1.2.4-1 ocfs2-2.6.9-55.EL-1.2.5-6 ocfs2console-1.2.4-1 warning: /etc/sysconfig/o2cb saved as /etc/sysconfig/o2cb.rpmsave # rm -f /etc/sysconfig/o2cb.rpmsave

# rm -rf /etc/ocfs2

Remove Node to be Deleted from the OCFS2 Cluster The next step is to remove the Oracle RAC node being deleted (linux3) from the "live" OCFS2 cluster. This entails running theo2cb_ctl command-line utility from the remaining two RAC nodes linux1 and linux2. As root, run the following from linux1 and then linux2:
[root@linux1 ~]# o2cb_ctl -D -u -n linux3 o2cb_ctl: Not yet supported [root@linux2 ~]# o2cb_ctl -D -u -n linux3 o2cb_ctl: Not yet supported

o2cb_ctl parameters:
-D -u : Delete an object from the existing OCFS2 Cluster Configuration. : Valid only with -D. When deleting something (node or cluster), it will also remove it from the live cluster (/config). If the parameter is not specified, then only update the /etc/ocfs2/cluster.conf. : Object name which is usually the node name or cluster name.

-n

At the time of this writing, the latest version of OCFS2 (namely the o2cb cluster stack)only supports "adding" nodes dynamically. Deleting a node requires the OCFS2 cluster to be down which means a database outage. This is a serious shortcoming with OCFS2 that hasn't been addressed for some time. Currently, an enhancement request (oss.oracle.com Bugzilla Bug 694) has been submitted to Oracle, however no target date or version has been set for this enhancement.

Removing linux3 from the OCFS2 cluster will have to be performed manually during the next scheduled database outage. After bringing the clustered database and all Oracle Clusterware services down on both linux1 and linux2, remove the following stanza from the /etc/ocfs2/cluster.conf file on both Oracle RAC nodes:
node: ip_port = 7777 ip_address = 192.168.2.107 number = 2 name = linux3 cluster = ocfs2 Before closing out the /etc/ocfs2/cluster.conf file, change the value

ofnode_count to 2 on both Oracle RAC nodes:

node_count = 2

After modifying the /etc/ocfs2/cluster.conf file on both Oracle RAC nodes, unmount the OCFS2 file system and restart the o2cb cluster stack on both nodes:
# umount /u02 # /etc/init.d/o2cb offline ocfs2 # /etc/init.d/o2cb unload # /etc/init.d/o2cb load # /etc/init.d/o2cb online ocfs2 # mount /u02

After mounting the OCFS2 file system on both Oracle RAC nodes, verify that linux3does not exist in the live OCFS2 cluster only linux1 and linux2 should be present:
[root@linux1 node]# ls -l /config/cluster/ocfs2/node total 0 drwxr-xr-x 2 root root 0 Mar 1 01:32 linux1 drwxr-xr-x 2 root root 0 Mar 1 01:32 linux2 [root@linux2 ~]# ls -l /config/cluster/ocfs2/node total 0 drwxr-xr-x 2 root root 0 Mar 1 01:33 linux1 drwxr-xr-x 2 root root 0 Mar 1 01:33 linux2

Remove OCFS2 Mount Options from /etc/fstab Finally, remove the following entry from the
LABEL=oracrsfiles /u02 ocfs2 /etc/fstab

from
0 0

linux3:

_netdev,datavolume,nointr

Remove Access Permissions on Openfiler for the Node to be Deleted

Remove network access permissions in Openfiler for linux3 and disable auto-discovery of iSCSI volumes from linux3!

Overview At this point, linux3 has been successfully removed from the Oracle RAC configuration. Remember that in the existing Oracle RAC configuration that all shared disk storage is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.2 (respin 2). When we added linux3 to the current Oracle RAC configuration in the article "Adding a Node to an Oracle RAC 10g Release 2 Cluster on Linux - (CentOS 4.5 / iSCSI)", we needed to configure network access in Openfiler so that linux3 had permissions to the shared iSCSI volumes. We also configured the iSCSI initiator on linux3 to automatically discover all available iSCSI volumes on each boot. In this final section of the article, we will be revoking network access in Openfiler for the node linux3 and lastly, modifying the iSCSI initiator on linux3 to disable automatic discovery of iSCSI volumes from Openfiler. For the purpose of this example, all iSCSI traffic was configured to use the private network interface eth1 which in this article is on the 192.168.2.0 network. Revoking Openfiler network access permissions from linux3 can be done using the Openfiler Storage Control Center a browser based tool over an https connection on port 446. For example:
https://openfiler1:446/

From the Openfiler Storage Control Center home page, login as an administrator. The default administration login credentials for Openfiler are:

Username: openfiler Password: password

The first page the administrator sees is the [Accounts] / [Authentication] screen. Configuring user accounts and groups is not necessary for this article and will therefore not be discussed. Revoke Access from linux3 to each iSCSI Volume

The first step in this process is to remove access from individual iSCSI volume within Openfiler.

linux3

to each

The current Openfiler configuration contains five logical iSCSI volumes in a single volume group named rac1 which is used for the existing Oracle RAC configuration.
iSCSI / Logical Volumes in Volume Group rac1 Volume Name crs Volume Description Oracle Clusterware Oracle ASM Volume 1 Oracle ASM Volume 2 Oracle ASM Volume 3 Oracle ASM Volume 4 Required Space Filesystem (MB) Type 2,048 iSCSI

asm1

118,720

iSCSI

asm2

118,720

iSCSI

asm3

118,720

iSCSI

asm4

118,720

iSCSI

To view the available iSCSI volumes from within the Openfiler Storage Control Center, navigate to [Volumes] / [List of Existing Volumes]. There we will see all five logical volumes within the volume group rac1:

Figure 2: Current Logical (iSCSI) Volumes For each of the five logical volumes, click on the 'Edit' link (under the Properties column). This will bring up the 'Edit properties' screen for that volume. Scroll to the bottom of this screen; change the access for host linux3-priv from 'Allow' to 'Deny' and click the 'Update' button. Perform this task for all five logical volumes.

Figure 3: Deny Host Access to Logical (iSCSI) Volumes Remove linux3 from Openfiler Local Network Configuration The next step is to modify the local network configuration within Openfiler by removing the node linux3 (which was actually entered as node linux3-priv). This task can be completed using the Openfiler Storage Control Center by navigating to [General] / [Local Networks]. The Local Networks screen allows an administrator to setup (add/remove) networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we want to remove linux3 from the Openfiler local network configuration. Simply, check the linux3-priv / 192.168.2.107 row and click the [Update] button at the bottom of the screen to remove linux3from the Openfiler local network configuration. The following image shows the step required to remove linux3 and update the local network configuration within Openfiler:

Figure 4: Configure Openfiler Host Access for deleted Oracle RAC Node Disable iSCSI Initiator Service on linux3 The final step in this process is to modify the iSCSI initiator service on linux3 so it will not automatically start and therefore will not attempt to discover iSCSI volumes from the Openfiler server. First, stop the iscsi initiator service on linux3:
# service iscsi stop Searching for iscsi-based multipath maps Found 0 maps Stopping iscsid: [ OK ] Removing iscsi driver: [ OK ]

Next, edit the file /etc/iscsi.conf on linux3 and comment out (or remove) the DiscoveryAddress entry which specifies the hostname

of the Openfiler network storage server. In our example, use a the beginning of DiscoveryAddress entry as shown below:
... # DiscoveryAddress=openfiler1-priv ...

at

Finally, configure the iSCSI service to not be active across machine reboots on linux3. The Linux command chkconfig can be used to achieve that as follows:
# chkconfig --level 345 iscsi off

About the Author Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX, Linux, and Windows server environment. Jeff's other interests include mathematical encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 18 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science.

Copyright (c) 1998-2012 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.

I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.
Last modified on Saturday, 18-Sep-2010 17:44:12 EDT Page Count: 32179

DBA Tips Archive for Oracle

Add a Node to an Existing Oracle RAC 10g R2 Cluster on Linux - (RHEL 4.5)
by Jeff Hunter, Sr. Database Administrator Contents Overview Hardware and Costs Install the Linux Operating System Network Configuration Configure Network Security on the Openfiler Storage Server 6. Configure the iSCSI Initiator 7. Create "oracle" User and Directories 8. Configure the Linux Server for Oracle 9. Configure the "hangcheck-timer" Kernel Module 10. Configure RAC Nodes for Remote Access using SSH 11. All Startup Commands for New Oracle RAC Node 12. Install and Configure Oracle Cluster File System (OCFS2) 13. Install and Configure Automatic Storage Management (ASMLib 2.0)
1. 2. 3. 4. 5.

14. Pre-Installation Tasks for Oracle10g Release 2 15. Extend Oracle Clusterware Software to the New Node 16. Extend Oracle Database Software to the New Node 17. Add Listener to New Node 18. Add Database Instance to the New Node 19. About the Author

Overview As your organization grows so too does your need for more application and database resources to support the company's IT systems. Oracle RAC 10g provides a scalable framework which allows DBA's to effortlessly extend the database tier to support this increased demand. As the number of users and transactions increase, additional Oracle instances can be added to the Oracle database cluster to distribute the extra load. This document is an extension to my article "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)". Contained in this new article are the steps required to add a single node to an already running and configured two-node Oracle RAC 10g Release 2 environment on the CentOS 32-bit (x86) platform. Although this article was written and tested on CentOS 4.5 Linux, it should work unchanged with Red Hat Enterprise Linux 4 Update 5. This article assumes the following:
The reader has already built and configured a two-node Oracle RAC 10g Release 2 environment using the article "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)". The article provides comprehensive instructions for building a two-node RAC cluster, each with a single processor running CentOS 4.5, Oracle RAC 10g Release 2, OCFS2, and ASMLib 2.0. The current two-node RAC environment actually consists of three machines two named linux1 and linux2 which each run an Oracle10g instance and a third node to run the network storage server named openfiler1.

Note: The current two-node Oracle RAC environment has been upgraded from its base release (10.2.0.1.0) to version 10.2.0.3.0 by applying the 5337014 patchset (p5337014_10203_LINUX.zip). The patchset was applied to Oracle Clusterware and the Oracle Database software. I also applied the

one-off patchset - "BUNDLE Patch for Base Bug 6000740" (MLR7 ON TOP OF 10.2.0.3) to the Oracle Clusterware and Oracle Database software. The procedures for installing both patchsets are not included in any of the parent article(s).
To maintain the current naming convention, the new Oracle RAC node to be added to the existing cluster will be named linux3 (running a new instance named orcl3) making it a three-node cluster. The new Oracle RAC node should have the same operating system version and installed patches as the current two-node cluster. Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and Oracle Database software installed on their local disks. The current two-node Oracle RAC environment does not use shared Oracle homes for the Clusterware or Database software. The software owner for the Oracle Clusterware and Oracle Database installs will be "oracle". It is important that the UID and GID of the oracle user account be identical to that of the existing RAC nodes. For the purpose of this example, the oracle user account will be defined as follows:
[oracle@linux1 ~]$ id oracle uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)

The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) to store the two files required to be shared by Oracle Clusterware; namely the Oracle Cluster Registry (OCR) file and the Voting Disk. Instructions for installing and adding the new Oracle RAC node to the "live" OCFS2 file system will be included.

Automatic Storage Management (ASM) is being used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area. In addition to ASM, we will also be configuring ASMLib on the new Oracle RAC node. To add instances to an existing RAC database, Oracle Corporation recommends using the Oracle cloning procedures which is described in the Oracle Universal Installer and OPatch User's Guide. This article, however, uses manual procedures to add nodes and instances to the existing Oracle RAC cluster. The manual procedures method described in this article involve extending the RAC database by first extending the Oracle Clusterware home to the new Oracle RAC node and then extending the Oracle Database home. In other words, you extend the software onto the new node in the same order as you installed the clusterware and Oracle database software components on the existing

two-node RAC. During the creation of the existing two-node cluster, the installation of Oracle Clusterware and the Oracle Database software were only performed from one node in the RAC cluster namely from linux1 as theoracle user account. The Oracle Universal Installer (OUI) on that particular node would then use the sshand scp commands to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. The oracle user account on the node running the OUI (runInstaller) had to betrusted by all other nodes in the RAC cluster. This meant that the oracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI (linux1) against all other Linux servers in the cluster without being prompted for a password. The same security requirements hold true for this article. User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securely copied from linux1 to the new Oracle RAC node (linux3) using ssh and scp without being prompted for a password. All shared disk storage for the existing Oracle RAC is based on iSCSI using a Network Storage Server; namely Openfiler Release 2.2 (respin 2). Powered by rPath Linux, Openfiler is a free browser-based network storage management utility that delivers file-based Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single framework. Openfiler supports CIFS, NFS, HTTP/DAV, FTP, however, we will only be making use of its iSCSI capabilities to implement an inexpensive SAN for the shared storage components required by Oracle RAC 10g. This solution offers a low-cost alternative to fibre channel for testing and educational purposes, but given the low-end hardware being used, it should not be used in a production environment. These articles provide a low cost alternative for those who want to become familiar with Oracle RAC 10gusing commercial off the shelf components and downloadable software. Bear in mind that these articles are provided for educational purposes only so the setup is kept simple to demonstrate ideas and concepts. For example, the disk mirroring configured in this article will be setup on one physical disk only, while in practice that should be performed on multiple physical drives. In addition, each Linux node will only be configured with two network cards one for the public network (eth0) and one for the private cluster interconnect "and" network storage server for shared iSCSI access (eth1). For a production RAC implementation, the private interconnect should be at least gigabit (or more) and "only" be used by Oracle to transfer Cluster Manager and Cache Fusion related data. A third dedicated network interface (i.e. eth2) should be configured on another gigabit network for access to the network storage server (Openfiler).

The following is a conceptual look at what the environment will look like after adding the third Oracle RAC node (linux3) to the cluster. Click on the graphic below to enlarge the image:

Figure 1: Adding linux3 to the current Oracle RAC 10g Release 2 Environment

While this article provides comprehensive instructions for successfully adding a node to an existing Oracle RAC 10gsystem, it is by no means a substitute for the official Oracle documentation. In addition to this article, users should also consult the following Oracle documents to gain a full understanding of alternative configuration options, installation, an administration with Oracle RAC 10g. Oracle's official documentation site is docs.oracle.com

Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) Linux

Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 1 Release 2 (10.2) 2 Day + Real Application Clusters Guide - 10g Release 2 (10.2)

Hardware and Costs The hardware used in this article to build the third node (linux3) consists of a Linux workstation and components which can be purchased at many local computer stores or over the Internet.
Oracle RAC Node 3 - (linux3)
Dell Dimension 3000 Series - Intel(R) Pentium(R) 4 Processor at 2.80GHz - 2GB DDR SDRAM (at 333MHz) - 60GB 7200 RPM Internal Hard Drive - Integrated Intel 3D AGP Graphics - Integrated 10/100 Ethernet - (Broadcom BCM4401) - CDROM (48X Max Variable) - 3.5" Floppy - No Keyboard, Monitor, or Mouse - (Connected to KVM Switch) 1 - Ethernet LAN Card Each Linux server for Oracle RAC should contain two NIC adapters. The Dell Dimension includes an integrated 10/100 Ethernet adapter that will be used to connect to the public network. The second NIC adapter will be used for the private network (RAC interconnect and Openfiler networked storage). Select the appropriate NIC adapter that is compatible with the maximum data transmission speed of the network switch to be used for the private network. For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network. Used for RAC interconnect to linux1, linux2 and Openfiler networked storage. Gigabit Ethernet Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT) 2 - Network Cables Category 5e patch cable - (Connect linux3 to public network) Category 5e patch cable - (Connect linux3 to interconnect ethernet switch) US$5 US$5 US$35

US$300

Total

US$345

We are about to start the installation process. As we start to go into the details of the installation, it should be noted that most of the tasks within this document will need to be performed on the new Oracle RAC node (linux3). I will indicate at the beginning of each section whether or not the task(s) should be performed on the new Oracle RAC node, the current Oracle RAC node(s), or on the network storage server (openfiler1).

Install the Linux Operating System

Perform the following installation on the new Oracle RAC node!

After procuring the required hardware, it is time to start the configuration process. The first task we need to perform is to install the Linux operating system. As already mentioned, this article will use CentOS 4.5. Although I have used Red Hat Fedora in the past, I wanted to switch to a Linux environment that would guarantee all of the functionality contained with Oracle. This is where CentOS comes in. The CentOS project takes the Red Hat Enterprise Linux 4 source RPMs and compiles them into a free clone of the Red Hat Enterprise Server 4 product. This provides a free and stable version of the Red Hat Enterprise Linux 4 (AS/ES) operating environment that I can now use for testing different Oracle configurations. I have moved away from Fedora as I need a stable environment that is not only free, but as close to the actual Oracle supported operating system as possible. While CentOS is not the only project performing the same functionality, I tend to stick with it as it is stable and reacts fast with regards to updates by Red Hat. Downloading CentOS

Use the links (below) to download CentOS 4.5. After downloading CentOS, you will then want to burn each of the ISO images to CD. CentOS.org

CentOS-4.5-i386-bin1of4.iso CentOS-4.5-i386-bin2of4.iso CentOS-4.5-i386-bin3of4.iso CentOS-4.5-i386-bin4of4.iso

(622 (636 (638 (313

MB) MB) MB) MB)

If you are downloading the above ISO files to a MS Windows machine, there are many options for burning these images (ISO files) to a CD. You may already be familiar with and have the proper software to burn images to CD. If you are not familiar with this process and do not have the required software to burn images to CD, here are just two (of many) software packages that can be used:

UltraISO Magic ISO Maker

Installing CentOS This section provides a summary of the screens used to install CentOS. For more detailed installation instructions, it is possible to use the manuals from Red Hat Linux http://www.redhat.com/docs/manuals/. I would suggest, however, that the instructions I have provided below be used for this Oracle RAC 10g configuration.
Before installing the Linux operating system on the new Oracle RAC node, you should have the two NIC interfaces (cards) installed.

After downloading and burning the CentOS images (ISO files) to CD, insert CentOS Disk #1 into the new Oracle RAC server (linux3in this example), power it on, and answer the installation screen prompts as noted below. Boot Screen

The first screen is the CentOS boot screen. At the boot: prompt, hit [Enter] to start the installation process. Media Test When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any errors, the media burning software would have warned us. After several seconds, the installer should then detect the video card, monitor, and mouse. The installer then goes into GUI mode. Welcome to CentOS At the welcome screen, click [Next] to continue. Language / Keyboard Selection The next two screens prompt you for the Language and Keyboard settings. In almost all cases, you can accept the defaults. Make the appropriate selection for your configuration and click [Next] to continue. Installation Type Choose the [Custom] option and click [Next] to continue. Disk Partitioning Setup Select [Automatically partition] and click [Next] continue. If there were a previous installation of Linux on this machine, the next screen will ask if you want to "remove" or "keep" old partitions. Select the option to [Remove all partitions on this system]. Also, ensure that the [hda] drive is selected for this installation. I also keep the checkbox [Review (and modify if needed) the partitions created] selected. Click [Next] to continue. You will then be prompted with a dialog window asking if you really want to remove all partitions. Click [Yes] to acknowledge this warning. Partitioning

The installer will then allow you to view (and modify if needed) the disk partitions it automatically selected. For most automatic layouts, the installer will choose 100MB for /boot, double the amount of RAM (systems with < 2GB RAM) or an amount equal to RAM (systems with > 2GB RAM) for swap, and the rest going to the root (/) partition. Starting with EL 4, the installer will create the same disk configuration as just noted but will create them using the Logical Volume Manager (LVM). For example, it will partition the first hard drive (/dev/hda for my configuration) into two partitions one for the /boot partition (/dev/hda1) and the remainder of the disk dedicate to a LVM named VolGroup00 (/dev/hda2). The LVM Volume Group (VolGroup00) is then partitioned into two LVM partitions - one for the root filesystem (/) and another for swap. The main concern during the partitioning phase is to ensure enough swap space is allocated as required by Oracle (which is a multiple of the available RAM). The following is Oracle's requirement for swap space:
Available RAM Swap Space Required

Between 1 GB and 1.5 times the size of 2 GB RAM Between 2 GB and Equal to the size of 8 GB RAM More than 8 GB .75 times the size of RAM

For the purpose of this install, I will accept all automatically preferred sizes. (Including 2GB for swap since I have 2GB of RAM installed.) If for any reason, the automatic layout does not configure an adequate amount of swap space, you can easily change that from this screen. To increase the size of the swap partition, [Edit] the volume group VolGroup00. This will bring up the "Edit LVM Volume Group: VolGroup00" dialog. First, [Edit] and decrease the size of the root file system (/) by the amount you want to add to the swap partition. For example, to add another 512MB to swap, you would decrease the size of the root file system by

512MB (i.e. 36,032MB - 512MB = 35,520MB). Now add the space you decreased from the root file system (512MB) to the swap partition. When completed, click [OK] on the "Edit LVM Volume Group: VolGroup00" dialog. Once you are satisfied with the disk layout, click [Next] to continue. Boot Loader Configuration The installer will use the GRUB boot loader by default. To use the GRUB boot loader, accept all default values and click [Next] to continue. Network Configuration I made sure to install both NIC interfaces (cards) in the new Linux machine before starting the operating system installation. This screen should have successfully detected each of the network devices. First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to not activate eth1 by default. Second, [Edit] both eth0 and eth1 as follows. You may choose to use different IP addresses for both eth0 and eth1 and that is OK. Configure eth1 (the interconnect and storage network) on a different subnet than eth0 (the public network): eth0: - Check OFF the option to [Configure using DHCP] - Leave the [Activate on boot] checked ON - IP Address: 192.168.1.107 - Netmask: 255.255.255.0 eth1: - Check OFF the option to [Configure using DHCP] - Leave the [Activate on boot] checked ON - IP Address: 192.168.2.107 - Netmask: 255.255.255.0 Continue by setting your hostname manually. I used "linux3" for this new Oracle RAC node. Finish this dialog off by supplying your gateway and DNS servers. Firewall

On this screen, make sure to select [No firewall]. Also under the option to "Enable SELinux?", select [Disabled] and click [Next] to continue. You will be prompted with a warning dialog about not setting the firewall. If this occurs, simply hit [Proceed] to continue. Additional Language Support / Time Zone The next two screens allow you to select additional language support and time zone information. In almost all cases, you can accept the defaults. Make the appropriate selection for your configuration and click [Next] to continue. Set Root Password Select a root password and click [Next] to continue. Package Group Selection Scroll down to the bottom of this screen and select [Everything] under the "Miscellaneous" section. Click [Next] to continue. Please note that the installation of Oracle does not require all Linux packages to be installed. My decision to install all packages was for the sake of brevity. Please see section "Pre-Installation Tasks for Oracle10g Release 2" for a more detailed look at the critical packages required for a successful Oracle installation. Also note that with some RHEL 4 distributions, you will not get the "Package Group Selection" screen by default. There, you are asked to simply "Install default software packages" or "Customize software packages to be installed". Select the option to "Customize software packages to be installed" and click [Next] to continue. This will then bring up the "Package Group Selection" screen. Now, scroll down to the bottom of this screen and select [Everything] under the "Miscellaneous" section. Click [Next] to continue. About to Install This screen is basically a confirmation screen. Click [Next] on this screen and then the [Continue] button on the dialog box to start the

installation. During the installation process, you will be asked to switch disks to Disk #2, Disk #3, and then Disk #4. Note that with CentOS 4.5, the installer would ask to switch to Disk #2, Disk #3, Disk #4, Disk #1, and then back to Disk #4. Graphical Interface (X) Configuration With most RHEL 4 distributions (not the case with CentOS 4.5), when the installation is complete, the installer will attempt to detect your video hardware. Ensure that the installer has detected and selected the correct video hardware (graphics card and monitor) to properly use the X Windows server. You will continue with the X configuration in the next serveral screens. Congratulations And that's it. You have successfully installed CentOS on the new Oracle RAC node (linux3). The installer will eject the CD from the CD-ROM drive. Take out the CD and click [Reboot] to reboot the system. When the system boots into Linux for the first time, it will prompt you with another Welcome screen. The following wizard allows you to configure the date and time, add any additional users, test the sound card, and to install any additional CDs. The only screen I care about is the time and date (and if you are using CentOS 4.x, the monitor/display settings). As for the others, simply run through them as there is nothing additional that needs to be installed (at this point anyways!). If everything was successful, you should now be presented with the login screen.

Network Configuration

Perform the following network configuration tasks on the new Oracle RAC node!

Introduction to Network Settings Although we configured several of the network settings during the installation of CentOS, it is important to not skip this section as it contains critical steps that are required for a successful RAC environment. During the Linux O/S install we already configured the IP address and host name for the new Oracle RAC node. We now need to configure the /etc/hosts file as well as adjusting several of the network settings for the interconnect. All nodes in the RAC cluster should have one static IP address for the public network and one static IP address for the private cluster interconnect. The private interconnect should only be used by Oracle to transfer Cluster Manager and Cache Fusion related data along with data for the network storage server (Openfiler). Note that Oracle does not support using the public network interface for the interconnect. You must have one network interface for the public network and another network interface for the private interconnect. For a production RAC implementation, the interconnect should be at least gigabit (or more) and only be used by Oracle as well as having the network storage server (Openfiler) on a separate gigabit network. Configuring Public and Private Network With the new Oracle RAC node, we need to configure the network for access to the public network as well as the private interconnect. The easiest way to configure network settings in Red Hat Linux is with the program Network Configuration. This application can be started from the commandline as the "root" user account as follows:
# su # /usr/bin/system-config-network &

Do not use DHCP naming for the public IP address or the interconnects we need static IP addresses!

Using the Network Configuration application, we will need to configure both NIC devices as well as the /etc/hosts file on all nodes in the RAC cluster. Both of these tasks can be completed using the Network Configuration GUI.

Please note that for the purpose of this example configuration the /etc/hosts entries will be the same for all three Oracle RAC nodes (linux1, linux2, and linux3) as well as the network storage server (openfiler1): Our example configuration will use the following settings for all nodes:
Oracle RAC Node 3 - (linux3)
Devi IP Address Subnet ce eth0 Gateway Purpose

192.168.1.1 255.255.25 192.168. Connects linux3 to the public network 07 5.0 1.1 192.168.2.1 255.255.25 07 5.0 Connects linux3 (interconnect) to linux1/linux2 (linux1-priv/linux2-priv)

eth1

/etc/hosts 127.0.0.1 # Public Network 192.168.1.100 192.168.1.101 192.168.1.107 localhost.localdomain localhost - (eth0) linux1 linux2 linux3

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv 192.168.2.107 linux3-priv # Public Virtual 192.168.1.200 192.168.1.201 192.168.1.207 IP (VIP) addresses - (eth0:1) linux1-vip linux2-vip linux3-vip

# Private Storage Network for Openfiler 192.168.1.195 openfiler1 192.168.2.195 openfiler1-priv

Oracle RAC Node 2 - (linux2)

Devi IP Address Subnet ce eth0 Gateway Purpose

192.168.1.1 255.255.25 192.168. Connects linux2 to the public network 01 5.0 1.1 192.168.2.1 255.255.25 01 5.0 Connects linux2 (interconnect) to linux1/linux3 (linux1-priv/linux3-priv)

eth1

/etc/hosts 127.0.0.1 # Public Network 192.168.1.100 192.168.1.101 192.168.1.107 localhost.localdomain localhost - (eth0) linux1 linux2 linux3

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv 192.168.2.107 linux3-priv # Public Virtual 192.168.1.200 192.168.1.201 192.168.1.207 IP (VIP) addresses - (eth0:1) linux1-vip linux2-vip linux3-vip

# Private Storage Network for Openfiler 192.168.1.195 openfiler1 192.168.2.195 openfiler1-priv

Oracle RAC Node 1 - (linux1)

Devi IP Address Subnet ce eth0 Gateway Purpose

192.168.1.1 255.255.25 192.168. Connects linux1 to the public network 00 5.0 1.1 192.168.2.1 255.255.25 00 5.0 Connects linux1 (interconnect) to linux2/linux3 (linux2-priv/linux3-priv)

eth1

/etc/hosts 127.0.0.1 # Public Network 192.168.1.100 192.168.1.101 192.168.1.107 localhost.localdomain localhost - (eth0) linux1 linux2 linux3

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv 192.168.2.107 linux3-priv # Public Virtual 192.168.1.200 192.168.1.201 192.168.1.207 IP (VIP) addresses - (eth0:1) linux1-vip linux2-vip linux3-vip

# Private Storage Network for Openfiler

192.168.1.195 192.168.2.195

openfiler1 openfiler1-priv

In the screen shots below, only the new Oracle RAC node (linux3) is shown. Ensure that the /etc/hosts file is updated on all participating nodes to access the new Oracle RAC node!

Figure 2: Network Configuration Screen - Node 3 (linux3)

Figure 3: Ethernet Device Screen - eth0 (linux3)

Figure 4: Ethernet Device Screen - eth1 (linux3)

Figure 5: Network Configuration Screen - /etc/hosts (linux3) Once the network is configured, you can use the ifconfig command to verify everything is working. The following example is from the new Oracle RAC node linux3:
# /sbin/ifconfig -a eth0 Link encap:Ethernet HWaddr 00:1E:2A:37:6B:9E inet addr:192.168.1.107 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::21e:2aff:fe37:6b9e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1167677 errors:0 dropped:0 overruns:0 frame:0 TX packets:1842517 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:576629131 (549.9 MiB) TX bytes:2143836310 (1.9 GiB) Interrupt:209 Base address:0xef00 eth1 Link encap:Ethernet HWaddr 00:0E:0C:C0:78:64 inet addr:192.168.2.107 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20e:cff:fec0:7864/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:48 errors:0 dropped:0 overruns:0 frame:0 TX packets:59 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4782 (4.6 KiB) TX bytes:5564 (5.4 KiB) Base address:0xdd80 Memory:fe9c0000-fe9e0000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2034 errors:0 dropped:0 overruns:0 frame:0 TX packets:2034 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2386686 (2.2 MiB) TX bytes:2386686 (2.2 MiB) Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

sit0

Verify Network Access to All Nodes Verify that the new Oracle RAC node has access to the public and private network for all current nodes. From linux3:
# ping -c 1 linux1 | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms # ping -c 1 linux1-priv | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms # ping -c 1 linux2 | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms # ping -c 1 linux2-priv | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms # ping -c 1 openfiler1 | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms # ping -c 1 openfiler1-priv | grep '1 packets transmitted' 1 packets transmitted, 1 received, 0% packet loss, time 0ms

Confirm the RAC Node Name is Not Listed in Loopback Address Ensure that the new Oracle RAC node (linux3) is not included for the loopback address in the /etc/hosts file. If the machine name is listed in the in the loopback address entry as below:

it will need to be removed as shown below:

127.0.0.1

127.0.0.1

linux3 localhost.localdomain localhost

localhost.localdomain localhost

If the RAC node name is listed for the loopback address, you will receive the following error during the RAC installation:
ORA-00603: ORACLE server session terminated by fatal error

or

ORA-29702: error occurred in Cluster Group Service operation

Confirm localhost is defined in the /etc/hosts file for the loopback address Ensure that the entry for localhost.localdomain and localhost are included for the loopback address in the /etc/hosts file for the new Oracle RAC node:
127.0.0.1 localhost.localdomain localhost

If an entry does not exist for localhost in the /etc/hosts file, Oracle Clusterware will be unable to start the application resources notably the ONS process. The error would indicate "Failed to get IP for localhost" and will be written to the log file for ONS. For example:
CRS-0215 could not start resource 'ora.linux3.ons'. Check log file "/u01/app/crs/log/linux3/racg/ora.linux3.ons.log" for more details.

The ONS log file will contain lines similar to the following:

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved. 2007-04-14 13:10:02.729: [ RACG][3086871296][13316][3086871296] [ora.linux3.ons]: Failed to get IP for localhost (1) Failed to get IP for localhost (1) Failed to get IP for localhost (1) onsctl: ons failed to start ...

Adjusting Network Settings With Oracle 9.2.0.1 and onwards, Oracle now makes use of UDP as the default protocol on Linux for inter-process communication (IPC), such as Cache Fusion and Cluster Manager buffer transfers between instances within the RAC cluster.

Oracle strongly suggests to adjust the default and maximum send buffer size (SO_SNDBUF socket option) to 256 KB, and the default and maximum receive buffer size (SO_RCVBUF socket option) to 256 KB. The receive buffers are used by TCP and UDP to hold received data until it is read by the application. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This means that datagrams will be discarded if they don't fit in the socket receive buffer. This could cause the sender to overwhelm the receiver. The default and maximum window size can be changed without a reboot. Add the following entries to the /etc/sysctl.conf file on the new Oracle RAC node:
# # # # # # # # # # # # # # # # # # # # +---------------------------------------------------------+ | ADJUSTING NETWORK SETTINGS | +---------------------------------------------------------+ | With Oracle 9.2.0.1 and onwards, Oracle now makes use | | of UDP as the default protocol on Linux for | | inter-process communication (IPC), such as Cache Fusion | | and Cluster Manager buffer transfers between instances | | within the RAC cluster. Oracle strongly suggests to | | adjust the default and maximum receive buffer size | | (SO_RCVBUF socket option) to 256 KB, and the default | | and maximum send buffer size (SO_SNDBUF socket option) | | to 256 KB. The receive buffers are used by TCP and UDP | | to hold received data until it is read by the | | application. The receive buffer cannot overflow because | | the peer is not allowed to send data beyond the buffer | | size window. This means that datagrams will be | | discarded if they don't fit in the socket receive | | buffer. This could cause the sender to overwhelm the | | receiver. | +---------------------------------------------------------+

# +---------------------------------------------------------+ # | Default setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. | # +---------------------------------------------------------+ net.core.rmem_default=262144 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. | # +---------------------------------------------------------+ net.core.rmem_max=262144 # +---------------------------------------------------------+ # | Default setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.core.wmem_default=262144

# +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.core.wmem_max=262144

Then, ensure that each of these parameters are truly in effect by running the following command on the new Oracle RAC node:
# sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.core.rmem_default = 262144 net.core.rmem_max = 262144 net.core.wmem_default = 262144 net.core.wmem_max = 262144

Check and turn off UDP ICMP rejections: During the Linux installation process, I indicated to not configure the firewall option. (By default the option to configure a firewall is selected by the installer.) This has burned me several times so I like to do a double-check that the firewall option is not configured and to ensure udp ICMP filtering is turned off. If UDP ICMP is blocked or rejected by the firewall, the Oracle Clusterware software will crash after several minutes of running. When the Oracle Clusterware process fails, you will have something similar to the following in the <machine_name>_evmocr.log file:
08/29/2005 22:17:19 oac_init:2: Could not connect to server, clsc retcode = 9 08/29/2005 22:17:19 a_init:12!: Client init unsuccessful : [32] ibctx:1:ERROR: INVALID FORMAT proprinit:problem reading the bootblock or superbloc 22

When experiencing this type of error, the solution is to remove the udp ICMP (iptables) rejection rule - or to simply have the firewall option turned off. The Oracle Clusterware software will then start to operate normally and not crash. The following commands should be executed as the root user account:

1. Check to ensure that the firewall option is turned off. If the firewall option is stopped (like it is in my example below) you do not have to proceed with the following steps.
2. # /etc/rc.d/init.d/iptables status Firewall is stopped.

3. If the firewall option is operating you will need to first manually disable UDP ICMP rejections:
4. # /etc/rc.d/init.d/iptables stop 5. 6. Flushing firewall rules: [ OK ] 7. Setting chains to policy ACCEPT: filter [ Unloading iptables modules: [ OK ]

OK

8. Then, to turn UDP ICMP rejections off for next server reboot (which should always be turned off):
# chkconfig iptables off

Configure Network Security on the Openfiler Storage Server

Perform the following configuration tasks on the network storage server (openfiler1)!

With the network now setup, the next step is to configure network access in Openfiler so that the new Oracle RAC node (linux3) has permissions to the shared iSCSI volumes used in the current Oracle RAC 10g environment. For the purpose of this example, all iSCSI traffic will use the private network interface eth1 which in this article is on the 192.168.2.0 network. Openfiler administration is performed using the Openfiler Storage Control Center a browser based tool over an https connection on port 446. For example:
https://openfiler1:446/

From the Openfiler Storage Control Center home page, login as an administrator. The default administration login credentials for Openfiler are:

Username: openfiler Password: password

The first page the administrator sees is the [Accounts] / [Authentication] screen. Configuring user accounts and groups is not necessary for this article and will therefore not be discussed. Services This article assumes that the current Oracle RAC 10g environment is operational and therefore the iSCSI services should already be enabled within Openfiler. To verify the iSCSI services are running, use the Openfiler Storage Control Center and navigate to [Services] / [Enable/Disable]:

Figure 6: Verify iSCSI Services are Enabled Another method is to SSH into the Openfiler server and verify the iscsi-target service is running:
[root@openfiler1 ~]# service iscsi-target status ietd (pid 3784) is running...

Network Access Restriction The next step is to configure network access in Openfiler so that the new Oracle RAC node (linux3) has permissions to the shared iSCSI volumes used in the current Oracle RAC 10g environment.

Again, this task can be completed using the Openfiler Storage Control Center by navigating to [General] / [Local Networks]. The Local Networks screen allows an administrator to setup networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we will want to add the new Oracle RAC node individually rather than allowing the entire 192.168.2.0 network have access to Openfiler resources. When entering the new Oracle RAC node, note that the 'Name' field is just a logical name used for reference only. As a convention when entering nodes, I simply use the node name defined for that IP address. Next, when entering the actual node in the 'Network/Host' field, always use it's IP address even though its host name may already be defined in your /etc/hosts file or DNS. Lastly, when entering actual hosts in our Class C network, use a subnet mask of 255.255.255.255. It is important to remember that you will be entering the IP address of the private network (eth1) for the new Oracle RAC node. The following image shows the results of adding the new Oracle RAC node linux3 to the local network configuration:

Figure 7: Configure Openfiler Host Access for new Oracle RAC Node Current Logical iSCSI Volumes The current Openfiler configuration contains five logical iSCSI volumes in a single volume group named rac1.
iSCSI / Logical Volumes in Volume Group rac1 Volume Name crs Volume Description Oracle Clusterware Oracle ASM Volume 1 Required Space Filesystem (MB) Type 2,048 iSCSI

asm1

118,720

iSCSI

asm2

Oracle ASM Volume 2 Oracle ASM Volume 3 Oracle ASM Volume 4

118,720

iSCSI

asm3

118,720

iSCSI

asm4

118,720

iSCSI

To view the available iSCSI volumes from within the Openfiler Storage Control Center, navigate to [Volumes] / [List of Existing Volumes]. There we will see all five logical volumes within the volume group rac1:

Figure 8: Current Logical (iSCSI) Volumes Grant Access Rights to New Logical Volumes Before an iSCSI client can have access to any of the iSCSI volumes, it needs to be granted the appropriate permissions. In this section, we need to grant access to each of the five local iSCSI volumes to the new Oracle RAC node linux3.

From the Openfiler Storage Control Center, navigate to [Volumes] / [List of Existing Volumes]. This will present the screen shown in the previous section. For each of the five logical volumes, click on the 'Edit' link (under the Properties column). This will bring up the 'Edit properties' screen for that volume. Scroll to the bottom of this screen; change the access for host linux3-priv from 'Deny' to 'Allow' and click the 'Update' button. Perform this task for all five logical volumes.

Figure 9: Grant Host Access to Logical (iSCSI) Volumes

Configure the iSCSI Initiator

Configure the iSCSI initiator on the new Oracle RAC node!

An iSCSI client can be any system (Linux, Unix, MS Windows, Apple Mac, etc.) for which iSCSI support (a driver) is available. In our case, the clients are the three Oracle RAC nodes, (linux1, linux2, and linux3), running Red Hat 4. In this section we will be configuring the iSCSI initiator on the new Oracle RAC node linux3. This involves configuring the /etc/iscsi.conf file on the new Oracle RAC node with the name of the network storage server (openfiler1) so it can discover the current iSCSI volumes.

iSCSI (initiator) service On the new Oracle RAC node, we have to make sure the iSCSI (initiator) service is up and running. If not installed as part of the operating system setup, the iscsi-initiator-utils RPM (i.e. iscsiinitiator-utils-4.0.3.0-5.i386.rpm) should be downloaded and installed on the new Oracle RAC node.
The new Oracle RAC node must have the iscsi-initiator-utils RPM installed. To determine if this package is installed, perform the following:
# rpm -qa | grep iscsi iscsi-initiator-utils-4.0.3.0-5

If not installed, the iscsi-initiator-utils RPM package can be found on disk 3 of 4 of the RHEL4 Update 5 distribution or downloaded from one of the Internet RPM resources.

Use the following command to install the iscsi-initiator-utils RPM package if not present:
# rpm -Uvh iscsi-initiator-utils-4.0.3.0-5.i386.rpm warning: iscsi-initiator-utils-4.0.3.0-5.i386.rpm: V3 DSA signature: NOKEY, key ID 443e1821 Preparing... ########################################### [100%] 1:iscsi-initiator-utils ########################################### [100%]

After verifying that the iscsi-initiator-utils RPM is installed, the only configuration step required on the new Oracle RAC node (iSCSI client) is to specify the network storage server (iSCSI server) in the /etc/iscsi.conf file. Edit the /etc/iscsi.conf file and include an entry for DiscoveryAddress which specifies the hostname of the Openfiler network storage server. In our case that was:
... DiscoveryAddress=openfiler1-priv ...

After making the change to the /etc/iscsi.conf file on the new Oracle RAC node, we can start (or restart) the iscsi initiator service on that node:
# service iscsi restart Searching for iscsi-based multipath maps Found 0 maps

Stopping iscsid: iscsid not running Checking iscsi config: [ OK ] Loading iscsi driver: [ OK ] Starting iscsid: [ OK ]

We should also configure the iSCSI service to be active across machine reboots for the new Oracle RAC node. The Linux commandchkconfig can be used to achieve that as follows:
# chkconfig --level 345 iscsi on

Discovering iSCSI Targets The iSCSI initiator service should now be configured and started on the new Oracle RAC node. In the parent to this article ("Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)"), we needed go through the arduous task of mapping the iSCSI target names discovered from Openfiler to the local SCSI device name on one of the Oracle RAC nodes. Given that all five logical iSCSI volumes were partitioned and formatted with labels in that article, we don't have to perform that task again. Note that one of the iSCSI volumes was formatted and labeled using OCFS2 while the other four were labeled for use by ASM. In this section, we simply want to verify that the new Oracle RAC node was able to successfully discover the five logical iSCSI volumes on the Openfiler server. When the Openfiler server publishes available iSCSI targets, configured clients get the message that new iSCSI disks are now available. This happens when the iscsitarget service gets started/restarted on the Openfiler server or when the iSCSI initiator service is started/restarted on the client. We would see something like this in the client's /var/log/messages file:
... Jan 21 16:41:29 Jan 21 16:41:29 192.168.2.195 Jan 21 16:41:29 Jan 21 16:41:29 Jan 21 16:41:29 Jan 21 16:41:29 Jan 21 16:41:29 Jan 21 16:41:29 linux3 iscsi: iscsid startup succeeded linux3 iscsid[13822]: Connected to Discovery Address linux3 linux3 linux3 linux3 linux3 linux3 kernel: kernel: kernel: kernel: kernel: kernel: iscsi-sfnet:host0: Session established iscsi-sfnet:host2: Session established iscsi-sfnet:host1: Session established scsi0 : SFNet iSCSI driver scsi2 : SFNet iSCSI driver scsi1 : SFNet iSCSI driver

Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0 Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04 Jan 21 16:41:29 linux3 kernel: SCSI device sda: 243138560 512-byte hdwr sectors (124487 MB) Jan 21 16:41:29 linux3 kernel: SCSI device sda: drive cache: write through Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0 Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04 Jan 21 16:41:29 linux3 kernel: SCSI device sda: 243138560 512-byte hdwr sectors (124487 MB) Jan 21 16:41:29 linux3 kernel: iscsi-sfnet:host3: Session established Jan 21 16:41:29 linux3 kernel: iscsi-sfnet:host4: Session established Jan 21 16:41:29 linux3 kernel: scsi3 : SFNet iSCSI driver Jan 21 16:41:29 linux3 kernel: SCSI device sda: drive cache: write through Jan 21 16:41:29 linux3 kernel: sda: unknown partition table Jan 21 16:41:29 linux3 kernel: Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0 Jan 21 16:41:29 linux3 scsi.agent[13934]: disk at /devices/platform/host0/target0:0:0/0:0:0:0 Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04 Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0 Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04 Jan 21 16:41:29 linux3 kernel: scsi4 : SFNet iSCSI driver Jan 21 16:41:29 linux3 kernel: SCSI device sdb: 243138560 512-byte hdwr sectors (124487 MB) Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0 Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04 Jan 21 16:41:29 linux3 kernel: SCSI device sdb: drive cache: write through Jan 21 16:41:29 linux3 scsi.agent[13983]: disk at /devices/platform/host2/target2:0:0/2:0:0:0 Jan 21 16:41:29 linux3 scsi.agent[13996]: disk at /devices/platform/host3/target3:0:0/3:0:0:0 Jan 21 16:41:30 linux3 kernel: SCSI device sdb: 243138560 512-byte hdwr sectors (124487 MB) Jan 21 16:41:30 linux3 kernel: SCSI device sdb: drive cache: write through Jan 21 16:41:30 linux3 kernel: sdb: unknown partition table Jan 21 16:41:30 linux3 kernel: Attached scsi disk sdb at scsi2, channel 0, id 0, lun 0 Jan 21 16:41:30 linux3 kernel: SCSI device sdc: 243138560 512-byte hdwr sectors (124487 MB) Jan 21 16:41:30 linux3 kernel: SCSI device sdc: drive cache: write through Jan 21 16:41:30 linux3 kernel: SCSI device sdc: 243138560 512-byte hdwr sectors (124487 MB) Jan 21 16:41:30 linux3 kernel: SCSI device sdc: drive cache: write through Jan 21 16:41:30 linux3 kernel: sdc: unknown partition table Jan 21 16:41:30 linux3 kernel: Attached scsi disk sdc at scsi3, channel 0, id 0, lun 0

Jan 21 16:41:30 linux3 kernel: SCSI device sdd: 243138560 512-byte hdwr sectors (124487 MB) Jan 21 16:41:30 linux3 kernel: SCSI device sdd: drive cache: write through Jan 21 16:41:30 linux3 kernel: SCSI device sdd: 243138560 512-byte hdwr sectors (124487 MB) Jan 21 16:41:30 linux3 kernel: SCSI device sdd: drive cache: write through Jan 21 16:41:30 linux3 kernel: sdd: unknown partition table Jan 21 16:41:30 linux3 kernel: Attached scsi disk sdd at scsi1, channel 0, id 0, lun 0 Jan 21 16:41:30 linux3 kernel: SCSI device sde: 4194304 512-byte hdwr sectors (2147 MB) Jan 21 16:41:30 linux3 scsi.agent[14032]: disk at /devices/platform/host4/target4:0:0/4:0:0:0 Jan 21 16:41:30 linux3 scsi.agent[14045]: disk at /devices/platform/host1/target1:0:0/1:0:0:0 Jan 21 16:41:30 linux3 kernel: SCSI device sde: drive cache: write through Jan 21 16:41:30 linux3 kernel: SCSI device sde: 4194304 512-byte hdwr sectors (2147 MB) Jan 21 16:41:30 linux3 kernel: SCSI device sde: drive cache: write through Jan 21 16:41:30 linux3 kernel: sde: unknown partition table Jan 21 16:41:30 linux3 kernel: Attached scsi disk sde at scsi4, channel 0, id 0, lun 0 ...

The above entries show that the client (linux3) was able to establish the iSCSI sessions with the iSCSI storage server (openfiler1-priv at 192.168.2.195). Another method not only checks for the existence of the iSCSI volumes, but also displays how the local SCSI device names map to iSCSI targets' host IDs and LUNs. Use the following script which was provided by Martin Jones to display these mappings:
iscsi-ls-map.sh # --------------------# FILE: iscsi-ls-map.sh # --------------------RUN_USERID=root export RUN_USERID RUID=`id | awk -F\( '{print $2}'|awk -F\) '{print $1}'` if [[ ${RUID} != "$RUN_USERID" ]];then echo " " echo "You must be logged in as $RUN_USERID to run this script." echo "Exiting script." echo " " exit 1 fi dmesg | grep "^Attach" \ | awk -F" " '{ print "/dev/"$4 " " $6 }' \

| sed -e 's/,//' | sed -e 's/scsi//' \ | sort -n -k2 \ | sed -e '/disk1/d' > /tmp/tmp_scsi_dev iscsi-ls | egrep -e "TARGET NAME" -e "HOST ID" \ | awk -F" " '{ if ($0 ~ /^TARGET.*/) printf $4; if ( $0 ~ /^HOST/) printf " %s\n",$4}' \ | sort -n -k2 \ | cut -d':' -f2- \ | cut -d'.' -f2- > /tmp/tmp_scsi_targets join -t" " -1 2 -2 2 /tmp/tmp_scsi_dev /tmp/tmp_scsi_targets > MAP echo "Host / SCSI ID echo "---------------cat MAP | sed -e 's/ / rm -f MAP SCSI Device Name ----------------------/g' iSCSI Target Name" -----------------"

Example run:
# ./iscsi-ls-map.sh Host / SCSI ID SCSI Device Name ---------------- -----------------------0 /dev/sda 1 /dev/sdd 2 /dev/sdb 3 /dev/sdc 4 /dev/sde iSCSI Target Name ----------------asm4 asm3 asm2 asm1 crs

Create "oracle" User and Directories

Perform the following tasks on the new Oracle RAC node!

I will be using the Oracle Cluster File System, Release 2 (OCFS2) to store the files required to be shared for the Oracle Clusterware software. When using OCFS2, the UID of the UNIX user "oracle" and GID of the UNIX group "oinstall" must be the same on all of the Oracle RAC nodes in the cluster. If either the UID or GID are different, the files on the OCFS2 file system will show up as "unowned" or may even

be owned by a different user. For this article and its parent article, I will use 501 for the "oracle" UID and 501 for the "oinstall" GID. Note that members of the UNIX group oinstall are considered the "owners" of the Oracle software. Members of the dba group can administer Oracle databases, for example starting up and shutting down databases. In this article, we are creating the oracle user account to have both responsibilities!
This guide adheres to the Optimal Flexible Architecture (OFA) for naming conventions used in creating the directory structure.

Create Group and User for Oracle Lets start this section by creating the UNIX and oracle user account:
oinstall

and

dba

group

# groupadd -g 501 oinstall # groupadd -g 502 dba # useradd -m -u 501 -g oinstall -G dba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle # id oracle uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)

Set the password for the

oracle

account:

# passwd oracle Changing password for user oracle. New UNIX password: xxxxxxxxxxx Retype new UNIX password: xxxxxxxxxxx passwd: all authentication tokens updated successfully.

Verify That the User nobody Exists Before installing the Oracle software, complete the following procedure to verify that the user nobody exists on the system: 1. To determine if the user exists, enter the following command:
2. # id nobody uid=99(nobody) gid=99(nobody) groups=99(nobody)

If this command displays information about the nobody user, then you do not have to create that user.

3.

If the user nobody does not exist, then enter the following command to create it:
# /usr/sbin/useradd nobody

Create the Oracle Base Directory The next step is to create a new directory that will be used to store the Oracle Database software. When configuring the oracle user's environment (later in this section) we will be assigning the location of this directory to the $ORACLE_BASE environment variable. The following assumes that the directories are being created in the root file system. Please note that this is being done for the sake of simplicity and is not recommended as a general practice. Normally, these directories would be created on a separate file system. After the directory is created, you must then specify the correct owner, group, and permissions for it. Perform the following on the new Oracle RAC node:
# mkdir -p /u01/app/oracle # chown -R oracle:oinstall /u01/app/oracle # chmod -R 775 /u01/app/oracle

At the end of this procedure, you will have the following:

owned by root. /u01/app owned by root. /u01/app/oracle owned by oracle:oinstall with 775 permissions. This ownership and permissions enables the OUI to create the oraInventory directory, in the path /u01/app/oracle/oraInventory.
/u01

Create the Oracle Clusterware Home Directory Next, create a new directory that will be used to store the Oracle Clusterware software. When configuring the oracle user's environment (later in this section) we will be assigning the location of this directory to the $ORA_CRS_HOME environment variable.

As noted in the previous section, the following assumes that the directories are being created in the root file system. This is being done for the sake of simplicity and is not recommended as a general practice. Normally, these directories would be created on a separate file system. After the directory is created, you must then specify the correct owner, group, and permissions for it. Perform the following on the new Oracle RAC node:
# mkdir -p /u01/app/crs # chown -R oracle:oinstall /u01/app/crs # chmod -R 775 /u01/app/crs

At the end of this procedure, you will have the following:

owned by root. /u01/app owned by root. /u01/app/crs owned by oracle:oinstall with 775 permissions. These permissions are required for Oracle Clusterware installation and are changed during the installation process.
/u01

Create Mount Point for OCFS2 / Clusterware Let's now create the mount point for the Oracle Cluster File System, Release 2 (OCFS2) that will be used to store the two Oracle Clusterware shared files. Perform the following on the new Oracle RAC node:
# mkdir -p /u02 # chown -R oracle:oinstall /u02 # chmod -R 775 /u02

Create Login Script for oracle User Account To ensure that the environment is setup correctly for the "oracle" UNIX userid on the new Oracle RAC node, use the following.bash_profile:
When you are setting the Oracle environment variables for each Oracle RAC node, ensure to assign each RAC node a unique Oracle SID!

For this example, I used:

linux1 : ORACLE_SID=orcl1 linux2 : ORACLE_SID=orcl2 linux3 : ORACLE_SID=orcl3

Login to the new Oracle RAC node as the oracle user account:
# su - oracle .bash_profile for Oracle User # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi alias ls="ls -FA" alias s="screen -DRRS iPad -t iPad" export JAVA_HOME=/usr/local/java # User export export export export export specific environment and startup programs ORACLE_BASE=/u01/app/oracle ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1 ORA_CRS_HOME=/u01/app/crs ORACLE_PATH=$ORACLE_BASE/dba_scripts/sql:.:$ORACLE_HOME/rdbms/admin CV_JDKHOME=/usr/local/java

# Each RAC node must have a unique ORACLE_SID. (i.e. orcl1, orcl2, orcl3,...) export ORACLE_SID=orcl3 export export export export export export export export export export export export export export export export export PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin PATH=${PATH}:$ORACLE_BASE/dba_scripts/bin ORACLE_TERM=xterm TNS_ADMIN=$ORACLE_HOME/network/admin ORA_NLS10=$ORACLE_HOME/nls/data NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS" LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib CLASSPATH=$ORACLE_HOME/JRE CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib THREADS_FLAG=native TEMP=/tmp TMPDIR=/tmp

Configure the Linux Server for Oracle

Perform the following tasks on the new Oracle RAC node!

The kernel parameters and shell limits discussed in this section will need to be defined on the new Oracle RAC node every time the machine is booted. This section will not go into great depth in explaining the purpose of those kernel parameters that are required by Oracle (These parameters are described in detail in the parent to this article). Provided in this section, however, are instructions on how to set all required kernel parameters for Oracle and how to have them enabled when the node boots. Further instructions for configuring kernel parameters in a startup script (/etc/sysctl.conf) is included in the section "All Startup Commands for New Oracle RAC Node". Swap Space Considerations

Installing Oracle10g Release 2 requires a minimum of 512MB of memory. (An inadequate amount of swap during the installation will cause the Oracle Universal Installer to either "hang" or "die") To check the amount of memory you have, type:
# cat /proc/meminfo | grep MemTotal MemTotal: 2074428 kB

To check the amount of swap you have allocated, type:

# cat /proc/meminfo | grep SwapTotal SwapTotal: 2031608 kB

If you have less than 512MB of memory (between your RAM and SWAP), you can add temporary swap space by creating a temporary swap file. This way you do not have to use a raw device or even more drastic, rebuild your system.

As root, make a file that will act as additional swap space, let's say about 500MB:
# dd if=/dev/zero of=tempswap bs=1k count=500000

Now we should change the file permissions:

# chmod 600 tempswap

Finally we format the "partition" as swap and add it to the swap space:
# mke2fs tempswap # mkswap tempswap # swapon tempswap

Configuring Kernel Parameters and Shell Limits The kernel parameters and shell limits presented in this section are recommended values only as documented by Oracle. For production database systems, Oracle recommends that you tune these values to optimize the performance of the system. On the new Oracle RAC node, verify that the kernel parameters described in this section are set to values greater than or equal to the recommended values. Also note that when setting the four semaphore values that all four values need to be entered on one line. Setting Shared Memory / Semaphores / File Handles / Local IP Range Set the following kernel parameters in the new Oracle RAC node.
/etc/sysctl.conf

file on the

cat >> /etc/sysctl.conf <<EOF # +---------------------------------------------------------+ # | ADJUSTING ADDITIONAL KERNEL PARAMETERS FOR ORACLE | # +---------------------------------------------------------+ # | Configure the kernel parameters for all Oracle Linux | # | servers by setting shared memory and semaphores, | # | setting the maximum amount of file handles, and setting | # | the IP local port range. | # +---------------------------------------------------------+ # +---------------------------------------------------------+ # | SHARED MEMORY | # +---------------------------------------------------------+ kernel.shmmax=2147483648 # +---------------------------------------------------------+

# | SEMAPHORES | # | ---------| # | | # | SEMMSL_value SEMMNS_value SEMOPM_value SEMMNI_value | # | | # +---------------------------------------------------------+ kernel.sem=250 32000 100 128 # +---------------------------------------------------------+ # | FILE HANDLES | # ----------------------------------------------------------+ fs.file-max=65536 # +---------------------------------------------------------+ # | LOCAL IP RANGE | # ----------------------------------------------------------+ net.ipv4.ip_local_port_range=1024 65000 EOF

Setting Shell Limits for the oracle User To improve the performance of the software on Linux systems, Oracle recommends you increase the following shell limits for the oracle user:
Shell Limit Maximum number of open file descriptors Item in limits.conf nofile Hard Limit 65536 16384

Maximum number of processes available to a nproc single user

To make these changes, run the following as root:

cat >> oracle oracle oracle oracle EOF /etc/security/limits.conf <<EOF soft nproc 2047 hard nproc 16384 soft nofile 1024 hard nofile 65536

cat >> /etc/pam.d/login <<EOF session required /lib/security/pam_limits.so EOF

Update the default shell startup file for the "oracle" UNIX account.

For the Bourne, Bash, or Korn shell, add the following lines to the /etc/profile file by running the following command:
cat >> /etc/profile <<EOF if [ \$USER = "oracle" ]; then if [ \$SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi EOF

For the C shell (csh or tcsh), add the following lines to the /etc/csh.login file by running the following command:
cat >> /etc/csh.login <<EOF if ( \$USER == "oracle" ) then limit maxproc 16384 limit descriptors 65536 endif EOF

Activating All Kernel Parameters for the System At this point, we have covered all of the required Linux kernel parameters needed for a successful Oracle installation and configuration. The sections above configured the Linux system to persist each of the kernel parameters through reboots on system startup by placing them all in the /etc/sysctl.conf file. We could reboot at this point to ensure all of these parameters are set in the kernel or we could simply "run" the /etc/sysctl.conf file by running the following command as root on the new Oracle RAC node:
# sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.core.rmem_default = 262144 net.core.rmem_max = 262144 net.core.wmem_default = 262144 net.core.wmem_max = 262144

kernel.shmmax = 2147483648 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000

Setting the Correct Date and Time on the new Oracle RAC Node When adding the new Oracle RAC node to the cluster, the Oracle Universal Installer (OUI) copies the Oracle Clusterware and Oracle Database software from the source RAC node (linux1 in this article) to the new node in the cluster (linux3). During the remote copy process, the OUI will execute the UNIX "tar" command on the remote node (linux3) to extract the files that were archived and copied over. If the date and time on the node performing the install is greater than that of the node it is copying to, the OUI will throw an error from the "tar" command indicating it is attempting to extract files stamped with a time in the future:
Error while copying directory /u01/app/crs with exclude file list 'null' to nodes 'linux3'. [PRKC-1002 : All the submitted commands did not execute successfully] --------------------------------------------linux3: /bin/tar: ./bin/lsnodes: time stamp 2008-02-13 09:21:34 is 735 s in the future /bin/tar: ./bin/olsnodes: time stamp 2008-02-13 09:21:34 is 735 s in the future ...(more errors on this node)

Please note that although this would seem like a severe error from the OUI, it can safely be disregarded as a warning. The "tar" command DOES actually extract the files; however, when you perform a listing of the files (using ls -l) on the remote node (the new Oracle RAC node), they will be missing the time field until the time on the remote server is greater than the timestamp of the file. Before attempting to add the new node, ensure that all nodes in the cluster are set as closely as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server. Accessing a Network Time Protocol server, however, may not always be an option. In this case, when manually setting the date and time for the nodes in the cluster, ensure that the date and time of the node you are performing the software installations from

(linux1) is less than the new node being added to the cluster (linux3). I generally use a 20 second difference as shown in the following example: Show the date and time from linux1:
# date Thu Feb 14 00:17:00 EST 2008

Setting the date and time on the new Oracle RAC node linux3:
# date -s "2/14/2008 00:17:20"

The RAC configuration described in this article does not make use of a Network Time Protocol server.

Configure the "hangcheck-timer" Kernel Module

Perform the following tasks on the new Oracle RAC node!

Oracle 9.0.1 and 9.2.0.1 used a userspace watchdog daemon called watchdogd to monitor the health of the cluster and to restart a RAC node in case of a failure. Starting with Oracle 9.2.0.2 (and still available in Oracle10g Release 2), the watchdog daemon has been deprecated by a Linux kernel module named hangcheck-timer which addresses availability and reliability problems much better. The hang-check timer is loaded into the Linux kernel and checks if the system hangs. It will set a timer and check the timer after a certain amount of time. There is a configurable threshold to hang-check that, if exceeded will reboot the machine. Although the hangchecktimer module is not required for Oracle Clusterware (Cluster Manager) operation, it is highly recommended by Oracle. The hangcheck-timer.ko Module The hangcheck-timer module uses a kernel-based timer that periodically checks the system task scheduler to catch delays in order to determine the health of the system. If the system hangs or

pauses, the timer resets the node. The hangcheck-timer module uses theTime Stamp Counter (TSC) CPU register which is a counter that is incremented at each clock signal. The TCS offers much more accurate time measurements since this register is updated by the hardware automatically. Much more information about the hangcheck-timer project can be found here. Installing the hangcheck-timer.ko Module The hangcheck-timer was normally shipped only by Oracle, however, this module is now included with Red Hat Linux AS starting with kernel versions 2.4.9-e.12 and higher. The hangcheck-timer should already be included. Use the following to ensure that you have the module included:
# find /lib/modules -name "hangcheck-timer.ko" /lib/modules/2.6.9-55.EL/kernel/drivers/char/hangcheck-timer.ko

In the above output, we care about the hangcheck timer object (hangcheck-timer.ko) in the /lib/modules/2.6.955.EL/kernel/drivers/char directory. Configuring and Loading the hangcheck-timer Module There are two key parameters to the
hangcheck-tick: hangcheck-timer

module:

This parameter defines the period of time between checks of system health. The default value is 60 seconds; Oracle recommends setting it to 30 seconds. hangcheck-margin: This parameter defines the maximum hang delay that should be tolerated before hangcheck-timer resets the RAC node. It defines the margin of error in seconds. The default value is 180 seconds; Oracle recommends setting it to 180 seconds.
The two hangcheck-timer module parameters indicate how long a RAC node must hang before it will reset the system. A node reset will occur when the following is true:

system hang time > (hangcheck_tick + hangcheck_margin)

Configuring Hangcheck Kernel Module Parameters Each time the hangcheck-timer kernel module is loaded (manually or by Oracle) it needs to know what value to use for each of the two parameters we just discussed: (hangcheck-tick and hangcheckmargin). These values need to be available after each reboot of the Linux server. To do this, make an entry with the correct values to the /etc/modprobe.conf file as follows:
# su # echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf

Each time the hangcheck-timer kernel module gets loaded, it will use the values defined by the entry I made in the/etc/modprobe.conf file. Manually Loading the Hangcheck Kernel Module for Testing Oracle is responsible for loading the hangcheck-timer kernel module when required. It is for this reason that it is not required to perform a modprobe or insmod of the hangcheck-timer kernel module in any of the startup files (i.e. /etc/rc.local). It is only out of pure habit that I continue to include a modprobe of the hangchecktimer kernel module in the/etc/rc.local file. Someday I will get over it, but realize that it does not hurt to include a modprobe of the hangcheck-timer kernel module during startup. So to keep myself sane and able to sleep at night, I always configure the loading of the hangcheck-timer kernel module on each startup as follows:
# echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local

You don't have to manually load the hangcheck-timer kernel module using modprobe orinsmod after each reboot. The hangcheck-timer module will be loaded by Oracle (automatically) when needed.

Now, to test the hangcheck-timer kernel module to verify it is picking up the correct parameters we defined in the/etc/modprobe.conf file, use the modprobe command. Although you could load the hangcheck-timer kernel module by passing it the appropriate parameters (e.g. insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180), we want to verify that it is picking up the options we set in the /etc/modprobe.conf file. To manually load the hangcheck-timer kernel module and verify it is using the correct values defined in the/etc/modprobe.conf file, run the following command:
# su # modprobe hangcheck-timer # grep Hangcheck /var/log/messages | tail -2 Feb 14 01:22:52 linux3 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds). Feb 14 01:22:52 linux3 kernel: Hangcheck: Using monotonic_clock().

Configure RAC Nodes for Remote Access using SSH

Perform the following configuration procedures on linux1 and the new Oracle RAC node!

During the creation of the existing two-node cluster, the installation of Oracle Clusterware and the Oracle Database software were only performed from one node in the RAC cluster namely from linux1 as the oracle user account. The Oracle Universal Installer (OUI) on that particular node then would use the ssh and scp commands to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. Theoracle user account on the node running the OUI (runInstaller) had to be trusted by all other nodes in the RAC cluster. This meant that theoracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI against all other Linux servers in the cluster without being prompted for a password. The same security requirements hold true for this article. User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securely copied from linux1 to the new Oracle RAC node (linux3) using ssh and scpwithout being prompted for a password.

As was the case when configuring the existing two-node cluster, this article assumes the Oracle software installation to the new Oracle RAC node will be performed from linux1. This section provides the methods required for configuring SSH1, an RSA key, and user equivalence for the new Oracle RAC node. Configuring the Secure Shell To determine if SSH is installed and running on the new Oracle RAC node, enter the following command:
# pgrep sshd 3695

If SSH is running, then the response to this command is a list of process ID number(s). Creating the RSA Keys on the new Oracle RAC Node The first step in configuring SSH is to create an RSA public/private key pair on the new Oracle RAC node. An RSA public/private key should already exist on both of the two nodes in the current twonode cluster. The command to do this will create a public and private key for RSA (for a total of two keys per node). The content of the RSA public keys will then be copied into an authorized key file onlinux1 which is then distributed to all other Oracle RAC nodes in the cluster. Use the following steps to create the RSA key pair from the new Oracle RAC node (linux3); 1. Logon as the "oracle" UNIX user account.
# su - oracle

2.

If necessary, create the .ssh directory in the "oracle" user's home directory and set the correct permissions on it:

3. $ mkdir -p ~/.ssh $ chmod 700 ~/.ssh

4. Enter the following command to generate an RSA key pair (public and private key) for the SSH protocol:

$ /usr/bin/ssh-keygen -t rsa

At the prompts:
o o

Accept the default location for the key files. Enter and confirm a pass phrase. This should be different from the "oracle" UNIX user account password however it is not a requirement.

This command will write the public key to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file. Note that you should never distribute the private key to anyone! Updating and Distributing the "authorized key file" from linux1 Now that the new Oracle RAC node contains a public and private key for RSA, you will need to update the authorized key file onlinux1 to add (append) the new RSA public key from linux3. An authorized key file is nothing more than a single file that contains a copy of everyone's (every node's) RSA public key. Once the authorized key file contains all of the public keys, it is then distributed to all other nodes in the cluster. Complete the following steps on linux1 to update and then distribute the authorized key file to all nodes in the Oracle RAC cluster.
1.

As the "oracle" UNIX user account, verify that an authorized key file currently exists on the node linux1(~/.ssh/authorized_keys). The authorized key file should already exist from the initial installation of Oracle RAC.
2 01:18 id_rsa.pub

2. $ cd ~/.ssh 3. $ ls -l *.pub -rw-r--r-- 1 oracle oinstall 223 Sep

4.

In this step, use SCP (Secure Copy) or SFTP (Secure FTP) to copy the content of the ~/.ssh/id_rsa.pub public key from the new Oracle RAC node to the authorized key file on linux1. You will be prompted for the oracle UNIX user account password for the new Oracle RAC node. Again, this task will be performed from linux1.

$ ssh linux3 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys The authenticity of host 'linux3 (192.168.1.107)' can't be established. RSA key fingerprint is f5:38:37:e8:84:4e:bd:6d:6b:25:f7:94:58:e8:b2:7a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'linux3,192.168.1.107' (RSA) to the list of known hosts. oracle@linux3's password: xxxxx

The first time you use SSH to connect to a node from a particular system, you will see a message similar to the following:
The authenticity of host 'linux3 (192.168.1.107)' can't be established. RSA key fingerprint is f5:38:37:e8:84:4e:bd:6d:6b:25:f7:94:58:e8:b2:7a. Are you sure you want to continue connecting (yes/no)? yes Enter yes at the prompt to continue. You should not see this message

again when you connect from this system to the same node.

5.

At this point, we have the RSA public key from every node in the cluster (including the new Oracle RAC node) in the authorized key file on linux1. We now need to copy it to all other nodes in the cluster. Use the scp command to copy the authorized key file to all remaining nodes in the RAC cluster:

6. $ scp ~/.ssh/authorized_keys linux2:.ssh/authorized_keys 7. Enter passphrase for key '/home/oracle/.ssh/id_rsa': xxxxx 8. authorized_keys 100% 669 0.7KB/s 00:00 9. 10. $ scp ~/.ssh/authorized_keys linux3:.ssh/authorized_keys 11. oracle@linux3's password: xxxxx authorized_keys 100% 669 0.7KB/s 00:00

12. Change the permission of the authorized key file on all Oracle RAC nodes in the cluster by logging into the node and running the following:
$ chmod 600 ~/.ssh/authorized_keys

13. At

this point, if you use ssh to log in to or run a command on the new Oracle RAC node, you are prompted for the pass phrase that you specified when you created the RSA key. For example, test the following from linux1:

14. $ ssh linux3 hostname 15. Enter passphrase for key '/home/oracle/.ssh/id_rsa': xxxxx linux3

If you see any other messages or text, apart from the host name, then the Oracle installation can fail. Make any changes required to ensure that only the host name is displayed when you enter these commands. You should ensure that any part of a login script(s) that generate any output, or ask any questions, are modified so that they act only when the shell is an interactive shell.

Enabling SSH User Equivalency for the Current Shell Session When running the addNode.sh script from linux1 (which runs the OUI), it will need to run the secure shell tool commands (ssh andscp) on the new Oracle RAC node without being prompted for a pass phrase. Even though SSH is now configured on all Oracle RAC nodes in the cluster, using the secure shell tool commands will still prompt for a pass phrase. Before running the addNode.sh script, you need to enable user equivalence for the terminal session you plan to run the script from. For the purpose of this article, the addNode.shscript will be run from linux1. User equivalence will need to be enabled on any new terminal shell session on linux1 before attempting to run the addNode.sh script. If you log out and log back in to the node you will be performing the Oracle installation from, you must enable user equivalence for the terminal shell session as this is not done by default. To enable user equivalence for the current terminal shell session, perform the following steps:
1.

Logon to the node where you want to run the addNode.sh script from (linux1) as the "oracle" UNIX user account.
# su - oracle

2. Enter the following commands:

3. $ exec /usr/bin/ssh-agent $SHELL 4. $ /usr/bin/ssh-add 5. Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

At the prompt, enter the pass phrase for each key that you generated.

6.

If SSH is configured correctly, you will be able to use the ssh and scp commands without being prompted for a password or pass phrase from this terminal session:

7. $ ssh linux1 "date;hostname" 8. Fri Feb 22 12:13:57 EST 2008 9. linux1 10. 11. $ ssh linux2 "date;hostname" 12. Fri Feb 22 12:14:43 EST 2008 13. linux2 14. 15. $ ssh linux3 "date;hostname" 16. Fri Feb 22 12:13:16 EST 2008 linux3

The commands above should display the date set on each Oracle RAC node along with its hostname. If any of the nodes prompt for a password or pass phrase then verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys.

Also, if you see any other messages or text, apart from the date and hostname, then the Oracle installation can fail. Make any changes required to ensure that only the date is displayed when you enter these commands. You should ensure that any part of a login script(s) that generate any output, or ask any questions, are modified so that they act only when the shell is an interactive shell.
17. The

Oracle Universal Installer is a GUI interface and requires the use of an X Server. From the terminal session enabled for user equivalence (the node you will be running the addNode.sh script from), set the environment variable DISPLAY to a valid X Windows display: Bourne, Korn, and Bash shells:
$ DISPLAY=<Any X-Windows Host>:0 $ export DISPLAY

C shell:
$ setenv DISPLAY <Any X-Windows Host>:0

After setting the DISPLAY variable to a valid X Windows display, you should perform another test of the current terminal session to ensure that X11 forwarding is not enabled:
$ ssh linux1 hostname linux1

$ ssh linux2 hostname linux2 $ ssh linux3 hostname linux3

18. You

must run the addNode.sh script from this terminal session or remember to repeat the steps to enable user equivalence (steps 2, 3, and 4 from this section) before you start the Oracle Universal Installer from a different terminal session.

For more information on configuring SSH and user equivalence in an Oracle RAC 10g environment, see the section "Configure RAC Nodes for Remote Access using SSH" in the parent article.

All Startup Commands for New Oracle RAC Node

Verify that the following startup commands are included on the new Oracle RAC node!

This section will recap all of the parameters, commands, and entries that were covered in previous sections of this document that need to happen on the new Oracle RAC node when the machine is booted. For each of the startup files below, I indicate in blue the entries that should be included in each of the startup files on the new Oracle RAC node.

/etc/modprobe.conf

All parameters and values to be used by kernel modules.

/etc/modprobe.conf alias eth0 b44 alias eth1 e1000

alias scsi_hostadapter ata_piix alias usb-controller ehci-hcd alias usb-controller1 uhci-hcd options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

/etc/sysctl.conf

We wanted to adjust the default and maximum send buffer size as well as the default and maximum receive buffer size for the interconnect. This file also contains those parameters responsible for configuring shared memory, semaphores, file handles, and local IP range for use by the Oracle instance.
/etc/sysctl.conf # Kernel sysctl configuration file for Red Hat Linux # # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and # sysctl.conf(5) for more details. # Controls IP packet forwarding net.ipv4.ip_forward = 0 # Controls source route verification net.ipv4.conf.default.rp_filter = 1 # Controls the System Request debugging functionality of the kernel kernel.sysrq = 0 # Controls whether core dumps will append the PID to the core filename. # Useful for debugging multi-threaded applications. kernel.core_uses_pid = 1 # # # # # # # # # # # # # # +---------------------------------------------------------+ | ADJUSTING NETWORK SETTINGS | +---------------------------------------------------------+ | With Oracle 9.2.0.1 and onwards, Oracle now makes use | | of UDP as the default protocol on Linux for | | inter-process communication (IPC), such as Cache Fusion | | and Cluster Manager buffer transfers between instances | | within the RAC cluster. Oracle strongly suggests to | | adjust the default and maximum receive buffer size | | (SO_RCVBUF socket option) to 256 KB, and the default | | and maximum send buffer size (SO_SNDBUF socket option) | | to 256 KB. The receive buffers are used by TCP and UDP | | to hold received data until it is read by the | | application. The receive buffer cannot overflow because |

# # # # # #

| the peer is not allowed to send data beyond the buffer | | size window. This means that datagrams will be | | discarded if they don't fit in the socket receive | | buffer. This could cause the sender to overwhelm the | | receiver. | +---------------------------------------------------------+

# +---------------------------------------------------------+ # | Default setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. | # +---------------------------------------------------------+ net.core.rmem_default=262144 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. | # +---------------------------------------------------------+ net.core.rmem_max=262144 # +---------------------------------------------------------+ # | Default setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.core.wmem_default=262144 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.core.wmem_max=262144 # # # # # # # # +---------------------------------------------------------+ | ADJUSTING ADDITIONAL KERNEL PARAMETERS FOR ORACLE | +---------------------------------------------------------+ | Configure the kernel parameters for all Oracle Linux | | servers by setting shared memory and semaphores, | | setting the maximum amount of file handles, and setting | | the IP local port range. | +---------------------------------------------------------+

# +---------------------------------------------------------+ # | SHARED MEMORY | # +---------------------------------------------------------+ kernel.shmmax=2147483648 # +---------------------------------------------------------+ # | SEMAPHORES | # | ---------| # | | # | SEMMSL_value SEMMNS_value SEMOPM_value SEMMNI_value | # | | # +---------------------------------------------------------+ kernel.sem=250 32000 100 128 # +---------------------------------------------------------+

# | FILE HANDLES | # ----------------------------------------------------------+ fs.file-max=65536 # +---------------------------------------------------------+ # | LOCAL IP RANGE | # ----------------------------------------------------------+ net.ipv4.ip_local_port_range=1024 65000

Verify that each of the required kernel parameters (above) are configured in the/etc/sysctl.conf file. Then, ensure that each of these parameters are truly in effect by running the following command on both Oracle RAC nodes in the cluster:
# sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.core.rmem_default = 262144 net.core.rmem_max = 262144 net.core.wmem_default = 262144 net.core.wmem_max = 262144 kernel.shmmax = 2147483648 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000

/etc/hosts

All machine/IP entries for nodes in the RAC cluster.

/etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 # Public Network 192.168.1.100 192.168.1.101 192.168.1.107 localhost.localdomain - (eth0) linux1 linux2 linux3 localhost

# Private Interconnect - (eth1) 192.168.2.100 linux1-priv 192.168.2.101 linux2-priv 192.168.2.107 linux3-priv

# Public Virtual 192.168.1.200 192.168.1.201 192.168.1.207

IP (VIP) addresses - (eth0:1) linux1-vip linux2-vip linux3-vip

# Private Storage Network for Openfiler - (eth1) 192.168.1.195 openfiler1 192.168.2.195 openfiler1-priv 192.168.1.106 192.168.1.102 192.168.1.105 192.168.1.120 melody alex bartman cartman

/etc/rc.local

Loading the hangcheck-timer kernel module.

/etc/rc.local #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local # # # # +---------------------------------------------------------+ | HANGCHECK TIMER | | (I do not believe this is required, but doesn't hurt) | +---------------------------------------------------------+

/sbin/modprobe hangcheck-timer

Install and Configure Oracle Cluster File System (OCFS2)

Perform the following tasks on the new Oracle RAC node!

Overview The current two-node Oracle RAC database makes use of the Oracle Cluster File System, Release 2 (OCFS2) to store the two files that are required to be shared by the Oracle Clusterware software. Note that for each of the two shared Oracle Clusterware shared files, a mirrored copy was created making for five files in total:

Oracle Cluster Registry (OCR) o File 1 : /u02/oradata/orcl/OCRFile o File 2 : /u02/oradata/orcl/OCRFile_mirror CRS Voting Disk o File 1 : /u02/oradata/orcl/CSSFile o File 2 : /u02/oradata/orcl/CSSFile_mirror1 o File 3 : /u02/oradata/orcl/CSSFile_mirror2

Along with these two groups of files (the OCR and Voting disk), we also used this space to store the shared ASM SPFILE for all Oracle RAC instances. In this section, we will download and install the release of OCFS2 used for the current two-node cluster on the new Oracle RAC node -(OCFS2 Release 1.2.5-6). See the following page for more information on OCFS2 (including Installation Notes) for Linux: OCFS2 Project Documentation Download OCFS2 The OCFS2 distribution comprises of two sets of RPMs; namely, the kernel module and the tools. Download the same OCFS2 distribution used for the current two-node RAC starting with the OCFS2 kernel module (the driver). With CentOS 4.5, I am using kernel release 2.6.9-55.EL. The available OCFS2 kernel modules for Linux kernel 2.6.955.EL are listed below. Always download the OCFS2 kernel module that matches the distribution, platform, kernel version and the kernel flavor (smp, hugemem, psmp, etc).

ocfs2-2.6.9-55.EL-1.2.5-6.i686.rpm - (for single processor) ocfs2-2.6.9-55.ELsmp-1.2.5-6.i686.rpm - (for multiple processors) ocfs2-2.6.9-55.ELhugemem-1.2.5-6.i686.rpm - (for hugemem) Next, download the OCFS2 tools and the OCFS2 console applications. ocfs2-tools-1.2.4-1.i386.rpm - (OCFS2 tools) ocfs2console-1.2.4-1.i386.rpm - (OCFS2 console)
The OCFS2 Console is optional but highly recommended. The ocfs2consoleapplication requires e2fsprogs, glib2 2.2.3 or later, vte 0.11.10 or later, pygtk2 (EL4) or python-gtk (SLES9) 1.99.16 or later, python 2.3 or later and ocfs2-tools. If you were curious as to which OCFS2 driver release you need, use the OCFS2 release that matches your kernel version. To determine your kernel release:
$ uname -a Linux linux3 2.6.9-55.EL #1 Wed May 2 13:52:16 EDT 2007 i686 i686 i386 GNU/Linux

In the absence of the string "smp" after the string "EL", we are running a single processor (Uniprocessor) machine. If the string "smp" were to appear, then you would be running on a multi-processor machine.

Install OCFS2 I will be installing the OCFS2 files onto the new Oracle RAC node (linux3) which is a single processor machine. The installation process is simply a matter of running the following command on the new Oracle RAC node as the root user account:
$ su # rpm -Uvh ocfs2-2.6.9-55.EL-1.2.5-6.i686.rpm \ ocfs2console-1.2.4-1.i386.rpm \ ocfs2-tools-1.2.4-1.i386.rpm Preparing... ########################################### 1:ocfs2-tools ########################################### 2:ocfs2-2.6.9-55.EL ########################################### 3:ocfs2console ###########################################

[100%] [ 33%] [ 67%] [100%]

Disable SELinux (RHEL4 U2 and higher) Users of RHEL4 U2 and higher (CentOS 4.5 is based on RHEL4 U5) are advised that OCFS2 currently does not work with SELinux enabled. If you are using RHEL4 U2 or higher (which includes us since we are using CentOS 4.5) you will need to disable SELinux (using tool system-config-securitylevel) to get the O2CB service to execute.
A ticket has been logged with Red Hat on this issue.

If you followed the installation instructions I provided for the CentOS operating system, the SELinux option should already be disabled in which case this section can be skipped. During the CentOS installation, the SELinux option was disabled in the Firewall section. If you did not follow the instructions to disable the SELinux option during the installation of CentOS (or if you simply want to verify it is truly disable), run the "Security Level Configuration" GUI utility:
# /usr/bin/system-config-securitylevel &

This will bring up the following screen:

Figure 10: Security Level Configuration Opening Screen Now, click the SELinux tab and check off the "Enabled" checkbox. After clicking [OK], you will be presented with a warning dialog. Simply acknowledge this warning by clicking "Yes". Your screen should now look like the following after disabling the SELinux option:

Figure 11: SELinux Disabled If modifications were made to disable SELinux on the new Oracle RAC node, it will need to be rebooted to implement the change. SELinux must be disabled before you can continue with configuring OCFS2!
# init 6

Configure OCFS2 The next step is to generate and configure the /etc/ocfs2/cluster.conf file on the new Oracle RAC node. The easiest way to accomplish this is to run the GUI tool ocfs2console. The /etc/ocfs2/cluster.conf file will contain hostnames and IP addresses for "all" nodes in the cluster. After creating the /etc/ocfs2/cluster.conf on the new Oracle RAC node, these

changes will then be distributed to the other two current RAC nodes using the o2cb_ctl command-line utility. In this section, we will not only create and configure the /etc/ocfs2/cluster.conf file using ocfs2console, but will also create and start the cluster stack O2CB. When the /etc/ocfs2/cluster.conf file is not present, (as will be the case in our example), theocfs2console tool will create this file along with a new cluster stack service (O2CB) with a default cluster name of ocfs2.
Note that OCFS2 will be configured to use the private network (192.168.2.0) for all of its network traffic as recommended by Oracle. While OCFS2 does not take much bandwidth, it does require the nodes to be alive on the network and sends regular keepalive packets to ensure that they are. To avoid a network delay being interpreted as a node disappearing on the net which could lead to a node-self-fencing, a private interconnect is recommended. It is safe to use the same private interconnect for both Oracle RAC and OCFS2.

A popular question then is what node name should be used and should it be related to the IP address? The node name needs to match the hostname of the machine. The IP address need not be the one associated with that hostname. In other words, any valid IP address on that node can be used. OCFS2 will not attempt to match the node name (hostname) with the specified IP address.
$ su # ocfs2console &

This will bring up the GUI as shown below:

Figure 12: ocfs2console Screen Using the ocfs2console GUI tool, perform the following steps: Select [Cluster] -> [Configure Nodes...]. This will start the OCFS2 Cluster Stack (Figure 13). Acknowledge this Information dialog box by clicking [Close]. You will then be presented with the "Node Configuration" dialog. 2. On the "Node Configurtion" dialog, click the [Add] button. o This will bring up the "Add Node" dialog. o In the "Add Node" dialog, enter the Host name and IP address for the first node in the cluster. Leave the IP Port set to its default value of 7777. In my example, I added all three nodes using linux1 / 192.168.2.100 for the first node, linux2 /192.168.2.101 for the second node and linux3 / 192.168.2.107 for the third node. Note: The node name you enter "must" match the
1.

hostname of the machine and the IP addresses will use the privateinterconnect. o Click [Apply] on the "Node Configuration" dialog - All nodes should now be "Active" as shown in Figure 14. o Click [Close] on the "Node Configuration" dialog. 3. After verifying all values are correct, exit the application using [File] -> [Quit]. This needs to be performed on both Oracle RAC nodes in the cluster.

Figure 13: Starting the OCFS2 Cluster Stack The following dialog shows the OCFS2 settings I used when configuring the new Oracle RAC node:

Figure 14: Configuring Nodes for OCFS2 After exiting the ocfs2console, you will have a /etc/ocfs2/cluster.conf similar to

the following. In the next section, this file (along with other changes) will be distributed to the current two RAC nodes:
/etc/ocfs2/cluster.conf node:

ip_port = 7777 ip_address = 192.168.2.100 number = 0 name = linux1 cluster = ocfs2 ip_port = 7777 ip_address = 192.168.2.101 number = 1 name = linux2 cluster = ocfs2

node:

node:

ip_port = 7777 ip_address = 192.168.2.107 number = 2 name = linux3 cluster = ocfs2 node_count = 3 name = ocfs2

cluster:

Add New Oracle RAC Node to the OCFS2 Cluster The next step is to add the new Oracle RAC node (linux3) to the current "live" OCFS2 cluster. This entails running the o2cb_ctlcommand-line utility from the current two RAC nodes linux1 and linux2. As root, run the following from linux1 and then linux2:
[root@linux1 ~]# o2cb_ctl -C -i -n linux3 -t node -a number=2 -a ip_address=192.168.2.107 -a ip_port=7777 -a cluster=ocfs2 Node linux3 created [root@linux2 ~]# o2cb_ctl -C -i -n linux3 -t node -a number=2 -a ip_address=192.168.2.107 -a ip_port=7777 -a cluster=ocfs2 Node linux3 created

o2cb_ctl parameters:

-C -i

-n -t -a

: Create an object in the OCFS2 Cluster Configuration. : Valid only with -C. When creating something (node or cluster), it will also install it in the live cluster (/config). If the parameter is not specified, then only update the /etc/ocfs2/cluster.conf. : Object name which is usually the node name or cluster name. : Type can be cluster, node or heartbeat. : Attribute in the format "parameter=value" which will be set in the file /etc/ocfs2/cluster.conf file. Since nodes are numbered starting with zero, the third node in the OCFS2 cluster will be "number=2". Set the IP address which in this example will be the private interconnect "ip_address=192.168.2.107". The port number used in the current two-node cluster is "ip_port=7777". Finally, identify which OCFS2 cluster to use which in our case is named "cluster=ocfs2".

Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold Next, configure the on-boot properties of the OC2B driver on the new Oracle RAC node so that the cluster stack services will start on each boot. We will also be adjusting the OCFS2 Heartbeat Threshold from its default setting of 7 to 61. Set the on-boot properties as follows:
# /etc/init.d/o2cb offline ocfs2 # /etc/init.d/o2cb unload # /etc/init.d/o2cb configure Configuring the O2CB driver. This will configure the on-boot properties of the O2CB driver. The following questions will determine whether the driver is loaded on boot. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Load O2CB driver on boot (y/n) [n]: y Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2 Specify heartbeat dead threshold (>=7) [7]: 61 Specify network idle timeout in ms (>=5000) [10000]: 10000 Specify network keepalive delay in ms (>=1000) [5000]: 5000 Specify network reconnect delay in ms (>=2000) [2000]: 2000 Writing O2CB configuration: OK Loading module "configfs": OK Mounting configfs filesystem at /config: OK Loading module "ocfs2_nodemanager": OK Loading module "ocfs2_dlm": OK Loading module "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting O2CB cluster ocfs2: OK

Mount the OCFS2 File System Since the clustered file system already exists, the next step is to simply mount it on the new Oracle RAC node. Let's first do it using the command-line, then I'll show how to include it in the /etc/fstab to have it mount on each boot. The current OCFS2 file system was created with the label oracrsfiles which will be used when mounting. First, here is how to manually mount the OCFS2 file system from the command-line. Remember that this needs to be performed as theroot user account on the new Oracle RAC node:
$ su # mount -t ocfs2 -o datavolume,nointr -L "oracrsfiles" /u02

If the mount was successful, you will simply get your prompt back. We should, however, run the following checks to ensure the file system is mounted correctly. Let's use the mount command to ensure that the clustered file system is really mounted:
# mount /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) /dev/hda1 on /boot type ext3 (rw) none on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) cartman:SHARE2 on /cartman type nfs (rw,addr=192.168.1.120) configfs on /config type configfs (rw) ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) /dev/sdc1 on /u02 type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)

Please take note of the datavolume option I am using to mount the clustered file system. Oracle database users must mount any volume that will contain the Voting Disk file, Cluster Registry (OCR), Data files, Redo logs, Archive logs and Control files with the datavolume mount option so as to ensure that the Oracle processes open the files with the o_direct flag. The nointr option ensures that the I/O's are not interrupted by signals.

Any other type of volume, including an Oracle home (which I will not be using for this article), should not be mounted with this mount option.

Configure OCFS2 to Mount Automatically at Startup This section provides the steps necessary to mount the OCFS2 file system each time the new Oracle RAC node is booted using its label. We start by adding the following line to the /etc/fstab file on the new Oracle RAC node:
LABEL=oracrsfiles /u02 ocfs2 _netdev,datavolume,nointr 0 0

Notice the "_netdev" option for mounting this file system. The _netdev mount option is a must for OCFS2 volumes. This mount option indicates that the volume is to be mounted after the network is started and dismounted before the network is shutdown.

Now, let's make sure that the ocfs2.ko kernel module is being loaded and that the file system will be mounted during the boot process. If you have been following along with the examples in this article, the actions to load the kernel module and mount the OCFS2 file system should already be enabled. However, we should still check those options by running the following as the root user account on the new Oracle RAC node:
$ su # chkconfig --list o2cb o2cb 0:off 1:off

The flags that I have marked in bold should be set to "on". Check Permissions on OCFS2 File System

2:on

3:on

4:on

5:on

6:off

From the new Oracle RAC node, use the ls command to check ownership. The permissions should be set to 0775 with owner "oracle" and group "oinstall".
# ls -ld /u02 drwxrwxr-x 4 oracle oinstall 4096 Dec 20 01:53 /u02

Verify Access to the Shared Clusterware Files

From the new Oracle RAC node as the oracle user account, use the ls command to verify access to the Oracle Clusterware shared files (OCR file and Voting Disk):
[oracle@linux3 ~]$ ls -l /u02/oradata/orcl total 14820 -rw-r--r-- 1 oracle oinstall 10240000 Dec -rw-r--r-- 1 oracle oinstall 10240000 Dec -rw-r--r-- 1 oracle oinstall 10240000 Dec drwxr-x--- 2 oracle oinstall 4096 Dec -rw-r----- 1 root oinstall 4931584 Dec -rw-r----- 1 root oinstall 4931584 Dec

20 20 20 20 20 20

02:33 02:33 02:33 15:34 14:49 14:49

CSSFile CSSFile_mirror1 CSSFile_mirror2 dbs/ OCRFile OCRFile_mirror

How to Determine OCFS2 Version To determine which version of OCFS2 is running, use:
# cat /proc/fs/ocfs2/version OCFS2 1.2.5 Mon Jul 30 13:22:57 PDT 2007 (build 4d201e17b1bc7db76d96570e328927c7)

Install and Configure Automatic Storage Management (ASMLib 2.0)

Perform the following tasks on the new Oracle RAC node!

Introduction The current two-node Oracle RAC database makes use of Automatic Storage Management (ASM) to be used as the file system and volume manager for all Oracle physical database files (data, online redo logs, control files, archived redo logs) and a Flash Recovery Area. In this section, we will download, install, and configure ASMLib on the new Oracle RAC node. Using this method, Oracle database files will be stored on raw block

devices managed by ASM using ASMLib calls.RAW devices are not required with this method as ASMLib works with block devices. Download the ASMLib 2.0 Packages The ASMLib distribution comprises of two sets of RPMs; namely, the kernel module and the ASMLib tools. Download the same ASMLib distribution used for the current two-node RAC starting with the ASMLib kernel module (the driver). With CentOS 4.5, I am using kernel release 2.6.9-55.EL while the ASMLib kernel driver used in the current two-node RAC environment is version 2.0.3-1. Like the Oracle Cluster File System, we need to download the version for the Linux kernel and number of processors on the new Oracle RAC node. We are using kernel 2.6.9-55.EL #1 while the machine I am using only has a single processor: Oracle ASMLib Downloads for Red Hat Enterprise Linux 4 AS oracleasm-2.6.9-55.EL-2.0.3-1.i686.rpm - (for single processor) oracleasm-2.6.9-55.ELsmp-2.0.3-1.i686.rpm - (for multiple processors) oracleasm-2.6.9-55.ELhugemem-2.0.3-1.i686.rpm - (for hugemem) You will also need to download the following ASMLib tools: oracleasmlib-2.0.2-1.i386.rpm - (Userspace library) oracleasm-support-2.0.3-1.i386.rpm - (Driver support files) Install ASMLib 2.0 Packages I will be installing the ASMLib files onto the new Oracle RAC node (linux3) which is a single processor machine. The installation process is simply a matter of running the following command on the new Oracle RAC node as the root user account:
$ su # rpm -Uvh oracleasm-2.6.9-55.EL-2.0.3-1.i686.rpm \ oracleasmlib-2.0.2-1.i386.rpm \

oracleasm-support-2.0.3-1.i386.rpm Preparing... ########################################### 1:oracleasm-support ########################################### 2:oracleasm-2.6.9-55.EL ########################################### 3:oracleasmlib ###########################################

[100%] [ 33%] [ 67%] [100%]

Configure and Loading the ASMLib 2.0 Packages After downloading and installing the ASMLib 2.0 Packages for Linux, we now need to configure and load the ASM kernel module. Run the following as root on the new Oracle RAC node:
$ su # /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracle Default group to own the driver interface []: oinstall Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: [ OK ] Creating /dev/oracleasm mount point: [ OK ] Loading module "oracleasm": [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ]

Scan for ASM Disks From the new Oracle RAC node, you can now perform a scandisk to recognize the current volumes. Even though the above configuration automatically ran the scandisk utility, I still like to manually perform this step!
# /etc/init.d/oracleasm scandisks Scanning system for ASM disks [ OK ]

We can now test that the ASM disks were successfully identified using the following command as the root user account:
# /etc/init.d/oracleasm listdisks VOL1

VOL2 VOL3 VOL4

Pre-Installation Tasks for Oracle10g Release 2

Perform the following checks on the new Oracle RAC node and run the Oracle Cluster Verification Utility (CVU) from linux1!

When installing the Linux O/S (CentOS or Red Hat Enterprise Linux 4), you should verify that all required RPMs for Oracle are installed. If you followed the instructions I used for installing Linux, you would have installed Everything, in which case you will have all of the required RPM packages. However, if you performed another installation type (i.e. "Advanced Server), you may have some packages missing and will need to install them. All of the required RPMs are on the Linux CDs/ISOs. The next pre-installation step is to run the Cluster Verification Utility (CVU) from linux1 against the new Oracle RAC node. CVU is a command-line utility provided on the Oracle Clusterware installation media. It is responsible for performing various system checks to assist you with confirming the Oracle RAC nodes are properly configured for Oracle Clusterware and Oracle Real Application Clusters installation. The CVU only needs to be run from the node you will be performing the Oracle installations from (linux1 in this article). Check Required RPMs The following packages must be installed on the new Oracle RAC node. Note that the version number for your Linux distribution may vary slightly.
binutils-2.15.92.0.2-21 compat-db-4.1.25-9 compat-gcc-32-3.2.3-47.3 compat-gcc-32-c++-3.2.3-47.3 compat-libstdc++-33-3.2.3-47.3 compat-libgcc-296-2.96-132.7.2 control-center-2.8.0-12.rhel4.5

cpp-3.4.6-3 gcc-3.4.6-3 gcc-c++-3.4.6-3 glibc-2.3.4-2.25 glibc-common-2.3.4-2.25 glibc-devel-2.3.4-2.25 glibc-headers-2.3.4-2.25 glibc-kernheaders-2.4-9.1.98.EL gnome-libs-1.4.1.2.90-44.1 libaio-0.3.105-2 libstdc++-3.4.6-3 libstdc++-devel-3.4.6-3 make-3.80-6.EL4 openmotif-2.2.3-10.RHEL4.5 openmotif21-2.1.30-11.RHEL4.6 pdksh-5.2.14-30.3 setarch-1.6-1 sysstat-5.0.5-11.rhel4 xscreensaver-4.18-5.rhel4.11

Note that the openmotif RPM packages are only required to install Oracle demos. This article does not cover the installation of Oracle demos.

To query package information (gcc and glibc-devel for example), use the "rpm <PackageName> [, <PackageName>]" command as follows:
# rpm -q gcc glibc-devel gcc-3.4.6-3 glibc-devel-2.3.4-2.25

-q

If you need to install any of the above packages (which you should not have to if you installed Everything), use the "rpm -Uvh <PackageName.rpm>" command. For example, to install the GCC gcc3.4.6-3 package, use:
# rpm -Uvh gcc-3.4.6-3.i386.rpm

Prerequisites for Using Cluster Verification Utility JDK 1.4.2 You must have JDK 1.4.2 installed on your system before you can run CVU. If you do not have JDK 1.4.2 installed on your system, and you attempt to run CVU, you will receive an error message similar to the following:
ERROR. Either CV_JDKHOME environment variable should be set or /stagepath/cluvfy/jrepack.zip should exist.

Note that during the creation of the current two-node Oracle RAC, the JDK was installed on to linux1 and the appropriate environment variable (CV_JDKHOME=/usr/local/java) was defined in the .bash_profile login script for the oracleuser account. No action needs to be performed in this section. Install cvuqdisk RPM (RHEL Users Only) The second pre-requisite for running the CVU is for Red Hat Linux users. If you are using Red Hat Linux, then you must download and install the Red Hat operating system package cvuqdisk on the new Oracle RAC node. Without cvuqdisk, CVU will be unable to discover shared disks, and you will receive the error message "Package cvuqdisk not installed" when you run CVU. The cvuqdisk RPM can be found on the Oracle Clusterware installation media in the rpm directory. For the purpose of this article, the Oracle Clusterware media was extracted to the ~oracle/orainstall/clusterware directory on linux1. Note that before installing the cvuqdisk RPM, we need to set an environment variable named CVUQDISK_GRP to point to the group that will own the cvuqdisk utility. The default group is oinstall which is the primary group we are using for theoracle UNIX user account in this article. If you are using a different primary group (i.e. dba), you will need to setCVUQDISK_GRP=<YOUR_GROUP> before attempting to install the cvuqdisk RPM. Locate and copy the cvuqdisk RPM from linux1 to the new Oracle RAC node (linux3). Next, perform the following steps as the root user account from linux3 to install:
# -- IF YOU ARE USING A PRIMARY GROUP OTHER THAN oinstall # CVUQDISK_GRP=<YOUR_GROUP>; export CVUQDISK_GRP # cd ~oracle/orainstall/clusterware/rpm # rpm -iv cvuqdisk-1.0.1-1.rpm Preparing packages for installation... cvuqdisk-1.0.1-1 # ls -l /usr/sbin/cvuqdisk -rwsr-x--- 1 root oinstall 4168 Jun 2 2005 /usr/sbin/cvuqdisk

Verify Remote Access / User Equivalence

The CVU should be run from linux1 the node we will be extending the Oracle software from. Before running CVU, login as the oracle user account and verify remote access / user equivalence is configured to all nodes in the cluster. When using the secure shell method, user equivalence will need to be enabled for the terminal shell session before attempting to run the CVU. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:
# su - oracle $ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

Checking Pre-Installation Tasks for CRS with CVU Once all prerequisites for using CVU have been met, we can start by checking that all pre-installation tasks for Oracle Clusterware (CRS) are completed by executing the following command as the "oracle" UNIX user account (with user equivalence enabled) from linux1:
$ cd ~oracle/orainstall/clusterware/cluvfy $ ./runcluvfy.sh stage -pre crsinst -n linux1,linux2,linux3 -verbose

Review the CVU report. Note that there are several errors you may ignore in this report. The first error is with regards to finding a suitable set of interfaces for VIPs which can be safely ignored. This is a bug documented in Metalink Note 338924.1:
Suitable interfaces for the private interconnect on subnet "192.168.2.0": linux3 eth1:192.168.2.107 linux2 eth1:192.168.2.101 linux1 eth1:192.168.2.100 ERROR: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check failed.

As documented in the note, this error can be safely ignored. The last set of errors that can be ignored deal with specific RPM package versions that do not exist in RHEL4 Update 5. For example:

compat-gcc-7.3-2.96.128 compat-gcc-c++-7.3-2.96.128 compat-libstdc++-7.3-2.96.128 compat-libstdc++-devel-7.3-2.96.128

While these specific packages are listed as missing in the CVU report, please ensure that the correct versions of the compat-* packages are installed on the new Oracle RAC node. For example, in RHEL4 Update 5, these would be:

compat-gcc-32-3.2.3-47.3 compat-gcc-32-c++-3.2.3-47.3 compat-libstdc++-33-3.2.3-47.3 compat-libstdc++-296-2.96-132.7.2

Checking the Hardware and Operating System Setup with CVU The next CVU check to run will verify the hardware and operating system setup. Again, run the following as the "oracle" UNIX user account from linux1:
$ cd ~oracle/orainstall/clusterware/cluvfy $ ./runcluvfy.sh stage -post hwos -n linux1,linux2,linux3 -verbose

Review the CVU report. As with the previous check (pre-installation tasks for CRS), the check for finding a suitable set of interfaces for VIPs will fail and can be safely ignored. Also note that the check for shared storage accessibility will fail.
Checking shared storage accessibility... WARNING: Unable to determine the sharedness of /dev/sdc on nodes: linux3,linux3,linux3,linux2,linux2,linux2,linux1,linux1,linux1 Shared storage check failed on nodes "linux3,linux2,linux1".

This too can be safely ignored. While we know the disks are visible and shared from all Oracle RAC nodes in the cluster, the check itself fails. Several reasons for this have been documented. The first came from Metalink indicating that cluvfy currently does not work with devices other than SCSI devices. This would include devices like EMC PowerPath and volume groups like those from Openfiler. At the

time of this writing, no workaround exists other than to use manual methods for detecting shared devices. Another reason for this error was documented by Bane Radulovic at Oracle Corporation. His research shows that CVU calls smartclt on Linux, and the problem is that smartclt does not return the serial number from our iSCSI devices. For example, a check against /dev/sde shows:
# /usr/sbin/smartctl -i /dev/sde smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen Home page is http://smartmontools.sourceforge.net/ Device: Openfile Virtual disk Version: 0 Serial number: Device type: disk Local Time is: Mon Sep 3 02:02:53 2007 EDT Device supports SMART and is Disabled Temperature Warning Disabled or Not Supported

At the time of this writing, it is unknown if the Openfiler developers have plans to fix this.

Extend Oracle Clusterware Software to the New Node

Extend the Oracle Clusterware software to the new Oracle RAC node from linux1!

Overview In this section, we will extend the current Oracle RAC database by adding the new Oracle RAC node linux3. The new node will need to be added to the cluster at the clusterware layer so that the other nodes in the RAC cluster consider it to be part of the cluster. When you extend an Oracle RAC database, you must first extend the Oracle Clusterware home to the new node and then extend the Oracle Database home. In other words, you extend the software onto the new node in the same order as you installed the clusterware and Oracle database components on the existing nodes.

Oracle Clusterware is already installed on the cluster. The task in this section is to add the new Oracle RAC node to the clustered configuration. This is done by executing the Oracle provided utility addNode.sh from one of the existing nodes in the cluster; namelylinux1. This script is located in the Oracle Clusterware's home oui/bin directory (/u01/app/crs/oui/bin). During the add nodeprocess, the shared Oracle Cluster Registry file and Voting Disk will be updated with the information regarding the new node. Verifying Terminal Shell Environment Before starting the Oracle Universal Installer, you should first verify you are logged onto the server you will be running the installer from (i.e. linux1) then run the xhost command as root from the console to allow X Server connections. Next, login as the oracle user account. If you are using a remote client to connect to the node performing the installation (SSH / Telnet to linux1 from a workstation configured with an X Server), you will need to set the DISPLAY variable to point to your local workstation. Finally, verify remote access / user equivalence to all nodes in the cluster: Verify Server and Enable X Server Access
# hostname linux1 # xhost + access control disabled, clients can connect from any host

Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer

from against the new Oracle RAC node without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) $ ssh linux1 "date;hostname" Sat Feb 23 16:48:05 EST 2008 linux1 $ ssh linux3 "date;hostname" Sat Feb 23 16:46:51 EST 2008 linux3

Configure Oracle Clusterware on the New Node The next step is to configure Oracle Clusterware on the new Oracle RAC node linux3. As previously mentioned, this is performed by executing the new addNode.sh utility located in the Oracle Clusterware's home oui/bin directory (/u01/app/crs/oui/bin) fromlinux1:
$ hostname linux1 $ id -a uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba) $ cd /u01/app/crs/oui/bin $ ./addNode.sh Screen Name Welcome Screen Specify Cluster Nodes to Add to Installation Response Click Next

In this screen, the OUI lists all existing nodes in the top portion labeled "Existing Nodes". the bottom half of the screen labeled "Specify New Nodes", enter the information for the new node in the appropriate fields: Public Node Name Private Node Name Virtual Node Name

linux3

linux3-priv

linux3-vip

Click Next to continue. Cluster Node Additional Summary Execute Configuration Scripts

Verify the new Oracle RAC node is listed under the "New Nodes" drilldown. Click Install to start the installation!

Once all of the required Oracle Clusterware components have been copied from linux1 to linux3, the OUI prompts to execute three files as described in the following sections.

From linux3

Navigate to the /u01/app/oracle/oraInventory directory on linux3 and run orainstRoot.sh as the "root" user acco

From linux1

Important: As documented in Metalink Note 392415.1, the rootaddnode.sh script (which is run in this section) error out at the end with "Connection refused" (PRKC-1044) when trying to trying to add a new node to the clu The reason this error occurs is because the "oracle" user account on the node running the rootaddnode.sh script setup with SSH for remote access to the new node and has a non-empty SSH passphrase. Note that for obvious security reasons, the "oracle" user account is typically setup with a non-empty pass phrase for SSH keys and wo thus succumb to this error. The rootaddnode.sh script uses SSH to check remote node connectivity from linux1 linux3. If it gives any prompt, it will consider ssh is not configured properly. The script will then use rsh instead rsh is not configured, then it will error out with "Connection refused". If you are using SSH for user equivalenc I am in this article), you will need to temporarily define an empty rsa passphrase for the "oracle" user account o linux1 as follows: [oracle@linux1 ~]$ ssh-keygen -p Enter file in which the key is (/home/oracle/.ssh/id_rsa): Enter old passphrase: [OLD PASSPHRASE] Key has comment '/home/oracle/.ssh/id_rsa' Enter new passphrase (empty for no passphrase): [JUST HIT ENTER WITHOUT ENTERING A PASSPHRASE] Enter same passphrase again: [JUST HIT ENTER WITHOUT ENTERING A PASSPHRASE] Your identification has been saved with the new passphrase.

Afer temporarily defining an empty rsa passphrase for the "oracle" user account, navigate to the/u01/app/crs/install directory on linux1 and run rootaddnode.sh as the "root" user account. The rootaddnode. script will add the new node information to the Oracle Cluster Registry (OCR) file using the srvctl utility.

After running the rootaddnode.sh script from linux1, you can set your passphrase back to the old passphrase usi the same "ssh-keygen -p" command.

From linux3 Finally, navigate to the /u01/app/crs directory on linux3 and run root.sh as the "root" user account.

If the Oracle Clusterware home directory is a subdirectory of the ORACLE_BASE directory (which should nev be!), you will receive several warnings regarding permissions while running the root.sh script on the new node. These warnings can be safely ignored. The root.sh may take awhile to run. With Oracle version 10.2.0.1, when running the root.sh on linux3, you will receive a critical error and the output should look like: ... Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. linux1 linux2 linux3 CSS is active on all nodes. Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps The given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs.

This issue is specific to Oracle 10.2.0.1 (noted in Metalink article 338924.1) and needs to be resolved before continuing. The easiest workaround is to re-run vipca (GUI) manually as root from linux3 (the node where the e occurred). Please keep in mind that vipca is a GUI and will need to set your DISPLAY variable accordingly to X server: # $ORA_CRS_HOME/bin/vipca When the "VIP Configuration Assistant" appears, this is how I answered the screen prompts: Welcome: Click Next Network interfaces: Select only the public interface - eth0 Virtual IPs for cluster nodes: Node Name: linux1 IP Alias Name: linux1-vip IP Address: 192.168.1.200 Subnet Mask: 255.255.255.0 Node Name: linux2 IP Alias Name: linux2-vip IP Address: 192.168.1.201 Subnet Mask: 255.255.255.0 Node Name: linux3 IP Alias Name: linux3-vip IP Address: 192.168.1.207 Subnet Mask: 255.255.255.0

Summary: Click Finish Configuration Assistant Progress Dialog: Click OK after configuration is complete. Configuration Results: Click Exit Go back to the OUI and acknowledge the "Execute Configuration scripts" dialog window. End of installation At the end of the installation, exit from the OUI.

Verify Oracle Clusterware Installation After extending Oracle Clusterware to the new node, we can run through several tests to verify the install was successful. Run the following commands on the new Oracle RAC node (linux3): Check Cluster Nodes
$ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2 linux3 3

Confirm Oracle Clusterware Function

$ $ORA_CRS_HOME/bin/crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------ora....SM1.asm application 0/5 0/0 ONLINE ONLINE linux1 ora....X1.lsnr application 0/5 0/0 ONLINE ONLINE linux1 ora.linux1.gsd application 0/5 0/0 ONLINE ONLINE linux1 ora.linux1.ons application 0/3 0/0 ONLINE ONLINE linux1 ora.linux1.vip application 0/0 0/0 ONLINE ONLINE linux1 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE linux2 ora....X2.lsnr application 0/5 0/0 ONLINE ONLINE linux2 ora.linux2.gsd application 0/5 0/0 ONLINE ONLINE linux2 ora.linux2.ons application 0/3 0/0 ONLINE ONLINE linux2 ora.linux2.vip application 0/0 0/0 ONLINE ONLINE linux2 ora.linux3.gsd application 0/5 0/0 ONLINE ONLINE linux3 ora.linux3.ons application 0/3 0/0 ONLINE ONLINE linux3 ora.linux3.vip application 0/0 0/0 ONLINE ONLINE linux3 ora.orcl.db application 0/1 0/1 ONLINE ONLINE linux2 ora....l1.inst application 0/5 0/0 ONLINE ONLINE linux1 ora....l2.inst application 0/5 0/0 ONLINE ONLINE linux2 ora...._taf.cs application 0/1 0/1 ONLINE ONLINE linux2 ora....cl1.srv application 0/1 0/0 ONLINE ONLINE linux1 ora....cl2.srv application 0/1 0/0 ONLINE ONLINE linux2

Check CRS Status

$ $ORA_CRS_HOME/bin/crsctl check crs

CSS appears healthy CRS appears healthy EVM appears healthy

Check Oracle Clusterware Auto-Start Scripts on New Node (linux3)

$ ls -l /etc/init.d/init.* -rwxr-xr-x 1 root root 2236 -rwxr-xr-x 1 root root 5252 -rwxr-xr-x 1 root root 44589 -rwxr-xr-x 1 root root 3669 Feb Feb Feb Feb 23 23 23 23 19:16 19:16 19:16 19:16 /etc/init.d/init.crs /etc/init.d/init.crsd /etc/init.d/init.cssd /etc/init.d/init.evmd

Extend Oracle Database Software to the New Node

Extend the Oracle Database software to the new Oracle RAC node from linux1!

Overview After copying and configuring the Oracle Clusterware software to the new node, we now need to copy the Oracle Database software from one of the existing nodes to linux3. This is done by executing the Oracle provided utility addNode.sh from one of the existing nodes in the cluster; namely linux1. This script is located in the $ORACLE_HOME/oui/bin directory (/u01/app/oracle/product/10.2.0/db_1/oui/bin). Verifying Terminal Shell Environment As discussed in the previous section, the terminal shell environment needs to be configured for remote access and user equivalence to the new Oracle RAC node before running the Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to

perform any of the actions described below with regards to setting up remote access and the DISPLAY variable: Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) $ ssh linux1 "date;hostname" Sat Feb 23 16:48:05 EST 2008 linux1 $ ssh linux3 "date;hostname" Sat Feb 23 16:46:51 EST 2008 linux3

Install Oracle Database Software on the New Node Copy the Oracle Database software to the new Oracle RAC node linux3. As previously mentioned, this is performed by executing the new addNode.sh utility located in the $ORACLE_HOME/oui/bin directory from linux1:
$ hostname

linux1 $ id -a uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba) $ cd /u01/app/oracle/product/10.2.0/db_1/oui/bin $ ./addNode.sh Screen Name Welcome Screen Specify Cluster Nodes to Add to Installation Cluster Node Additional Summary Response Click Next

In this screen, the OUI lists all of the nodes already part of the installation in the top port labeled "Existing Nodes". On the bottom half of the screen labeled "Specify New Nodes" list of new nodes which can be added. By default linux3 is selected. Verify linux3 is selec (checked) and Click Next to continue.

Verify the new Oracle RAC node is listed under the "New Nodes" drilldown. Click Install to start the installation!

Execute Configuration Scripts

Once all of the required Oracle Database components have been copied from linux1 to linux3, the OUI prompts to execute the root.sh on the new Oracle RAC node. Navigate to the /u01/app/oracle/product/10.2.0/db_1 directory on linux3 and run root.sh as the "root" user account.

After running the root.sh script on the new Oracle RAC node, go back to the OUI and acknowledge the "Execu Configuration scripts" dialog window. End of installation At the end of the installation, exit from the OUI.

Add Listener to New Node

Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Network Configuration Assistant (NETCA) will setup the TNS listener in a clustered configuration to include the new node in the cluster.

Overview In this section, you will use the Network Configuration Assistant (NETCA) to setup the TNS listener in a clustered configuration to include the new Oracle RAC node. The NETCA program will be run from linux1 with user equivalence enabled to all nodes in the cluster. Verifying Terminal Shell Environment As discussed in the previous section, the terminal shell environment needs to be configured for remote access and user equivalence to the new Oracle RAC node before running the NETCA. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and theDISPLAY variable: Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the NETCA from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the NETCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add

Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) $ ssh linux1 "date;hostname" Sat Feb 23 16:48:05 EST 2008 linux1 $ ssh linux3 "date;hostname" Sat Feb 23 16:46:51 EST 2008 linux3

Run the Network Configuration Assistant To start the NETCA, run the following from
$ netca & linux1:

The following table walks you through the process of reconfiguring the TNS listeners in a clustered configuration to include the new node.
Screen Name Select the Type of Oracle Net Services Configuration Select the nodes to configure Type of Configuration Response

Select Cluster configuration

Only select the new Oracle RAC node: linux3. Select Listener configuration. The following screens are now like any other normal listener configuration. You can simply accept the default parameters for the next six screens: What do you want to do: Add Listener name: LISTENER Selected protocols: TCP Port number: 1521 Configure another listener: No Listener configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen. Select Naming Methods configuration. The following screens are:

Listener Configuration Next 6 Screens

Type of Configuration Naming Methods

Configuration

Selected Naming Methods: Local Naming Naming Methods configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen. Click Finish to exit the NETCA.

Type of Configuration

Verify TNS Listener Configuration The Oracle TNS listener process should now be running on all three nodes in the RAC cluster:
$ hostname linux1 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX1 $ $ORA_CRS_HOME/bin/crs_stat ora.linux1.LISTENER_LINUX1.lsnr NAME=ora.linux1.LISTENER_LINUX1.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux1 ===================== $ hostname linux2 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX2 $ $ORA_CRS_HOME/bin/crs_stat ora.linux2.LISTENER_LINUX2.lsnr NAME=ora.linux2.LISTENER_LINUX2.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux2 ===================== $ hostname linux3 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX3 $ $ORA_CRS_HOME/bin/crs_stat ora.linux3.LISTENER_LINUX3.lsnr NAME=ora.linux3.LISTENER_LINUX3.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux3

Add Database Instance to the New Node

Add the new Oracle instance to the new Oracle RAC node using DBCA!

Overview The final step in extending the Oracle RAC database is to add a new database instance to the new Oracle RAC node. The database instance will be named orcl3 and hosted on the new node linux3. This process can be performed using either Enterprise Manager or the Database Configuration Assistant (DBCA). For the purpose of this article, I am opting to use the DBCA. Before executing the DBCA, make certain that $ORACLE_HOME and $PATH are set appropriately for the$ORACLE_BASE/product/10.2.0/db_1 environment. You should also verify that all services we have installed up to this point (Oracle TNS listener, Oracle Clusterware processes, etc.) are running before attempting to start the clustered database creation process. The DBCA program will be run from linux1 with user equivalence enabled to all nodes in the cluster. Verifying Terminal Shell Environment As discussed in the previous section, the terminal shell environment needs to be configured for remote access and user equivalence to the new Oracle RAC node before running the DBCA. Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the

actions described below with regards to setting up remote access and theDISPLAY variable: Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the DBCA from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the DBCA. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) $ ssh linux1 "date;hostname" Sat Feb 23 16:48:05 EST 2008 linux1 $ ssh linux3 "date;hostname" Sat Feb 23 16:46:51 EST 2008 linux3

Add Database Instance to New Node To start the database instance creation process for the new Oracle RAC node, run the following from linux1:
$ dbca & Screen Name Welcome Screen Response Select Oracle Real Application Clusters database.

Operations Instance Management

Select Instance Management. Select Add an instance. Provides a list of clustered databases running on the node. For the purpose of this example, the clustered database running on node linux1 is orcl. Select this clustered database.

List of cluster databases At the bottom of this screen, the DBCA requests you to "Specify a user with SYSDBA system privileges": Username: sys Password: <sys_password> Click Next to continue. List of cluster database instances This screen provides a list of all instances currently available on the cluster, their status, and which node they reside on. Verify this list is correct and Click Next to continue. This screen lists the next instance name in the series and requests the node on which to add the instance to. In this example, the next instance name is orcl3 and the node name to create it on is linux3. For this example, the default values are correct (instance name "orcl3" to be added to node "linux3"). After verifying these values, Click Next to continue. After clicking Next, there will be a small pause before the next screen appears as the DBCA determines the current state of the new node and what services (if any) are configured on the existing nodes. If the current clustered database has any database services defined, the next screen allows the DBA to configure those database services for the new instance. In this example, the existing clustered database has one service defined namedorcl_taf. With the "orcl_taf" database service selected, change the details toPreferred for the new instance (orcl3) and the "TAF Policy" set to Basic. By default, the DBCA does a good job of determining the instance specific files such as an UNDO tablespace (UNDOTBS3), database files for this tablespace, and two redo log groups. Verify the storage options and Click Finish to add the instance.

Instance naming and node selection

Database Services

Instance Storage

Database Configuration Assistant: Summary

After verifying the instance creation options in the summary dialog, Click OK to begin the instance management process. During the add instance step, the DBCA verifies the new node and then checks to determine if ASM is present on the existing cluster (which in this example, ASM is configured). The DBCA presents a dialog box indicating that "ASM is present on the cluster but needs to be extended to the following nodes: [linux3]. Do you want ASM to be extended?" Click on Yes to add the ASM instance to the new node. NOTE: In the previous section (Add Listener to New Node), I provided instructions to setup the TNS listener in a clustered configuration to include the new Oracle RAC node using NETCA. If the listener is not yet configured on the new Oracle RAC node, the DBCA will prompt the user with a dialog asking to configure a new listener using port 1521 and listener name "LISTENER_LINUX3". The TNS listener must be present and started on the new Oracle RAC node in order to create and start the ASM instance on the new node.

Extend ASM

Database Configuration Assistant Progress Screen

A progress bar is display while the new instance is being configured. Once the instance management process is complete, the DBCA prompts the user with a dialog and the message "Do you want to perform another operation?" Click No to end and exit the DBCA utility. The DBCA will automatically start the new instance (orcl3) on the node linux3. If any services were configured during the instance management process, however, they are left in an offline state. For the purpose of this example, I had to manually start the "orcl_taf" service for the database: $ srvctl start service -s orcl_taf -d orcl -i orcl3

Start New Database Services

When the Oracle Database Configuration Assistant has completed, you will have successfully extended the current Oracle RAC database! Verify New Database Environment Check Cluster Services
$ $ORA_CRS_HOME/bin/crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....SM1.asm application ONLINE ONLINE linux1 ora....X1.lsnr application ONLINE ONLINE linux1 ora.linux1.gsd application ONLINE ONLINE linux1 ora.linux1.ons application ONLINE ONLINE linux1 ora.linux1.vip application ONLINE ONLINE linux1

ora....SM2.asm ora....X2.lsnr ora.linux2.gsd ora.linux2.ons ora.linux2.vip ora....SM3.asm ora....X3.lsnr ora.linux3.gsd ora.linux3.ons ora.linux3.vip ora.orcl.db ora....l1.inst ora....l2.inst ora....l3.inst ora...._taf.cs ora....cl1.srv ora....cl2.srv ora....cl3.srv - or -

application application application application application application application application application application application application application application application application application application

ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

linux2 linux2 linux2 linux2 linux2 linux3 linux3 linux3 linux3 linux3 linux2 linux1 linux2 linux3 linux2 linux1 linux2 linux3

$ rac_crs_stat HA Resource ----------ora.linux1.ASM1.asm ora.linux1.LISTENER_LINUX1.lsnr ora.linux1.gsd ora.linux1.ons ora.linux1.vip ora.linux2.ASM2.asm ora.linux2.LISTENER_LINUX2.lsnr ora.linux2.gsd ora.linux2.ons ora.linux2.vip ora.linux3.ASM3.asm ora.linux3.LISTENER_LINUX3.lsnr ora.linux3.gsd ora.linux3.ons ora.linux3.vip ora.orcl.db ora.orcl.orcl1.inst ora.orcl.orcl2.inst ora.orcl.orcl3.inst ora.orcl.orcl_taf.cs ora.orcl.orcl_taf.orcl1.srv ora.orcl.orcl_taf.orcl2.srv ora.orcl.orcl_taf.orcl3.srv

Target -----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

State ----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on on on on on on on on on on on on on on on on on on

linux1 linux1 linux1 linux1 linux1 linux2 linux2 linux2 linux2 linux2 linux3 linux3 linux3 linux3 linux3 linux2 linux1 linux2 linux3 linux2 linux1 linux2 linux3

Verify New Instance Login to one of the instances and query the
gv$instance

view:

SQL> select inst_id, instance_name, status, to_char(startup_time, 'DD-MONYYYY HH24:MI:SS')

from gv$instance order by inst_id; INSTANCE_NAME ---------------orcl1 orcl2 orcl3 STATUS ---------OPEN OPEN OPEN TO_CHAR(STARTUP_TIME -------------------23-FEB-2008 00:10:16 23-FEB-2008 00:10:47 26-FEB-2008 22:52:51

INST_ID ---------1 2 3

Update TNSNAMES Login to all machines that will be accessing the new instance and update the tnsnames.ora file (if necessary). Verify Enterprise Manager - Database Control The DBCA should have updated and added the new node(s) to EM Database Control. Bring up a web browser and navigate to:
http://linux3:1158/em

About the Author Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX, Linux, and Windows server environment. Jeff's other interests include mathematical encryption theory, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 18 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science.

Copyright (c) 1998-2012 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info. I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.
Last modified on Sunday, 18-Mar-2012 22:20:27 EDT Page Count: 81605