Build Your Own Oracle RAC Cluster On Oracle Enterprise Linux and iSCSI

16/02/14
Build Your Own Oracle RAC Cluster on Oracle Enterprise Linux and iSCSI
Welcome BOUBA Account Sign Out Help Country Communities I am a... I want to...
Search
Products
Oracle Technology Network
Application Development Framew ork Application Express Big Data Business Intelligence Cloud Computing Communications Database Performance & Availability Data Warehousing .NET Dynamic Scripting Languages Embedded Digital Experience Enterprise Architecture Enterprise Management Identity & Security Java Linux Service-Oriented Architecture Solaris SQL & PL/SQL Systems - All Articles Virtualization
Solutions
Downloads
Store
Support
Training
Partners
About
OTN
Articles DBA: Linux Build Your Own Oracle RAC Cluster on Oracle Enterprise Linux and iSCSI by Jeffrey Hunter Learn how to set up and configure an Oracle RAC 10g Release 2 development cluster on Oracle Enterprise Linux for less than US$2,700. DOWNLOAD Oracle Database 10g TAGS linux, rac, clustering All
The information in this guide is not validated by Oracle, is not supported by Oracle, and should only be used at your own risk; it is for educational purposes only. Updated August 2009 Contents Introduction Oracle RAC 10g Overview Shared-Storage Overview iSCSI Technology Hardware & Costs Install the Linux Operating System Install Required Linux Packages for Oracle RAC Network Configuration Install Openfiler Configure iSCSI Volumes using Openfiler Configure iSCSI Volumes on Oracle RAC Nodes Create "oracle" User and Directories Configure the Linux Servers for Oracle Configure the hangcheck-timer Kernel Module Configure RAC Nodes for Remote Access using SSH All Startup Commands for Both Oracle RAC Nodes Install & Configure Oracle Cluster File System (OCFS2) Install & Configure Automatic Storage Management (ASMLib 2.0) Download Oracle RAC 10g Software Pre-Installation Tasks for Oracle Database 10g Release 2 Install Oracle 10g Clusterware Software Install Oracle Database 10g Software Install Oracle Database 10g Companion CD Software Create TNS Listener Process Create the Oracle Cluster Database Post-Installation Tasks - (Optional) Verify TNS Networking Files Create / Alter Tablespaces Verify the RAC Cluster & Database Configuration Starting / Stopping the Cluster Transparent Application Failover - (TAF) Troubleshooting Conclusion Acknowledgements Downloads for this guide: Oracle Enterprise Linux Release 5 Update 3 (Available for x86 and x86_64) Oracle Database 10g Release 2 EE, Clusterware, Companion CD - (10.2.0.1.0) (Available for x86 and x86_64) Openfiler 2.3 Respin (21-01-09) ( openfiler-2.3-x86-disc1.iso -OR- openfiler-2.3-x86_64-disc1.iso) ASMLib 2.0 Library RHEL5 - (2.0.4-1) ( oracleasmlib-2.0.4-1.el5.i386.rpm -OR- oracleasmlib-2.0.4-1.el5.x86_64.rpm) Support files
See the Oracle RAC 11g Release 1 version of this guide here 1. Introduction One of the most efficient ways to become familiar with Oracle Real Application Clusters (RAC) 10g technology is to have access to an actual Oracle RAC 10g cluster. There's no better way to understand its benefitsincluding fault tolerance, security, load balancing, and scalabilitythan to experience them directly. Unfortunately, for many shops, the price of the hardware required for a typical production RAC configuration makes this goal impossible. A small two-node cluster can cost from US$10,000 to well over US$20,000. That cost would not even include the heart of a production RAC environment typically a storage area networkwhich can start at US$10,000. For those who want to become familiar with Oracle RAC 10g without a major cash outlay, this guide provides a low-cost alternative to configuring an Oracle RAC 10g Release 2 system using commercial off-the-shelf components and downloadable software at an estimated cost of US$2,200 to US$2,700. The system will consist of a dual node cluster (two i386 nodes with a single processor), both running Oracle Enterprise Linux (OEL) Release 5 Update 3 for x86, Oracle10g Release 2 for Linux x86, OCFS2, and ASMLib 2.0. All shared disk storage for Oracle RAC will be based on iSCSI using Openfiler Release 2.3 x86_64 running on a third node (known in this article as the Network Storage Server). Although this article should work with Red Hat Enterprise Linux, Oracle Enterprise Linux (available for free) will provide the same if not better stability and will already include the OCFS2 and ASMLib software packages (with the exception of the ASMLib userspace libraries which is a separate download). Powered by rPath Linux, Openfiler is a free browser-based network storage management utility that delivers file-based Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single framework. The entire software stack interfaces with open source applications such as Apache, Samba, LVM2, ext3, Linux NFS and iSCSI Enterprise Target. Openfiler combines these ubiquitous technologies into a small, easy to manage solution fronted by a powerful web-based management interface. Openfiler supports CIFS, NFS, HTTP/DAV, FTP, however, we will only be making use of its iSCSI capabilities to implement an inexpensive SAN for the shared storage components required by Oracle RAC 10g. The operating system and Openfiler application will be installed on one internal SATA disk. A second internal 73GB 15K SCSI hard disk will be configured as a single "Volume Group" that will be used for all shared disk storage requirements. The Openfiler server will be configured to use this volume group for iSCSI based storage and will be used in our Oracle RAC 10g configuration to store the shared files required by Oracle Clusterware as well as all Oracle ASM volumes. Note: This article is provided for educational purposes only, so the setup is kept simple to demonstrate ideas and concepts. For example, the disk mirroring configured in this article will be setup on one physical disk only, while in practice that should be performed on multiple physical
www.oracle.com/technetwork/articles/hunter-rac10gr2-iscsi-101317.html
1/29
16/02/14
drives. Also note that while this article provides detailed instructions for successfully installing a complete Oracle RAC 10g system, it is by no means a substitute for the official Oracle documentation. In addition to this article, users should also consult the following Oracle documents to gain a full understanding of alternative configuration options, installation, and administration with Oracle RAC 10g. Oracle's official documentation site is docs.oracle.com. Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) for Linux Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 10g Release 2 (10.2) 2 Day + Real Application Clusters Guide - 10g Release 2 (10.2) The previous Oracle9i and Oracle 10g Release 1 guides used raw partitions for storing files on shared storage, but here we will make use of the Oracle Cluster File System Release 2 (OCFS2) and Oracle Automatic Storage Management (ASM) feature. The two Oracle RAC nodes will be configured as follows: Oracle Database Files RAC Node Name linux1 linux2 File Type Oracle Cluster Registry Instance Name racdb1 racdb2 File Name Database Name racdb racdb Oracle Clusterware Shared Files iSCSI Volume Name Mount Point /u02 File System OCFS2 /u02/oradata/racdb/OCRFile crs /u02/oradata/racdb/OCRFile_mirror /u02/oradata/racdb/CSSFile /u02/oradata/racdb/CSSFile_mirror1 crs /u02/oradata/racdb/CSSFile_mirror2 File System / $ORACLE_BASE Volume Manager for DB Files /u01/app/oracle ASM /u01/app/oracle ASM
CRS Voting Disk
/u02
OCFS2
Note that with Oracle Database 10g Release 2 (10.2), Cluster Ready Services, or CRS, is now called Oracle Clusterware. The Oracle Clusterware software will be installed to / u 0 1 / a p p / c r son both of the nodes that make up the RAC cluster. Starting with Oracle Database 10g Release 2 (10.2), Oracle Clusterware should be installed in a separate Oracle Clusterware home directory which is non-release specific ( / u 0 1 / a p p / o r a c l e / p r o d u c t / 1 0 . 2 . 0 / . . .for example) and must never be a subdirectory of the O R A C L E _ B A S Edirectory ( / u 0 1 / a p p / o r a c l efor example). This is a change to the Optimal Flexible Architecture (OFA) rules. Note that the Oracle Clusterware and Oracle Real Application Clusters installation documentation from Oracle incorrectly state that the Oracle Clusterware home directory can be a subdirectory of the O R A C L E _ B A S Edirectory. For example, in Chapter 2, "Preinstallation", in the section "Oracle Clusterware home directory", it incorrectly lists the path / u 0 1 / a p p / o r a c l e / p r o d u c t / c r sas a possible Oracle Clusterware home (or CRS home) path. This is incorrect. The default ORACLE_BASE path is / u 0 1 / a p p / o r a c l e , and the Oracle Clusterware home must never be a subdirectory of the ORACLE_BASE directory. This issue is tracked with Oracle documentation bug "5843155" - (B14203-08 HAS CONFLICTING CRS_HOME LOCATIONS ) and is fixed in Oracle 11g. The Oracle Clusterware software will be installed to / u 0 1 / a p p / c r son both of the nodes that make up the RAC cluster, however, the Clusterware software requires that two of its files, the "Oracle Cluster Registry (OCR)" file and the "Voting Disk" file be shared with both nodes in the cluster. These two files will be installed on shared storage using Oracle's Cluster File System, Release 2 (OCFS2). It is also possible to use RAW devices for these files, however, it is not possible to use ASM for these two shared Clusterware files. The Oracle10g Release 2 Database software will be installed into a separate Oracle Home; namely / u 0 1 / a p p / o r a c l e / p r o d u c t / 1 0 . 2 . 0 / d b _ 1 on both of the nodes that make up the RAC cluster. All of the Oracle physical database files (data, online redo logs, control files, archived redo logs) will be installed to shared volumes being managed by Automatic Storage Management (ASM). (The Oracle database files can just as easily be stored on OCFS2. Using ASM, however, makes the article that much more interesting!) Note: This article is only designed to work as documented with absolutely no substitutions. The only exception here is the choice of vendor hardware (i.e. machines, networking equipment, and internal / external hard drives). Ensure that the hardware you purchase from the vendor is supported on Oracle Enterprise Linux (Release 5 Update 3). I tend to stick with Dell hardware given their superb quality and compatibility with Linux. For a test system of this nature, I highly recommend purchasing pre-owned or refurbished Dell hardware from a reputable company like Stallard Technologies, Inc.. Stallard Technologies has a proven track record of delivering the best value on pre-owned hardware combined with a commitment to superior customer service. I base my recommendation on my own outstanding personal experience with their organization. To learn more about Stallard Technologies, visit their website or contact John Brauer. If you are looking for an example that takes advantage of Oracle RAC 11g Release 1 with OEL using iSCSI, click here. If you are looking for an example that takes advantage of Oracle RAC 10g Release 2 with RHEL 4 using FireWire, click here.
2. Oracle RAC 10g Overview Before introducing the details for building a RAC cluster, it might be helpful to first clarify what a cluster is. A cluster is a group of two or more interconnected computers or servers that appear as if they are one server to end users and applications and generally share the same set of physical disks. The key benefit of clustering is to provide a highly available framework where the failure of one node (for example a database server) does not bring down an entire application. In the case of failure with one of the servers, the other surviving server (or servers) can take over the workload from the failed server and the application continue to function normally as if nothing has happened. The concept of clustering computers actually started several decades ago. The first successful cluster product was developed by DataPoint in 1977 named ARCnet. The ARCnet product enjoyed much success by academia types in research labs, but didn't really take off in the commercial market. It wasn't until the 1980's when Digital Equipment Corporation (DEC) released its VAX cluster product for the VAX/VMS operating system. With the release of Oracle 6 for the Digital VAX cluster product, Oracle Corporation was the first commercial database to support clustering at the database level. It wasn't long, however, before Oracle realized the need for a more efficient and scalable distributed lock manager (DLM) as the one included with the VAX/VMS cluster product was not well suited for database applications. Oracle decided to design and write their own DLM for the VAX/VMS cluster product which provided the fine-grain block level locking required by the database. Oracle's own DLM was included in Oracle 6.2 which gave birth to Oracle Parallel Server (OPS) - the first database to run the parallel server. By Oracle 7, OPS was extended to included support for not only the VAX/VMS cluster product but also with most flavors of UNIX. This framework required vendor-supplied clusterware which worked well, but made for a complex environment to setup and manage given the multiple layers involved. By Oracle 8, Oracle introduced a generic lock manager which was integrated into the Oracle kernel. In later releases of Oracle, this became known as the Integrated Distributed Lock Manager (IDLM) and relied on an additional layer known as the Operating System Dependant (OSD) layer. This new model paved the way for Oracle to not only have their own DLM, but to also create their own clusterware product in future releases. Oracle Real Application Clusters (RAC), introduced with Oracle9i, is the successor to Oracle Parallel Server. Using the same IDLM, Oracle 9i could still rely on external clusterware but was the first release to include their own clusterware product named Cluster Ready Services (CRS). With Oracle 9i, CRS was only available for Windows and Linux. By Oracle 10g, Oracle's clusterware product was available for all operating systems. With the release of Oracle Database 10g Release 2 (10.2), Cluster Ready Services was renamed to Oracle Clusterware. When using Oracle 10g or higher, Oracle Clusterware is the only clusterware that you need for most platforms on which Oracle RAC operates (except for Tru cluster, where you need vendor clusterware). You can still use clusterware from other vendors if the clusterware is certified for Oracle RAC. This guide uses Oracle Clusterware 10g. Like OPS, Oracle RAC allows multiple instances to access the same database (storage) simultaneously. RAC provides fault tolerance, load balancing, and performance benefits by allowing the system to scale out, and at the same time since all instances access the same database, the failure of one node will not cause the loss of access to the database. At the heart of Oracle RAC is a shared disk subsystem. Each instance in the cluster must be able to access all of the data, redo log files, control files and parameter file for all other instances in the cluster. The data disks must be globally available in order to allow all instances to access the database. Each instance has its own redo log files and UNDO tablespace that are locally read-writeable. The other instances in the cluster must
2/29
16/02/14
be able to access them (read-only) in order to recover that instance in the event of a system failure. The redo log files for an instance are only writeable by that instance and will only be read from another instance during system failure. The UNDO, on the other hand, is read all the time during normal database operation (e.g. for CR fabrication). The biggest difference between Oracle RAC and OPS is the addition of Cache Fusion. With OPS a request for data from one instance to another required the data to be written to disk first, then the requesting instance can read that data (after acquiring the required locks). With cache fusion, data is passed along a high-speed interconnect using a sophisticated locking algorithm. Not all database clustering solutions use shared storage. Some vendors use an approach known as a Federated Cluster, in which data is spread across several machines rather than shared by all. With Oracle RAC, however, multiple instances use the same set of disks for storing data. Oracle's approach to clustering leverages the collective processing power of all the nodes in the cluster and at the same time provides failover security. Pre-configured Oracle RAC solutions are available from vendors such as Dell, IBM and HP for production environments. This article, however, focuses on putting together your own Oracle RAC 10g environment for development and testing by using Linux servers and a low cost shared disk solution; iSCSI. For more background about Oracle RAC, visit the Oracle RAC Product Center on OTN.
3. Shared-Storage Overview Today, fibre channel is one of the most popular solutions for shared storage. As mentioned earlier, fibre channel is a high-speed serial-transfer interface that is used to connect systems and storage devices in either point-to-point (FC-P2P), arbitrated loop (FC-AL), or switched topologies (FC-SW). Protocols supported by Fibre Channel include SCSI and IP. Fibre channel configurations can support as many as 127 nodes and have a throughput of up to 2.12 gigabits per second in each direction, and 4.25 Gbps is expected. Fibre channel, however, is very expensive. Just the fibre channel switch alone can start at around US$1,000. This does not even include the fibre channel storage array and high-end drives, which can reach prices of about US$300 for a 36GB drive. A typical fibre channel setup which includes fibre channel cards for the servers is roughly US$10,000, which does not include the cost of the servers that make up the cluster. A less expensive alternative to fibre channel is SCSI. SCSI technology provides acceptable performance for shared storage, but for administrators and developers who are used to GPL-based Linux prices, even SCSI can come in over budget, at around US$2,000 to US$5,000 for a two-node cluster. Another popular solution is the Sun NFS (Network File System) found on a NAS. It can be used for shared storage but only if you are using a network appliance or something similar. Specifically, you need servers that guarantee direct I/O over NFS, TCP as the transport protocol, and read/write block sizes of 32K. See the Certify page on Oracle Metalink for supported Network Attached Storage (NAS) devices that can be used with Oracle RAC. One of the key drawbacks that has limited the benefits of using NFS and NAS for database storage has been performance degradation and complex configuration requirements. Standard NFS client software (client systems that use the operating system provided NFS driver) is not optimized for Oracle database file I/O access patterns. With the introduction of Oracle 11g, a new feature known as Direct NFS Client integrates the NFS client functionality directly in the Oracle software. Through this integration, Oracle is able to optimize the I/O path between the Oracle software and the NFS server resulting in significant performance gains. Direct NFS Client can simplify, and in many cases automate, the performance optimization of the NFS client configuration for database workloads. To learn more about Direct NFS Client, see the Oracle White Paper entitled " Oracle Database 11g Direct NFS Client ". The shared storage that will be used for this article is based on iSCSI technology using a network storage server installed with Openfiler. This solution offers a low-cost alternative to fibre channel for testing and educational purposes, but given the low-end hardware being used, it is not recommended to be used in a production environment.
4. iSCSI Technology For many years, the only technology that existed for building a network based storage solution was a Fibre Channel Storage Area Network (FC SAN). Based on an earlier set of ANSI protocols called Fib er Distrib uted Data Interface (FDDI), Fibre Channel was developed to move SCSI commands over a storage network. Several of the advantages to FC SAN include greater performance, increased disk utilization, improved availability, better scalability, and most important to us support for server clustering! Still today, however, FC SANs suffer from three major disadvantages. The first is price. While the costs involved in building a FC SAN have come down in recent years, the cost of entry still remains prohibitive for small companies with limited IT budgets. The second is incompatible hardware components. Since its adoption, many product manufacturers have interpreted the Fibre Channel specifications differently from each other which has resulted in scores of interconnect problems. When purchasing Fibre Channel components from a common manufacturer, this is usually not a problem. The third disadvantage is the fact that a Fibre Channel network is not Ethernet! It requires a separate network technology along with a second set of skill sets that need to exist with the datacenter staff. With the popularity of Gigabit Ethernet and the demand for lower cost, Fibre Channel has recently been given a run for its money by iSCSI-based storage systems. Today, iSCSI SANs remain the leading competitor to FC SANs. Ratified on February 11, 2003 by the Internet Engineering Task Force (IETF), the Internet Small Computer System Interface, better known as iSCSI, is an Internet Protocol (IP)-based storage networking standard for establishing and managing connections between IP-based storage devices, hosts, and clients. iSCSI is a data transport protocol defined in the SCSI-3 specifications framework and is similar to Fibre Channel in that it is responsible for carrying block-level data over a storage network. Block-level communication means that data is transferred between the host and the client in chunks called blocks. Database servers depend on this type of communication (as opposed to the file level communication used by most NAS systems) in order to work properly. Like a FC SAN, an iSCSI SAN should be a separate physical network devoted entirely to storage, however, its components can be much the same as in a typical IP network (LAN). While iSCSI has a promising future, many of its early critics were quick to point out some of its inherent shortcomings with regards to performance. The beauty of iSCSI is its ability to utilize an already familiar IP network as its transport mechanism. The TCP/IP protocol, however, is very complex and CPU intensive. With iSCSI, most of the processing of the data (both TCP and iSCSI) is handled in software and is much slower than Fibre Channel which is handled completely in hardware. The overhead incurred in mapping every SCSI command onto an equivalent iSCSI transaction is excessive. For many the solution is to do away with iSCSI software initiators and invest in specialized cards that can offload TCP/IP and iSCSI processing from a server's CPU. These specialized cards are sometimes referred to as an iSCSI Host Bus Adaptor (HBA) or a TCP Offload Engine (TOE) card. Also consider that 10-Gigabit Ethernet is a reality today! As with any new technology, iSCSI comes with its own set of acronyms and terminology. For the purpose of this article, it is only important to understand the difference between an iSCSI initiator and an iSCSI target. iSCSI Initiator Basically, an iSCSI initiator is a client device that connects and initiates requests to some service offered by a server (in this case an iSCSI target). The iSCSI initiator software will need to exist on each of the Oracle RAC nodes ( l i n u x 1and l i n u x 2 ). An iSCSI initiator can be implemented using either software or hardware. Software iSCSI initiators are available for most major operating system platforms. For this article, we will be using the free Linux Open-iSCSI software driver found in the i s c s i i n i t i a t o r u t i l sRPM. The iSCSI software initiator is generally used with a standard network interface card (NIC) a Gigabit Ethernet card in most cases. A hardware initiator is an iSCSI HBA (or a TCP Offload Engine (TOE) card), which is basically just a specialized Ethernet card with a SCSI ASIC on-board to offload all the work (TCP and SCSI commands) from the system CPU. iSCSI HBAs are available from a number of vendors, including Adaptec, Alacritech, Intel, and QLogic. iSCSI Target o p e n f i l e r 1 So with all of this talk about iSCSI, does this mean the death of Fibre Channel anytime soon? Probably not. Fibre Channel has clearly
3/29
16/02/14
demonstrated its capabilities over the years with its capacity for extremely high speeds, flexibility, and robust reliability. Customers who have strict requirements for high performance storage, large complex connectivity, and mission critical reliability will undoubtedly continue to choose Fibre Channel. Before closing out this section, I thought it would be appropriate to present the following chart that shows speed comparisons of the various types of disk interfaces and network technologies. For each interface, I provide the maximum transfer rates in kilobits (kb), kilobytes (KB), megabits (Mb), megabytes (MB), gigabits (Gb), and gigabytes (GB) per second with some of the more common ones highlighted in grey. Disk Interface / Network / BUS Serial Parallel (standard) 10Base-T Ethernet IEEE 802.11b wireless Wi-Fi (2.4 GHz band) USB 1.1 Parallel (ECP/EPP) SCSI-1 IEEE 802.11g wireless WLAN (2.4 GHz band) SCSI-2 (Fast SCSI / Fast Narrow SCSI) 100Base-T Ethernet (Fast Ethernet) ATA/100 (parallel) IDE Fast Wide SCSI (Wide SCSI) Ultra SCSI (SCSI-3 / Fast-20 / Ultra Narrow) Ultra IDE Wide Ultra SCSI (Fast Wide 20) Ultra2 SCSI FireWire 400 - (IEEE1394a) USB 2.0 Wide Ultra2 SCSI Ultra3 SCSI FireWire 800 - (IEEE1394b) Gigabit Ethernet PCI - (33 MHz / 32-bit) Serial ATA I - (SATA I) Wide Ultra3 SCSI Ultra160 SCSI PCI - (33 MHz / 64-bit) PCI - (66 MHz / 32-bit) AGP 1x - (66 MHz / 32-bit) Serial ATA II - (SATA II) Ultra320 SCSI FC-AL Fibre Channel PCI-Express x1 - (bidirectional) PCI - (66 MHz / 64-bit) AGP 2x - (133 MHz / 32-bit) Serial ATA III - (SATA III) PCI-X - (100 MHz / 64-bit) PCI-X - (133 MHz / 64-bit) AGP 4x - (266 MHz / 32-bit) 10G Ethernet - (IEEE 802.3ae) PCI-Express x4 - (bidirectional) AGP 8x - (533 MHz / 32-bit) PCI-Express x8 - (bidirectional) PCI-Express x16 - (bidirectional) Speed Kb KB Mb MB Gb GB 115 14.375 0.115 0.014 920 115 0.92 0.115 10 11 12 24 40 54 80 100 100 160 160 264 320 320 400 480 640 640 800 1000 1064 1200 1280 1280 2128 2128 2128 2400 2560 3200 4000 4256 4264 4800 6400 1.25 1.375 1.5 3 5 6.75 10 12.5 12.5 20 20 33 40 40 50 60 80 80 100 125 133 150 160 160 266 266 266 300 320 400 500 532 533 600 800 1064 1066 1250 2000 2133 4000 8000
133.6 16.7
1 1.064 1.2 1.28 1.28 2.128 2.128 2.128 2.4 2.56 3.2 4 4.256 4.264 4.8 6.4 8.512 8.528 10 16 17.064 32 64
1 1 1.25 2 2.1 4 8
5. Hardware & Costs The hardware used to build our example Oracle RAC 10g environment consists of three Linux servers (two Oracle RAC nodes and one Network Storage Server) and components that can be purchased at many local computer stores or over the Internet (i.e. Stallard Technologies, Inc.). Oracle RAC Node 1 - (linux1) Dimension 2400 Series Intel(R) Pentium(R) 4 Processor at 2.80GHz 2GB DDR SDRAM (at 333MHz) 40GB 7200 RPM Internal Hard Disk Integrated Intel 3D AGP Graphics Integrated 10/100 Ethernet - (Broadcom BCM4401) CDROM (48X Max Variable) 3.5" Floppy No Keyboard, Monitor, or Mouse - (Connected to KVM Switch) 1 - Ethernet LAN Card Used for RAC interconnect to linux2 and Openfiler networked storage. Each Linux server for Oracle RAC should contain two NIC adapters. The Dell Dimension includes an integrated 10/100 Ethernet adapter that will be used to connect to the public network. The second NIC adapter will be used for the private network (RAC interconnect and Openfiler networked storage). Select the appropriate NIC adapter that is compatible with the maximum data transmission speed of the network switch to be used for the private network. For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network. Gigabit Ethernet Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT) Oracle RAC Node 2 - (linux2) Dimension 2400 Series Intel(R) Pentium(R) 4 Processor at 2.80GHz 2GB DDR SDRAM (at 333MHz) 40GB 7200 RPM Internal Hard Disk Integrated Intel 3D AGP Graphics Integrated 10/100 Ethernet - (Broadcom BCM4401) US$35
US$600
4/29
16/02/14
CDROM (48X Max Variable) 3.5" Floppy No Keyboard, Monitor, or Mouse - (Connected to KVM Switch) 1 - Ethernet LAN Card Used for RAC interconnect to linux1 and Openfiler networked storage. Each Linux server for Oracle RAC should contain two NIC adapters. The Dell Dimension includes an integrated 10/100 Ethernet adapter that will be used to connect to the public network. The second NIC adapter will be used for the private network (RAC interconnect and Openfiler networked storage). Select the appropriate NIC adapter that is compatible with the maximum data transmission speed of the network switch to be used for the private network. For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network. Gigabit Ethernet Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT) Network Storage Server - (openfiler1) Dell PowerEdge 1800 Dual 3.0GHz Xeon / 1MB Cache / 800FSB (SL7PE) 6GB of ECC Memory 500GB SATA Internal Hard Disk 73GB 15K SCSI Internal Hard Disk Single embedded Intel 10/100/1000 Gigabit NIC No Keyboard, Monitor, or Mouse - (Connected to KVM Switch) Note: The operating system and Openfiler application will be installed on the 500GB internal SATA disk. A second internal 73GB 15K SCSI hard disk will be configured for the database storage. The Openfiler server will be configured to use this second hard disk for iSCSI based storage and will be used in our Oracle RAC 10g configuration to store the shared files required by Oracle Clusterware as well as all Oracle ASM volumes. Please be aware that any type of hard disk (internal or external) should work for database storage as long as it can be recognized by the network storage server (Openfiler) and has adequate space. For example, I could have made an extra partition on the 500GB internal SATA disk for the iSCSI target, but decided to make use of the faster SCSI disk for this example. US$800 1 - Ethernet LAN Card Used for networked storage on the private network. The Network Storage Server (Openfiler server) should contain two NIC adapters. The Dell PowerEdge 1800 machine includes an integrated 10/100/1000 Ethernet adapter that will be used to connect to the public network. The second NIC adapter will be used for the private network (Openfiler networked storage). Select the appropriate NIC adapter that is compatible with the maximum data transmission speed of the network switch to be used for the private network. For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network. Gigabit Ethernet Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT) Miscellaneous Components 1 - Ethernet Switch Used for the interconnect between linux1-priv and linux2-priv. This switch will also be used for network storage traffic for Openfiler. For the purpose of this article, I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network. Gigabit Ethernet D-Link 8-port 10/100/1000 Desktop Switch - (DGS-2208) 6 - Network Cables Category 6 patch cable - (Connect linux1 to public network) Category 6 patch cable - (Connect linux2 to public network) Category 6 patch cable - (Connect openfiler1 to public network) Category 6 patch cable - (Connect linux1 to interconnect Ethernet switch) Category 6 patch cable - (Connect linux2 to interconnect Ethernet switch) Category 6 patch cable - (Connect openfiler1 to interconnect Ethernet switch) Optional Components KVM Switch This article requires access to the console of all nodes (servers) in order to install the operating system and perform several of the configuration tasks. When managing a very small number of servers, it might make sense to connect each server with its own monitor, keyboard, and mouse in order to access its console. However, as the number of servers to manage increases, this solution becomes unfeasible. A more practical solution would be to configure a dedicated computer which would include a single monitor, keyboard, and mouse that would have direct access to the console of each server. This solution is made possible using a Keyboard, Video, Mouse Switch better known as a KVM Switch. A KVM switch is a hardware device that allows a user to control multiple computers from a single keyboard, video monitor and mouse. Avocent provides a high quality and economical 4port switch which includes four 6' cables: SwitchView 1000 - (4SV1000BND1-001) For a detailed explanation and guide on the use and KVM switches, please see the article " KVM Switches For the Home and the Enterprise". Total US$340 US$2,555 US$50 US$10 US$10 US$10 US$10 US$10 US$10 US$35 US$35 US$600
We are about to start the installation process. Now that we have talked about the hardware that will be used in this example, let's take a conceptual look at what the environment would look like (click on the graphic b elow to view larger image):
5/29
16/02/14
Figure 1: Architecture As we start to go into the details of the installation, it should be noted that most of the tasks within this document will need to be performed on both Oracle RAC nodes (linux1 and linux2). I will indicate at the beginning of each section whether or not the task(s) should be performed on both Oracle RAC nodes or on the network storage server (openfiler1).
6. Install the Linux Operating System Perform the following installation on b oth Oracle RAC nodes in the cluster! This section provides a summary of the screens used to install the Linux operating system. This guide is designed to work with Oracle Enterprise Linux Release 5 Update 3 for x86 and follows Oracle's suggestion of performing a "default RPMs" installation type to ensure all expected Linux O/S packages are present for a successful Oracle RDBMS installation. Before installing the Oracle Enterprise Linux operating system on both Oracle RAC nodes, you should have the two NIC interfaces (cards) installed. Download the following ISO images for Oracle Enterprise Linux Release 5 Update 3 for either x86 or x86_64 depending on your hardware architecture. Oracle E-Delivery Web site for Oracle Enterprise Linux 32-bit (x86) Installations V15417-01.zip (581 MB) V15418-01.zip (617 MB) V15419-01.zip (618 MB) V15420-01.zip (611 MB) V15421-01.zip (253 MB) After downloading the Oracle Enterprise Linux operating system, unzip each of the files. You will then have the following ISO images which will need to be burned to CDs: Enterprise-R5-U3-Server-i386-disc1.iso Enterprise-R5-U3-Server-i386-disc2.iso Enterprise-R5-U3-Server-i386-disc3.iso Enterprise-R5-U3-Server-i386-disc4.iso Enterprise-R5-U3-Server-i386-disc5.iso
Note: If the Linux RAC nodes have a DVD installed, you may find it more convenient to make use of the single DVD image: V15414-01.zip (2.7 GB)
Unzip the single DVD image file and burn it to a DVD: Enterprise-R5-U3-Server-i386-dvd.iso
64-bit (x86_64) Installations V15427-01.zip (577 MB) V15428-01.zip (615 MB) V15429-01.zip (619 MB) V15430-01.zip (615 MB) V15431-01.zip (620 MB) V15432-01.zip (120 MB) After downloading the Oracle Enterprise Linux operating system, unzip each of the files. You will then have the following ISO images which will need to be burned to CDs: Enterprise-R5-U3-Server-x86_64-disc1.iso Enterprise-R5-U3-Server-x86_64-disc2.iso Enterprise-R5-U3-Server-x86_64-disc3.iso Enterprise-R5-U3-Server-x86_64-disc4.iso Enterprise-R5-U3-Server-x86_64-disc5.iso Enterprise-R5-U3-Server-x86_64-disc6.iso
Note: If the Linux RAC nodes have a DVD installed, you may find it more convenient to make use of the single DVD image: V15416-01.zip (3.2 GB)
6/29
16/02/14
Unzip the single DVD image file and burn it to a DVD: Enterprise-R5-U3-Server-x86_64-dvd.iso
If you are downloading the above ISO files to a MS Windows machine, there are many options for burning these images (ISO files) to a CD/DVD. You may already be familiar with and have the proper software to burn images to a CD/DVD. If you are not familiar with this process and do not have the required software to burn images to a CD/DVD, here are just two (of many) software packages that can be used: UltraISO Magic ISO Maker
After downloading and burning the Oracle Enterprise Linux images (ISO files) to CD/DVD, insert OEL Disk #1 into the first server ( l i n u x 1in this example), power it on, and answer the installation screen prompts as noted below. After completing the Linux installation on the first node, perform the same Linux installation on the second node while substituting the node name l i n u x 1for l i n u x 2and the different IP addresses where appropriate. Boot Screen The first screen is the Oracle Enterprise Linux boot screen. At the boot: prompt, hit [Enter] to start the installation process. Media Test When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any errors, the media burning software would have warned us. After several seconds, the installer should then detect the video card, monitor, and mouse. The installer then goes into GUI mode. Welcome to Oracle Enterprise Linux At the welcome screen, click [Next] to continue. Language / Keyboard Selection The next two screens prompt you for the Language and Keyboard settings. Make the appropriate selections for your configuration. Detect Previous Installation Note that if the installer detects a previous version of Oracle Enterprise Linux, it will ask if you would like to "Install Enterprise Linux" or "Upgrade an existing Installation". Always select to "Install Enterprise Linux". Disk Partitioning Setup Keep the default selection to [Remove linux partitions on selected drives and create default layout] and check the option to [Review and modify partitioning layout]. Click [Next] to continue. You will then be prompted with a dialog window asking if you really want to remove all Linux partitions. Click [Yes] to acknowledge this warning. Partitioning The installer will then allow you to view (and modify if needed) the disk partitions it automatically selected. For most automatic layouts, the installer will choose 100MB for /boot, double the amount of RAM (systems with <= 2,048MB RAM) or an amount equal to RAM (systems with > 2,048MB RAM) for swap, and the rest going to the root ( / ) partition. Starting with RHEL 4, the installer will create the same disk configuration as just noted but will create them using the Logical Volume Manager (LVM). For example, it will partition the first hard drive ( / d e v / h d afor my configuration) into two partitions one for the /boot partition ( / d e v / h d a 1 ) and the remainder of the disk dedicate to a LVM named VolGroup00 ( / d e v / h d a 2 ). The LVM Volume Group (VolGroup00) is then partitioned into two LVM partitions - one for the root filesystem ( / ) and another for swap. The main concern during the partitioning phase is to ensure enough swap space is allocated as required by Oracle (which is a multiple of the available RAM). The following is Oracle's requirement for swap space: Available RAM Swap Space Required Between 1,024MB and 2,048MB 1.5 times the size of RAM Between 2,049MB and 8,192MB Equal to the size of RAM More than 8,192MB 0.75 times the size of RAM
For the purpose of this install, I will accept all automatically preferred sizes. (Including 4,096MB for swap since I have 2,048MB of RAM installed.) If for any reason, the automatic layout does not configure an adequate amount of swap space, you can easily change that from this screen. To increase the size of the swap partition, [Edit] the volume group VolGroup00. This will bring up the "Edit LVM Volume Group: VolGroup00" dialog. First, [Edit] and decrease the size of the root file system ( / ) by the amount you want to add to the swap partition. For example, to add another 512MB to swap, you would decrease the size of the root file system by 512MB (i.e. 36,032MB - 512MB = 35,520MB). Now add the space you decreased from the root file system (512MB) to the swap partition. When completed, click [OK] on the "Edit LVM Volume Group: VolGroup00" dialog. Once you are satisfied with the disk layout, click [Next] to continue. Boot Loader Configuration The installer will use the GRUB boot loader by default. To use the GRUB boot loader, accept all default values and click [Next] to continue. Network Configuration I made sure to install both NIC interfaces (cards) in each of the Linux machines before starting the operating system installation. This screen should have successfully detected each of the network devices. Since we will be using this machine to host an Oracle database, there will be several changes that need to be made to the network configuration. The settings you make here will, of course, depend on your network configuration. The key point to make is that the machine should never be configured with DHCP since it will be used to host the Oracle database server. You will need to configure the machine with static IP addresses. You will also need to configure the server with a real host name. First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to not activate eth1 by default. Second, [Edit] both eth0 and eth1 as follows. Verify that the option "Enable IPv4 support" is selected. Click off the option for "Use dynamic IP configuration (DHCP)" and configure a static IP address and Netmask for your environment. Click off the option to "Enable IPv6 support". You may choose to use different IP addresses for both eth0 and eth1 that I have documented in this guide and that is OK. Put eth1 (the interconnect) on a different subnet than eth0 (the public network): eth0: - Check ON the option to [Enable IPv4 support] - Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration) IPv4 Address: 192.168.1.100 Prefix (Netmask): 255.255.255.0 - Check OFF the option to [Enable IPv6 support] eth1: - Check ON the option to [Enable IPv4 support] - Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration) IPv4 Address: 192.168.2.100 Prefix (Netmask): 255.255.255.0 - Check OFF the option to [Enable IPv6 support] Continue by manually setting your hostname. I used "linux1" for the first node and "linux2" for the second. Finish this dialog off by supplying your gateway and DNS servers. Time Zone Selection Select the appropriate time zone for your environment and click [Next] to continue.
7/29
16/02/14
Set Root Password Select a root password and click [Next] to continue. Package Installation Defaults By default, Oracle Enterprise Linux installs most of the software required for a typical server. There are several other packages (RPMs), however, that are required to successfully install the Oracle Database software. The installer includes a "Customize software" selection that allows the addition of RPM groupings such as "Development Libraries" or "Legacy Library Support". The ADDITION of such RPM groupings is NOT an issue. De-selecting any "default RPM" groupings or individual RPMs, however, can result in failed Oracle Clusterware and Oracle Database installation attempts. For the purpose of this article, select the radio button [Customize now] and click [Next] to continue. This is where you pick the packages to install. Most of the packages required for the Oracle software are grouped into "Package Groups" (i.e. Application -> Editors). Since these nodes will be hosting the Oracle Clusterware and Oracle RAC software, verify that at least the following package groups are selected for install. For many of the Linux package groups, not all of the packages associated with that group get selected for installation. (Note the "Optional packages" button after selecting a package group.) So although the package group gets selected for install, some of the packages required by Oracle do not get installed. In fact, there are some packages that are required by Oracle that do not belong to any of the available package groups (i.e. libaio-devel). Not to worry. A complete list of required packages for Oracle Clusterware 10g and Oracle RAC 10g for Oracle Enterprise Linux 5 will be provided at the end of this section. These packages will need to be manually installed from the Oracle Enterprise Linux CDs after the operating system install. For now, install the following package groups: Desktop Environments GNOME Desktop Environment Applications Editors Graphical Internet Text-based Internet Development Development Libraries Development Tools Legacy Software Development Servers Server Configuration Tools Base System Administration Tools Base Java Legacy Software Support System Tools X Window System
In addition to the above packages, select any additional packages you wish to install for this node keeping in mind to NOT de-select any of the "default" RPM packages . After selecting the packages to install click [Next] to continue. About to Install This screen is basically a confirmation screen. Click [Next] to start the installation. If you are installing Oracle Enterprise Linux using CDs, you will be asked to switch CDs during the installation process depending on which packages you selected. Congratulations And that's it. You have successfully installed Oracle Enterprise Linux on the first node (linux1). The installer will eject the CD/DVD from the CDROM drive. Take out the CD/DVD and click [Reboot] to reboot the system. Post Installation Wizard Welcome Screen When the system boots into Oracle Enterprise Linux for the first time, it will prompt you with another Welcome screen for the "Post Installation Wizard". The post installation wizard allows you to make final O/S configuration settings. On the "Welcome" screen, click [Forward] to continue. License Agreement Read through the license agreement. Choose "Yes, I agree to the License Agreement" and click [Forward] to continue. Firewall On this screen, make sure to select the [Disabled] option and click [Forward] to continue. You will be prompted with a warning dialog about not setting the firewall. When this occurs, click [Yes] to continue. SELinux On the SELinux screen, choose the [Disabled] option and click [Forward] to continue. You will be prompted with a warning dialog warning that changing the SELinux setting will require rebooting the system so the entire file system can be relabeled. When this occurs, click [Yes] to acknowledge a reboot of the system will occur after firstboot (Post Installation Wizard) is completed. Kdump Accept the default setting on the Kdump screen (disabled) and click [Forward] to continue. Date and Time Settings Adjust the date and time settings if necessary and click [Forward] to continue. Create User Create any additional (non-oracle) operating system user accounts if desired and click [Forward] to continue. For the purpose of this article, I will not be creating any additional operating system accounts. I will be creating the "oracle" user account during the Oracle database installation later in this guide. If you chose not to define any additional operating system user accounts, click [Continue] to acknowledge the warning dialog. Sound Card This screen will only appear if the wizard detects a sound card. On the sound card screen click [Forward] to continue. Additional CDs On the "Additional CDs" screen click [Finish] to continue. Reboot System Given we changed the SELinux option (to disabled), we are prompted to reboot the system. Click [OK] to reboot the system for normal use. Login Screen After rebooting the machine, you are presented with the login screen. Login using the "root" user account and the password you provided during the installation. Perform the same installation on the second node After completing the Linux installation on the first node, repeat the above steps for the second node (linux2). When configuring the machine name and networking, ensure to configure the proper values. For my installation, this is what I configured for linux2: First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to not activate eth1. Second, [Edit] both eth0 and eth1 as follows. Verify that the option "Enable IPv4 support" is selected. Click off the option for "Use dynamic IP configuration (DHCP)" and configure a static IP address and Netmask for your environment. Click off the option to "Enable IPv6 support". You may choose to use different IP addresses for both eth0 and eth1 that I have documented in this guide and that is OK. Put eth1 (the interconnect) on a
8/29
16/02/14
different subnet than eth0 (the public network): eth0: - Check ON the option to [Enable IPv4 support] - Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration) IPv4 Address: 192.168.1.101 Prefix (Netmask): 255.255.255.0 - Check OFF the option to [Enable IPv6 support] eth1: - Check ON the option to [Enable IPv4 support] - Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration) IPv4 Address: 192.168.2.101 Prefix (Netmask): 255.255.255.0 - Check OFF the option to [Enable IPv6 support] Continue by setting your hostname manually. I used "linux2" for the second node. Finish this dialog off by supplying your gateway and DNS servers.
7. Install Required Linux Packages for Oracle RAC Install the following required Linux packages on b oth Oracle RAC nodes in the cluster! After installing Oracle Enterprise Linux, the next step is to verify and install all packages (RPMs) required by both Oracle Clusterware and Oracle RAC. The Oracle Universal Installer (OUI) performs checks on your machine during installation to verify that it meets the appropriate operating system package requirements. To ensure that these checks complete successfully, verify the software requirements documented in this section before starting the Oracle installs. Although many of the required packages for Oracle were installed in Section 6 (Install the Linux Operating System), several will be missing either because they were considered optional within the package group or simply didn't exist in any package group! The packages listed in this section (or later versions) are required for Oracle Clusterware 10g Release 2 and Oracle RAC 10g Release 2 running on the Oracle Enterprise Linux 5 platform. 32-bit (x86) Installations binutils-2.17.50.0.6-2.el5 compat-libstdc++-296-2.96-138 compat-libstdc++-33-3.2.3-61 elfutils-libelf-0.125-3.el5 elfutils-libelf-devel-0.125 gcc-4.1.1-52 gcc-c++-4.1.1-52 glibc-2.5-12 glibc-common-2.5-12 glibc-devel-2.5-12 glibc-headers-2.5-12 libaio-0.3.106 libaio-devel-0.3.106 libgcc-4.1.1-52 libstdc++-4.1.1 libstdc++-devel-4.1.1-52.e15 libXp-1.0.0-8 make-3.81-1.1 openmotif-2.2.3 sysstat-7.0.0 unixODBC-2.2.11 unixODBC-devel-2.2.11
Note that the openmotif RPM packages are required to install Oracle demos. This article does not cover the installation of Oracle demos. Each of the packages listed above can be found on CD #1, CD #2, and CD #3 on the Oracle Enterprise Linux 5 - (x86) CDs. While it is possible to query each individual package to determine which ones are missing and need to be installed, an easier method is to run the r p mU v h P a c k a g e N a m e command from the five CDs as follows. For packages that already exist and are up to date, the RPM command will simply ignore the install and print a warning message to the console that the package is already installed. # F r o mO r a c l eE n t e r p r i s eL i n u x5 . 3( x 8 6 ) -[ C D# 1 ] m k d i rp/ m e d i a / c d r o m m o u n tr/ d e v / c d r o m/ m e d i a / c d r o m c d/ m e d i a / c d r o m / S e r v e r r p mU v hb i n u t i l s 2 . * r p mU v he l f u t i l s l i b e l f 0 . * r p mU v hg l i b c 2 . * r p mU v hg l i b c c o m m o n 2 . * r p mU v hl i b a i o 0 . * r p mU v hl i b g c c 4 . * r p mU v hl i b s t d c + + 4 . * r p mU v hm a k e 3 . * c d/ e j e c t
# F r o mO r a c l eE n t e r p r i s eL i n u x5 . 3( x 8 6 )-[ C D# 2 ] m o u n tr/ d e v / c d r o m/ m e d i a / c d r o m c d/ m e d i a / c d r o m / S e r v e r r p mU v he l f u t i l s l i b e l f d e v e l 0 . * r p mU v hg l i b c d e v e l 2 . * r p mU v hg l i b c h e a d e r s 2 . * r p mU v hg c c 4 . * r p mU v hg c c c + + 4 . * r p mU v hl i b s t d c + + d e v e l 4 . * r p mU v hu n i x O D B C 2 . * c d/ e j e c t
9/29
16/02/14
F r o mO r a c l eE n t e r p r i s eL i n u x5 . 3( x 8 6 )-[ C D# 3 ] m o u n tr/ d e v / c d r o m/ m e d i a / c d r o m c d/ m e d i a / c d r o m / S e r v e r r p mU v hc o m p a t l i b s t d c + + 2 9 6 * r p mU v hc o m p a t l i b s t d c + + 3 3 * r p mU v hl i b a i o d e v e l 0 . * r p mU v hl i b X p 1 . * r p mU v ho p e n m o t i f 2 . * r p mU v hs y s s t a t 7 . * r p mU v hu n i x O D B C d e v e l 2 . * c d/ e j e c t
64-bit (x86_64) Installations binutils-2.17.50.0.6-2.el5 compat-libstdc++-296-2.96-138 compat-libstdc++-33-3.2.3-61 compat-libstdc++-33-3.2.3-61 (32 bit) elfutils-libelf-0.125-3.el5 elfutils-libelf-devel-0.125 gcc-4.1.1 gcc-c++-4.1.1 glibc-2.5-12 glibc-2.5-12 (32 bit) glibc-common-2.5 glibc-devel-2.5-12 glibc-devel-2.5-12 (32 bit) glibc-headers-2.5-12 libaio-0.3.106 libaio-0.3.106 (32 bit) libaio-devel-0.3.106 libgcc-4.1.1 libgcc-4.1.1 (32 bit) libstdc++-4.1.1 libstdc++-4.1.1 (32 bit) libstdc++-devel-4.1.1-52.e15 libXp-1.0.0-8 make-3.81-1.1 openmotif-2.2.3 sysstat-7.0.0 unixODBC-2.2.11 unixODBC-2.2.11 (32 bit) unixODBC-devel-2.2.11
Note that the openmotif RPM packages are required to install Oracle demos. This article does not cover the installation of Oracle demos. Each of the packages listed above can be found on CD #1, CD #2, CD #3, and CD #4 on the Oracle Enterprise Linux 5 - (x86_64) CDs. While it is possible to query each individual package to determine which ones are missing and need to be installed, an easier method is to run the r p mU v hP a c k a g e N a m e command from the five CDs as follows. For packages that already exist and are up to date, the RPM command will simply ignore the install and print a warning message to the console that the package is already installed. # F r o mO r a c l eE n t e r p r i s eL i n u x5 . 3( x 8 6 _ 6 4 ) -[ C D# 1 ] m k d i rp/ m e d i a / c d r o m m o u n tr/ d e v / c d r o m/ m e d i a / c d r o m c d/ m e d i a / c d r o m / S e r v e r r p mU v hb i n u t i l s 2 . * r p mU v he l f u t i l s l i b e l f 0 . * r p mU v hg l i b c 2 . * r p mU v hg l i b c c o m m o n 2 . * r p mU v hl i b a i o 0 . * r p mU v hl i b g c c 4 . * r p mU v hl i b s t d c + + 4 . * r p mU v hm a k e 3 . * c d/ e j e c t
# F r o mO r a c l eE n t e r p r i s eL i n u x5 . 3( x 8 6 _ 6 4 )-[ C D# 2 ] m o u n tr/ d e v / c d r o m/ m e d i a / c d r o m c d/ m e d i a / c d r o m / S e r v e r r p mU v he l f u t i l s l i b e l f d e v e l 0 . * r p mU v hg l i b c d e v e l 2 . * r p mU v hg l i b c h e a d e r s 2 . * r p mU v hg c c 4 . * r p mU v hg c c c + + 4 . * r p mU v hl i b s t d c + + d e v e l 4 . * r p mU v hu n i x O D B C 2 . * c d/ e j e c t
# F r o mO r a c l eE n t e r p r i s eL i n u x5 . 3( x 8 6 _ 6 4 )-[ C D# 3 ] m o u n tr/ d e v / c d r o m/ m e d i a / c d r o m c d/ m e d i a / c d r o m / S e r v e r r p mU v hc o m p a t l i b s t d c + + 2 9 6 * r p mU v hc o m p a t l i b s t d c + + 3 3 * r p mU v hl i b a i o d e v e l 0 . * r p mU v hl i b X p 1 . * r p mU v ho p e n m o t i f 2 . * r p mU v hu n i x O D B C d e v e l 2 . * c d/ e j e c t
10/29
16/02/14
#
F r o mO r a c l eE n t e r p r i s eL i n u x5 . 3( x 8 6 _ 6 4 )-[ C D# 4 ] m o u n tr/ d e v / c d r o m/ m e d i a / c d r o m c d/ m e d i a / c d r o m / S e r v e r r p mU v hs y s s t a t 7 . * c d/ e j e c t
8. Network Configuration Perform the following network configuration on b oth Oracle RAC nodes in the cluster! Note: Although we configured several of the network settings during the installation of Oracle Enterprise Linux, it is important to not skip this section as it contains critical steps that are required for a successful RAC environment. Introduction to Network Settings During the Linux O/S install we already configured the IP address and host name for both of the Oracle RAC nodes. We now need to configure the / e t c / h o s t sfile as well as adjusting several of the network settings for the interconnect. Both of the Oracle RAC nodes should have one static IP address for the public network and one static IP address for the private cluster interconnect. Do not use DHCP naming for the public IP address or the interconnects; you need static IP addresses! The private interconnect should only be used by Oracle to transfer Cluster Manager and Cache Fusion related data along with data for the network storage server (Openfiler). Note that Oracle does not support using the public network interface for the interconnect. You must have one network interface for the public network and another network interface for the private interconnect. For a production RAC implementation, the interconnect should be at least gigabit (or more) and only be used by Oracle as well as having the network storage server (Openfiler) on a separate gigabit network. Configuring Public and Private Network In our two node example, we need to configure the network on both Oracle RAC nodes for access to the public network as well as their private interconnect. The easiest way to configure network settings in Oracle Enterprise Linux is with the program "Network Configuration". This application can be started from the command-line as the "root" user account as follows: # s u-
# / u s r / b i n / s y s t e m c o n f i g n e t w o r k& Note: Do not use DHCP naming for the public IP address or the interconnects - we need static IP addresses! Using the Network Configuration application, you need to configure both NIC devices as well as the / e t c / h o s t sfile. Both of these tasks can be completed using the Network Configuration GUI. Notice that the / e t c / h o s t ssettings are the same for both nodes and that I removed any entry that has to do with IPv6 (for example, : : 1l o c a l h o s t 6 . l o c a l d o m a i n 6l o c a l h o s t 6 ). Our example configuration will use the following settings: Oracle RAC Node 1 - (linux1) Device IP Address Subnet Gateway Purpose eth0 192.168.1.100 255.255.255.0 192.168.1.1 Connects linux1 to the public network eth1 192.168.2.100 255.255.255.0 Connects linux1 (interconnect) to linux2 (linux2-priv) /etc/hosts 1 2 7 . 0 . 0 . 1 l o c a l h o s t . l o c a l d o m a i nl o c a l h o s t #P u b l i cN e t w o r k-( e t h 0 ) 1 9 2 . 1 6 8 . 1 . 1 0 0 l i n u x 1 1 9 2 . 1 6 8 . 1 . 1 0 1 l i n u x 2 #P r i v a t eI n t e r c o n n e c t-( e t h 1 ) 1 9 2 . 1 6 8 . 2 . 1 0 0 l i n u x 1 p r i v 1 9 2 . 1 6 8 . 2 . 1 0 1 l i n u x 2 p r i v #P u b l i cV i r t u a lI P( V I P )a d d r e s s e s-( e t h 0 : 1 ) 1 9 2 . 1 6 8 . 1 . 2 0 0 l i n u x 1 v i p 1 9 2 . 1 6 8 . 1 . 2 0 1 l i n u x 2 v i p #P r i v a t eS t o r a g eN e t w o r kf o rO p e n f i l e r-( e t h 1 ) 1 9 2 . 1 6 8 . 1 . 1 9 5 o p e n f i l e r 1 1 9 2 . 1 6 8 . 2 . 1 9 5 o p e n f i l e r 1 p r i v Oracle RAC Node 2 - (linux2) Device IP Address Subnet Gateway Purpose eth0 192.168.1.101 255.255.255.0 192.168.1.1 Connects linux2 to the public network eth1 192.168.2.101 255.255.255.0 Connects linux2 (interconnect) to linux1 (linux1-priv) /etc/hosts 1 2 7 . 0 . 0 . 1 l o c a l h o s t . l o c a l d o m a i nl o c a l h o s t #P u b l i cN e t w o r k-( e t h 0 ) 1 9 2 . 1 6 8 . 1 . 1 0 0 l i n u x 1 1 9 2 . 1 6 8 . 1 . 1 0 1 l i n u x 2 #P r i v a t eI n t e r c o n n e c t-( e t h 1 ) 1 9 2 . 1 6 8 . 2 . 1 0 0 l i n u x 1 p r i v 1 9 2 . 1 6 8 . 2 . 1 0 1 l i n u x 2 p r i v #P u b l i cV i r t u a lI P( V I P )a d d r e s s e s-( e t h 0 : 1 ) 1 9 2 . 1 6 8 . 1 . 2 0 0 l i n u x 1 v i p 1 9 2 . 1 6 8 . 1 . 2 0 1 l i n u x 2 v i p #P r i v a t eS t o r a g eN e t w o r kf o rO p e n f i l e r-( e t h 1 ) 1 9 2 . 1 6 8 . 1 . 1 9 5 o p e n f i l e r 1 1 9 2 . 1 6 8 . 2 . 1 9 5 o p e n f i l e r 1 p r i v
11/29
16/02/14
Note that the virtual IP addresses only need to be defined in the / e t c / h o s t sfile (or your DNS) for both Oracle RAC nodes. The public virtual IP addresses will be configured automatically by Oracle Clusterware when you run the Oracle Universal Installer, which starts Oracle's Virtual Internet Protocol Configuration Assistant (VIPCA). All virtual IP addresses will be activated when the s r v c t ls t a r tn o d e a p p sn< n o d e _ n a m e > command is run. This is the Host Name/IP Address that will be configured in the client(s) tnsnames.ora file (more details later). Although I am getting ahead of myself, this is the Host Name/IP Address that will be configured in the client(s) t n s n a m e s . o r afile for each Oracle Net Service Name. All of this will be explained much later in this article! In the screen shots below, only Oracle RAC Node 1 (linux1) is shown. Be sure to make all the proper network settings to both Oracle RAC nodes.
Figure 2: Network Configuration Screen, Node 1 (linux1)
Figure 3: Ethernet Device Screen, eth0 (linux1)
12/29
16/02/14
Figure 4: Ethernet Device Screen, eth1 (linux1)
Figure 5: Network Configuration Screen, /etc/hosts (linux1) Once the network is configured, you can use the i f c o n f i gcommand to verify everything is working. The following example is from l i n u x 1 : # / s b i n / i f c o n f i ga e t h 0 L i n ke n c a p : E t h e r n e t H W a d d r0 0 : 1 4 : 6 C : 7 6 : 5 C : 7 1 i n e ta d d r : 1 9 2 . 1 6 8 . 1 . 1 0 0 B c a s t : 1 9 2 . 1 6 8 . 1 . 2 5 5 M a s k : 2 5 5 . 2 5 5 . 2 5 5 . 0 i n e t 6a d d r :f e 8 0 : : 2 1 4 : 6 c f f : f e 7 6 : 5 c 7 1 / 6 4S c o p e : L i n k U PB R O A D C A S TR U N N I N GM U L T I C A S T M T U : 1 5 0 0 M e t r i c : 1 R Xp a c k e t s : 4 7 4 1 7 6e r r o r s : 0d r o p p e d : 0o v e r r u n s : 0f r a m e : 0 T Xp a c k e t s : 2 9 5 4 2 4e r r o r s : 0d r o p p e d : 0o v e r r u n s : 0c a r r i e r : 0 c o l l i s i o n s : 0t x q u e u e l e n : 1 0 0 0 R Xb y t e s : 6 4 6 9 6 0 8 6 8( 6 1 6 . 9M i B ) T Xb y t e s : 1 3 8 0 6 8 3 6 4( 1 3 1 . 6M i B ) I n t e r r u p t : 1 7 7B a s ea d d r e s s : 0 x 6 f 0 0 e t h 1 L i n ke n c a p : E t h e r n e t H W a d d r0 0 : 0 E : 0 C : 6 4 : D 1 : E 5 i n e ta d d r : 1 9 2 . 1 6 8 . 2 . 1 0 0 B c a s t : 1 9 2 . 1 6 8 . 2 . 2 5 5 M a s k : 2 5 5 . 2 5 5 . 2 5 5 . 0 i n e t 6a d d r :f e 8 0 : : 2 0 e : c f f : f e 6 4 : d 1 e 5 / 6 4S c o p e : L i n k U PB R O A D C A S TR U N N I N GM U L T I C A S T M T U : 1 5 0 0 M e t r i c : 1 R Xp a c k e t s : 1 4 5 2 6 0 2e r r o r s : 0d r o p p e d : 0o v e r r u n s : 0f r a m e : 0 T Xp a c k e t s : 1 8 0 4 2 6 3e r r o r s : 0d r o p p e d : 0o v e r r u n s : 0c a r r i e r : 0 c o l l i s i o n s : 0t x q u e u e l e n : 1 0 0 0 R Xb y t e s : 1 1 6 5 7 2 4 7 7 1( 1 . 0G i B ) T Xb y t e s : 2 2 0 5 8 2 6 7 9 2( 2 . 0G i B ) B a s ea d d r e s s : 0 x d d c 0M e m o r y : f e 9 c 0 0 0 0 f e 9 e 0 0 0 0 L i n ke n c a p : L o c a lL o o p b a c k i n e ta d d r : 1 2 7 . 0 . 0 . 1 M a s k : 2 5 5 . 0 . 0 . 0 i n e t 6a d d r :: : 1 / 1 2 8S c o p e : H o s t U PL O O P B A C KR U N N I N G M T U : 1 6 4 3 6 M e t r i c : 1 R Xp a c k e t s : 1 3 7 1e r r o r s : 0d r o p p e d : 0o v e r r u n s : 0f r a m e : 0 T Xp a c k e t s : 1 3 7 1e r r o r s : 0d r o p p e d : 0o v e r r u n s : 0c a r r i e r : 0
l o
13/29
16/02/14
c o l l i s i o n s : 0t x q u e u e l e n : 0 R Xb y t e s : 2 1 6 9 8 1 4( 2 . 0M i B ) T Xb y t e s : 2 1 6 9 8 1 4( 2 . 0M i B ) s i t 0 L i n ke n c a p : I P v 6 i n I P v 4 N O A R P M T U : 1 4 8 0 M e t r i c : 1 R Xp a c k e t s : 0e r r o r s : 0d r o p p e d : 0o v e r r u n s : 0f r a m e : 0 T Xp a c k e t s : 0e r r o r s : 0d r o p p e d : 0o v e r r u n s : 0c a r r i e r : 0 c o l l i s i o n s : 0t x q u e u e l e n : 0 R Xb y t e s : 0( 0 . 0b ) T Xb y t e s : 0( 0 . 0b )
About Virtual IP Why is there a Virtual IP (VIP) in 10g? Why does it just return a dead connection when its primary node fails? It's all about availability of the application. When a node fails, the VIP associated with it is supposed to be automatically failed over to some other node. When this occurs, two things happen. The new node re-arps the world indicating a new MAC address for the address. For directly connected clients, this usually causes them to see errors on their connections to the old address. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the clients getting errors immediately. This means that when the client issues SQL to the node that is now down, or traverses the address list while connecting, rather than waiting on a very long TCP/IP time-out (~10 minutes), the client receives a TCP reset. In the case of SQL, this is O R A 3 1 1 3 . In the case of connect, the next address in tnsnames is used. Going one step further is making use of Transparent Application Failover (TAF). With TAF successfully configured, it is possible to completely avoid O R A 3 1 1 3errors all together! TAF will be discussed in more detail in Section 31 ("Transparent Application Failover - (TAF)"). Without using VIPs, clients connected to a node that died will often wait a 10-minute TCP timeout period before getting an error. As a result, you don't really have a good HA solution without using VIPs (Source - Metalink Note 220970.1). Confirm the RAC Node Name is Not Listed in Loopback Address Ensure that the node names ( l i n u x 1or l i n u x 2 ) are not included for the loopback address in the / e t c / h o s t sfile. If the machine name is listed in the in the loopback address entry as below: 1 2 7 . 0 . 0 . 1
l i n u x 1 l o c a l h o s t . l o c a l d o m a i nl o c a l h o s t
it will need to be removed as shown below: 1 2 7 . 0 . 0 . 1l o c a l h o s t . l o c a l d o m a i nl o c a l h o s t
If the RAC node name is listed for the loopback address, you will receive the following error during the RAC installation: O R A 0 0 6 0 3 :O R A C L Es e r v e rs e s s i o nt e r m i n a t e db yf a t a le r r o r
or O R A 2 9 7 0 2 :e r r o ro c c u r r e di nC l u s t e rG r o u pS e r v i c eo p e r a t i o n
Confirm localhost is defined in the /etc/hosts file for the loopback address Ensure that the entry for l o c a l h o s t . l o c a l d o m a i nand l o c a l h o s tare included for the loopback address in the / e t c / h o s t sfile for each of the Oracle RAC nodes: 1 2 7 . 0 . 0 . 1 l o c a l h o s t . l o c a l d o m a i nl o c a l h o s t If an entry does not exist for l o c a l h o s tin the / e t c / h o s t sfile, Oracle Clusterware will be unable to start the application resources notably the ONS process. The error would indicate " Failed to get IP for localhost " and will be written to the log file for ONS. For example: C R S 0 2 1 5c o u l dn o ts t a r tr e s o u r c e' o r a . l i n u x 1 . o n s ' .C h e c kl o gf i l e " / u 0 1 / a p p / c r s / l o g / l i n u x 1 / r a c g / o r a . l i n u x 1 . o n s . l o g " f o rm o r ed e t a i l s . The ONS log file will contain lines similar to the following: O r a c l eD a t a b a s e1 0 gC R SR e l e a s e1 0 . 2 . 0 . 1 . 0P r o d u c t i o nC o p y r i g h t1 9 9 6 ,2 0 0 5O r a c l e .A l lr i g h t sr e s e r v e d . 2 0 0 7 0 4 1 41 3 : 1 0 : 0 2 . 7 2 9 :[R A C G ] [ 3 0 8 6 8 7 1 2 9 6 ] [ 1 3 3 1 6 ] [ 3 0 8 6 8 7 1 2 9 6 ] [ o r a . l i n u x 1 . o n s ] :F a i l e dt og e tI Pf o rl o c a l h o s t ( 1 ) F a i l e dt og e tI Pf o rl o c a l h o s t( 1 ) F a i l e dt og e tI Pf o rl o c a l h o s t( 1 ) o n s c t l :o n sf a i l e dt os t a r t . . . Adjusting Network Settings With Oracle 9.2.0.1 and later, Oracle makes use of UDP as the default protocol on Linux for inter-process communication (IPC), such as Cache Fusion and Cluster Manager buffer transfers between instances within the RAC cluster. Oracle strongly suggests to adjust the default and maximum receive buffer size (SO_RCVBUF socket option) to 1MB and the default and maximum send buffer size ( S O _ S N D B U Fsocket option) to 256KB. The receive buffers are used by TCP and UDP to hold received data until it is read by the application. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This means that datagrams will be discarded if they don't fit in the socket receive buffer, potentially causing the sender to overwhelm the receiver. The default and maximum window size can be changed in the / p r o cfile system without reboot: # s u-r o o t # s y s c t lwn e t . c o r e . r m e m _ d e f a u l t = 1 0 4 8 5 7 6 n e t . c o r e . r m e m _ d e f a u l t=1 0 4 8 5 7 6
14/29
16/02/14
#
s y s c t lwn e t . c o r e . r m e m _ m a x = 1 0 4 8 5 7 6 n e t . c o r e . r m e m _ m a x=1 0 4 8 5 7 6 # s y s c t lwn e t . c o r e . w m e m _ d e f a u l t = 2 6 2 1 4 4 n e t . c o r e . w m e m _ d e f a u l t=2 6 2 1 4 4 # s y s c t lwn e t . c o r e . w m e m _ m a x = 2 6 2 1 4 4 n e t . c o r e . w m e m _ m a x=2 6 2 1 4 4 The above commands made the changes to the already running OS. You should now make the above changes permanent (for each reboot) by adding the following lines to the / e t c / s y s c t l . c o n ffile for both nodes in your RAC cluster: #+ + #|A D J U S T I N GN E T W O R KS E T T I N G S | #+ + #|W i t hO r a c l e9 . 2 . 0 . 1a n do n w a r d s ,O r a c l en o wm a k e su s e | #|o fU D Pa st h ed e f a u l tp r o t o c o lo nL i n u xf o r | #|i n t e r p r o c e s sc o m m u n i c a t i o n( I P C ) ,s u c ha sC a c h eF u s i o n| #|a n dC l u s t e rM a n a g e rb u f f e rt r a n s f e r sb e t w e e ni n s t a n c e s | #|w i t h i nt h eR A Cc l u s t e r .O r a c l es t r o n g l ys u g g e s t st o | #|a d j u s tt h ed e f a u l ta n dm a x i m u mr e c e i v eb u f f e rs i z e | #|( S O _ R C V B U Fs o c k e to p t i o n )t o1 0 2 4K B ,a n dt h ed e f a u l t | #|a n dm a x i m u ms e n db u f f e rs i z e( S O _ S N D B U Fs o c k e to p t i o n ) | #|t o2 5 6K B .T h er e c e i v eb u f f e r sa r eu s e db yT C Pa n dU D P | #|t oh o l dr e c e i v e dd a t au n t i li ti sr e a db yt h e | #|a p p l i c a t i o n .T h er e c e i v eb u f f e rc a n n o to v e r f l o wb e c a u s e| #|t h ep e e ri sn o ta l l o w e dt os e n dd a t ab e y o n dt h eb u f f e r | #|s i z ew i n d o w .T h i sm e a n st h a td a t a g r a m sw i l lb e | #|d i s c a r d e di ft h e yd o n ' tf i ti nt h es o c k e tr e c e i v e | #|b u f f e r .T h i sc o u l dc a u s et h es e n d e rt oo v e r w h e l mt h e | #|r e c e i v e r . | #+ + #+ + #|D e f a u l ts e t t i n gi nb y t e so ft h es o c k e t" r e c e i v e "b u f f e r| #|w h i c hm a yb es e tb yu s i n gt h eS O _ R C V B U Fs o c k e to p t i o n . | #+ + n e t . c o r e . r m e m _ d e f a u l t = 1 0 4 8 5 7 6 #+ + #|M a x i m u ms e t t i n gi nb y t e so ft h es o c k e t" r e c e i v e "b u f f e r| #|w h i c hm a yb es e tb yu s i n gt h eS O _ R C V B U Fs o c k e to p t i o n . | #+ + n e t . c o r e . r m e m _ m a x = 1 0 4 8 5 7 6 #+ + #|D e f a u l ts e t t i n gi nb y t e so ft h es o c k e t" s e n d "b u f f e r | #|w h i c hm a yb es e tb yu s i n gt h eS O _ S N D B U Fs o c k e to p t i o n . | #+ + n e t . c o r e . w m e m _ d e f a u l t = 2 6 2 1 4 4 #+ + #|M a x i m u ms e t t i n gi nb y t e so ft h es o c k e t" s e n d "b u f f e r | #|w h i c hm a yb es e tb yu s i n gt h eS O _ S N D B U Fs o c k e to p t i o n . | #+ + n e t . c o r e . w m e m _ m a x = 2 6 2 1 4 4 Check and turn off UDP ICMP rejections During the Linux installation process, I indicated to not configure the firewall option. By default the option to configure a firewall is selected by the installer. This has burned me several times so I like to do a double-check that the firewall option is not configured and to ensure udp ICMP filtering is turned off. If UDP ICMP is blocked or rejected by the firewall, the Oracle Clusterware software will crash after several minutes of running. When the Oracle Clusterware process fails, you will have something similar to the following in the < m a c h i n e _ n a m e > _ e v m o c r . l o gfile: 0 8 / 2 9 / 2 0 0 52 2 : 1 7 : 1 9
o a c _ i n i t : 2 :C o u l dn o tc o n n e c tt os e r v e r ,c l s cr e t c o d e=9
0 8 / 2 9 / 2 0 0 52 2 : 1 7 : 1 9
a _ i n i t : 1 2 ! :C l i e n ti n i tu n s u c c e s s f u l:[ 3 2 ]
i b c t x : 1 : E R R O R :I N V A L I DF O R M A T
p r o p r i n i t : p r o b l e mr e a d i n gt h eb o o t b l o c ko rs u p e r b l o c2 2 When experiencing this type of error, the solution is to remove the udp ICMP (iptables) rejection rule - or to simply have the firewall option turned off. The Oracle Clusterware software will then start to operate normally and not crash. The following commands should be executed as the r o o t user account: Check to ensure that the firewall option is turned off. If the firewall option is stopped (like it is in my example below) you do not have to proceed with the following steps. # / e t c / r c . d / i n i t . d / i p t a b l e ss t a t u s
15/29
16/02/14
F i r e w a l li ss t o p p e d . If the firewall option is operating you will need to first manually disable UDP ICMP rejections: # / e t c / r c . d / i n i t . d / i p t a b l e ss t o p
F l u s h i n gf i r e w a l lr u l e s :[ O K]
S e t t i n gc h a i n st op o l i c yA C C E P T :f i l t e r[ O K]
U n l o a d i n gi p t a b l e sm o d u l e s :[ O K] Then, to turn UDP ICMP rejections off for next server reboot (which should always be turned off): # c h k c o n f i gi p t a b l e so f f
9. Install Openfiler Perform the following installation on the network storage server (openfiler1)! With the network configured on both Oracle RAC nodes, the next step is to install the Openfiler software to the network storage server ( o p e n f i l e r 1 ). Later in this article, the network storage server will be configured as an iSCSI storage device for all Oracle RAC 10g shared storage requirements. Powered by rPath Linux, Openfiler is a free browser-based network storage management utility that delivers file-based Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single framework. The entire software stack interfaces with open source applications such as Apache, Samba, LVM2, ext3, Linux NFS and iSCSI Enterprise Target. Openfiler combines these ubiquitous technologies into a small, easy to manage solution fronted by a powerful web-based management interface. Openfiler supports CIFS, NFS, HTTP/DAV, FTP, however, we will only be making use of its iSCSI capabilities to implement an inexpensive SAN for the shared storage components required by Oracle RAC 10g. The operating system and Openfiler application will be installed on one internal SATA disk. A second internal 73GB 15K SCSI hard disk will be configured as a single "Volume Group" that will be used for all shared disk storage requirements. The Openfiler server will be configured to use this volume group for iSCSI based storage and will be used in our Oracle RAC 10g configuration to store the shared files required by Oracle Clusterware as well as all Oracle ASM volumes. Please be aware that any type of hard disk (internal or external) should work for database storage as long as it can be recognized by the network storage server (Openfiler) and has adequate space. For example, I could have made an extra partition on the 500GB internal SATA disk for the iSCSI target, but decided to make use of the faster SCSI disk for this example. To learn more about Openfiler, please visit their website at http://www.openfiler.com/ Download Openfiler Use the links below to download Openfiler NAS/SAN Appliance, version 2.3 (Final Release) for either x86 or x86_64 depending on your hardware architecture. This example uses x86_64. After downloading Openfiler, you will then need to burn the ISO image to CD. Openfiler x86: openfiler-2.3-x86-disc1.iso (322 MB) - OR x86_64: openfiler-2.3-x86_64-disc1.iso (336 MB) If you are downloading the above ISO file to a MS Windows machine, there are many options for burning these images (ISO files) to a CD. You may already be familiar with and have the proper software to burn images to CD. If you are not familiar with this process and do not have the required software to burn images to CD, here are just two (of many) software packages that can be used: UltraISO Magic ISO Maker
Install Openfiler This section provides a summary of the screens used to install the Openfiler software. For the purpose of this article, I opted to install Openfiler with all default options. The only manual change required was for configuring the local network settings. Once the install has completed, the server will reboot to make sure all required components, services and drivers are started and recognized. After the reboot, any external hard drives (if connected) will be discovered by the Openfiler server. For more detailed installation instructions, please visit http://www.openfiler.com/learn/. I would suggest, however, that the instructions I have provided below be used for this Oracle RAC 10g configuration. Before installing the Openfiler software to the network storage server, you should have both NIC interfaces (cards) installed and any external hard drives connected and turned on (if you will be using external hard drives). After downloading and burning the Openfiler ISO image (ISO file) to CD, insert the CD into the network storage server ( o p e n f i l e r 1in this example), power it on, and answer the installation screen prompts as noted below. Boot Screen The first screen is the Openfiler boot screen. At the boot: prompt, hit [Enter] to start the installation process. Media Test When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any errors, the media burning software would have warned us. After several seconds, the installer should then detect the video card, monitor, and mouse. The installer then goes into GUI mode. Welcome to Openfiler NSA At the welcome screen, click [Next] to continue. Keyboard Configuration The next screen prompts you for the Keyboard settings. Make the appropriate selection for your configuration. Disk Partitioning Setup
16/29
16/02/14
The next screen asks whether to perform disk partitioning using "Automatic Partitioning" or "Manual Partitioning with Disk Druid". Although the official Openfiler documentation suggests to use Manual Partitioning, I opted to use "Automatic Partitioning" given the simplicity of my example configuration. Select [Automatically partition] and click [Next] continue. Automatic Partitioning If there were a previous installation of Linux on this machine, the next screen will ask if you want to "remove" or "keep" old partitions. Select the option to [Remove all partitions on this system]. For my example configuration, I selected ONLY the 500GB SATA internal hard drive [sdb] for the operating system and Openfiler application installation. I de-selected the 73GB SCSI internal hard drive since this disk will be used exclusively in the next section to create a single "Volume Group" that will be used for all iSCSI based shared disk storage requirements for Oracle RAC 10g. I also keep the checkbox [Review (and modify if needed) the partitions created] selected. Click [Next] to continue. You will then be prompted with a dialog window asking if you really want to remove all partitions. Click [Yes] to acknowledge this warning. Partitioning The installer will then allow you to view (and modify if needed) the disk partitions it automatically chose for hard disks selected in the previous screen. In almost all cases, the installer will choose 100MB for /boot, an adequate amount of swap, and the rest going to the root ( / ) partition for that disk (or disks). In this example, I am satisfied with the installers recommended partitioning for / d e v / s d b . The installer will also show any other internal hard disks it discovered. For my example configuration, the installer found the 73GB SCSI internal hard drive as / d e v / s d a . For now, I will "Delete" any and all partitions on this drive (there was only one, / d e v / s d a 1 ). In the next section, I will create the required partition for this particular hard disk. Network Configuration I made sure to install both NIC interfaces (cards) in the network storage server before starting the Openfiler installation. This screen should have successfully detected each of the network devices. First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to not activate eth1 by default. Second, [Edit] both eth0 and eth1 as follows. You may choose to use different IP addresses for both eth0 and eth1 and that is OK. You must, however, configure eth1 (the storage network) to be on the same subnet you configured for eth1 on linux1 and linux2: eth0: - Check off the option to [Configure using DHCP] - Leave the [Activate on boot] checked - IP Address: 192.168.1.195 - Netmask: 255.255.255.0 eth1: - Check off the option to [Configure using DHCP] - Leave the [Activate on boot] checked - IP Address: 192.168.2.195 - Netmask: 255.255.255.0 Continue by setting your hostname manually. I used a hostname of "openfiler1". Finish this dialog off by supplying your gateway and DNS servers. Time Zone Selection The next screen allows you to configure your time zone information. Make the appropriate selection for your location. Set Root Password Select a root password and click [Next] to continue. About to Install This screen is basically a confirmation screen. Click [Next] to start the installation. Congratulations And that's it. You have successfully installed Openfiler on the network storage server. The installer will eject the CD from the CD-ROM drive. Take out the CD and click [Reboot] to reboot the system. If everything was successful after the reboot, you should now be presented with a text login screen and the URL to use for administering the Openfiler server. Modify /etc/hosts File on Openfiler Server Although not mandatory, I typically copy the contents of the /etc/hosts file from one of the Oracle RAC nodes to the new Openfiler server. This allows convenient name resolution when testing the network for the cluster.
10. Configure iSCSI Volumes using Openfiler Perform the following configuration tasks on the network storage server (openfiler1)! Openfiler administration is performed using the Openfiler Storage Control Center a browser based tool over an https connection on port 446. For example: h t t p s : / / o p e n f i l e r 1 . i d e v e l o p m e n t . i n f o : 4 4 6 /
The first page the administrator sees is the [Status] / [System Information] screen. To use Openfiler as an iSCSI storage server, we have to perform six major tasks; set up iSCSI services, configure network access, identify and partition the physical storage, create a new volume group, create all logical volumes, and finally, create new iSCSI targets for each of the logical volumes. Services To control services, we use the Openfiler Storage Control Center and navigate to [Services] / [Manage Services]: From the Openfiler Storage Control Center home page, login as an administrator. The default administration login credentials for Openfiler are: Username: o p e n f i l e r Password: p a s s w o r d
17/29
16/02/14
Figure 6: Enable iSCSI Openfiler Service To enable the iSCSI service, click on the 'Enable' link under the 'iSCSI target server' service name. After that, the 'iSCSI target server' status should change to 'Enabled'. The ietd program implements the user level part of iSCSI Enterprise Target software for building an iSCSI storage system on Linux. With the iSCSI target enabled, we should be able to SSH into the Openfiler server and see the iscsi-target service running: [ r o o t @ o p e n f i l e r 1~ ] # s e r v i c ei s c s i t a r g e ts t a t u s i e t d( p i d1 4 2 4 3 )i sr u n n i n g . . . Network Access Configuration The next step is to configure network access in Openfiler to identify both Oracle RAC nodes ( l i n u x 1and l i n u x 2 ) that will need to access the iSCSI volumes through the storage (private) network. Note that iSCSI volumes will be created later on in this section. Also note that this step does not actually grant the appropriate permissions to the iSCSI volumes required by both Oracle RAC nodes. That will be accomplished later in this section by updating the ACL for each new logical volume. As in the previous section, configuring network access is accomplished using the Openfiler Storage Control Center by navigating to [System] / [Network Setup]. The "Network Access Configuration" section (at the bottom of the page) allows an administrator to setup networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we will want to add both Oracle RAC nodes individually rather than allowing the entire 192.168.2.0 network have access to Openfiler resources. When entering each of the Oracle RAC nodes, note that the 'Name' field is just a logical name used for reference only. As a convention when entering nodes, I simply use the node name defined for that IP address. Next, when entering the actual node in the 'Network/Host' field, always use its IP address even though its host name may already be defined in your / e t c / h o s t sfile or DNS. Lastly, when entering actual hosts in our Class C network, use a subnet mask of 255.255.255.255. It is important to remember that you will be entering the IP address of the private network (eth1) for each of the RAC nodes in the cluster. The following image shows the results of adding both Oracle RAC nodes:
Figure 7: Configure Openfiler Network Access for Oracle RAC Nodes Physical Storage In this section, we will be creating the five iSCSI volumes to be used as shared storage by both of the Oracle RAC nodes in the cluster. This involves multiple steps that will be performed on the internal 73GB 15K SCSI hard disk connected to the Openfiler server. Storage devices like internal IDE/SATA/SCSI/SAS disks, storage arrays, external USB drives, external FireWire drives, or ANY other storage can be connected to the Openfiler server and served to the clients. Once these devices are discovered at the OS level, Openfiler Storage Control Center can be used to set up and manage all of that storage.
18/29
16/02/14
In our case, we have a 73GB internal SCSI hard drive for our shared storage needs. On the Openfiler server this drive is seen as / d e v / s d b (MAXTOR ATLAS15K2_73SCA). To see this and to start the process of creating our iSCSI volumes, navigate to [Volumes] / [Block Devices] from the Openfiler Storage Control Center:
Figure 8: Openfiler Physical Storage - Block Device Management Partitioning the Physical Disk The first step we will perform is to create a single primary partition on the / d e v / s d binternal hard disk. By clicking on the / d e v / s d blink, we are presented with the options to 'Edit' or 'Create' a partition. Since we will be creating a single primary partition that spans the entire disk, most of the options can be left to their default setting where the only modification would be to change the ' Partition Type' from 'Extended partition' to ' Physical volume'. Here are the values I specified to create the primary partition on / d e v / s d b : Mode: Primary Partition Type: Physical volume Starting Cylinder: 1 Ending Cylinder: 8924 The size now shows 68.36 GB. To accept that we click on the "Create" button. This results in a new partition ( / d e v / s d b 1 ) on our internal hard disk:
Figure 9: Partition the Physical Volume Volume Group Management The next step is to create a Volume Group. We will be creating a single volume group named r a c 1that contains the newly created primary partition. From the Openfiler Storage Control Center, navigate to [Volumes] / [Volume Groups]. There we would see any existing volume groups, or none as in our case. Using the Volume Group Management screen, enter the name of the new volume group (rac1), click on the checkbox in front of / d e v / s d b 1to select that partition, and finally click on the 'Add volume group' button. After that we are presented with the list that now shows our newly created volume group named "rac1":
19/29
16/02/14
Figure 10: New Volume Group Created Logical Volumes We can now create the five logical volumes in the newly created volume group (rac1). From the Openfiler Storage Control Center, navigate to [Volumes] / [Add Volume]. There we will see the newly created volume group (rac1) along with its block storage statistics. Also available at the bottom of this screen is the option to create a new volume in the selected volume group (Create a volume in "rac1"). Use this screen to create the following five logical (iSCSI) volumes. After creating each logical volume, the application will point you to the "Manage Volumes" screen. You will then need to click back to the "Add Volume" tab to create the next logical volume until all five iSCSI volumes are created: iSCSI / Logical Volumes Volume Name Volume Description Required Space (MB) Filesystem Type r a c d b c r s racdb - Oracle Clusterware 2,048 iSCSI r a c d b a s m 1 racdb - ASM Volume 1 16,984 iSCSI r a c d b a s m 2 racdb - ASM Volume 2 16,984 iSCSI r a c d b a s m 3 racdb - ASM Volume 3 16,984 iSCSI r a c d b a s m 4 racdb - ASM Volume 4 16,984 iSCSI In effect we have created five iSCSI disks that can now be presented to iSCSI clients ( l i n u x 1and l i n u x 2 ) on the network. The "Manage Volumes" screen should look as follows:
20/29
16/02/14
Figure 11: New Logical (iSCSI) Volumes iSCSI Targets At this point we have five iSCSI logical volumes. Before an iSCSI client can have access to them, however, an iSCSI target will need to be created for each of these five volumes. Each iSCSI logical volume will be mapped to a specific iSCSI target and the appropriate network access permissions to that target will be granted to both Oracle RAC nodes. For the purpose of this article, there will be a one-to-one mapping between an iSCSI logical volume and an iSCSI target. There are three steps involved in creating and configuring an iSCSI target; create a unique Target IQN (basically, the universal name for the new iSCSI target), map one of the iSCSI logical volumes (created in the previous section) to the newly created iSCSI target, and finally, grant both of the Oracle RAC nodes access to the new iSCSI target. Please note that this process will need to be performed for each of the five iSCSI logical volumes created in the previous section. For the purpose of this article, the following table lists the new iSCSI target names (the Target IQN) and which iSCSI logical volume it will be mapped to: iSCSI Target / Logical Volume Mappings Target IQN iSCSI Volume Name Volume Description i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s r a c d b c r s racdb - Oracle Clusterware i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1r a c d b a s m 1 racdb - ASM Volume 1 i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2r a c d b a s m 2 racdb - ASM Volume 2 i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3r a c d b a s m 3 racdb - ASM Volume 3 i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4r a c d b a s m 4 racdb - ASM Volume 4 We are now ready to create the five new iSCSI targets - one for each of the iSCSI logical volumes. The example below illustrates the three steps required to create a new iSCSI target by creating the Oracle Clusterware / racdb-crs target ( i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s ). This three step process will need to be repeated for each of the five new iSCSI targets listed in the table above. Create New Target IQN From the Openfiler Storage Control Center, navigate to [Volumes] / [iSCSI Targets]. Verify the grey sub-tab "Target Configuration" is selected. This page allows you to create a new iSCSI target. A default value is automatically generated for the name of the new iSCSI target (better known as the "Target IQN"). An example Target IQN is " i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : t s n . a e 4 6 8 3 b 6 7 f d 3 ":
Figure 12: Create New iSCSI Target : Default Target IQN
21/29
16/02/14
t s n . a e 4 6 8 3 b 6 7 f d 3 r a c d b . c r s Figure 13
Figure 13: Create New iSCSI Target : Replace Default Target IQN Once you are satisfied with the new Target IQN, click the "Add" button. This will create a new iSCSI target and then bring up a page that allows you to modify a number of settings for the new iSCSI target. For the purpose of this article, none of settings for the new iSCSI target need to be changed. LUN Mapping After creating the new iSCSI target, the next step is to map the appropriate iSCSI logical volumes to it. Under the "Target Configuration" sub-tab, verify the correct iSCSI target is selected in the section "Select iSCSI Target". If not, use the pull-down menu to select the correct iSCSI target and hit the "Change" button. Next, click on the grey sub-tab named "LUN Mapping" (next to "Target Configuration" sub-tab). Locate the appropriate iSCSI logical volume ( / d e v / r a c 1 / r a c d b c r sin this case) and click the "Map" button. You do not need to change any settings on this page. Your screen should look similar to Figure 14 after clicking the "Map" button:
Figure 14: Create New iSCSI Target : Map LUN Network ACL Before an iSCSI client can have access to the newly created iSCSI target, it needs to be granted the appropriate permissions. Awhile back, we configured network access in Openfiler for two hosts (the Oracle RAC nodes). These are the two nodes that will need to access the new iSCSI targets through the storage (private) network. We now need to grant both of the Oracle RAC nodes access to the new iSCSI target. Click on the grey sub-tab named "Network ACL" (next to "LUN Mapping" sub-tab). For the current iSCSI target, change the "Access" for both hosts from 'Deny' to 'Allow' and click the 'Update' button:
22/29
16/02/14
Figure 15: Create New iSCSI Target : Update Network ACL Go back to the Create New Target IQN section and perform these three tasks for the remaining four ISCSI logical volumes while substituting the values found in the " iSCSI Target / Logical Volume Mappings" table .
11. Configure iSCSI Volumes on Oracle RAC Nodes Configure the iSCSI initiator on b oth Oracle RAC nodes in the cluster! Creating partitions, however, should only b e executed on one of nodes in the RAC cluster. An iSCSI client can be any system (Linux, Unix, MS Windows, Apple Mac, etc.) for which iSCSI support (a driver) is available. In our case, the clients are two Linux servers, l i n u x 1and l i n u x 2 , running Oracle Enterprise Linux 5.3. In this section we will be configuring the iSCSI software initiator on both of the Oracle RAC nodes. Oracle Enterprise Linux 5.3 includes the OpeniSCSI iSCSI software initiator which can be found in the i s c s i i n i t i a t o r u t i l sRPM. This is a change from previous versions of Oracle Enterprise Linux (4.x) which included the Linux iscsi-sfnet software driver developed as part of the Linux-iSCSI Project. All iSCSI management tasks like discovery and logins will use the command-line interface i s c s i a d mwhich is included with Open-iSCSI. The iSCSI software initiator will be configured to automatically login to the network storage server ( o p e n f i l e r 1 ) and discover the iSCSI volumes created in the previous section. We will then go through the steps of creating persistent local SCSI device names (i.e. / d e v / i s c s i / a s m 1 ) for each of the iSCSI target names discovered using u d e v . Having a consistent local SCSI device name and which iSCSI target it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM. Before we can do any of this, however, we must first install the iSCSI initiator software! Installing the iSCSI (initiator) service With Oracle Enterprise Linux 5.3, the Open-iSCSI iSCSI software initiator does not get installed by default. The software is included in the iscsiinitiator-utils package which can be found on CD #1. To determine if this package is installed (which in most cases, it will not be), perform the following on both Oracle RAC nodes: # r p mq a|g r e pi s c s i i n i t i a t o r u t i l s If the iscsi-initiator-utils package is not installed, load CD #1 into each of the Oracle RAC nodes and perform the following: # m o u n tr/ d e v / c d r o m/ m e d i a / c d r o m # c d/ m e d i a / c d r o m / S e r v e r # r p mU v hi s c s i i n i t i a t o r u t i l s 6 . 2 . 0 . 8 6 8 0 . 1 8 . e l 5 . i 3 8 6 . r p m # c d/ # e j e c t Configure the iSCSI (initiator) service After verifying that the i s c s i i n i t i a t o r u t i l spackage is installed on both Oracle RAC nodes, start the i s c s i dservice and enable it to automatically start when the system boots. We will also configure the i s c s iservice to automatically start which logs into iSCSI targets needed at system startup. # s e r v i c ei s c s i ds t a r t T u r n i n go f fn e t w o r ks h u t d o w n .S t a r t i n gi S C S Id a e m o n :[ O K ] [ O K ] # c h k c o n f i gi s c s i do n # c h k c o n f i gi s c s io n Now that the iSCSI service is started, use the i s c s i a d mcommand-line interface to discover all available targets on the network storage server. This should be performed on both Oracle RAC nodes to verify the configuration is functioning properly:
23/29
16/02/14
#
i s c s i a d mmd i s c o v e r yts e n d t a r g e t spo p e n f i l e r 1 p r i v 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 , 1i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 , 1i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 , 1i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 , 1i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 , 1i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s Manually Login to iSCSI Targets At this point the iSCSI initiator service has been started and each of the Oracle RAC nodes were able to discover the available targets from the network storage server. The next step is to manually login to each of the available targets which can be done using the i s c s i a d mcommand-line interface. This needs to be run on both Oracle RAC nodes. Note that I had to specify the IP address and not the host name of the network storage server (openfiler1-priv) - I believe this is required given the discovery (above) shows the targets using the IP address. # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1p1 9 2 . 1 6 8 . 2 . 1 9 5l # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2p1 9 2 . 1 6 8 . 2 . 1 9 5l # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3p1 9 2 . 1 6 8 . 2 . 1 9 5l # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4p1 9 2 . 1 6 8 . 2 . 1 9 5l # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r sp1 9 2 . 1 6 8 . 2 . 1 9 5l Configure Automatic Login The next step is to ensure the client will automatically login to each of the targets listed above when the machine is booted (or the iSCSI initiator service is started/restarted). As with the manual login process described above, perform the following on both Oracle RAC nodes: # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1p1 9 2 . 1 6 8 . 2 . 1 9 5o pu p d a t enn o d e . s t a r t u pva u t o m a t i c # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2p1 9 2 . 1 6 8 . 2 . 1 9 5o pu p d a t enn o d e . s t a r t u pva u t o m a t i c # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3p1 9 2 . 1 6 8 . 2 . 1 9 5o pu p d a t enn o d e . s t a r t u pva u t o m a t i c # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4p1 9 2 . 1 6 8 . 2 . 1 9 5o pu p d a t enn o d e . s t a r t u pva u t o m a t i c # i s c s i a d mmn o d eTi q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r sp1 9 2 . 1 6 8 . 2 . 1 9 5o pu p d a t enn o d e . s t a r t u pva u t o m a t i c Create Persistent Local SCSI Device Names In this section, we will go through the steps to create persistent local SCSI device names for each of the iSCSI target names. This will be done using u d e v . Having a consistent local SCSI device name and which iSCSI target it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM. When either of the Oracle RAC nodes boot and the iSCSI initiator service is started, it will automatically login to each of the targets configured in a random fashion and map them to the next available local SCSI device name. For example, the target i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1may get mapped to / d e v / s d a . I can actually determine the current mappings for all targets by looking at the / d e v / d i s k / b y p a t hdirectory: # ( c d/ d e v / d i s k / b y p a t h ;l sl* o p e n f i l e r *|a w k' { F S = "" ;p r i n t$ 9""$ 1 0""$ 1 1 } ' ) i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 l u n 0>. . / . . / s d a i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2 l u n 0>. . / . . / s d b i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3 l u n 0>. . / . . / s d c i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4 l u n 0>. . / . . / s d d i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s l u n 0>. . / . . / s d e Using the output from the above listing, we can establish the following current mappings: Current iSCSI Target Name to local SCSI Device Name Mappings iSCSI Target Name SCSI Device Name iqn.2006-01.com.openfiler:racdb.asm1 iqn.2006-01.com.openfiler:racdb.asm2 iqn.2006-01.com.openfiler:racdb.asm3 iqn.2006-01.com.openfiler:racdb.asm4 / d e v / s d a / d e v / s d b / d e v / s d c / d e v / s d d
iqn.2006-01.com.openfiler:racdb.crs / d e v / s d e This mapping, however, may change every time the Oracle RAC node is rebooted. For example, after a reboot it may be determined that the iSCSI target i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1gets mapped to the local SCSI device / d e v / s d d . It is therefore impractical to rely on using the local SCSI device name given there is no way to predict the iSCSI target mappings after a reboot. What we need is a consistent device name we can reference (i.e. / d e v / i s c s i / a s m 1 ) that will always point to the appropriate iSCSI target through reboots. This is where the Dynamic Device Management tool named u d e vcomes in. u d e vprovides a dynamic device directory using symbolic links that point to the actual device using a configurable set of rules. When u d e vreceives a device event (for example, the client logging in to an iSCSI target), it matches its configured rules against the available device attributes provided in sysfs to identify the device. Rules that match may provide additional device information or specify a device node name and multiple symlink names and instruct u d e vto run additional programs (a SHELL script for example) as part of the device event handling process. The first step is to create a new rules file. The file will be named / e t c / u d e v / r u l e s . d / 5 5 o p e n i s c s i . r u l e sand contain only a single line of name=value pairs used to receive events we are interested in. It will also define a call-out SHELL script ( / e t c / u d e v / s c r i p t s / i s c s i d e v . s h ) to handle the event. Create the following rules file / e t c / u d e v / r u l e s . d / 5 5 o p e n i s c s i . r u l e son both Oracle RAC nodes: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . #/ e t c / u d e v / r u l e s . d / 5 5 o p e n i s c s i . r u l e s K E R N E L = = " s d * " ,B U S = = " s c s i " ,P R O G R A M = " / e t c / u d e v / s c r i p t s / i s c s i d e v . s h% b " , S Y M L I N K + = " i s c s i / % c / p a r t % n " . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24/29
16/02/14
We now need to create the UNIX SHELL script that will be called when this event is received. Let's first create a separate directory on both Oracle RAC nodes where u d e vscripts can be stored: # m k d i rp/ e t c / u d e v / s c r i p t s Next, create the UNIX shell script / e t c / u d e v / s c r i p t s / i s c s i d e v . s hon both Oracle RAC nodes: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . # ! / b i n / s h #F I L E :/ e t c / u d e v / s c r i p t s / i s c s i d e v . s h B U S = $ { 1 } H O S T = $ { B U S % % : * } [e/ s y s / c l a s s / i s c s i _ h o s t]| |e x i t1 f i l e = " / s y s / c l a s s / i s c s i _ h o s t / h o s t $ { H O S T } / d e v i c e / s e s s i o n * / i s c s i _ s e s s i o n * / t a r g e t n a m e " t a r g e t _ n a m e = $ ( c a t$ { f i l e } ) #T h i si sn o ta no p e n s c s id r i v e i f[z" $ { t a r g e t _ n a m e } "] ;t h e n e x i t1 f i e c h o" $ { t a r g e t _ n a m e # # * . } " . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . After creating the UNIX SHELL script, change it to executable: # c h m o d7 5 5/ e t c / u d e v / s c r i p t s / i s c s i d e v . s h Now that u d e vis configured, restart the iSCSI service on both Oracle RAC nodes: # s e r v i c ei s c s is t o p L o g g i n go u to fs e s s i o n[ s i d :1 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g g i n go u to fs e s s i o n[ s i d :2 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g g i n go u to fs e s s i o n[ s i d :3 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g g i n go u to fs e s s i o n[ s i d :4 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g g i n go u to fs e s s i o n[ s i d :5 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g o u to f[ s i d :1 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l L o g o u to f[ s i d :2 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l L o g o u to f[ s i d :3 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l L o g o u to f[ s i d :4 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l L o g o u to f[ s i d :5 ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l S t o p p i n gi S C S Id a e m o n : # s e r v i c ei s c s is t a r t i s c s i dd e a db u tp i df i l ee x i s t s T u r n i n go f fn e t w o r ks h u t d o w n .S t a r t i n gi S C S Id a e m o n :[ O K ] [
O K ] S e t t i n gu pi S C S It a r g e t s :L o g g i n gi nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 L o g g i n gi nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g g i n gi nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g g i n gi nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g g i n gi nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] L o g i nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l L o g i nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l L o g i nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l L o g i nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l L o g i nt o[ i f a c e :d e f a u l t ,t a r g e t :i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2 ,p o r t a l :1 9 2 . 1 6 8 . 2 . 1 9 5 , 3 2 6 0 ] :s u c c e s s f u l [ O K ] Let's see if our hard work paid off: # l sl/ d e v / i s c s i / * / d e v / i s c s i / a s m 1 : t o t a l0 l r w x r w x r w x1r o o tr o o t9J u l2 91 6 : 0 9p a r t>. . / . . / s d b / d e v / i s c s i / a s m 2 : t o t a l0 l r w x r w x r w x1r o o tr o o t9J u l2 91 6 : 0 9p a r t>. . / . . / s d d / d e v / i s c s i / a s m 3 : t o t a l0 l r w x r w x r w x1r o o tr o o t9J u l2 91 6 : 0 9p a r t>. . / . . / s d c / d e v / i s c s i / a s m 4 : t o t a l0 l r w x r w x r w x1r o o tr o o t9J u l2 91 6 : 0 9p a r t>. . / . . / s d a / d e v / i s c s i / c r s : t o t a l0 l r w x r w x r w x1r o o tr o o t9J u l2 91 6 : 0 9p a r t>. . / . . / s d e The listing above shows that u d e vdid the job it was suppose to do! We now have a consistent set of local device names that can be used to reference the iSCSI targets. For example, we can safely assume that the device name / d e v / i s c s i / a s m 1 / p a r twill always reference the iSCSI
25/29
16/02/14
target i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 . We now have a consistent iSCSI target name to local device name mapping which is described in the following table: iSCSI Target Name to Local Device Name Mappings iSCSI Target Name Local Device Name iqn.2006-01.com.openfiler:racdb.asm1 / d e v / i s c s i / a s m 1 / p a r t iqn.2006-01.com.openfiler:racdb.asm2 / d e v / i s c s i / a s m 2 / p a r t iqn.2006-01.com.openfiler:racdb.asm3 / d e v / i s c s i / a s m 3 / p a r t iqn.2006-01.com.openfiler:racdb.asm4 / d e v / i s c s i / a s m 4 / p a r t iqn.2006-01.com.openfiler:racdb.crs / d e v / i s c s i / c r s / p a r t Create Partitions on iSCSI Volumes We now need to create a single primary partition on each of the iSCSI volumes that spans the entire size of the volume. As mentioned earlier in this article, I will be using Oracle's Cluster File System, Release 2 (OCFS2) to store the two files to be shared for Oracle's Clusterware software. We will then be using Automatic Storage Management (ASM) to create four ASM volumes; two for all physical database files (data/index files, online redo log files, and control files) and two for the Flash Recovery Area (RMAN backups and archived redo log files). The following table lists the five iSCSI volumes and what file systems they will support: Oracle Shared Drive Configuration Size Mount Point ASM Diskgroup Name 2GB /u02 17.8GB ORCL:VOL1 +RACDB_DATA1 17.8GB ORCL:VOL2 +RACDB_DATA1
File System Type iSCSI Target (short) Name OCFS2 ASM ASM ASM ASM crs asm1 asm2 asm3 asm4
File Types Oracle Cluster Registry (OCR) File - (~250 MB) Voting Disk - (~20MB) Oracle Database Files Oracle Database Files
17.8GB ORCL:VOL3 +FLASH_RECOVERY_AREA Oracle Flash Recovery Area 17.8GB ORCL:VOL4 +FLASH_RECOVERY_AREA Oracle Flash Recovery Area
Total 73.2GB As shown in the table above, we will need to create a single Linux primary partition on each of the five iSCSI volumes. The f d i s kcommand is used in Linux for creating (and removing) partitions. For each of the five iSCSI volumes, you can use the default values when creating the primary partition as the default action is to use the entire disk. You can safely ignore any warnings that may indicate the device does not contain a valid DOS partition (or Sun, SGI or OSF disklabel). In this example, I will be running the f d i s kcommand from l i n u x 1to create a single primary partition on each iSCSI target using the local device names created by u d e vin the previous section: / d e v / i s c s i / a s m 1 / p a r t / d e v / i s c s i / a s m 2 / p a r t / d e v / i s c s i / a s m 3 / p a r t / d e v / i s c s i / a s m 4 / p a r t / d e v / i s c s i / c r s / p a r t
Note: Creating the single partition on each of the iSCSI volumes must only be run from one of the nodes in the Oracle RAC cluster! (i.e. l i n u x 1 ) ## f d i s k/ d e v / i s c s i / a s m 1 / p a r t C o m m a n d( mf o rh e l p ) : n C o m m a n da c t i o n e e x t e n d e d p p r i m a r yp a r t i t i o n( 1 4 ) p P a r t i t i o nn u m b e r( 1 4 ) : 1 F i r s tc y l i n d e r( 1 1 6 9 9 2 ,d e f a u l t1 ) : 1 L a s tc y l i n d e ro r+ s i z eo r+ s i z e Mo r+ s i z e K( 1 1 6 9 9 2 ,d e f a u l t1 6 9 9 2 ) : 1 6 9 9 2 C o m m a n d( mf o rh e l p ) : p D i s k/ d e v / i s c s i / a s m 1 / p a r t :1 7 . 8G B ,1 7 8 1 7 4 0 3 3 9 2b y t e s 6 4h e a d s ,3 2s e c t o r s / t r a c k ,1 6 9 9 2c y l i n d e r s U n i t s=c y l i n d e r so f2 0 4 8*5 1 2=1 0 4 8 5 7 6b y t e s D e v i c eB o o t / d e v / i s c s i / a s m 1 / p a r t 1 C o m m a n d( mf o rh e l p ) : w T h ep a r t i t i o nt a b l eh a sb e e na l t e r e d ! C a l l i n gi o c t l ( )t or e r e a dp a r t i t i o nt a b l e . S y n c i n gd i s k s . ## f d i s k/ d e v / i s c s i / a s m 2 / p a r t C o m m a n d( mf o rh e l p ) : n C o m m a n da c t i o n e e x t e n d e d p p r i m a r yp a r t i t i o n( 1 4 ) p S t a r t 1 E n d 1 6 9 9 2 B l o c k s 1 7 3 9 9 7 9 2 I d S y s t e m 8 3 L i n u x
26/29
16/02/14
P a r t i t i o nn u m b e r( 1 4 ) : 1 F i r s tc y l i n d e r( 1 1 6 9 9 2 ,d e f a u l t1 ) : 1 L a s tc y l i n d e ro r+ s i z eo r+ s i z e Mo r+ s i z e K( 1 1 6 9 9 2 ,d e f a u l t1 6 9 9 2 ) : 1 6 9 9 2 C o m m a n d( mf o rh e l p ) : p D i s k/ d e v / i s c s i / a s m 2 / p a r t :1 7 . 8G B ,1 7 8 1 7 4 0 3 3 9 2b y t e s 6 4h e a d s ,3 2s e c t o r s / t r a c k ,1 6 9 9 2c y l i n d e r s U n i t s=c y l i n d e r so f2 0 4 8*5 1 2=1 0 4 8 5 7 6b y t e s D e v i c eB o o t / d e v / i s c s i / a s m 2 / p a r t 1 C o m m a n d( mf o rh e l p ) : w T h ep a r t i t i o nt a b l eh a sb e e na l t e r e d ! C a l l i n gi o c t l ( )t or e r e a dp a r t i t i o nt a b l e . S y n c i n gd i s k s . ## f d i s k/ d e v / i s c s i / a s m 3 / p a r t C o m m a n d( mf o rh e l p ) : n C o m m a n da c t i o n e e x t e n d e d p p r i m a r yp a r t i t i o n( 1 4 ) p P a r t i t i o nn u m b e r( 1 4 ) : 1 F i r s tc y l i n d e r( 1 1 6 9 9 2 ,d e f a u l t1 ) : 1 L a s tc y l i n d e ro r+ s i z eo r+ s i z e Mo r+ s i z e K( 1 1 6 9 9 2 ,d e f a u l t1 6 9 9 2 ) : 1 6 9 9 2 C o m m a n d( mf o rh e l p ) : p D i s k/ d e v / i s c s i / a s m 3 / p a r t :1 7 . 8G B ,1 7 8 1 7 4 0 3 3 9 2b y t e s 6 4h e a d s ,3 2s e c t o r s / t r a c k ,1 6 9 9 2c y l i n d e r s U n i t s=c y l i n d e r so f2 0 4 8*5 1 2=1 0 4 8 5 7 6b y t e s D e v i c eB o o t / d e v / i s c s i / a s m 3 / p a r t 1 C o m m a n d( mf o rh e l p ) : w T h ep a r t i t i o nt a b l eh a sb e e na l t e r e d ! C a l l i n gi o c t l ( )t or e r e a dp a r t i t i o nt a b l e . S y n c i n gd i s k s . ## f d i s k/ d e v / i s c s i / a s m 4 / p a r t C o m m a n d( mf o rh e l p ) : n C o m m a n da c t i o n e e x t e n d e d p p r i m a r yp a r t i t i o n( 1 4 ) p P a r t i t i o nn u m b e r( 1 4 ) : 1 F i r s tc y l i n d e r( 1 1 6 9 6 0 ,d e f a u l t1 ) : 1 L a s tc y l i n d e ro r+ s i z eo r+ s i z e Mo r+ s i z e K( 1 1 6 9 6 0 ,d e f a u l t1 6 9 6 0 ) : 1 6 9 6 0 C o m m a n d( mf o rh e l p ) : p D i s k/ d e v / i s c s i / a s m 4 / p a r t :1 7 . 7G B ,1 7 7 8 3 8 4 8 9 6 0b y t e s 6 4h e a d s ,3 2s e c t o r s / t r a c k ,1 6 9 6 0c y l i n d e r s U n i t s=c y l i n d e r so f2 0 4 8*5 1 2=1 0 4 8 5 7 6b y t e s S t a r t 1 E n d 1 6 9 9 2 B l o c k s 1 7 3 9 9 7 9 2 I d S y s t e m 8 3 L i n u x S t a r t 1 E n d 1 6 9 9 2 B l o c k s 1 7 3 9 9 7 9 2 I d S y s t e m 8 3 L i n u x
27/29
16/02/14
D e v i c eB o o t / d e v / i s c s i / a s m 4 / p a r t 1 C o m m a n d( mf o rh e l p ) : w T h ep a r t i t i o nt a b l eh a sb e e na l t e r e d ! C a l l i n gi o c t l ( )t or e r e a dp a r t i t i o nt a b l e . S y n c i n gd i s k s . ## f d i s k/ d e v / i s c s i / c r s / p a r t C o m m a n d( mf o rh e l p ) : n C o m m a n da c t i o n e e x t e n d e d p p r i m a r yp a r t i t i o n( 1 4 ) p P a r t i t i o nn u m b e r( 1 4 ) : 1 F i r s tc y l i n d e r( 1 1 0 0 9 ,d e f a u l t1 ) : 1 L a s tc y l i n d e ro r+ s i z eo r+ s i z e Mo r+ s i z e K( 1 1 0 0 9 ,d e f a u l t1 0 0 9 ) : 1 0 0 9 C o m m a n d( mf o rh e l p ) : p D i s k/ d e v / i s c s i / c r s / p a r t :2 1 4 7M B ,2 1 4 7 4 8 3 6 4 8b y t e s 6 7h e a d s ,6 2s e c t o r s / t r a c k ,1 0 0 9c y l i n d e r s U n i t s=c y l i n d e r so f4 1 5 4*5 1 2=2 1 2 6 8 4 8b y t e s D e v i c eB o o t / d e v / i s c s i / c r s / p a r t 1 C o m m a n d( mf o rh e l p ) : w T h ep a r t i t i o nt a b l eh a sb e e na l t e r e d ! C a l l i n gi o c t l ( )t or e r e a dp a r t i t i o nt a b l e . S y n c i n gd i s k s . S t a r t 1 E n d 1 0 0 9 B l o c k s 2 0 9 5 6 6 2 I d S y s t e m 8 3 L i n u x S t a r t 1 E n d 1 6 9 6 0 B l o c k s 1 7 3 6 7 0 2 4 I d S y s t e m 8 3 L i n u x
Page 1 Verify New Partitions After creating all required partitions from l i n u x 1 , you should now inform the kernel of the partition changes using the following command as the " r o o t " user account from all remaining nodes in the Oracle RAC cluster ( l i n u x 2 ). Note that the mapping of iSCSI target names discovered from Openfiler and the local SCSI device name will be different on both Oracle RAC nodes. This is not a concern and will not cause any problems since we will not be using the local SCSI device names but rather the local device names created by u d e vin the previous section. From l i n u x 2 , run the following commands: # p a r t p r o b e # f d i s kl D i s k/ d e v / h d a :4 0 . 0G B ,4 0 0 0 0 0 0 0 0 0 0b y t e s 2 5 5h e a d s ,6 3s e c t o r s / t r a c k ,4 8 6 3c y l i n d e r s U n i t s=c y l i n d e r so f1 6 0 6 5*5 1 2=8 2 2 5 2 8 0b y t e s D e v i c eB o o t / d e v / h d a 1 * / d e v / h d a 2 S t a r t 1 1 4 E n d 1 3 4 8 6 3 B l o c k s 1 0 4 3 9 1 3 8 9 5 7 6 2 5 I d S y s t e m 8 3 L i n u x 8 e L i n u xL V M
D i s k/ d e v / s d e :2 1 4 7M B ,2 1 4 7 4 8 3 6 4 8b y t e s 6 7h e a d s ,6 2s e c t o r s / t r a c k ,1 0 0 9c y l i n d e r s U n i t s=c y l i n d e r so f4 1 5 4*5 1 2=2 1 2 6 8 4 8b y t e s D e v i c eB o o t / d e v / s d e 1 S t a r t 1 E n d 1 0 0 9 B l o c k s 2 0 9 5 6 6 2 I d S y s t e m 8 3 L i n u x
D i s k/ d e v / s d b :1 7 . 7G B ,1 7 7 8 3 8 4 8 9 6 0b y t e s 6 4h e a d s ,3 2s e c t o r s / t r a c k ,1 6 9 6 0c y l i n d e r s U n i t s=c y l i n d e r so f2 0 4 8*5 1 2=1 0 4 8 5 7 6b y t e s D e v i c eB o o t / d e v / s d b 1 S t a r t 1 E n d 1 6 9 6 0 B l o c k s 1 7 3 6 7 0 2 4 I d S y s t e m 8 3 L i n u x
D i s k/ d e v / s d a :1 7 . 8G B ,1 7 8 1 7 4 0 3 3 9 2b y t e s 6 4h e a d s ,3 2s e c t o r s / t r a c k ,1 6 9 9 2c y l i n d e r s U n i t s=c y l i n d e r so f2 0 4 8*5 1 2=1 0 4 8 5 7 6b y t e s D e v i c eB o o t / d e v / s d a 1 S t a r t 1 E n d 1 6 9 9 2 B l o c k s 1 7 3 9 9 7 9 2 I d S y s t e m 8 3 L i n u x
28/29
16/02/14
D i s k/ d e v / s d d :1 7 . 8G B ,1 7 8 1 7 4 0 3 3 9 2b y t e s 6 4h e a d s ,3 2s e c t o r s / t r a c k ,1 6 9 9 2c y l i n d e r s U n i t s=c y l i n d e r so f2 0 4 8*5 1 2=1 0 4 8 5 7 6b y t e s D e v i c eB o o t / d e v / s d d 1 S t a r t 1 E n d 1 6 9 9 2 B l o c k s 1 7 3 9 9 7 9 2 I d S y s t e m 8 3 L i n u x
D i s k/ d e v / s d c :1 7 . 8G B ,1 7 8 1 7 4 0 3 3 9 2b y t e s 6 4h e a d s ,3 2s e c t o r s / t r a c k ,1 6 9 9 2c y l i n d e r s U n i t s=c y l i n d e r so f2 0 4 8*5 1 2=1 0 4 8 5 7 6b y t e s D e v i c eB o o t / d e v / s d c 1 S t a r t 1 E n d 1 6 9 9 2 B l o c k s 1 7 3 9 9 7 9 2 I d S y s t e m 8 3 L i n u x
As a final step you should run the following command on both Oracle RAC nodes to verify that u d e vcreated the new symbolic links for each new partition: # ( c d/ d e v / d i s k / b y p a t h ;l sl* o p e n f i l e r *|a w k' { F S = "" ;p r i n t$ 9""$ 1 0""$ 1 1 } ' ) i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 l u n 0>. . / . . / s d b i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 1 l u n 0 p a r t 1>. . / . . / s d b 1 i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2 l u n 0>. . / . . / s d d i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 2 l u n 0 p a r t 1>. . / . . / s d d 1 i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3 l u n 0>. . / . . / s d c i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 3 l u n 0 p a r t 1>. . / . . / s d c 1 i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4 l u n 0>. . / . . / s d a i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . a s m 4 l u n 0 p a r t 1>. . / . . / s d a 1 i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s l u n 0>. . / . . / s d e i p 1 9 2 . 1 6 8 . 2 . 1 9 5 : 3 2 6 0 i s c s i i q n . 2 0 0 6 0 1 . c o m . o p e n f i l e r : r a c d b . c r s l u n 0 p a r t 1>. . / . . / s d e 1 The listing above shows that u d e vdid indeed create new device names for each of the new partitions. We will be using these new device names when configuring the volumes for OCFS2 and ASMlib later in this guide: / d e v / i s c s i / a s m 1 / p a r t 1 / d e v / i s c s i / a s m 2 / p a r t 1 / d e v / i s c s i / a s m 3 / p a r t 1 / d e v / i s c s i / a s m 4 / p a r t 1 / d e v / i s c s i / c r s / p a r t 1 Page 2Page 3
E-mail this page Printer View
ORACLE CLOUD Learn About Oracle Cloud Get a Free Trial Learn About PaaS Learn About SaaS Learn About IaaS
JAVA Learn About Java Download Java for Consumers Download Java for Developers Java Resources for Developers Java Cloud Service Java Magazine
CUSTOMERS AND EVENTS Explore and Read Customer Stories All Oracle Events Oracle OpenWorld JavaOne
COMMUNITIES Blogs Discussion Forums Wikis Oracle ACEs User Groups Social Media Channels
SERVICES AND STORE Log In to My Oracle Support Training and Certification Become a Partner Find a Partner Solution Purchase from the Oracle Store CONTACT AND CHAT Phone: +1.800.633.0738 Global Contacts Oracle Support Partner Support
Subscribe Careers Contact Us Site Maps Legal Notices Terms of Use Privacy Cookie Preferences Oracle Mobile
29/29

Build Your Own Oracle RAC Cluster On Oracle Enterprise Linux and iSCSI

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Build Your Own Oracle RAC Cluster On Oracle Enterprise Linux and iSCSI

Transféré par

Droits d'auteur :

Formats disponibles

16/02/14

CRS Voting Disk

Figure 2: Network Configuration Screen, Node 1 (linux1)

Figure 3: Ethernet Device Screen, eth0 (linux1)

Figure 4: Ethernet Device Screen, eth1 (linux1)

it will need to be removed as shown below: 1 2 7 . 0 . 0 . 1l o c a l h o s t . l o c a l d o m a i nl o c a l h o s t

Figure 12: Create New iSCSI Target : Default Target IQN

Vous aimerez peut-être aussi