<Insert Picture Here>

Oracle Real Application Clusters (RAC) on Oracle VM

Michael Timpanaro-Perrotta, Daniel Dibbets

Oracle RAC Overview

RAC is the cornerstone of Oracles Grid infrastructure
RAC is the enabler of Grid

Higher Availability
RAC systems can be configured to have no single point of failure

Scalability
RAC allows database systems to scale-out rather than having to scale-up

Flexibility
RAC allows users to easily adapt the hardware infrastructure that supports their database systems

Affordability
Can be built using low-cost servers and low-cost storage

Manageability
RAC with EM, allows users to manage a cluster of database servers as if they were single systems.

Oracle RAC Overview

RAC is the cornerstone of Oracles Grid infrastructure RAC is an option to the Oracle Database
Inherits benefits from the manageability, reliability, and security features built into the Oracle Database

Pooling of resources across applications Flexible resource sharing and management Highest possible scalability and availability Can be built using low-cost servers and low-cost storage Works for existing applications with no changes

Oracle RAC Architecture

//
public network

Node1

VIP1 Service Listener instance 1 ASM

Oracle Clusterware

VIP2 Service Listener instance 2 ASM

Oracle Clusterware

Node 2

VIPn Service Listener instance n ASM

Oracle Clusterware

Node n

Operating System

Operating System shared storage

Operating System

Managed by ASM RAW / Block Devices

Redo / Archive logs all instances Database / Control files OCR and Voting Disks

Benefits of RAC on Oracle VM

Improved server consolidation
RAC databases with underutilized CPU resources or peaky CPU utilization can be consolidated with other similar workloads

Rapid provisioning of RAC servers and resources

RAC nodes can be quickly deployed including fully patched and configured system images

Sub-capacity licensing
Oracle VM can be configured using hard partitioning Hard partitioning allows you to license only CPUs used by the partition instead of all CPUs on a physical server

Virtual Cluster
Create RAC environment on a single physical server Perfect environment for test, Q/A, or demo clusters

Oracle VM Single Instance DB Support

Configuration
Oracle Database 10.2.0.3 or higher Guest OS: Oracle Enterprise Linux 4.x, 5,x Red Hat Linux 4.x, 5.x Support for both 32 and 64 bit Oracle VM 2.1 or higher Support for both para-virtualized (PV) and hardware virtualized (HV) modes Use PV drivers in both modes Windows supported with Oracle Database 11.1.0.7 Live migration supported with Oracle Database 11.1.0.7

Oracle VM RAC Database Support

Configuration
Oracle Database 10.2.0.4 32bit Guest OS : Oracle Enterprise Linux 5.1 or higher Requires Oracle VM 2.1.2 Requires para-virtualized mode guests Static mode support only Live Migration, dynamic resource changes (CPU / Memory) currently not supported

Planned : RAC-OVM Template - coming soon 64bit support coming soon 11g support Support for Dynamic resources changes HA Restart and Live Migration support

OVM Storage Configurations

Managed Storage
Provisioned through DOM0 SAN or NAS attached

Dedicated Storage
Directly mounted to guest IP storage only

DOM0
Mgmt & Control Domain NFS Client OCFS Storage Block Devices

1 2

Workload Apps, etc.

Workload Apps, etc.

NFS

NFS

Guest OS HV Guest VM

Guest OS PV Guest VM

iSCSI

Hypercall API

iSCSI

Hypervisor X86/64 Bare-Metal Server

SAN
CPU

Mem

I/O

I/O

Mem

CPU

1. 2. 3.

Enhanced NFS driver Block Device Loop-back driver Block Device driver

Server Virtualization Certification

Why RAC on OVM Certification is needed CRS-RAC is very time and scheduling sensitive
Follow best practices and guidelines to avoid surprises

Validation/Certification - Areas of focus:

Clock drift Scheduling Timeout sensitivity Dynamic changes Image migration Shared I/O service times OS profile and statistics Integration with new APIs CPU, Memory affinity

Sizing and Configuration Guest and Dom0

Size as you would a native server Same traditional questions
How much memory do I need? How many CPUs do I need? How much I/O and network bandwidth should I dedicate?

RAC on OVM Server Configuration

Minimum of 2 Oracle VM servers
OVM (Manager) version is 2.1.2

Ensure enough memory to accommodate DomU components (OS, CRS, and RAC databases) Do not overload Dom0 with unnecessary packages, products and workload

RAC on OVM Server Configuration contd

CPU configuration
Dom0 Keep the default OVM VCPU settings, which is in most cases 1 VCPU assigned to Dom0 for each real CPU/core DomU The total amount of VCPUs assigned to all guest domains on one server should not exceed 2x the amount of real CPUs Over-committing CPUs/cores is still not recommended

Dynamic VCPU and memory changes are not currently supported

RAC on OVM Server Configuration

contd Configure Time synchronization
Configure NTPD to run in Dom0 and DomU Set guests to manage their own system clocks by setting xen.independent_wallclock=1 in /etc/sysctl.conf file. DomU time should not be synchronized with Dom0

Installation / Depoyment methods

Feasable deployment methods are:
Normal installation Supported RAC Cloning Supported Template installation Currently not supported; Oracle will deliver OVM Templates for RAC in the future

RAC on OVM Clusterware Configuration

Set diagwait to 13
Will set the oprocd margin to 10 seconds instead of 500ms Will prevent unneccesary node evictions under high load Is a general recommendation for servers under high load, not specific to Oracle VM Please review Metalink note 559365.1 on how to set this

RAC on Oracle VM Non-prod Deployment

G u e s t D o m a in 1 A S M In s ta n c e +ASM 1 D a ta b a s e 1 In s ta n c e 1

O r a c le C lu s te r W a r e

O r a c le E n te r p r is e L in u x

D a ta b a s e 1

G u e s t D o m a in 2

A S M In s ta n c e +ASM 2

D a ta b a s e 1 In s ta n c e 2

O r a c le C lu s te r W a r e

O r a c le E n t e r p r is e L in u x

D o m a in - 0

RAC on OVM Production Deployment

Guest Domain 1 ASM Instance +ASM1 Database 1 Instance 1 Database 1 Database 1 Instance 2 Guest Domain 1 ASM Instance +ASM2

Oracle ClusterWare Oracle Enterprise Linux

Oracle ClusterWare Oracle Enterprise Linux

Guest Domain 2 ASM Instance +ASM1 Database 2 Instance 1 Database 2 Database 2 Instance 2

Guest Domain 2 ASM Instance +ASM2

Oracle ClusterWare Oracle Enterprise Linux

Oracle ClusterWare Oracle Enterprise Linux

Domain-0

Domain-0

RAC on OVM Disk Configuration

Two or more HBAs need for redundancy
HBAs should be configured as multipath-ed devices in Dom0 All multipathing tools and drivers supported

Only physical disks configured as VBD are supported

File or block based storage configuration okay for single instance

Configure device persistency in Dom0

Vendor multipathing driver or dm and udev in Dom0

Optionally configure physical block devices (in DomU) as ASMLIB disks

RAC on OVM Disk Configuration

# xen config file example for RAC Guest Domain name = vmrac1" memory = 8192" disk = [ 'phy:/dev/mapper/mpath3p1,xvda,w!', 'phy:/dev/mapper/mpath4p1,xvdb,w!', 'phy:/dev/mapper/mpath5p1,xvdc,w!', ] vif = [ 'mac=00:16:3E:00:00:08, bridge=xenbr0', 'mac=00:16:3E:10:A5:96, bridge=xenbr1', ] vfb = ["type=vnc,vncunused=1"] uuid = "3d6f1de4-626c-e02a-42a1-458c9c17e728" bootloader="/usr/bin/pygrub" vcpus=8 on_reboot = 'restart' on_crash = 'restart'

RAC on OVM ASM Disk Configuration

O r a c le V M
G uest 1 G uest 2

A S M In s ta n c e +A SM 1

A S M In s ta n c e +A SM 2

/d e v /x v d a

V B D D r iv e r

V B D D r iv e r

/d e v /x v d a

D o m a in -0 /d e v /s d a

M u ltip a th in g D r iv e r S A N H o s t B u s A d a p te r p o rt-0 p o rt-1

S A N S w itc h 1

S A N S w itc h 2

P ro c e s s o r 0

S A N S to ra g e

P ro c e s s o r 1

L U N -1

RAC on OVM OCFS Disk Configuration

Oracle VM
Guest 1 Guest 2

OCFS filesystem

/dev/xvda

VBD Driver

VBD Driver

/dev/xvda

Domain-0 /dev/sda

Multipathing Driver SAN Host Bus Adapter port-0 port-1

SAN Switch 1

SAN Switch 2

Processor 0

SAN Storage

Processor 1

LUN-1

RAC on OVM Network Configuration

Use dedicated network devices
For each host a minimum of two 1Gb Ethernet NICs is recommended For production RAC environments provide network traffic separation for public and private Extra Ethernet NICs required if network based storage connectivity (iSCSI or NFS) is used

These networks can be shared with multiple RAC VM guests, but network bandwidth and capacity needs to reviewed and correctly sized For complete network redundancy, 4-6 NICs recommended Bond ethx interfaces in Dom0 for HA and bandwidth

RAC on OVM Network Configuration

# xen config file example for RAC Guest Domain name = vmrac1" memory = 8192" disk = [ 'phy:/dev/mapper/mpath3p1,xvda,w!', 'phy:/dev/mapper/mpath4p1,xvdb,w!', 'phy:/dev/mapper/mpath5p1,xvdc,w!', ] vif = [ 'mac=00:16:3E:00:00:08, bridge=xenbr0', 'mac=00:16:3E:10:A5:96, bridge=xenbr1', ] vfb = ["type=vnc,vncunused=1"] uuid = "3d6f1de4-626c-e02a-42a1-458c9c17e728" bootloader="/usr/bin/pygrub" vcpus=8 on_reboot = 'restart' on_crash = 'restart'

RAC on OVM Network Configuration

O r a c le V M
G uest 1
P u b lic n e tw o rk P r iv a te n e tw o r k P u b lic n e tw o rk

G uest 2
P r iv a t e n e tw o rk

e th 0

e th 1

e th 0

e th 1

D o m a in -0

x e n b r0

xen b r1

bond0

bond1

e th 0

e th 1

e th 2

e th 3

P u b lic N e tw o rk S w itc h 1

P u b lic N e tw o r k S w itc h 2

P riv a te N e tw o r k S w itc h 1

P riv a te N e tw o rk S w itc h 2

RAC on OVM NFS based Virtual Cluster

Oracle VM
Guest 1
/u01/oradata Public network Private network filer01.us.oracle.com:/volume01 Public network Private network

Guest 2
/u01/oradata Storage network

Storage network

eth0

eth1

eth2

eth0

eth1

eth2

Domain-0

xenbr0

xenbr1

xenbr2

bond0

bond1

bond2

eth0

eth1

eth2

eth3

eth4

eth5

Public Network Switch 1

Public Network Switch 2

Private Network Switch 1

Private Network Storage Network Switch 2 Switch 1

Storage Network Switch 2

NetApp Filer: filer01.us.oracle.com Exported volume: volume01

Typical Installation issues

Disk Config
Make sure device persistancy is correctly setup with udev in Dom0 Make sure permissions are set persistently and correct using udev in DomU

Network Config
Make sure bridges are configured correctly in Dom0 Ensure order and MAC address bonding to interfaces in the vm.cfg config files Use brctl show to verify Ensure no firewall exists in DomU The default firewall in the Oracle supplied Linux templates blocks RAC traffic. Prevent duplicate MAC addresses on the network

Verifying Network Setup

[root@strab03 ~]# xm list Name Domain-0 OVM_EL5U1_X86_64_PVM_10GB [root@strab03 ~]# brctl show bridge name bridge id xenbr0 8000.feffffffffff ID Mem VCPUs 0 751 4 2 10240 2 STP enabled no State r-----b---Time(s) 7225.2 13.9

xenbr1

8000.feffffffffff

no

xenbr2

8000.feffffffffff

no

xenbr3

8000.feffffffffff

no

interfaces vif2.0 peth0 vif0.0 vif2.1 peth1 vif0.1 vif2.2 peth2 vif0.2 peth3 vif0.3

[root@strab03 ~]#

vif2.2 means interface eth2 running in domain with id 2 so in this case eth2 in domain OVM_EL5U1_X86_64_PVM_10GB

Verifying Network Setup

[root@localhost ~]# service iptables status Table: filter Chain INPUT (policy ACCEPT) num target prot opt source 1 RH-Firewall-1-INPUT all -- 0.0.0.0/0 Chain FORWARD (policy ACCEPT) num target prot opt source 1 RH-Firewall-1-INPUT all -Chain OUTPUT (policy ACCEPT) num target prot opt source Chain RH-Firewall-1-INPUT (2 references) num target prot opt source 1 ACCEPT all -- 0.0.0.0/0 2 ACCEPT icmp -- 0.0.0.0/0 3 ACCEPT esp -- 0.0.0.0/0 4 ACCEPT ah -- 0.0.0.0/0 5 ACCEPT udp -- 0.0.0.0/0 6 ACCEPT udp -- 0.0.0.0/0 7 ACCEPT tcp -- 0.0.0.0/0 8 ACCEPT all -- 0.0.0.0/0 9 ACCEPT tcp -- 0.0.0.0/0 10 REJECT all -- 0.0.0.0/0 [root@localhost ~]# destination 0.0.0.0/0

0.0.0.0/0

destination 0.0.0.0/0

destination

destination 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 224.0.0.251 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0

icmp type 255

udp dpt:5353 udp dpt:631 tcp dpt:631 state RELATED,ESTABLISHED state NEW tcp dpt:22 reject-with icmp-host-prohibited

This default template firewall will block RAC and ClusterwareTrafic. Stop with service iptables stop and disable with chkconfig iptables off

RAC on OVM Best Practices Summary

Each RAC node must be deployed on a separate physical server for production environments Be carefull with over-committing CPUs when running RAC on OVM Configure disk multi-pathing and NIC bonding in Dom0 To sustain required I/O throughput, ensure sufficient CPU resources are allocated to hypervisor

RAC on OVM Templates

Gold Image includes
EL 5.2 update 2 RAC stack - CRS/ASM/Db 11.1.0.7 plus current bundle Configured ASM diskgroup with seed database RAC Cloning to enable/provision additional nodes

Boot script will prompt for:

Hostname IP address Network devices

Requires 6 block devices to be used for ASM and OCR/Vote Deployment script will perform the following:
ssh setup and configured (password-less logins) Update /etc/hosts, define VIP addresses Updates Ifcfg-ethx files chmod/ownership for ASM devices and OCR/Vote

More Information
Oracle RAC on OVM Technical White Paper:
http://www.oracle.com/technology/products/database/clusterware/pdf/ oracle_rac_in_oracle_vm_environments.pdf

Oracle RAC on oracle.com:

http://otn.oracle.com/rac

Oracle Clusterware on oracle.com:

http://otn.oracle.com/clusterware

Oracle ASM on oracle.com:

http://otn.oracle.com/asm