Académique Documents
Professionnel Documents
Culture Documents
sales@mokumsolutions.com
Copyright 2014 Mokum Solutions, Inc. All rights reserved.
Distribution of the Oracle Cloud Cookbook or derivative of the work in any form
is prohibited unless prior permission is obtained from the Copyright holder.
About Mokum Solutions, Inc.
Founded in March 2011, Mokum Solutions, Inc. specializes in the implementation,
delivery and support of Oracle technologies in private and public clouds. Mokum
corporate headquarters are located in San Francisco, CA http://mokumsolutions.com
or call 1 415 252 9164
About the Author
The author of the Oracle Cloud Cookbook is none other than the owner of
Mokum Solutions, Inc., Roddy Rodstein. Roddy is one of the most respected
Oracle Cloud Computing experts, having designed and managed many of the
worlds largest and most complex Oracle private clouds. Before establishing
Mokum in March 2011, Roddy spent three years at Oracle on the Oracle VM
and Oracle Linux team designing and supporting Oracle's largest and most
complex customer environments. Before Oracle, Roddy spent six years at Citrix,
designing and supporting Citrix's largest and most complex customer environments,
Including Oracle's. With Mr. Rodsteins rich background and knowledge, there
can be no better resource for revealing the Oracle Cloud recipe.
Audience
The Oracle Cloud Cookbook is a comprehensive, field tested reference design that
guides you through each step to move to your Oracle software portfolio to an elastic
Oracle cloud using the Oracle VM product line, Oracle Linux, Oracle Engineered
Systems managed by Oracle Enterprise Manager 12c, with total control over Oracle
processor licensing.
http://mokumsolutions.com
Table of Contents
Oracle
Oracle
Oracle
Oracle
Oracle
VM
VM
VM
VM
VM
Change Log
Revision
Change Description
Updated By
Date
1.0
Document Creation
Roddy Rodstein
01/03/14
4 of 14
http://mokumsolutions.com
Network.
Oracle VM facilitates centralized server pool management using an agent-based architecture. The Oracle
VM agent is a python application that is installed by default with Oracle VM Server. Oracle VM Manager
dispatches commands using XML RPC over a dedicated network called the Server Management network
channel using TCP/8899 to each server pool's Master Server agent. Each Master Server agent dispatch
commands to subordinate agent servers using TCP/8899. The Oracle VM agent is also responsible for
propagating the /etc/ocfs2/cluster.conf le to subordinate agent servers. There is only one Master Server in
a server pool at any one point in time. The Master Server is the only server in a server pool to communicate
with Oracle VM Manager.
Note: When Oracle VM Server is installed, the IP address entered during the installation is assigned to the
Server Management network channel.
Once an Oracle VM server pool is created, two cluster conguration les are shared across all nodes in the
server pool that maintain the cluster layout and cluster timeouts congurations. The /etc/ocfs2/cluster.conf
le maintains the cluster layout and the /etc/syscong/o2cb le maintains the cluster timeouts. Both
conguration les are read by the user-space utility congfs. congfs communicates the list of nodes in the
/etc/ocfs2/cluster.conf le to the in-kernel node manager, along with the resource used for the heartbeat to
the in-kernel heartbeat thread.
An Oracle VM server must be online to be a member of an Oracle VM pool/cluster. Once the cluster is
on-line, each pool member starts a process, o2net. The o2net process creates TCP/IP intra-cluster node
communication channels on port 7777 and sends regular keepalive packages to each node in the cluster to
validate if the nodes are alive. The intra-cluster node communication uses the Cluster Heartbeat network
channel. If a pool member loses network connectivity the keepalive connection becomes silent causing the
node to self-fence. The keepalive connection time out value is managed in each nodes /etc/syscong/o2cb
le's O2CB_IDLE_TIMEOUT_MS setting.
Along with the keepalive packages that check for node connectivity, the cluster stack also employs a disk
heartbeat check. o2hb is the process that is responsible for the disk heartbeat component of cluster stack
that actively monitors the status of all pool members. The heartbeat system uses a le on the OCSF2 le
system, that each pool member periodically writes a block to, along with a time stamp. The time stamps are
read by each pool member and are used to check if a pool member is alive or dead. If a pool members
block stops getting updated, the node is considered dead, and self-fences. The disk heartbeat time out value
is managed in each nodes /etc/syscong/o2cb le's O2CB_HEARTBEAT_THRESHOLD setting.
The OCFS2 network and storage heartbeat time out values are managed in each Oracle VM Servers
/etc/syscong/o2cb le. Each pool member must have the same /etc/syscong/o2cb values. The default
timeout values should be tested and tuned to your network and storage infrastructure to provide predicable
failure response.
Tip: If a SAN storage controller fail over takes 120 seconds, and OCFS2 is set to the default value of 60
seconds, Oracle VM Servers will reboot halfway through the controller fail over. The
O2CB_HEARTBEAT_THRESHOLD timeout value must longer then the SAN storage controller fail over
timeout value.
The next example shows the default Oracle VM 3.2.x /etc/syscong/o2cb timeout values.
O2CB_IDLE_TIMEOUT_MS=60000 (60 secs)
O2CB_HEARTBEAT_THRESHOLD=91 (180 secs)
O2CB_RECONNECT_DELAY_MS=2000 (2 secs)
O2CB_KEEPALIVE_DELAY_MS=2000 (2 secs)
The next list explains each Oracle VM 3.2.x o2cb timeout value.
1- O2CB_IDLE_TIMEOUT_MS: Default settings is 60000 = 60 secs
Time in ms before a network connection is considered dead.
2- O2CB_HEARTBEAT_THRESHOLD: Default 91 = 180 secs
The disk heartbeat timeout is the number of two-second iterations before a node is considered dead. The
exact formula used to convert the timeout in seconds to the number of iterations is:
5 of 14
http://mokumsolutions.com
O2CB_HEARTBEAT_THRESHOLD = (((timeout in seconds) / 2) + 1)
For example, to specify a 60 sec timeout, set it to 31. For 120 secs, set it to 61. The default for Oracle VM
3.2.x is 91 (180 secs), the default for Oracle VM 3.1.x is 31 (60 secs).
Note: The O2CB_HEARTBEAT_THRESHOLD should be congured using the Oracle VM Manager GUI. The
max setting for 3.2.x and above in Oracle VM Manager is 300, i.e. O2CB_HEARTBEAT_THRESHOLD = 151
in /etc/syscong/o2cb.
3- O2CB_RECONNECT_DELAY_MS: Default 2000 = 2 secs
Min time in ms between connection attempts
4- O2CB_KEEPALIVE_DELAY_MS: Default 2000 = 2 secs
Max time in ms before a keepalive packet is sent
Note: If reboots are occurring and the root cause has not yet been identied, the following time-out values
may provide a temporary solution.
O2CB_IDLE_TIMEOUT_MS=90000 (90 secs)
O2CB_HEARTBEAT_THRESHOLD=91 (180 secs)
O2CB_RECONNECT_DELAY_MS=4000 (4 secs)
O2CB_KEEPALIVE_DELAY_MS=4000 (4 secs)
A minimum of one Ethernet network interface (NIC) card is required to install Oracle VM, although at least
four 10G or four or more 1G NICs are recommended for fault testing. Trunk ports/802.1Q and/or access
ports with NIC bonding mode 1, 4 and 6 are supported and congured post Oracle VM Server installation
with Oracle VM Manager and/or Oracle Enterprise Manager. Oracle VM supports two NIC ports per
network bond and a total of ve network bonds per Oracle VM Server.
The next Figures shows two dierent Oracle VM networking strategies.
Figure 1 shows a four port 10G 802.1q/LACP trunk port design with two mode 4 bonds.
Figure 2 shows a ten port 1G or 10G access port design with ve mode 1, 4 or 6 bonds.
6 of 14
http://mokumsolutions.com
Tip: I highly recommend the four port 10G 802.1q/LACP trunk port design with two mode 4 bonds. Trunk
ports can have two or more VLANs per port. An access port is limited to one VLAN per port.
The Cluster Heartbeat, Storage and Virtual Machine network channels NIC bond modes (1, 4 and 6) should
be fault tested with various network switch settings to conrm which NIC bonding mode and network
switch setting combination provides predicable failure response. Table 1 shows each of the Oracle VM
network channels, NIC bonding modes, and a variety of network switch options that should be fault tested.
Network
Channel
Description
Network
Type
iLO
(Integrated
Lights Out
Manager)
Bond Modes
Not
applicable
Note: iLo is
iLO ports should be on a
not managed network isolated from the
by Oracle VM server payload networks.
Manager. ilo
is included in
this table for
completeness.
Server
Management
Mode 1, 4 or
6
7 of 14
http://mokumsolutions.com
as well as administrative
ssh access to Oracle VM
Servers, and
http/https/VNC access to
and from Oracle VM
Manager.
special
network
switch
settings.
Mode 4
requires the
network
switch to
support
802.3ad/LACP.
Oracle VM Manager
dispatches commands
using XML RPC over the
Server Management
network using TCP/8899 to
each server pool's master
agent server. Each Master
Server agent server
dispatch commands to
subordinate agent servers
using the Server
Management network with
TCP/8899.
one VLAN.
A trunk port can have two or
more VLANs the port, i.e. the
port can carry trac for
multiple simultaneous VLANs.
Most network switches support
the following standards:
EtherChannel, Port Channels,
Link Aggregation Control
Protocol (LACP) / 802.3ad, etc...
Consult Cisco for network
switch conguration details:
http://www.cisco.com/en/US
/tech/tk389/tk213
/tsd_technology_support_proto...
Class A, B
or C
*The
network
could be a
private
non-routable
network.
Mode 1, 4 or
6
8 of 14
http://mokumsolutions.com
O2CB_IDLE_TIMEOUT_MS
setting.
Live
Migration
Storage
Class A, B
or C *The
network
could be a
private
non-routable
network.
Class A, B,
C or
FC/SAN
Mode 1, 4 or
6
Mode 1, 4 or
6
9 of 14
http://mokumsolutions.com
settings.
Mode 4
requires the
network
switch to
support
802.3ad/LACP.
Virtual
Machine(s)
Class A, B
or C
Mode 1, 4 or
6
Tip: A best practice is use a minimum of three nodes per cluster to ensure quorum and to be able to
generate meaningful network heartbeat (O2CB_IDLE_TIMEOUT_MS timeout) fault tests. A known
limitation with two node clusters and network failures causes the node with the higher node number to
self-fence.
10 of 14
http://mokumsolutions.com
bond modes and network switch congurations can trigger node evictions and unexpected server reboots.
Tip: When fault testing a two node cluster's O2CB_IDLE_TIMEOUT_MS time out value, the node with the
higher node number will reboot when the network fails. A best practice is use a minimum of three nodes
per cluster to ensure quorum and to be able to fault test the O2CB_IDLE_TIMEOUT_MS timeout value.
The next tables shows each of the fault test with the expected failure results. The example is four a four
port 10G 802.1q/LACP trunk port design with two mode 4 bonds. Modify the table to reect your design,
and your fault tests.
1. Disable switch ports using the following fault patterns to text NIC, Bond and Switch failures and
OCFS2 compatibility.
2. Use the following commands to conrm the tests results.
watch cat /proc/net/bonding/bond0
watch cat /proc/net/bonding/bond1
tail -f /var/log/messages
Note: While fault testing, run a HA enabled VM on the tested Oracle VM server to conrm that the VM is
not interrupted during the single port fault test. When both server management bond ports fail at the same
time, the Oracle VM server should fence, the HA enabled VM should restart on a live node.
Complete and document each test to conrm which settings provide the expected failure results.
Use Case
1- Disable the port and
wait +20 secs over the
O2CB_IDLE_TIMEOUT_MS
time out value.
Port(s)
Mode
x
LACP
eth1 Mode
bond0
x
LACP
LACP
eth3 Mode
bond1
x
LACP
LACP
eth0
bond0
Bond Switch
Expected Results
Mode Mode
eth2
bond1
Mode
x
Results
11 of 14
http://mokumsolutions.com
connectivity.
eth1
eth3 Mode
bond0 bond1
x
LACP
eth2
eth3 Mode
bond1 bond1
x
LACP
LACP
1) At the
O2CB_IDLE_TIMEOUT_MS
timeout value the node
self-fences and reboots.
2) After ~60 the VM
restartes on a live node
3) The node successfully
reboots and joins the pool
eth0
eth1 Mode
bond0 bond0
x
12 of 14
http://mokumsolutions.com
quorum disk to nd a suitable O2CB_HEARTBEAT_THRESHOLD o2cb timeout value, SAN and FC Switch
congurations that provides predicable failure response. For example, Oracle VM Servers should be able to
lose one HBA without node evictions. Incompatible o2cb timeout values, SAN and FC Switch congurations
can trigger node evictions and unexpected server reboots.
Note: While fault testing, run a HA enabled VM on the Oracle VM server that will be tested to conrm that
the VM is not interrupted during the single port fault testes. When both HBAs fail at the same time, the
Oracle VM server should fence, the HA enabled VM should restart on a live node.
The next tables shows each of the fault test with the expected failure results. Modify the table to reect
your design, and yourr fault tests.
Use the following commands to conrm the tests results.
watch multipath -ll
tail -f /var/log/messages
In the following example, each host has two HBAs installed with a single path to each fabric. Each fabric
then has two connections (one to each SP) to the array for a total of four paths on the array-side of the
fabric.
Complete and document each test to conrm which settings provide the expected failure results.
Use Case
Port(s)
Bond
Expected Results
Mode
HBA0
HBA1
HBA0
HBA1
After the
ALUA
O2CB_HEARTBEAT_THRESHOLD
Mode
timeout value the node
4
self-fences.
Results
13 of 14
http://mokumsolutions.com
service ovs-agent stop. After 60 seconds, ssh to the Virtual IP address to conrm that the Master Server
agent failed over to a new node.
14 of 14