Ha Vcs 410 101a 2 10 srtpg3 130918134134 Phpapp02

VERITAS Cluster Server for
UNIX, Example Application

Configurations
HA-VCS-410-101A-2-10-SRT (100-002147)
COURSE DEVELOPERS Disclaimer
Bilge Gerrits
The information contained in this publication is subject to
Siobhan Seeger
change without notice. VERITAS Software Corporation
Dawn Walker
makes no warranty of any kind with regard to this guide,
including, but not limited to, the implied warranties of
merchantability and fitness for a particular purpose.
VERITAS Software Corporation shall not be liable for errors
LEAD SUBJECT MATTER
EXPERTS contained herein or for incidental or consequential damages in
Geoff Bergren connection with the furnishing, performance, or use of this
Connie Economou manual.
Dave Rogers
Copyright
Jim Senicka
Pete Toemmes Copyright 2005 VERITAS Software Corporation. All
rights reserved. No part of the contents of this training
material may be reproduced in any form or by any means or
be used for the purposes of training or education without the
TECHNICAL written permission of VERITAS Software Corporation.
CONTRIBUTORS AND
REVIEWERS Trademark Notice
Billie Bachra
Barbara Ceran VERITAS, the VERITAS logo, and VERITAS FirstWatch,
Bob Lucas VERITAS Cluster Server, VERITAS File System, VERITAS
Gene Henriksen Volume Manager, VERITAS NetBackup, and VERITAS
HSM are registered trademarks of VERITAS Software
Corporation. Other product names mentioned herein may be
trademarks and/or registered trademarks of their respective
companies.
VERITAS Cluster Server for UNIX, Example Application
Configurations
April 2005 Release
VERITAS Software Corporation

350 Ellis Street
Mountain View, CA 94043
Phone 6505278000
www.veritas.com
Table of Contents
Course Introduction
VERITAS Cluster Server Curriculum ................................................................ Intro-2
Guide Prerequisites........................................................................................... Intro-3
Guide Objectives............................................................................................... Intro-4
Legend .............................................................................................................. Intro-6
Example 1: Clustering Applications

Application Service Overview.................................................................................. 1-3
VCS Agents for Managing Applications .................................................................. 1-6
The Application Agent ........................................................................................... 1-11
Lab 1: Creating an Application Resource.............................................................. 1-15
Example 2: Clustering Databases

Database Service Example..................................................................................... 2-3
VCS Database Agents ............................................................................................ 2-5
Database Preparation ............................................................................................. 2-9
The Enterprise Agent for Oracle ........................................................................... 2-15
Database Failover Behavior.................................................................................. 2-30
Additional Oracle Agent Functions........................................................................ 2-32
Lab 2: Creating Oracle Resources........................................................................ 2-38
Example 3: Clustering NFS

NFS Service Overview ............................................................................................ 3-3
Preparing NFS for High Availability......................................................................... 3-5
Testing the NFS Service ....................................................................................... 3-12
Configuring an NFS Service Group....................................................................... 3-16
NFS Lock Failover in VCS 4.1 .............................................................................. 3-19
Lab 3: Creating NFS Resources ........................................................................... 3-29
Appendix A: Lab Synopses

Lab 1 Synopsis: Creating an Application Resource ................................................ A-2
Lab 2 Synopsis: Creating Oracle Resources .......................................................... A-6
Lab 3 Synopsis: Creating NFS Resources............................................................ A-16
Appendix B: Lab Details

Lab 1 Details: Creating an Application Resource.................................................... B-3
Lab 2 Details: Creating Oracle Resources............................................................ B-11
Lab 3 Details: Creating NFS Resources ............................................................... B-31
Appendix C: Lab Solutions

Lab 1 Solutions: Creating an Application Resource................................................ C-3
Lab 2 Solutions: Creating Oracle Resources ........................................................ C-11
Lab 3 Solutions: Creating NFS Resources ........................................................... C-35
Appendix D: Design Worksheet: Template
Table of Contents i
Copyright 2005 VERITAS Software Corporation. All rights reserved.
ii VERITAS Cluster Server for UNIX, Example Application Configurations
Introduction
VERITAS Cluster Server Curriculum
Courses
Optional
VERITAS Guide
Cluster Server
Fundamentals
VERITAS Cluster
Server, Example
Application
Configurations
High Availability
Fundamentals
VERITAS Cluster Server Curriculum

The VERITAS Cluster Server curriculum is a collection of courses that are
designed to provide a full range of expertise with VERITAS Cluster Server (VCS)
high availability solutions.
VERITAS Cluster Server for UNIX Fundamentals

This course covers installation and configuration of common VCS configurations,
focusing on two-node clusters running application and database services.
High Availability Fundamentals

This course covers VERITAS Volume Manager and VERITAS Cluster Server
Fundamentals.
VERITAS Cluster Server for UNIX, Example Application

Configurations
This guide provides optional supplementary material for VERITAS Cluster Server
Fundamentals and High Availability Fundamentals. The instructor may opt to
present some of this material in class, depending on time constraints and student
interest.
Intro2 VERITAS Cluster Server for UNIX, Example Application Configurations

Guide Prerequisites
To successfully use the material in this
guide, you should have the following
expertise:
VERITAS Cluster Server fundamentals
VERITAS Volume Manager basic
configuration
System and network device configuration
Guide Prerequisites
This guide assumes that you have mastered the fundamentals of VERITAS Cluster
Server. You should also have an administrator-level understanding of one or more
UNIX platforms, including the ability to configure systems, storage devices, and
networking in multiserver environments. A basic understanding of VERITAS
Volume Manager is also required.
Introduction Intro3
Guide Objectives
After completing the VERITAS Cluster Server for
UNIX, Example Application Configurations
guide, you will be able to:
Configure an Application resource to manage an
application in a VCS environment.
Configure Oracle and Netlsnr resources and set
up detail monitoring of an Oracle database
instance.
Configure NFS and Share resources to make an
NFS service highly available.
Guide Objectives
The VERITAS Cluster Server for UNIX, Example Application Configurations
guide simulates the job tasks you perform to configure common applications in a
VCS environment.

Guide Resources
Participant Guide
Modules
Appendix A: Lab Synopses
Appendix B: Lab Details
Appendix C: Lab Solutions
Appendix D: Design Worksheet Template
Supplements
VCS Simulator: van.veritas.com
Tips & Tricks: www.veritas.com/education
Guide Resources
This guide contains modules with conceptual and configuration information for
each topic. Lab exercises enable you to practice your new skills, and solutions are
provided to guide you through the exercises.
Lab materials are provided in three forms, with increasing levels of detail to suit a
range of student expertise levels.
Appendix A: Lab Synopses has high-level task descriptions and design
worksheets.
Appendix B: Lab Details includes the lab procedures and detailed steps.
Appendix C: Lab Solutions includes the lab procedures and steps with the
corresponding command lines required to perform each step.
Appendix D: Design Worksheet Template provides a blank design
worksheet.
Additional supplements may be used in the classroom or provided to you by your
instructor.
Introduction Intro5
Legend
These are common symbols used in this guide.
Symbol Description
Server, node, or cluster system (terms
used interchangeably)
Server or cluster system that has faulted
Storage
Application service
Cluster interconnect
Wide area network (WAN) cloud

Symbol Description
Client systems on a network
VCS service group
Offline service group
VCS resource
Introduction Intro7
Example 1
Clustering Applications
Lesson Topics and Objectives
Topic After completing this lesson, you

will be able to:
Application Service Describe how application services
Overview work in an HA environment.
VCS Agents for Managing Describe how agents manage
Applications applications in a VCS environment.
The Application Agent Configure the Application agent.
Introduction
Overview
This module describes how agents manage resources in a VCS cluster. In this
module you are also introduced to the VCS bundled Application agent.
Importance
Understanding how agents work helps you to determine the best way to manage
your application service in a high availability environment.
Outline of Topics
Application Service Overview
VCS Agents for Managing Applications
The Application Agent
12 VERITAS Cluster Server for UNIX, Example Application Configurations

1
IP Address
Network
Network NIC
Daemons
File System Processes
Volume
Application
Application
S1 Disk Group
S2
Storage
Storage

Recall that application services consist of three basic building blocks or
components:
Applicationthe processes or daemons that handle client requests
Storagedata storage resources
Networkresources that enable clients to access the service from the network
In order to make an application service highly available using VCS, you configure
the applicable resources corresponding to the objects that are logically contained
within these components.
Lessons in the VERITAS Cluster Server Fundamentals guide focused on
configuring the storage and network components in VCS. This module
concentrates on the VCS agents you can use to manage the application component.
Note: Before bringing VCS into the environment, ensure that all components are
properly configured, as described in the Preparing Services for High Availability
lesson in the VERITAS Cluster Server Fundamentals Participant Guide.
Example 1 Clustering Applications 13

Application Requirements
In order to control an application using a VCS
agent, the application must:
Have a defined start and stop procedure.
Provide the ability to clean up after operational failures.
Provide the ability to monitor the health of the application
using an automated method.
Provide the ability to store its data to external shared
disks.
Be crash-tolerant; that is, it must be capable of being
restarted to a known state after sudden failures.
Be able to run on all servers designated as potential
hosts, and must have:
No license issues tied to individual servers
No hostname dependencies
Application Requirements
The most important requirements for an application to run in a cluster are crash
tolerance and host independence. This means that the application should be able to
recover after a crash to a known state, in a predictable and reasonable time, on two
or more hosts.
Most commercial applications today satisfy this requirement. More specifically, an
application is considered well-behaved and can be controlled using a VCS agent if
the application meets the following criteria:
There must be a defined start and stop procedure that can be automated.
If the application fails, it must be possible to clean up anything left behind,
such as defunct processes, reserved memory segments, and so on.
It should be possible to check the status of the application using an automated
method. Monitoring an application can be performed in many ways. In general,
the closer a test emulates what the users of the application do, the better the test
is in discovering problems. However, this may impact the performance of the
application. VCS may also detect problems that you want administrator
intervention to solve, such as running out of tablespace in a database.
Therefore, the monitoring should be carefully balanced to ensure that the
application is up while minimizing the overhead and limiting VCS to
appropriate monitoring actions.

If the application has any data that is updated during operation, the data should
be stored on an external shared disk. This also means that the application
should not keep any data in memory. Because the application is started on
another host after operational failures, any data that other hosts cannot reach is
lost. This precludes the use of any NVRAM accelerator boards or other disk-
1
caching mechanisms contained in a local host. Disk caching for performance
should only be performed on the external array.
The application must be able to restart to a known state without manual
intervention after operational failures, such as a system failure. For example,
an Oracle database guarantees this by acknowledging the clients only after
writing the changes to a log file on the shared disk (the redo logs) when a
commit changes request is received. Anything that has not been committed or
acknowledged is the responsibility of the client and is kept there. So when the
system fails, the database can be brought up on another system because the
other system has access to the redo logs and can determine what has been
committed and written to disk. The database can be brought to the exact point
of time of the crash by applying these redo logs.
The application must be host-independent. Some applications are licensed to
the host name, which means that the application does not start on another host
unless the host name is changed. A highly available application should not
have any features that tie it to a specific system in the cluster.

You can purchase Enterprise agents for popular
applications. Examples include:
DB2
Oracle, DB2, Informix, Sybase
Oracle
Sun ONE, Apache, SAP Sun ONE
You can also manage some applications using bundled
agents:
Application Process
Process Application
Some software vendors provide VERITAS-Enabled agents
for their applications, such as:
MySQL, NCR, and webMethods
Customized agents can be developed in-house or by
VERITAS Consulting Services.

Agents are programscombinations of binaries, libraries, scriptsthat function
as the intermediary between resources and the VCS engine. Every resource type
has an agent that interacts with all resources of that type on behalf of VCS.
You have several options for selecting an agent to manage an application.
You can purchase VERITAS Enterprise agent packages as add-ons to the basic
VERITAS Cluster Server product. Some advantages of using Enterprise agents
are that they are:
Easy to install and configure
Fully tested and supported for the documented versions of the operating
system and application
Enabled for detail monitoring (most agents) to perform additional
verification of the application status
You can use the Application or Process agents that are bundled with the base
VCS product. Depending on your application and requirements, you may need
to write custom code or scripts when using the Application agent. For example,
if you want to change the way that the agent monitors your application, you
can create your own script and configure the agent to use that script.
VERITAS has created partnerships with many application vendors to enable
them to create agents for managing their software.
You can modify an existing agent or create a new agent if you have specialized
requirements for managing your application in a high availability environment.
VERITAS provides separate documentation and training to enable you to
develop VCS agents.

Key Properties of VCS Agents
Only one agent daemon runs on a system for each
configured resource type.
1
An agent runs a single operation on a resource at one
time.
Agents are multithreaded so that operations can be
carried out in parallel on multiple resources of the
same type.
A resource cannot be managed without an agent.
If there are no resources of a particular type
anywhere in the cluster, the agent for that type is not
started.
Key Properties of VCS Agents

Key properties of VCS agents include:
Only one agent process runs on a system for each configured resource type.
That agent manages all resources of that type on that system.
An agent runs a single operation on a resource at one time. This ensures that
agents on different systems cannot be affecting a resource simultaneously on
different systems.
Agents are multithreaded so that operations can be carried out in parallel on
multiple resources of the same type. For example, the IP agent can be bringing
the IP1 resource online at the same time it is monitoring the IP2 resource.
A resource cannot be managed without an agent.
If there are no resources of a particular type anywhere in the cluster, the agent
for that type is not started.
VCS agents are located within subdirectories of the VCS bin directory,
/opt/VRTSvcs/bin.

VCS Agent Functions
To manage an application, the VCS agent must be
able to:
Start a specified program.
Stop the specified program.
Monitor the program.
Some agents have multilevel monitoring functions
that can be configured.
Basic-level monitoring may only check the process
table for applicable processes.
Second-level monitoring, if enabled, can execute
code, simulating client access to the application.
Clean up after a fault.
VCS Agent Functions

Most VCS agents perform this common set of tasks:
Bring resources online when requested by the VCS engine.
Take resources offline upon request.
Periodically monitor resources and send status information to the VCS engine.
Clean up if a resource fails to go offline properly.
Agents also:
Restart resources when they fault (depending on the resource configuration).
Send a message to the VCS engine and the agent log when errors are detected.
Some agents support multiple levels of monitoring. For example, most database
agents can be configured to perform two levels of monitoring:
Basic monitoring: Scans the process table for database processes
Detail monitoring: Performs a transaction on a test table in the database
Agents may also perform other functions, depending on the type of resource they
manage. The documentation for each agent describes which functions each one
supports. The VERITAS Cluster Server Agent Developers Guide provides a
comprehensive description of each supported agent function.
For information about resource states and transitions, see the Job Aid appendix
in this Participant Guide.

The VCS Agent Framework
Agents consist of an agent binary file plus a
collection of scripts or C++ functions that
1
correspond to each agent function.
The agent functions are referred to as entry
points.
Most agents include the online, offline, monitor,
and clean entry points.
Entry points are often implemented as Perl or
shell scripts, in which case they can easily be
customized.
Agents can also be implemented as compiled
executable files.
The VCS Agent Framework

The modular design of VCS provides an agent framework to enable additional
agents to be added without affecting the core engine functions. Agents manage
resources by calling the appropriate function or entry point, and HAD manages the
agents.
When an event occurs and HAD determines that an action must be taken, it calls
the appropriate agent. The agent then invokes the script or function for the
applicable entry point, passing a set of input parameters as arguments to perform
the resource management task requested. The scripts called by the agent are
referred to as entry points into the agent.
Entry points that are common to most agents are:
Online: Starts or creates the resource
Offline: Stops or removes a resource
Monitor: Determines resource status
Clean: Forcibly takes a resource offline
Entry points are determined by the type of resource that an agent is managing. For
example, a NIC resource is persistent, so it does not have online and offline entry
points. Online-only resources, such as NFS, do not have an offline entry point.
However, all agents are required to have monitor and clean entry points.
In most cases, entry points are executed as well-known UNIX shell commands and
are easily customizable for developers experienced with creating shell scripts.

How Agents Work
vxrecover -g AppDG bs AppVol

In-memory configuration
types.cf
types.cf App
Volume
type Volume (
static str ArgList[] =
{ Volume, DiskGroup }
. . . Volume Agent Online
) online
Entry Point
offline
$1 AppVolume
monitor
main.cf
main.cf $2 AppVol
clean
$3 AppDG
Volume AppVolume (
Volume = AppVol
Online AppVolume
DiskGroup = AppDG
)
HAD
How Agents Work

When the VCS engine requires an action to be taken for a resourceas the result
of an administrative command or in response to an event, such as a faultHAD
calls an agent with a set of parameters. The agent takes a set of parameters as input
from HAD. These parameters are the values specified by resource attributes in the
main.cf file. The order in which they are passed is specified by the ArgList
attribute of the corresponding resource type defined in types.cf.
The agent calls the entry point with the name of the resource followed by
argumentsvalues of the resource attributes (main.cf)in the order specified
in the ArgList attribute for the resource type (types.cf).
Note: The diagram shows a conceptual view of the communication and
relationships among VCS components. The agent does not read the configuration
files directly. The VCS engine has the configuration in memory and passes the
configuration information to the agent when it starts or a new resource is created.
The agent then stores the configuration information in memory.

The Application agent:
1
Simplifies managing applications with multiple
processes and enables detailed monitoring.
Requires no additional licensing; is bundled with
VCS.
Is easily customized by creating scripts that the
agent uses within its entry points.
Manages applications using these entry points:
Online Brings an application online using StartProgram
Offline Takes an application offline using StopProgram
Monitor Monitors the status of the application
Clean Takes the application offline using CleanProgram
or kills all the processes specified for the
application

If you have an application that does not have an available Enterprise agent, you
may be able to use the Application agent to make the application highly available.
The Application agent can be used to control more complex applications and can
be easily customized. Whereas the Process agent can be used to monitor a single
process, the Application agent can manage applications consisting of multiple
processes.
Entry Points
Online: Runs StartProgram with the specified parameters in the specified user
context
Offline: Runs StopProgram with the specified parameters in the specified user
context
Monitor: If no MonitorProgram is specified, verifies that all processes
specified in PidFiles and MonitorProcesses are running
If MonitorProgram is defined, the agent executes the MonitorProgram.
Clean: Kills all the processes specified in PidFiles or in MonitorProcesses
If CleanProgram is defined, the agent executes the CleanProgram.

Application Resource Definition
Required Attributes:
StartProgram Name of executable to start application
StopProgram Name of executable to stop application
Plus one or more of the following:
MonitorProgram Name of executable to monitor application
MonitorProcesses List of processes to be monitored
PidFiles List of files that contain the process ID of
the processes to be monitored
Optional Attributes:
CleanProgram Executable file that forcibly kills the
application and performs cleanup tasks
User UNIX account used as the execute
environment for entry points
The default is root.
Application Resource Definition

The required attributes listed in the slide must be defined to enable the Application
agent to manage a resource.
The optional attributes, CleanProgram and User, can be configured to provide
additional control of the application.

Application Resource Configuration
Configuration prerequisites:
The application should have its own start and stop
1
programs.
It must be possible to monitor the application by either
running a program that returns 0 for failure and 1 for
success or by checking a list of processes.
Sample configuration:
Application samba_app (
StartProgram = /usr/sbin/samba start
StopProgram = /usr/sbin/samba stop
PidFiles = { /var/lock/samba/smbd.pid }
MonitorProcesses = { smbd nmbd }
)
Application Resource Configuration

The example in the slide shows how an Application resource is configured to make
the Samba file sharing application highly available.
Note: The MonitorProcess values must match the output displayed by the ps
command exactly. For example, if the processes are displayed with full path
names, you must include the full path name when specifying the processes to
monitor.

Lesson Summary
Key Points
VCS provides a variety of agents to manage all
types of applications.
The Application agent is easily configurable for
basic management of an application.
Reference Materials
VERITAS Cluster Server Bundled Agents
Reference Guide
VERITAS Cluster Server Agent Developer's Guide
VERITAS Cluster Server Agent Development
Course
VERITAS Cluster Server Fundamentals Course
http://www.veritas.com
Summary
This lesson described how agents manage resources in a VCS cluster. In this lesson
you are also introduced to the VCS bundled Application agent.
Next Steps
Now that you understand how agents manage applications, you can configure
other common types of high availability services.
Additional Resources
VERITAS Cluster Server Bundled Agents Reference Guide
This guide describes each bundled agent in detail.
VERITAS Cluster Server Agent Developers Guide
This guide provides detailed information about the agent framework and
creating or customizing agents.
VERITAS Cluster Server Agent Development Course
This course provides detailed information about the agent framework and
creating or customizing agents. The course includes labs to customize existing
agents and create new agents.
VERITAS Cluster Server Fundamentals Course
This course provides detailed information about configuring VCS service
groups and resources.
http://www.veritas.com
A list of agents available from VERITAS and partners is available by selecting
Products>Cluster Server>Agents and Options link.

Lab 1: Creating an Application Resource
nameSG1
nameSG1
1
nameApp1
Delete
Delete the
the Process
Process name
nameIP1
resource
resource and
and add
add Mount1
an
an Application
Application
resource.
resource.
nameVol1 nameProxy1
Appendix
AppendixA:
A:Lab
LabSynopses
Synopses
nameDG1 Appendix
AppendixB:
B:Lab
LabDetails
Details
Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
See
Seethe
thenext
nextslide
slidefor
forclassroom
classroomvalues.
values.

Labs and solutions for this lesson are located on the following pages.
Appendix A provides brief lab instructions for experienced students.
Lab 1 Synopsis: Creating an Application Resource, page A-2
Appendix B provides step-by-step lab instructions.
Lab 1 Details: Creating an Application Resource, page B-3
Appendix C provides complete lab instructions and solutions.
Lab 1 Solutions: Creating an Application Resource, page C-3
Goal
The purpose of this lab is to configure the Application agent to manage a service
with multiple processes.
Prerequisites
Obtain any classroom-specific values needed for your classroom lab environment
and record these values in your design worksheet included with the lab exercise
instructions.
Results
Each students service can be started, monitored, and stopped on each cluster
system.

Example 2
Clustering Databases

will be able to:
Database Service Example Describe common database service
examples.
VCS Database Agents Describe VCS agents for managing
databases.
Database Preparation Prepare a database for use in a VCS
environment.
The Enterprise Agent for Install and configure the VCS Oracle
Oracle agent.
Database Failover Configure database failover behavior.
Behavior
Additional Oracle Agent Implement additional functionality
Functions provided by the Oracle agent.
Introduction
Overview
This module describes how to manage database services in a VCS environment.
An Oracle service group is used as an example showing specific agents and
resource types that are provided with VCS enterprise agent for Oracle.
Importance
Database services are a key component of most high availability environments.
Understanding how you can use VCS to manage your database software is critical
to implementing a complete high availability solution.
Outline of Topics
Database Service Example
VCS Database Agents
Database Preparation
The Enterprise Agent for Oracle
Database Failover Behavior
Additional Oracle Agent Functions

Generic Database Service Example
Database/
Database/ Application
Listener
Listener Component
File
File System
System IP
IP Address
Address
/db
/db 10.10.21.54
10.10.21.54
Contains
Contains data
data
2
and
and log
log files
files
Storage Network
Network Port
Port
Component NIC_device
NIC_device
VxVM
VxVM Volume
Volume Network
dbvol
dbvol Component
VxVM
VxVM Disk
Disk
Group
Group dbdg
dbdg
Database Service Example

As with most application services, database services generally include three basic
components:
Applicationthe processes providing the service
The diagram shows a simple example of a database service.
Example 2 Clustering Databases 23

Corresponding Database Service Group
Database/Listener
Database/Listener DB/Lsnr
/db
/db 10.10.21.54
10.10.21.54
Mount IP
dbvol
dbvol NIC_device
NIC_device
Volume NIC
dbdg
dbdg
Disk
Group
VCS manages database services in the same way it manages any other application
service. Resources are contained within the service group for each storage and
network and dependencies are defined to ensure that VCS brings the supporting
resources online before attempting to start the database.
The type of resource you use to manage the database depends on your database
software and your specific management requirements.

Agents for Managing Databases
You can purchase Enterprise agents for:
Oracle
DB2 Oracle Informix MySQL
Informix
Sybase DB2 Sybase
2
MySQL
Note:
Note: Not
Not all
all agents are supported
supported on all platforms.
platforms.
You can also manage some databases using the

bundled Application agent or customized
agents.
VCS Database Agents

You have several options for selecting an agent to manage the database
application.
You can purchase VERITAS enterprise agent packages as add-ons to the basic
VERITAS Cluster Server product. The advantages of using an enterprise agent
are that it is:
Easy to install and configure
Fully tested and supported for the documented versions of the operating
system and database application
Enterprise agents are the primary focus of this lesson.
You can use the Application agent that is bundled with the base VCS product.
Depending on your database and your requirements, you may need to write
custom code or scripts. For example, if you want to change the way the agent
monitors your database, you modify the monitor entry point script.
You can modify an existing agent or create a new agent if you have specialized
requirements for managing your database in a high availability environment.
VERITAS provides separate documentation and training to enable you to
develop VCS agents.

Agent Functions
To manage a database, the VCS agent must be
able to:
Start a specified database instance.
Stop the specified instance.
Monitor the instance.
Basic-level monitoring may check the process
table for applicable database processes.
Detail monitoring may write to a test table in the
specified database.
Clean up after a database fault to prevent possible
concurrency violations to enable failover.
Agent Functions
All agents perform the same basic functionsstart, stop, monitor, clean. The
enterprise agents for databases each perform similar functions.
The agent starts an instance using the database-specific utility (such as sqlplus
for Oracle), usually with a startup profile file name. The instance is likewise
stopped with the database shutdown utility.
Two levels of monitoring can be configured:
Basic monitoring: Scans the process table for database daemon processes
Detail monitoring: Performs a transaction on a test table in the database
Primary Monitoring
The instance can be monitored by scanning the process table for the process IDs
(PIDs) for critical database processes. The processes monitored vary by database.
For example, the Oracle agent monitors the ora_smon, ora_dbw, ora_pmon,
and ora_lgwr processes.

Detail Monitoring
Agents require this information to access and
write to the table:
A database user account with update privileges
The password for the database user account
The name of the database table
2
The name of the file containing the SQL
statements used to write to the table
Detail Monitoring
When detail monitoring is enabled, the monitor script attempts to write to a test
table in a specified database. If this script fails and failover is enabled, the entire
service group fails over to the next available system.
Writing to the database ensures that failures, such as full logs, are detected.
Reading from the database does not detect this type of error.
In order to write to a database table, you must create the test table in the database
that you are monitoring. Then, provide the agent with the access and configuration
information needed to write to the table:
The database user account with update privileges
The password for the database user account
The name of the database table
SQL statements used to write to the table
Note: Detail monitoring may detect errors that cannot be handled by failover or
restarts, such as a database running out of tablespace. Some agents, such as the
Enterprise Agent for Oracle, provide optional methods of handling these types of
errors. These additional error-handling methods are described later in this lesson.

Monitoring Listener Processes
Some databases have listeners built-in and do not require
additional VCS agents for monitoring.
The Enterprise agent for Oracle includes an agent to
monitor the listener process, named Netlsnr, which
provides two levels of monitoring:
Primary: Scans the process table for the tnslsnr
process
Secondary: Uses lsnrctl to test the listener
Oracle
Server
Net8 Data
User ONS Listener
Client Protocol
Server
Monitoring Listener Processes

Some database products, such as DB2, Sybase, and Informix, have the database
network connection mechanism built into the database processes and do not
require additional agents.
The Enterprise Agent for Oracle product contains an additional agent for
managing the Oracle Net Services (ONS) listener process for Oracle, which
controls client network connections to Oracle databases.
The Netlsnr agent starts and stops the Oracle Net Services listener process using
the lsnrctl command. Two levels of monitoring can be configured:
Primary monitoring: Scans the process table for the tnslsnr process
Secondary monitoring: Uses the lsnrctl command to test the listener
process

Verifying Software Compatibility
Use the latest database agent available for the VCS
version in use.
Verify that the agent supports the version of the
database used in the cluster by checking the agent
release notes.
Examples:
2
DB2 on AIX
Oracle on
on Solaris
DB2 Agent 4.0
Oracle Agent 4.1
VCS 4.0
VCS
VCS 4.1
4.1 AIX 5.2
Solaris 8,
8, 9, 10
10 DB2 ESE 7.2, 8.1
Oracle 8.0.x, 8i (8.1.x), 9i,
10g
Database Preparation
Verifying Software Compatibility
Ensure that each system that can host the database instance meets the operating
system and database requirements specified in the agent documentation.
Also, ensure that each system has adequate resources, such as shared memory.
Each system in the SystemList of the service group must be properly configured as
a database server.

Database Program (Executable) Files
The database program files can be configured either
on the local disks or on shared storage.
Local disk:
Provides redundancy; offers additional protection in
case the binaries are damaged
The database must be installed and configured
identically on each system that can run the database
service group.
Shared storage:
All systems must have the same mount point directory
for the database home directory.
Simplifies configuration, especially if there are many
systems that can run the same database instance
Database Program (Executable) Files

The database binaries can be configured either on the local disks or on the shared
storage. Select the location based on your high availability requirements.
Configuring the binaries on the local disks may enable you to minimize
downtime due to maintenance. You can update the database software on one
system while the database is running on another.
Installing binaries on a shared storage disk simplifies configuration, especially
if there are many systems that can run the same database instance and
participate in the service group. If you have many systems, applying updates
on each system can be a significant burden.
Note: Some databases, such as Oracle, install updates in a new directory. This
directory can be on shared storage, which provides a way to use the advantages
of updating one system while the database is running on the other. If binaries
are storage on local disks, you have to update each individual system.

Data Files
If using file system-based data files, the file systems
that contain these files must be located on shared
disks.
Create the same file system mount point on each
system in the cluster.
If using raw devices, such as Volume Manager
volumes, set the permissions for the volumes to be
2
owned permanently by the database. For example:
vxedit g dgname set group=dba user=dba mode=660 vol
Note: You cannot change permission only in UNIX.
VxVM sets volume permissions upon import.
Data Files
You must locate the data files on shared storage so that each failover target system
can access them. Use these guidelines for locating data files in a VCS
environment:
If using file system-based data files, the file systems that contain these files
must be located on shared disks. Create the same file system mount point on
each system in the cluster.
If using raw devices, such as Volume Manager volumes, set the permissions for
the volumes to be owned permanently by the database account. For example:
vxedit g dgname set group=dba user=dba mode=660
volume_name
Note: Volume Manager overrides UNIX permissions. It is not sufficient to
change permissions in UNIX because Volume Manager sets volume
permissions when importing a volume (with default owner is root). You must
also change the Volume Manager permissions, as shown in the slide.

Database-Related System Files
The following database-related system files must be
consistent on all systems listed in the SystemList
attribute for the service group:
Database user accounts
Passwords for database user accounts
Database group entries
Listener entries
Database shared memory settings
The database service group (virtual) IP address
Solaris AIX HP-UX Linux
Database-Related System Files

Although it is not necessary for the system files listed on the slide to be exactly the
same on all systems that participate in the service group, you must ensure that any
database-related configurations are the same or consistent on all of the systems.
Note: Changes to /etc/system require a reboot to take effect.
Solaris
/etc/passwd
/etc/shadow
/etc/group
/etc/services
/etc/system
/etc/hosts
AIX
/etc/passwd
/etc/security/passwd
/etc/group
/etc/services
/usr/samples/kernel/vmtune
/etc/hosts

HP-UX
/etc/passwd
/etc/group
/etc/services
/stand/system
/etc/hosts
Linux
/etc/passwd
2
/etc/shadow
/etc/group
/etc/services
Shared memory settings:
For drivers built into the kernel, append parameters to the kernel command
line using the boot loader.
For kernel modules, use /etc/modules.conf.
For tunable parameters, use sysctl and /etc/sysctl.conf.
/etc/hosts

Network Configuration
For listener service failover, include at least one IP
resource in the service group for the database.
Use the host name for this service group IP address
for the host field in the listener configuration file.
LISTENER=
(DESCRIPTION=
listener.ora
listener.ora
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=salesDB)(PORT=1521)
)
(ADDRESS=(PROTOCOL=ipc)(KEY=extproc))))
. . .
/etc/hosts IP
/etc/hosts
. . .
10.10.21.54 salesDB
. . . 10.10.21.54
Network Configuration
Each database service group requires at least one IP address for client connections.
This IP address should fail over together with the database in case of any major
faults.
Therefore, you need to use an IP resource (or an IPMultiNIC resource) and
configure the host name of the service group IP address in the database. The clients
should connect to the host name corresponding to this virtual IP address and not to
the local host names of the servers.

Installing Agents
Use the VERITAS VPI or the platform-specific
installation utility to install the VRTSvcs**
package:
Solaris: pkgadd
AIX: installp
HP-UX: swinstall
Linux: rpm
2
Install the agents on each system in the cluster
that can run the database service group.
Sample
SampleOracle
Oracle
Agent
AgentGuides
Guides
The Enterprise Agent for Oracle

Installing Agents
The enterprise agents are provided as a standard software package that can be
installed using the VERITAS Common Product Installation Utility or using the
platform-specific standard installation utility.
Install the agent on each system using this basic installation procedure:
1 Log on as root.
2 Mount the installation CD.
3 Change to the directory containing the agent software packages.
4 Unzip and extract the files from the tar archive.
5 Type the platform-specific command to install the agent.
For example, to install the Oracle enterprise agent, type:
Solaris
pkgadd d . VRTSvcsor
AIX
installp -ac -d /cdrom/VRTSvcsor.rte.bff VRTSvcsor.rte
HP-UX
swinstall -s /cdrom/oracle_agent/pkgs VRTSvcsor
Linux
rpm -ihv VRTSvcsor-2.0-Linux.i386.rpm

Adding Resource Type Definitions
You can use an online or offline method to add
resource type definitions.
To add types while VCS is running, use Cluster
Manager. Select File>Import Types.
To add types offline:
1. Copy the databaseTypes.cf file to the
/etc/VRTSvcs/conf/config directory.
2. Use the offline configuration process described
earlier in the VERITAS Cluster Server Fundamentals
course to add this line to main.cf and restart the
cluster using the modified configuration files.
include databaseTypes.cf
Adding Resource Type Definitions

Before you can create database resources, you must add the resource type
definitions to the VCS configuration.
You can add resource types while VCS is running or take VCS offline and modify
the configuration files.
Adding Resource Types to a Running Configuration

These methods are provided for adding resource types while VCS is running:
Using Cluster Manager:
1 Select File>Import Types.
2 Browse to select the file containing the type definition.
Use hatype to add an include statement to the main.cf file that references
the databasetypes.cf file. You can also use hatype commands to add
and modify resource type definitions in types.cf.
Create and run a command file containing hatype commands.
Adding Resource Types to an Offline Configuration

A simple method for adding resource types requires you to stop VCS, edit the
main.cf or types.cf file, and restart VCS. This is described in detail in the
Offline Configuration of Service Groups lesson of the VERITAS Cluster Server
Fundamentals course.

High Availability Database Configuration
Overview
Binaries
Binaries on
on Local
Local Storage
Prepare
PrepareOS.
OS. Y
Install
Installdatabase.
database. Y More
More
Systems?
Systems?
Configure
Configuredatabase.
database. N Add
Addresource
resourcetypes.
types.
2
Test
Testdatabase.
database. Add
Addservice
servicegroup.
group.
Stop
Stopresources.
resources. Add/test
Add/testresources.
resources.
Install
Installagent.
agent. Test
Testservice
servicegroup.
group.
High Availability Database Configuration Overview

Before you begin configuring VCS to manage your database services, prepare your
environment, including equipment, software, and staff.
Coordinate with the database and systems administrators responsible for the
database servers you are adding to the cluster environment. After database services
are placed under control of VCS, the databases cannot be managed separately,
outside of VCS.
As with any high availability service, configure and test each component on the
startup system and each failover system before you perform any VCS
configuration of the service.
This procedure applies to configurations where the database binaries are installed
locally on each system, which reflects the cluster design decisions assumed for this
course. Examples of other configurations are shown later in the lesson. The High
Availability Design Using VERITAS Cluster Server course provides a complete
discussion of design considerations for VCS environments.

Configuration Example: Oracle Service Group
Netlsnr
IP Oracle
Mount
NIC
Volume
Disk
Group
Oracle Service Group Configuration Example

A service group example for an Oracle database instance contains these types of
resources:
Netlsnr: Monitors the listener process
Oracle: Monitors the database and processes
Mount: Monitors one or more file systems containing:
Oracle application files (if installed on shared storage)
Oracle data files
Oracle log files
Volume: Monitors one or more volumes for the file systems
DiskGroup: Monitors one or more disk groups for the volumes
IP: Monitors the service group IP for the listener process
NIC: Monitors one or more network interface cards for remote client
connection
The example shown on the slide assumes that the Oracle binaries are located on
local storage. The data files are located on a file system (rather than raw volumes).
The clients access Oracle services using the service group IP address defined by
the IP resource.

Configuring an Oracle Resource
Starts and stops an
Oracle instance
Resource Definition Sample Value
Monitors the process
Service Group Name OraSG table
Resource Name OraSG_Ora
Resource Type Oracle
Required Attributes
main.cf
main.cf
2
SID AP Oracle
Oracle OraSG_Ora
OraSG_Ora ((
Owner oracle Sid
Sid == AP
AP
Home /ora Owner
Owner == oracle
oracle
Home
Home == ""/ora
/ora""
))
Note: Pfile is a required attribute for versions of the agent prior to 4.1.

The required attributes for an Oracle resource are shown in the example excerpt of
the main.cf file.
Note: In version 4.1 of the Oracle agent, Pfile is not a required attribute. However,
Pfile is required for versions prior to 4.1.
Agent Entry Points

Online: Uses the sqldba/svrmgrl/sqlplus startup -f $Pfile
command to start the database
The $Pfile argument is the startup profile file for the Oracle database
instance and is specified in the Pfile attribute.
Offline: Uses the shutdown command to stop the database
Monitor: Scans the process table for ora_dbw, ora_smon, ora_pmon, and
ora_lgwr
The ora_pmon process monitors all other Oracle processes.
If detail monitoring is enabled, as shown in the Configuring Oracle Detail
Monitoring section, the monitor script attempts to write to the database.
Clean: Forcibly stops the Oracle database by using the following svrmgrl or
sqlplus shutdown abort command:
If the process does not respond to the shutdown command, the agent scans
the process table and kills processes associated with the configured instance.
Info: Provides static and dynamic information about the state of the database
Action: Performs predefined actions on a resource

Optional Oracle Attributes: The StartUpOpt
Attribute
You can use the StartUpOpt attribute to specify
how the Oracle instance is started.
Value Action
STARTUP If the Pfile attribute is empty, run: startup
(default) If Pfile is specified, run:
startup pfile=pfile_location
RESTRICTED Starts the database in restricted mode
RECOVERDB Performs a recovery of database files upon instance

startup
CUSTOM Runs the SQL script in a specified location; an
example might be to start up in standby mode and
apply logs
Optional Oracle Attributes
The StartUpOpt Attribute

The Oracle agent supports an optional StartUpOpt attribute to enable you to
control how VCS starts a database instance.
Examples:
RECOVERDB: Assume that in the event the database node fails for a
nondatabase issue (for example, if the system panics due to a hardware issue),
you do not want the database to start up (come online) if any data file recovery
needs to take place. If you set StartUpOpt to RECOVERDB and the system
running the database faults, VCS starts the database in recovery mode when it
fails over to another node.
CUSTOM: You might want to start the database up in standby mode and apply
logs. In this case, you can create an SQL script to perform these actions, and
this script is called when you set StartUpOpt to CUSTOM.
You must create the script in /opt/VRTSvcs/bin/oracle with the name
of start_custom_SID.sql, where SID is the same as the value of the SID
attribute.

The ShutDownOpt Attribute
You can use the ShutdownOpt attribute to specify
how the Oracle instance is stopped.
Attribute Action
IMMEDIATE Runs shutdown immediate
(default)
2
TRANSACTIONAL Performs a transactional stop
SRVCTLSTOP Uses srvctl to stop the database
CUSTOM Runs the SQL script in a specified location
The ShutDownOpt Attribute

You can configure the ShutDownOpt attribute for the Oracle resource to control
how VCS stops a database instance when the Oracle resource is taken offline.
Note: If CUSTOM is specified, a script must exist in /opt/VRTSvcs/bin/
oracle with the name of shut_custom_SID.sql, where SID is the same as
the value of the SID attribute for this Oracle resource.

Other Optional Oracle Attributes
AutoEndBkup: Takes the data files in the
database out of the backup mode when
coming online (The default is 1, true.)
EnvFile: Specifies the location of a file
sourced by Oracle agent entry point scripts
containing environment variables
Encoding: Specifies encoding for other
language sets
Other Optional Oracle Attributes

The Oracle agent supports three other optional attributes described in the slide.
Example main.cf entries showing sample values for these attributes are shown
in the following resource definition. For more information, see the Enterprise
Agent for Oracle Users Guide for your platform.
Oracle OraSG_Ora (
Sid = AP
Owner = oracle
Home = /oracle
Pfile = "/oracle/dbs/initVRT.ora"
EnvFile = "/oracle/.ora_envfile"
AutoEndBkup = 0
Encoding = eucJP
)
The example value for the Encoding attribute sets encoding to the Japanese
language set.

Configuring Oracle Detail Monitoring
Oracle OraSG_Ora ( main.cf

main.cf
Sid = AP
Owner = oracle
Home = /ora
Pfile = /ora/dbs/initAP.ora
EnvFile = /ora/.ora_envfile
Enable
Enable DetailMonitor = 1
2
User
User defined
defined in
in AP
AP
User = scott database
database with
with update
update
Pword = lgLimHmmKum privileges
privileges for
for testtable
testtable
Table = testtable
MonScript = ./bin/Oracle/SqlTest.pl
)
The
The table
table created
created in The
The executable
executable script
script
AP database
AP database before
before containing
containing SQL
SQL write
write
enabling
enabling the
the VCS
VCS resource
resource statements
statements
Configuring Oracle Detail Monitoring

The example configuration shows an Oracle resource with detail monitoring
configured.
These optional Oracle attributes are used to configure detail monitoring:
DetailMonitor: A flag to enable and disable detail monitoring (VCS 4.x)
The default is 0, do not perform detail monitoring. You can set this value
higher than zero to control how often detail monitoring is performed. For
example, if set to 5, detail monitoring is performed every fifth monitor interval.
User: A database user account with update privileges
Pword: The password for the database user account
You must encrypt the password as described in a later section.
Table: The name of the database table VCS uses for additional monitoring
MonScript: The executable script file containing the SQL statements VCS uses
when writing to the table
EnvFile: The file containing environment variables sourced by the agent
Configuration Prerequisites
Create the database user and password for use by VCS.
Create a test table within the monitored database.
Create an executable script with SQL statements.
In this example, the user scott with the password tiger should be defined in
the VRT database with update privileges to the table called testtable. This table
should be created in the database before the additional monitoring is enabled.

Encrypting Passwords
You can use the VCS encryption utility to encrypt database passwords before
configuring the Pword attribute in the Oracle agent configuration.
Note: The value of Pword is automatically encrypted when you use the Oracle
configuration wizard or one of the VCS GUIs to configure the resource.
To encrypt a password, run the vcsencrypt command. For example, type:
# root:/opt/VRTSvcs/bin> vcsencrypt -agent

Error Handling for Detail Monitoring
You can configure the optional Oracle error handling file when
using detail monitoring to specify VCS response based on the
type of database errors. See
/opt/VRTSvcs/bin/Oracle/oraerror.dat.
Action Description
IGNORE Error is not critical; take no action
2
UNKNOWN Marks the resource state as UNKNOWN
WARN Marks the resource state as ONLINE and notifies

the administrator (when notification is configured)
FAILOVER Marks the resource offline when a serious error
occurs; resource faults and SG fails over
NOFAILOVER Freezes the service group temporarily and marks
the resource OFFLINE
Error Handling for Detail Monitoring

You can configure how VCS responds to different types of errors when detail
monitoring is configured to prevent the database from failing over in cases where
that action is not appropriate.
For example, if the user name and password specified for an Oracle resource is
incorrect, an error of the form ORA-01017 invalid username/password;
logon denied occurs. This error indicates that the configuration is wrong, but
the database may be functioning properly. In this case, you do not want VCS to
bring down the database and restart it on another system. The same failure occurs
wherever the database is brought online.
Rather, you want VCS to notify the appropriate administrator to fix the
configuration problem.
You can customize error handling by modifying the oraerror.dat file in
/opt/VRTSvcs/bin/Oracle. Entries in this file have the format:
Oracle_error_string:action:confidence_level
This example entry leaves the Oracle resource online and sets the confidence level
to 10 when an invalid user name error code is returned.
01017:WARN:10

Configuring a Netlsnr Resource
Resource Definition Sample Value Starts and stops a

Service Group Name OraSG listener process
Resource Name OraSG_Listen using lsnrctl
Resource Type Netlsnr Monitors the process
table
Required Attributes
Owner oracle
Home /ora
TNSAdmin /ora/net/admin
Listener LISTENER
LsnrPwd S2cEjcD5s3Cbc
Optional Attributes Detail Monitoring
EnvFile /oracle/dbs/envfile
Monscript ./bin/Netlsnr/Lsnr
Test.pl
Configuring the Netlsnr Resource and Agent

The following main.cf excerpt corresponds to the Netlsnr example resource
shown in the design worksheet in the slide.
Netlsnr OraSG_Listen (
Owner = oracle
Home = "/oracle"
TnsAdmin = "/ora/net/admin"
Listener= LISTENER
LsnrPwd = S2cEjcD5s3Cbc
EnvFile = "/ora/dbs/envfile
MonScript = "./bin/Netlsnr/LsnrTest.pl"
)
Note: The Sqlnet resource type manages the listener for versions of the Oracle
agent prior to 4.0. The attributes are the same, except LsnrPwd, which is not
supported prior to 4.0.

Agent Entry Points
Online: Uses the following command to start the listener process:
lsnrctl -start $LISTENER
Offline: Uses the following command to stop the listener process:
lsnrctl -stop $LISTENER
Monitor: Scans the process table for tnslsnr $LISTENER and runs
MonScript as the option
If secondary monitoring is enabled, the monitor script runs the command:
lsnrctl -stat $LISTENER
Clean: Scans the process table for tnslsnr $LISTENER and kills it
2
Action: Performs predefined actions on a resource

Common Oracle Configurations
Netlsnr Oracle
Netlsnr
IP Oracle
IP Mount Mount
Mount
NIC
Binary Data Log Data
NIC
Volume Volume Volume Volume
Disk Disk
Group Group
Common Oracle Configurations

The diagram shows some common Oracle VCS configurations.
In the resource diagram on the left:
Oracle data files are written to raw volumes. A Mount resource is not
required.
Oracle binary files are located on shared storage on a file system with a
separate volume that is part of the service group configuration, so they are
highly available.
The Oracle instance must be online before the listener is started.
In the resource diagram on the right:
The Oracle resource must come online last.
The log and data files are located on separate file systems and volumes, but
they use the same disk group.
The binaries are on local storage and so are not part of the service group
configuration.
Note: As a best practice, minimize the number of volumes and disk groups used in
database service groups. As a general rule, use one disk group per database. Large
numbers of volume management objects complicate administration and can impact
performance during startup.

Alternative Oracle Configurations
You can use variations of resources and dependencies in an Oracle service group to
accommodate different configurations. Consider these alternatives:
The Oracle resource is not required to be dependent on the IP resource.
However, there is little purpose in bringing the database online unless the IP
address is available and clients can connect to the database.
Regardless of whether you link the Oracle resource to the IP resources, the
Netlsnr resource should have a dependency on the IP resource.
If Net8 is used, a Netlsnr resource can be configured with no dependency on
the database. The Net8 listener can be started independently of the database.

Database Failover Scenarios
If you want the listener to be restarted and never
cause failover if it faults:
Set the listener resource to non-critical.
Do not make the listener a child of a critical resource.
Set RestartLimit = x.
If you do not want VCS to forcibly clean or fail over if
the database or listener faults:
Set ManageFaults=None for the service group.
Notification on fault, no clean, no failover
Even if a database resource is non-critical, the

! clean entry point is run if the resource faults.
This shuts down the database and leaves it offline.
Database Failover Behavior

Database Failover Scenarios
You can use resource and service group attributes to control how VCS responds
when resources or systems fault. A complete discussion of failover behavior is
provided in the VCS Response to Resource Faults and Handling System and
Communication Failures lessons in the VERITAS Cluster Server Fundamentals
course.
The scenarios in the slide show the practical application of some of the more
common failover configurations.

Failover During Oracle Hot Backup
If a fault occurs while a database is in hot
backup mode, you can prevent the service
group from coming online elsewhere.
Set the AutoEndBkUp attribute to 0 to prevent
the database from being opened on another
node.
2
By default, AutoEndBkUp is set to 1, which
takes data files out of backup mode when the
Oracle resource is brought online, enabling
failover.
Failover During Oracle Hot Backup

If a cluster node fails while running a hot backup of an Oracle database, failover to
another node can succeed only if the Oracle resource attribute, AutoEndBkUp, is
set to a nonzero value.
Otherwise, the database in the backup mode on the failover system cannot be
opened and VCS cannot bring the Oracle resource online.
The following errors are displayed to indicate this condition:
$ ORA-1110 "data file %s: %s"
or
$ ORA-1113 "file %s needs media recovery"
Before VCS can bring the Oracle resource online on the failover system, you must
take the data files in the database out of the backup mode and shut down the
database so that it can be reopened. Refer to the Oracle documentation for
instructions on how to change the state of the database files.

Other Oracle Agent Functions
You can use the info and action entry points of
the Oracle agent to perform additional tasks.
Info: Provides static and dynamic information about
the state of an online resource (such as a database)
Action: Enables you to perform predefined actions on
a resource
These entry points are supported in other resource
types as well.
Their uses related to Oracle are described in more
detail in the following sections.
Additional Oracle Agent Functions

In 4.x versions, the Oracle agent supports two additional entry points you can use
to manage database functions from within VCS:
Info: Checks the status of the instance
Action: Performs specified actions, such as backing up the Oracle database,
changing the database state, and suspending and resuming a database
You can initiate these types of operations using the hares command, as shown on
the following pages.
This can be useful for scripting common database administration tasks that can be
initiated from the VCS operator or administrator.

Predefined Actions for Oracle
Keyword (Action) Description

VRTS_GetInstance Retrieves the name of the configured instance
Name*
VRTS_GetRunning Retrieves the list of processes monitored by the agent
Services*
DBRestrict Enables RESTRICTED mode for the database session
2
DBUndoRestrict Disables RESTRICTED mode for the database session
DBSuspend Suspends a database (Oracle 8i and later)
DBResume Resumes a suspended database
DBTbspBackup Places a tablespace in backup mode
*Also applies to Netlsnr resources
Predefined Actions for Oracle

The Oracle agent supports the action entry point, which enables you to perform
predefined actions on a resource.
The Oracle agent supports the actions shown in the slide. The Netlsnr agent
supports two of these actions:
VRTS_GetInstanceName
VRTS_GetRunningServices

Initiating Actions
Use the hares command to initiate predefined
actions on a resource.
Example usage:
hares -action OraSG_Ora DBRestrict
hares action OraSG_Ora DBTbspBackup \
-actionargs mytblspace
You can also develop custom actions for an
agent.
Initiating Actions
To perform an action on a resource, use the hares command with the following
syntax:
hares -action res token [-actionargs <arg1 ...>] [-sys
system]
The keyword represented as token in the command line example, must be one of
the keywords shown in the left column on the Predefined Actions for Oracle
slide. Example command lines are shown in the slide.
Custom actions can be developed enabling you to perform additional actions. See
the VERITAS Cluster Server Agent Development Guide for more information.

Retrievable Information Using the Info Entry
Point
Static Information
Version InstanceNo InstanceName

DatabaseName HostName StartupTime
Parallel Thread InstanceRole
2
Dynamic Information
InstanceStatus Logins OpenMode
LogMode ShutdownPending DatabaseStatus
TotalDBfileSize FreeDBfileSize Shared Pool

Percent free
Buffer Hits Percent
Retrievable Information Using the Info Entry Point

The Oracle agent info entry point provides static and dynamic information about
the state of the database. As described further in the next section, you can use the
hares command to obtain predefined types of information shown in the tables on
the slide.
When invoked, the info entry point maps one of these keywords to a SQL
statement and runs that statement on the database specified by the Oracle resource.

Retrieving Information
Use the hares command to retrieve information
about an online resource.
Example usage:
hares refreshinfo OraSG_Ora sys sys_name
hares value OraSG_Ora ResourceInfo
You can also add SQL statements to the
following file to customize information retrieval
using the info entry point:
/opt/VRTSvcs/bin/Oracle/resinfo.sql
Retrieving Information
You can manually invoke the info entry point using the hares command.
The first command line shown in the slide causes the agent to update the value
of the ResourceInfo attribute for the specified resource on the specified system
if the resource is online. If the Info entry point is successful, no output is
displayed.
Note: ResourceInfo is a temporary attributeits values are never dumped to
the main.cf file.
The second command line displays the value of the ResourceInfo attribute.
As an alternative to manually running hares -refreshinfo, you can set the
InfoInterval resource type attribute to a nonzero value. This tells the agent to run
the info entry point periodically, according to the value of the InfoInterval
attribute.
To set the InfoInterval type attribute, use the following syntax:
hatype -modify Oracle InfoInterval seconds
Note: You can add SQL statements to the /opt/VRTSvcs/bin/Oracle/
resinfo.sql file to obtain additional information using the info entry point. For
information on how to customize this file, see the VERITAS Cluster Server
Enterprise Agent for Oracle, Installation and Configuration Guide.

Lesson Summary
Key Points
VCS provides a variety of agents to manage
databases.
The Oracle agent has specialized functions for
managing Oracle databases.
Reference Materials
2
Reference Guide
VERITAS Cluster Server Enterprise Agent for
Oracle, Installation and Configuration Guide
VERITAS Cluster Server Agent Developer's Guide
High Availability Design Using VERITAS Cluster
Server Course
Summary
This lesson described how to put a database service under control of VCS. An
Oracle example was presented to show the resource types and agent functions that
are provided with enterprise agents.
Next Steps
Now that you understand how Oracle agents manage exported file systems, you
can configure other common types of high availability services.
VERITAS Cluster Server Enterprise Agent for Oracle, Installation and Users
Guide
This guide provides detailed information on procedures and concepts for
configuring and managing Oracle using the VCS enterprise agent.
VERITAS Cluster Server Agent Developers Guide
This guide describes how to customize agents, including the info and action
entry points.
High Availability Design Using VERITAS Cluster Server Course
This provides guidance on designing local and global clusters to manage
database environments.

Lab 2: Creating Oracle Resources
name name nameSG1

nameSG1
App1* Listener1
name
nameIP1
*This may be a Oracle1
Process resource if
the Application lab name
was not completed. nameProxy1
Mount1
1.
1. Add
Add an
an Oracle
Oracle
resource. nameVol1 Appendix
AppendixA:
A:Lab
LabSynopses
Synopses
resource.
Appendix
AppendixB:
B:Lab
LabDetails
Details
2.
2. Add
Add aa Netlsnr
Netlsnr Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
resource.
resource.
nameDG1
See
Seethe
thenext
nextslide
slidefor
forclassroom
classroomvalues.
values.

Lab 2 Synopsis: Creating Oracle Resources, page A-6
Lab 2 Details: Creating Oracle Resources, page B-11
Lab 2 Solutions: Creating Oracle Resources, page C-11
Goal
The purpose of this lab is to make an Oracle database service highly available.
Results
Each students database service can be started, monitored, and stopped on each
cluster system.
Prerequisites
instructions.

Example 3
Clustering NFS

will be able to:
NFS Service Overview Describe how NFS services work in a
high availability environment.
Preparing NFS for High Prepare NFS components for the VCS
Availability environment.
Testing an NFS Service Test the NFS service on each target
system.
Configuring an NFS Configuring NFS and Share resources
Service Group in an NFS service group.
NFS Lock Failover in VCS Configure an NFSLock resource to fail
4.1 over record locks.
Introduction
Overview
This module describes a recommended practice to prepare resources and manually
test NFS services before configuring an NFS service group. In this lesson, you are
introduced to the VCS Share and NFS resource types.
Importance
Although understanding how agents work enables you to use any VCS resource
type, you need to have an understanding of the specific application service at hand
to successfully configure the application in a high availability environment.
Understanding how you can use VCS to manage your NFS shares helps you
implement highly available NFS servers.
Outline of Topics
NFS Service Overview
Preparing NFS for High Availability
Testing the NFS Service
Configuring an NFS Service Group
NFS Lock Failover in VCS 4.1

IP Address
Network
Network NIC
Client Network
NFS daemons Requests
Share
File System
Application
Application Volume
Disk Group
3
Storage
Storage

Network File System (NFS) provides file system services to remote clients over
the public network. As with most application services, NFS service components
can be categorized into three basic components:
Applicationthe NFS processes (daemons) that handle file system mount
requests (mountd), client requests (nfsd), and the share name configuration
The diagram shows a simple example of an NFS service.
Example 3 Clustering NFS 33

NFS Configuration and Migration Process
Create or prepare NFS-related resources initially on each
system for each NFS service that will be put under VCS control;
this is not part of VCS operation.
Manually test NFS shared file systems on each system that is a
startup or failover target for that service group before putting
the file systems under VCS control.
Perform
Performone-time
one-time
configuration
configurationtasks
taskson
on
each
eachsystem.
system.
Start,
Start,verify,
verify,and
and Y
N
More
More Ready
Readyfor
for
stop
stopNFS
NFSservices
servicesonon Systems?
Systems? VCS
VCS
one
onesystem
systemat ataatime.
time.
NFS Configuration and Migration Process Overview

Use the process shown in the diagram to prepare and test NFS services on each
system before putting them under VCS control. Use the design worksheet to obtain
and record information about the service group and each resource. This is the
information you need to configure VCS to control these resources.
Details are provided in the following sections.

Identifying NFS Components and Relationships
Shared storage
resources:
IP Disks or components
of a logical volume
manager
Share NIC File systems to be
mounted
Mount NFS Network resources:
IP addresses
Volume Network interfaces
Application (NFS)
resources:
3
DiskGroup
NFS (nfsd, mountd)
Share names
Preparing NFS for High Availability

Identifying NFS Components and Relationships
VCS manages NFS services in the same way it manages any other application
service. Resources are contained within the service group for components, and
dependencies are defined to ensure that VCS brings the supporting resources
online in the necessary order.
The diagram shows the resources in an NFS service and their relationships. You
can use this diagram as a guide to configure and test NFS resources before
bringing them under VCS control.
Notice that unlike many other resource dependency diagrams, the IP resource is
the topmost parent. If the IP address came up before the NFS shared file system is
available, a client attempting to connect to the share gets a stale file handle and
never tries to mount the file system again. However, if the IP address is not
configured yet, the client retries to connect to the file system.

Configuring Shared Storage for NFS
Major and minor numbers for block devices used
for NFS services must be the same on each system.
Stale File Handle

NFS Error
Response
NFS NFS
Request Request
Before failover After failover
Configuring Shared Storage for NFS

After you create the disk group, volume, and file system to be shared, verify that
the major and minor devices number for storage devices are the same on each
system.
NFS services use the major and minor number of the block device to identify the
exported file system. For the identity of the exported file system to be the same no
matter which system in the cluster provides NFS services, the major and minor
numbers of the storage device must be the same on each system in the cluster.
Generally, major and minor numbers match. However, the values are affected by
circumstances, such as the order in which software is installed. For example, if you
install VxVM and VxFS in different orders on different systems, the major
numbers for devices can be different.

Major Numbers for Volumes
Verify that each system has the same major number
for the block device of the shared volume.
Verify that major numbers are also unique within a
system.
Verify that the major numbers match on all systems.
If major numbers do not match, reset one systems
numbers to match the other system.
grep vx /etc/name_to_major grep vx /etc/name_to_major

vxdmp 87 vxdmp 89
vxio 88 vxio 90
vxspec 89 S1 vxspec 91 S2
3
Major Numbers for Volumes

To display and change the major numbers on the VERITAS Volume Manager
(VxVM) volumes on shared storage, use the following platform-specific
instructions.
Solaris
1 Find the number corresponding to vxio and vxspec in the
/etc/name_to_major file. Type:
grep vx /etc/name_to_major
2 Select major numbers that are not in use by any other device by viewing the
/etc/name_to_major file on each system.
3 To make the major numbers the same on both systems use the haremajor
command to change the major numbers for vxio and vxspec on S1 to match
S2. For example, type:
haremajor -vx 90 91
Note: It may be necessary to change the major numbers on more than one
system to make them unique for all systems.
4 Reboot each system on which haremajor was successful changing major
numbers.
Repeat this procedure for each volume used for an NFS file system.

AIX
Use the lvlstmajor command to find out what the available major numbers are
for the file system. On each system:
1 Check the vxio driver numbers. Major numbers match the driver number.
# haremajor -v
47
2 Find an unused driver number on all systems.
# haremajor a
50...
3 Change each system to the same driver number.
# haremajor -s 50
vxio major number changed to 50
4 Reboot each system.
# reboot
HP-UX
See the HP-UX system documentation for information about changing LVM
device numbers.
Linux
Major and minor device numbers on Linux systems depend entirely on the order of
the SCSI devicesinternal and external are found during system startup.
Therefore, you cannot ensure that major and minor numbers match on different
Linux systems.
Because of this platform-specific limitation, your clients may have problems with
stale file handles to NFS-mounted file systems after a failover. To mitigate this
issue, follow these guidelines:
Ensure that shared storage is on a different controller (than the internal
controller) on each node.
Verify that the SCSI driver for the mentioned controller is loaded after the
internal controller (if such a driver exists).
If both controllers are driven by the same driver then ensure that the internal
controller is on a lower PCI slot.

Minor Numbers for Volumes
Use the ls command to determine whether the minor numbers for volumes match
on each system.
ls l /dev/vx/rdsk/dgname/volume_name
For example:
ls l /dev/vx/rdsk/datadg/vol01
crw------- 1 root root 99 0x000005 Aug 17 07:30 /dev/vx/
rdsk/datadg/vol01
If the minor numbers of the volume devices do not match, change the base minor
number for the disk group to a common value on all systems:
vxdg reminor dgname new_base_number
For detailed information on using haremajor and vxdg reminor, refer to the
VERITAS Cluster Server Installation Guide. You can also refer to the Volume
Manager command vxdg manual page for instructions on reconciling the
3
VERITAS Volume Manager minor numbers.

Configuring Networking for NFS
An administrative IP address must be
configured during system startup for the
interface used by this service.
Add entries to the name server to enable
clients to access the virtual IP address and
host name used to connect to the shared file
system.
Set up any other applicable network
configuration files.
Configuring Networking for NFS

In a high availability environment, the IP address and host name that are used by
the client to access the NFS service should not be tied to a specific system because
the same service can be provided by any system in the cluster.
Ensure that you have an administrative IP address configured on the network
interface that clients will use to access the NFS shared file systems. The
administrative IP address must be configured to enable VCS to set up the virtual IP
address for the NFS service group.
Set up any other network configuration files needed for your public network
environment. Also, configure the name service to enable clients to connect to the
shared file system using a virtual host name and IP address that will migrate
among the systems that can host the NFS service.

Checking the Startup Configuration
Verify that the file system is not mounted or shared
at system startup to enable VCS control:
Solaris:
No file system mount point entry is placed in
/etc/dfs/dfstab.
The /etc/vfstab entry should say no or manual
mount at boot.
AIX: No entry in /etc/exports; a mount=false entry
/etc/filesystems
HP-UX: No entries in /etc/exports or /etc/fstab
Linux: No entries in /etc/exports; noauto entry in
/etc/fstab
3
Configuring NFS (Application) Components
Checking the Startup Configuration

To share a file system in a high availability environment where the file system can
be mounted and shared from any one of the systems, you must prevent the
operating system from controlling the NFS resources:
Solaris
The file system mount point should not have an entry in the /etc/dfs/
dfstab file.
The file system should not have an entry in the /etc/vfstab file or the entry
should say no to mount at boot.
AIX
The file system mount point should not have an entry in /etc/exports.
The file system should have an entry in /etc/filesystems with
mount=false.
HP-UX
The file system should not have an entry in the /etc/fstab file.
Linux
The file system should have an entry with noauto in /etc/fstab.

Testing Procedure
N
Ready
Readyfor
for
Bring
Bringup
upresources.
resources. S1 VCS
VCS
More
More
Start up all resources
Systems?
Systems?
in dependency order.
Y
Shared
Sharedstorage
storage
NFS S2Sn
NFSdaemons,
daemons,exports
exports
Stop
Stopresources.
resources.
Virtual
VirtualIP
IPaddress
address
Test
Testremote
remoteaccess.
access.
Test
Testremote
remoteaccess.
access.
Stop
Stopresources.
resources. Bring
Bringup
upresources.
resources.
Testing the NFS Service

Before configuring a service group in VCS to manage an NFS service, test it on
each system that can be a startup or failover target for the service group. Following
this best practice recommendation ensures that VCS will successfully manage the
shared file systems when you configure the service group.
Detailed information about setting up shared storage and network resources was
provided earlier in the course. This lesson focuses on the additional steps needed
for testing NFS services.

Starting NFS Daemons
Verify that the NFS daemons are running or start
them if necessary:
1. Determine whether the appropriate NFS daemons
are running:
Solaris: nfsd, mountd, statd, lockd
AIX: nfsd, rpc.mountd, rpc.lockd, rpc.statd
HP-UX: nfsd, rpc.mountd, rpc.lockd, rpc.statd
Linux: nfsd, mountd
2. Start the NFS daemons, if they are not already
running.
3
Starting NFS Daemons

The NFS daemons are usually configured to start during system boot. To verify the
appropriate daemons are running, and start them, if necessary:
Solaris
ps -ef | grep nfs mountd statd lockd
/usr/lib/nfs/nfsd -a nserver
/usr/lib/nfs/mountd
/usr/lib/nfs/statd
/usr/lib/nfs/lockd
AIX
lssrc g nfs
startsrc g nfs
HP-UX
ps -ef | grep nfs
/usr/sbin/nfsd -a nserver
/usr/sbin/rpc.mountd
Linux
/sbin/service nfs status
/sbin/service nfs start
/usr/sbin/rpc.mountd

Exporting and Testing the File System
After verifying that the file system is mounted and
NFS daemons are running:
1. Export the file system.
2. Verify that the file system is exported.
3. Configure the virtual IP address.
4. Mount the file system from a remote client on
the public network.
Exporting the File System

After you have verified that the file system is mounted and the NFS daemons are
running, export (share) the mount point of the file system with specified share
options, if any. Then, verify that the share is configured.
Solaris
share mount_point
share | grep mount_point
AIX
smitty mknfsexp: creates/updates /etc/exports
exportfs a
showmount
HP-UX
/usr/sbin/exportfs [F fstype] mount_point
exportfs | grep mount_point
Linux
/usr/sbin/exportfs [F fstype] mount_point
exportfs a
Note: Whenever you change /etc/exports, you must notify the NFS daemon
using either of these methods:
/sbin/service nfs reload
/sbin/exportfs -r

Migrating a Share
IP Address
Network
Network NIC
NFS daemons
File System Share
Volume
Application
Application
S1 Disk Group
S2
3
Storage
Storage
Migrating a Share
Before you place an NFS service under VCS control, it is recommended that you
manually migrate the service between all intended target systems. By doing so,
you:
Ensure that your operating system resources are properly configured on all
potential target cluster systems.
Observe the types of tasks VCS performs to bring a service online, monitor the
status of each resource, and switch a service to another system.
When you have completed the preparation of the testing tasks, you can document
or verify that your design worksheet is complete, and then configure VCS to
manage the NFS services.

The NFS Resource
The NFS agent:
Starts the NFS processes if they are not already running
No offline; OnOnly resource
Checks for active NFS-related processes (based on platform
type)

Service Group Name APExportSG
Resource Name APExportNFS
Resource Type NFS main.cf
main.cf
NFS
NFS APExportNFS
APExportNFS ((
Optional Attributes Nservers
Nservers == 32
32
Nservers 32 ))
Configuring an NFS Service Group

Use the online or offline configuration process described earlier in the course to
create a service group and add resources after you have manually tested the NFS
configuration on each system.
The NFS Resource
Entry Points
Online: Checks whether the nfsd and mountd processes are running
If processes are not running, the online entry point starts them and exits.
Offline: None
Monitor:
Solaris, HP-UX, Linux: Monitors TCP and UDP versions of the processes
by sending RPC calls clnt_create and clnt_call to the RPC server
If calls succeed, the resource is reported as online.
AIX: Determines whether processes are active by querying the SRC
Optional Attributes
Nservers: Specifies the number of concurrent NFS requests the server can
handle
Defaults: Solaris 16; HP-UX 4; AIX 0, obtains the value from the SRC
Nproc (Linux-only): Specifies the number of processes of nfsd to start
The default value is 8.
Protocol (HP-UX only): Specifies the UDP or TCP protocol

Configuring a Share Resource
The Share agent:
Shares and unshares an NFS file system
Reads the share table to check for an entry for the file system

Resource Name APExportShare
Resource Type Share
Required Attributes
main.cf
main.cf
PathName /AP/export1
Share
Share APExportShare
APExportShare ((
Optional Attributes PathName == "/AP/export1"
PathName "/AP/export1"
Options ))
3
Configuring a Share Resource
Entry Points
Online: Shares an NFS file system
Offline: Unshares an NFS file system
Monitor: If there is an entry in the shared table for the file system specified by
PathName, the resource is reported as online.
Optional Attributes
Options: Specifies options to be used with the share command while sharing the
file system. Defaults are:
Solaris, AIX, HP-UX
No options are specified by default.
Linux
ro, async, wdelay, root_squash
Note: Linux has an additional required attribute, Client, which specifies the host to
which the PathName directory is exported.
Configuration Prerequisites
The file system and share should not be configured automatically during
system startup.
The Mount and NFS resources must be configured in VCS.

NFS Considerations in a VCS Environment
Scenarios:
Consider setting RestartLimit to a higher value than
the default of 1 so that VCS attempts to restart the
NFS daemons before failing over to another system.
Increase the value of Nservers (Nproc on Linux) if
you need to increase the number of client
connections.
Note: If you increase this attribute while the resource
is online, freeze the service group, stop and restart
NFS outside of VCS, probe, and then unfreeze the
service group.
Best practice recommends that you do not use
cluster nodes as NFS clients.
NFS Considerations in a VCS Environment

NFS-specific issues to consider when configuring a high availability NFS service
group include:
Increase RestartLimit to avoid failover when daemons fail sporadically.
Increase the Nservers value if the NFS daemon needs more threads. If you
need to increase this value after the service group is running:
a Freeze the service group.
b Change the Nservers attribute value.
c Stop and restart NFS daemons outside of VCS to restart with the increased
Nservers value.
Note: You cannot simply take the NFS resource offline and bring it back
online. The NFS daemons are not stopped when you take the NFS resource
offline because the NFS resource type is on-only.
d Probe the service group to clear the faulted NFS resource.
e Unfreeze the service group.
Avoid using a cluster system that can be a failover target for the NFS service as
a client to mount the shared file system.
If you have the file system mounted and the NFS service group fails over to
that system, the client is stopped as a prerequisite to starting the NFS server.
Also, it is generally good practice to avoid relying on services outside the
cluster.

NFS Locking Overview
RPC lock requests

lockd lockd
statd statd
Status messages
/var/statmon/sm
NFS Client
client1
client2
3
. . .
NFS Server
NFS Lock Failover in VCS 4.1

VCS 4.1 introduced support for NFS lock failover by implementing a new bundled
agent, NFSLock. This agent works with the existing recovery mechanisms
inherent in the Network Lock Manager component of NFS.
NFS Locking Overview

NFS record locking is handled by lockd and statd, two cooperating daemons,
which run on both NFS server and client systems.
lockd
NFS client: The lockd process on an NFS client accepts locking requests
from applications for NFS mounted file systems.
NFS server: The lockd process on an NFS server accepts locking requests
from lockd processes on NFS clients.
statd
NFS client: The statd process on an NFS client system keeps track of the
list of servers where it holds locks.
NFS server: The statd process on an NFS server system tracks the clients
holding locks on that server. The client information is stored on the server
in /var/statmon/sm.

Basic Lock Process
To illustrate how NFS lock works, this example describes locking between one
NFS client and one NFS server, where an application running on the client
requests a remote lock:
1 lockd registers the server name hosting the requested remote lock with the
local statd process.
2 statd on the client contacts statd on the server to determine whether the
server is running.
3 statd notifies lockd that the server is running.
4 lockd on the client then communicates with lockd on the server to request
the lock.
5 statd on both the client and server systems periodically checks the status of
each statd process for registered systems

NFS Lock Recovery After Server Reboot
Reclaim grace period
d
c "Reclaim lock"
lockd lockd
b
statd statd
"Server rebooted" a
/var/statmon/sm
client2
client1
client2
3
. . .
server1 (after reboot)
Lock Recovery
The lockd and statd processes also provide recovery for locks in the event that
either a client or service crashes and reboots.
Client failure: When an NFS client crashes (as determined by statd):
a lockd on the server holds the locks the client established before the crash.
b When the client reboots, statd on the client sends new state information
to statd on the server, which passes the information to lockd.
c Applications on the client restart and request locks again as needed.
Note: If a client with locks on a server crashes and is not restarted, the locks
are held indefinitely.
Server failure: When an NFS server crashes and reboots:
a statd on the server sends a message to statd on each registered client
(listed in /var/statmon/sm) indicating that the server has rebooted.
b The client statd process informs the client lockd process that the server
has rebooted.
c The client lockd process attempts to reclaim the locks from the server.
d The server lockd process waits for a defined grace period (default 45
seconds) to grant reclaimed locks before the server revokes the locks and
resumes normal operation.

VCS Failover and NFS File Lock Recovery
VCS 4.1 provides several cooperating mechanisms to
enable a failover system to acquire NFS lock
information for recovery:
The system monitor synchronization daemon, smsyncd,
copies /var/statmon/sm entries to shared storage once
per second.
The NFSLock agent:
Kills lockd and statd on the failover system to initiate lock
recovery (online) and release locks (offline)
Monitors lockd, statd, and smsyncd
The nfs_restart trigger:
Runs smsyncd in oneshot mode to copy entries from shared
storage to /var/statmon/sm on the failover system
Restarts nfsd, statd, and lockd and starts smsyncd
VCS Failover and NFS Lock Recovery

VCS 4.1 includes several cooperating mechanisms that work with the NFS lock
recovery mechanisms provided by lockd and statd.
The smsyncd Daemon

The smsyncd process provides two functions:
In oneshot mode, smsyncd copies the list of clients holding locks from a
specified shared storage location to /var/statmon/sm. statd then uses
these entries to contact clients for recovery.
In daemon mode, smsyncd copies entries from /var/statmon/sm to a
specified shared storage location once per second. Each online NFSLock
resource has a corresponding instance of smsyncd running.
The NFSLock Agent

The NFSLock agent performs these functions:
online: Kills lockd to start the grace period and kills statd to initiate
recovery
offline: Kills lockd to release locks and kills statd to break the client/
server connection
monitor: Determines whether lockd, statd, and smsyncd are running and
restarts them if necessary

The nfs_restart Trigger
The nfs_restart trigger is configured by default and runs for every service group
when the service group is brought online. The trigger determines whether a service
group contains an NFS resource and takes no action if no NFS resources are
present.
If a service group has an NFS resource, the nfs_restart trigger runs smsyncd (in
oneshot mode) to copy lock entries from shared storage to /var/statmon/sm.
Then the trigger restarts nfsd, statd, and lockd and starts smsyncd (in
daemon mode).

The NFSLock Resource
IP
VCS
VCS4.1
4.1only
only
Share NIC
NFSLock NFS

Mount
Resource Name APExportNFSLock
Volume Resource Type NFSLock
Required Attributes
DiskGroup PathName /AP/export1
The NFSLock Resource

The NFSLock resource has one required attribute, PathName, which specifies the
root directory of the file system for which the resource is managing record locks.
The resource diagram in the slide shows how the NFSLock resource fits in the
service group dependency tree.

NFS Service Group Startup Process
VCS performs these actions during startup to enable lock recovery.
11
Run the nfs_restart trigger (runs smsyncd).
22
Bring DiskGroup, Volume, and Mount online.
33
Bring NFS online (monitors NFS daemons).
44
Bring NFSLock online (kills lockd, statd).
55
Bring remaining resources online.
3
NFS Service Group Startup Process
This process describes how an NFS service group is started, showing the
relationships between the NFS lock recovery components. The same events occur
when an NFS service group is initially brought online, switched from another
system, or failed over due to a fault.
1 Run the nfs_restart trigger.
For a service group with an NFS resource, the trigger:
a Runs smsyncd in oneshot mode to copy entries from shared storage to the
local /var/statmon/sm directory
b Starts nfsd, statd, and lockd if they are not already running
Note: These daemons are usually configured to start during system boot.
c Starts smsyncd in daemon mode, which copies /var/statmon/sm
entries to shared storage once per second
2 Bring the DiskGroup, Volume, and Mount resources online.
3 Bring the NFS resource online. This starts monitoring nfsd, mountd, statd,
and lockd.
4 Bring NFSLock online. The online entry point:
a Kills lockd to start the grace period
At this point, lockd allows 45 seconds (default) for clients to reclaim
locks.

b Kills statd to initiate recovery
statd on the server contacts statd on all registered clients indicating
that NFS has (re)started and locks can be reclaimed.
c After the grace period expires, lockd resumes normal operation and
accepts locking requests from any client.
5 Bring remaining resources online, in dependency order.

NFS Lock Recovery Limitations
Failover of one service group with an NFSLock
resource disrupts locking for all other online service
groups with NFSLock.
Clients may receive lock errors during the period
when the virtual IP address is up but locking services
are not registered on the server.
The one-second intervals of smsyncd writes leave a
window where a client added during that write period
may not be notified after for recovery.
Clients must use FQDN host names for the server.
3
NFS Lock Failover Limitations
Consider these limitations when implementing NFSLock resources in an NFS
service group:
If there are multiple service groups online on each node in the cluster, failing
over a single service group disrupts the locking service for all other groups
online on both the failing-from and failing-to nodes.
There is a time period where the virtual IP address is online but locking
services are not registered on the server. Any NFS client trying to acquire a
lock in this interval would fail and receive an ENOLCK error.
Every second, the smsyncd daemon copies the list of clients that hold the
locks on the shared file system in the service group. If the service group fails
before smsyncd has an opportunity to copy the client list, the clients may not
receive a notification after the service group is brought up. In this case, the
locks cannot be recovered.
You must provide a fully qualified hostname (nfsserver.princeton.edu) for the
NFS server while mounting the file system on the NFS client. If you do not use
a fully qualified hostname, or if you use a virtual IP address (10.122.12.25) or
partial hostname (nfsserver), NFS lock recovery fails.
Ensure that the NFS client stores the same information for the NFS server as
the client uses while mounting the file system. For example, if the NFS client
mounts the file system using fully qualified domain names for the NFS server,
the NFS client directory, /var/statmon/sm, should also have a fully
qualified domain name after the acquisition of locks.

Lesson Summary
Key Points
You can use the NFS and Share agents to
manage NFS exported file systems.
Prepare and test your NFS-related services
before configuring service groups.
Reference Materials
Reference Guide
VERITAS Cluster Server User's Guide
VERITAS Cluster Server Installation Guide
Summary
This lesson described how to put an NFS service under control of VCS.
Next Steps
Now that you understand how NFS agents manage exported file systems, you can
configure other common types of high availability services.
VERITAS Cluster Server Users Guide
This guide provides detailed information on procedures and concepts for
configuring and managing VCS clusters.
VERITAS Cluster Server Installation Guide
This guide provides detailed information about configuring NFS in preparation
for clustering NFS resources.

Lab 3: Creating NFS Resources
name nameSG1
nameSG1
IP1
name
nameShare1 nameProxy1
Process1*
*This may be an
Application name
resource if you nameNFS1
Mount1
completed that
lab exercise.
1. Add nameVol1
Add an
an NFS
NFS Appendix
resource. AppendixA:
A:Lab
LabSynopses
Synopses
resource. Appendix
AppendixB:
B:Lab
LabDetails
Details
3
2. Add a Share Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
resource. nameDG1
resource.

Lab 3 Synopsis: Creating NFS Resources, page A-16
Lab 3 Details: Creating NFS Resources, page B-31
Lab 3 Solutions: Creating NFS Resources, page C-35
Goal
The purpose of this lab is make an NFS service highly available.
Prerequisites
instructions.
Results
Each students NFS service can be started, monitored, and stopped on each cluster
system.

Appendix A
Lab Synopses
nameSG1
nameSG1
nameApp1
Delete
Delete the
the Process
Process name
nameIP1
resource
resource and
and add
add Mount1
an
an Application
Application
resource.
resource.
nameVol1 nameProxy1
Appendix
AppendixA:
A:Lab
LabSynopses
Synopses
nameDG1 Appendix
AppendixB:
B:Lab
LabDetails
Details
Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
See
Seethe
thenext
nextslide
slidefor
forclassroom
classroomvalues.
values.
Lab 1 Synopsis: Creating an Application Resource

Students work separately to modify their own nameSG1 service groups to add
NFS and Share resources.
Step-by-step instructions for this lab are located on the following page:
Solutions for this exercise are located on the following page:
Lab Assignments
Obtain the values for your classroom.
Object Values
Location of application
script files: loopy_dir
A2 VERITAS Cluster Server for UNIX, Example Application Configurations

Adding an Application Resource
Each student configures their own Application resources.
A
Resource Definition Sample Value Your Value
Service Group nameSG1
Resource Name nameApp1
Resource Type Application
Required Attributes
StartProgram /name1/startloopy name 1
/name1
StopProgram /name1/stoploopy /name1
PidFiles /name1/loopypid
Critical? No (0)
Enabled? Yes (1)
1 Delete the nameProcess1 resource.
2 Set up the application.
a Copy the scripts used by the application resource to your /name1 file
system from the location provided by your instructor:
Script location:
_________________________________________________________
loopy_dir
Copy these scripts:
startloopy
stoploopy
loopy
b Verify that the scripts are executable.
Appendix A Lab Synopses A3

c View the startloopy and stoploopy scripts to determine how they are
used.
3 Add the Application resource to the service group using the information in the
design worksheet and bring it online.

Required Attributes
/name1
Critical? No (0)
Enabled? Yes (1)
4 Save the cluster configuration.

Linking the Resources
Link the resources according to the design worksheet.
A
Resource Dependency Definition
Parent Resource Requires Child Resource
nameApp1 nameIP1
nameApp1 nameMount1
Testing the Application Service Group
1 Test the service group by switching it between systems in the cluster.
2 Set the Application resource to critical.
3 Save the cluster configuration and when all students have completed the lab,
close the cluster configuration.
4 Fault resources in the group to test failover.
Resetting the Service Group
Remove the link from the nameApp1 resource to the nameIP1 resource. An IP
address is not strictly needed for the loopy process to run. Removing the link
provides a simplified service group for any subsequent labs your instructor may
ask you to complete.
1 Unlink the nameApp1 and nameIP1 resources.
2 Save the cluster configuration and when all students using the cluster are
finished, close the cluster configuration.

name name nameSG1

nameSG1
App1* Listener1
name
nameIP1
Process resource if
Mount1
1.
1. Add
Add an
an Oracle
Oracle
AppendixA:
A:Lab
LabSynopses
Synopses
resource.
Appendix
AppendixB:
B:Lab
LabDetails
Details
2.
2. Add
Add aa Netlsnr
Netlsnr Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
resource.
resource.
nameDG1
See
Seethe
thenext
nextslide
slidefor
forclassroom
classroomvalues.
values.
Lab 2 Synopsis: Creating Oracle Resources

Students work separately in most sections of this lab to modify their own
nameSG1 service groups to add Oracle and Netlsnr resources.

Lab Assignments
Obtain the preparation information for your classroom.
Object Sample Value Your Value
A
Oracle instance name train1, train5, train9: ORCL1
for your service train2, train6, train10: ORCL2
group
train3, train7, train11: ORCL3
Location of Oracle
data files
Script used for data ORCL1: ORCL1_files.sh
file extraction (train1, train5, train9)
ORCL2: ORCL2_files.sh
(train2, train6, train10)
Preparing an Oracle Instance
1 Verify that your nameSG1 service group is online on your system; if it is not,
switch it or bring it online on your system.
2 Create a link for your SID to the /name1 file system on your system. You are
assigned a SID based on your system, as shown in the design worksheet.
For SID ORCL1: /orcl1 > /name1

3 Create a link for your SID to the /name1 file system on the other system in
the cluster.
Note: Be sure to complete this step. If you do not create the link on both systems,
you cannot switch the service group later.
4 Change to the directory containing the scripts for extracting Oracle data files.
Obtain that location from your instructor and record the directory.
Location of scripts:
________________________________________________________
5 Extract the Oracle datafiles by running the script for your SID, supplied in the
worksheet and recorded in the follow space. The script runs tar to copy the
Oracle data files to the shared storage volume you have imported.
Script name:_________________________________
6 Modify the /etc/hosts file on each system in the cluster to add an entry for
the Oracle SID (ORCL#) corresponding to the IP address used for the nameIP1
resource.
7 Ensure that the virtual IP address is configured.

Testing Oracle and the Listener
1 Start Oracle on your system.
A
2 After the Oracle instance is started, start the Listener. Your listener name is the
same name as your SID.
3 Stop the Listener (as user oracle).
4 Stop the Oracle instance (as user oracle).
5 Switch your nameSG1 service group to the other system and perform the same
test.
6 Switch your nameSG1 service group to your system and exit from their
system.
Installing the Oracle Agent
All lab participants install the Oracle Enterprise agent on their own systems. If you
have no lab partner, install the agent on each system in the cluster.
1 Record the installation directory provided by your instructor.
Installation directory:
________________________________________________________
install_dir
2 Copy the agent package to /tmp on your system.
3 Install the VRTSvcsor agent package.
4 If you do not have a lab partner, install the agent on each system.

.
Adding a Resource Type Definition
Work with your lab partner to add the resource types for Oracle to the cluster.
Use the information in the design worksheet to add an Oracle resource to your
nameSG1 service group.

Resource Name nameOracle1
Required Attributes
Home /oracle
Owner oracle
Sid ORCL#
Optional Attributes
Pfile /orcl#/admin/ORCL#/pfile/
initORCL#.ora
Critical? No (0)
Enabled? Yes (1)

Adding a Netlsnr Resource
Use the information in the design worksheet to add a Netlsnr resource to your
A
Note: If using a version of VCS earlier than 4.0, skip this section and proceed to
the 3.5 Alternate: Adding a Sqlnet Resource section.

Resource Name nameListener1
Resource Type Netlsnr
Required Attributes
Home /oracle
Owner oracle
TnsAdmin /oracle/network/admin
Listener ORCL#
Critical? No (0)
Enabled? Yes (1)

3.5 Alternate: Adding a Sqlnet Resource
If your class is using a version of VCS earlier than 4.0, use the information in the
design worksheet to add a Sqlnet resource to your nameSG1 service group.

Resource Type Sqlnet
Required Attributes
Home /oracle
Owner oracle
Listener ORCL#
Critical? No (0)
Enabled? Yes (1)
Link the Oracle and Listener resources according to the design worksheet.

Service Group ClusterService
nameOracle1 nameMount1
nameListener1 nameOracle1
nameListener1 nameIP1

Testing the Service Group
Test the service group by switching it between the systems. Then, set the Oracle
and Netlsnr resources to critical and test failover.
A
In this lab section, each student works on their own to remove the Oracle-related
resources. This provides a simplified service group for any subsequent labs your
instructor may ask you to complete.
Note: If you are configuring detail monitoring as shown in the optional lab section,
skip this section and return when you have completed configuring and
testing detail monitoring.
Optional Lab: Detail Monitoring of Oracle
1 Verify that Oracle is online on a system. Switch the nameSG1 service group to
your local system, if it is not already online.
2 Log in as the oracle user on that system and start sqlplus as internal.
3 Create an Oracle user named vcs with the password vcs.
4 Grant create session permissions to vcs while in sqlplus.
5 Create a table named vcs.vcstest and insert a row. This table is used to test
Oracle from VCS.
6 Select the data in the vcs.vcstest table to verify that a row is present.
7 Freeze the nameSG1 service group temporarily.

Modifying the Oracle Resource
1 Encrypt the vcs password using /opt/VRTSvcs/bin/vcsencrypt. Use

the output when setting the Pword attribute in the next step.
2 Modify up the nameOracle1 resource for detail monitoring. Use the values
shown in the outlined portion of the worksheet.

A
Required Attributes
Home /oracle
Owner oracle
SID ORCL#
Optional Attributes
initORCL#.ora
User vcs
Pword vcs
DetailMonitor 1
Table vcstest
MonScript ./bin/Oracle/SqlTest.pl
Critical? No (0)
Enabled? Yes (1)
3 Unfreeze the nameSG1 service group.
5 Start sqlplus and verify that the vcstest table is being updated at the monitor
interval.

name nameSG1
nameSG1
IP1
name
Process1*
*This may be an
Application name
Mount1
completed that
lab exercise.
1. Add nameVol1
Add an
an NFS
NFS Appendix
A:Lab
LabSynopses
Synopses
resource. Appendix
AppendixB:
B:Lab
LabDetails
Details
AppendixC:
C:Lab
LabSolutions
Solutions
resource. nameDG1
resource.
Lab 3 Synopsis: Creating NFS Resources

Preparing an NFS Service
Note: Skip this section if you are working on HP-UX or Linux systems.
1 Determine the block device name that will be shared by looking at the
nameMount1 resource attributes in the nameSG1 service group.
2 Verify that the major and minor numbers are the same for each system in the
cluster. If the major numbers do not match, change the numbers for vxio and
vxspec to make them unique and match.

Adding an NFS Resource
Using the information in the design worksheet, add an NFS resource to your
A
Resource Name nameNFS1
Resource Type NFS
Required Attributes
Critical? No (0)
Enabled? Yes (1)
Adding a Share Resource
Using the information in the design worksheet, add a Share resource to your

Resource Name nameShare1
Resource Type Share
Required Attributes
PathName /name1
Critical? No (0)
Enabled? Yes (1)

Testing an NFS Service Group
Link resources, switch the group between systems, and then verify that clients can
connect.

nameIP1 nameShare1
nameShare1 nameMount1
nameShare1 nameNFS1
In this lab section, each student works on their own to remove the NFS-related
1 Delete the nameShare1 and nameNFS1 resources.
2 Link the nameProcess1|App1 resource (parent) back to the nameIP1 resource

(child).

Appendix B
Lab Details
B2 VERITAS Cluster Server for UNIX, Example Application Configurations
B
Lab 1 Details: Creating an Application
Resource
Lab 1 Details: Creating an Application Resource B3

nameSG1
nameSG1
nameApp1
Delete
Delete the
the Process
Process name
nameIP1
resource
resource and
and add
add Mount1
an
an Application
Application
resource.
resource.
nameVol1 nameProxy1
Appendix
AppendixA:
A:Lab
LabSynopses
Synopses
nameDG1 Appendix
AppendixB:
B:Lab
LabDetails
Details
Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
See
Seethe
thenext
nextslide
slidefor
forclassroom
classroomvalues.
values.
Lab 1 Details: Creating an Application Resource

Students work separately to modify their own nameSG1 service groups.
Brief instructions for this lab are located on the following page:
Lab Assignments
Object Values


B
Required Attributes
/name1
Critical? No (0)
Enabled? Yes (1)
1 Verify that the nameSG1 service group is online and running on your system. If
the service group is not online, bring it online on your system.
2 Open the cluster configuration.
3 Take the nameProcess1 resource offline.
4 Disable the nameProcess1 resource.
5 Unlink the nameProcess1 resource from nameIP1 and nameMount1.

Script location:
_________________________________________________________
loopy_dir
Copy these scripts:
startloopy
stoploopy
loopy
used.
9 Add the resource to the service group.
10 Set the resource to not critical.
11 Set the required attributes for this resource and any optional attributes, if
needed.
12 Enable the resource.
13 Bring the resource online on your system.
14 Verify that the resource is online in VCS and at the operating system level.

Each student configures the dependencies for their own service group.

B
nameApp1 nameIP1
nameApp1 nameMount1
1 Link the resources according to the design worksheet.
2 Verify that the resources are linked properly.

1 Test the service group by switching it to their_sys in the cluster.
2 Verify that the service group came online properly on the other system.
3 Test the service group by switching it back to your system.
4 Verify that the service group came online properly on the original system.
6 Save the cluster configuration and view the configuration file to verify your
changes.
7 When all students have completed the lab, close the cluster configuration.

B
4 When all students using the cluster are finished, close the cluster configuration.

B
Lab 2 Details: Creating Oracle Resources
Lab 2 Details: Creating Oracle Resources B11

name name nameSG1

nameSG1
App1* Listener1
name
nameIP1
Process resource if
Mount1
1.
1. Add
Add an
an Oracle
Oracle
AppendixA:
A:Lab
LabSynopses
Synopses
resource.
Appendix
AppendixB:
B:Lab
LabDetails
Details
2.
2. Add
Add aa Netlsnr
Netlsnr Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
resource.
resource.
nameDG1
See
Seethe
thenext
nextslide
slidefor
forclassroom
classroomvalues.
values.
Lab 2 Details: Creating Oracle Resources


Lab Assignments

group
Location of Oracle
data files
B

2 Create a link for your SID to the /name1 file system on each system as
shown.
Note: Be sure to complete this step. If you do not create the link on both
systems, you cannot switch the service group later.
3 Obtain the location of the scripts provided for extracting Oracle data files from
your instructor and record the directory.
Location of scripts:_________________________________
script_dir
4 Change to the script directory.
cd script_dir
5 Extract the Oracle data files by running the script for your SID, supplied in the
worksheet and recorded in the following space. The script runs tar to copy
the Oracle data files to the shared storage volume you have imported.
Script name:_________________________________

resource.
a Record the IP address value of your nameIP1 resource.
IP address:_________________________________
b Edit the /etc/hosts files on all systems in the cluster to add your Oracle
SID as the host name (ORCL#) with this IP address. Coordinate edits with
B
your lab partner.

1 Use the following commands to start Oracle on your system.
su - oracle
Select the instance for your system, such as ORCL1.
sqlplus "/as sysdba"
startup
exit
same name as your SID. Use the following command (as user oracle):
lsnrctl start ORCL#
3 Stop the Listener using the following command (as user oracle):
lsnrctl stop ORCL#
4 Stop the Oracle instance using the following commands (as user oracle):

shutdown
exit
exit (back to root)
5 Switch your nameSG1 service group to their system.
6 Log in to their system and perform the same test for Oracle on their system.
rlogin their_sys
a Start Oracle.
su - oracle
startup
exit

b After the Oracle instance is started, start the Listener.
lsnrctl start ORCL#
c Stop the Listener.
lsnrctl stop ORCL#
d Stop the Oracle instance.
B
shutdown
exit
exit (back to root)
7 Switch your nameSG1 service group to your system.
8 Exit from their system.

All lab participants install the Oracle Enterprise agent on their own systems. If you
____________________________________________________
install_dir
3 Change to the /tmp directory.
4 Unzip the compressed package file.
5 Extract the compressed file from the tar file.
7 If you do not have a lab partner, install the agents on each system.

1 Use the GUI to open the cluster configuration, import the Oracle resource
types, and save the configuration.
2 View the main.cf file to see the addition of the include directive for the
B
OracleTypes.cf file.
3 Check the configuration directory for VCS for the presence of the

All students configure their own Oracle resources.

Required Attributes
Home /oracle
Owner oracle
Sid ORCL#
Optional Attributes
initORCL#.ora
Critical? No (0)
Enabled? Yes (1)
1 Add the nameOracle1 resource to the service group.
needed.
5 Bring the resource online on your system and verify that the resource is online.

Each student configures their own Netlsnr resources.


B
Required Attributes
Home /oracle
Owner oracle
Listener ORCL#
Critical? No (0)
Enabled? Yes (1)
needed.

Note: Each student completes this section if your class is using a version of VCS
earlier than 4.0.

Required Attributes
Home /oracle
Owner oracle
Listener ORCL#
Critical? No (0)
Enabled? Yes (1)
needed.

Each student configures their own dependencies.

B
1 Link the Oracle and Listener resources according to the design worksheet.

Test the service group by switching it between the systems. Then set the Oracle
1 Switch the nameSG1 service group to their system and verify that it comes
online.
2 Switch the nameSG1 service group to your system and verify that it comes
online.
3 Set the nameOracle1 and nameListener1 resources to critical.
5 When all students working in the cluster are finished configuring their Oracle
service groups, close the cluster configuration.

In this lab section, each student works on their own to remove the Oracle-related
B
2 Take the nameOracle1 and nameListener1 resources offline.
3 Disable the nameOracle1 and nameListener1 resources.
4 Unlink the nameOracle1 and nameListener1 resources.
5 Remove the nameOracle1 and nameListener1 resources.

2 Log in as the oracle user on that system.
su - oracle
3 Start sqlplus as internal.
4 Create an Oracle user named vcs with the password vcs. Type the following
commands from sqlplus:
create user vcs

identified by vcs
default tablespace users
quota 500K on users;
5 Grant permissions to vcs while in sqlplus:
grant create session to vcs;
Oracle from VCS:
create table vcs.vcstest (tstamp date);

insert into vcs.vcstest (tstamp) values (SYSDATE);
7 Select the data in the vcs.vcstest table to verify that a row is present:
select TO_CHAR(tstamp, 'MON DD, YYYY HH:MI:SS

AM')tstamp from vcs.vcstest;
exit (from sqlplus)
exit (back to root)


Use the values in the table to configure the nameOracle1 resource to perform
detailed monitoring.

Required Attributes
Home /oracle
Owner oracle
SID ORCL#
initORCL#.ora
Optional Attributes
initORCL#.ora
User vcs
Pword vcs
DetailMonitor 1
Table vcstest
MonScript ./bin/Oracle/SqlTest.pl
Critical? No (0)
Enabled? Yes (1)

2 Modify up the nameOracle1 resource for detail monitoring. Use the values
shown in the outlined portion of the worksheet.
Note: Use the encrypted string from the previous step when setting the Pword
attribute.
B
interval.
su - oracle
sqlplus "/ as sysdba"
select TO_CHAR(tstamp,'MON DD, YYYY HH:MI:SS
exit (from sqlplus)
exit (back to root)

B
Lab 3 Details: Creating NFS Resources
Lab 3 Details: Creating NFS Resources B31

name nameSG1
nameSG1
IP1
name
Process1*
*This may be an
Application name
Mount1
completed that
lab exercise.
1. Add nameVol1
Add an
an NFS
NFS Appendix
A:Lab
LabSynopses
Synopses
resource. Appendix
AppendixB:
B:Lab
LabDetails
Details
AppendixC:
C:Lab
LabSolutions
Solutions
resource. nameDG1
resource.
Lab: Creating NFS Resources


B
cluster.
3 If the major numbers do not match:
a Select major numbers that are not in use by any other device by viewing
the /etc/name_to_major file on each system.
b To make the major numbers the same on both systems, use the
haremajor command to change the major numbers for vxio and
vxspec.
Note: You may need to change the major numbers on more than one

Add an NFS resource to your nameSG1 service group.

Resource Type NFS
Required Attributes
Critical? No (0)
Enabled? Yes (1)
5 Bring the resource online if it is not already online. If the NFS daemons were
started by the operating system and running, the resource is marked as online.
Note: NFS is a persistent resource type; however, it is of type on-only.

Add a Share resource to your nameSG1 service group.

Resource Type Share
B
Required Attributes
PathName /name1
Critical? No (0)
Enabled? Yes (1)
3 Set the required attributes for this resource, and any optional attributes, if
needed.

connect.

nameIP1 nameShare1
nameShare1 nameNFS1
1 Unlink nameProcess1|App1 from nameIP1, if it is not already unlinked.
3 Test the service group by switching it between the systems.
4 Verify NFS functionality from a client system using the trainer system in the
classroom, or another system outside of your cluster.
a Log in to the client system.
b Create a mount point on the client system using your name as the mount
point.
c Mount the NFS file system on the trainer (or other) system using the virtual
IP address of your nameSG1 service group.
d Verify that the file system mounted.
5 In another window, perform a switch of the nameSG1 service group to the

other system (their_sys) and verify that the NFS mounted file system remains
mounted on the client machine.

6 Switch the nameSG1 service group back to your system and verify that the
NFS mounted file system remains mounted on the client machine.
7 Unmount the NFS file system on the client system.
8 Log off of the client system.
9 Set the NFS and Share resources to critical.
B
11 When all students using the cluster are finished configuring the NFS service
group, close the cluster configuration.

1 Clear all faulted resources in the service group.
3 Take the nameIP1 and nameShare1 resources offline.
4 Disable the nameShare1 and nameNFS1 resources.
5 Unlink the nameIP1 and nameShare1 resources.
6 Unlink the nameShare1 and nameNFS1 resources.
7 Unlink the nameShare1 and nameMount1 resources.

(child).

Appendix C
Lab Solutions
C2 VERITAS Cluster Server for UNIX, Example Application Configurations
Lab 1 Solutions: Creating an Application
Resource
Lab 1 Solutions: Creating an Application Resource C3

nameSG1
nameSG1
nameApp1
Delete
Delete the
the Process
Process name
nameIP1
resource
resource and
and add
add Mount1
an
an Application
Application
resource.
resource.
nameVol1 nameProxy1
Appendix
AppendixA:
A:Lab
LabSynopses
Synopses
nameDG1 Appendix
AppendixB:
B:Lab
LabDetails
Details
Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
See
Seethe
thenext
nextslide
slidefor
forclassroom
classroomvalues.
values.
Lab 1 Solutions: Creating an Application Resource

Students work separately to modify their own nameSG1 service groups.
Lab Assignments
Object Values


Required Attributes
/name1
C
Critical? No (0)
Enabled? Yes (1)
1 Verify that the nameSG1 service group is online and running on your system. If
this service group is not online, bring it online on your system.
hastatus -sum
hagrp -switch nameSG1 -to your_sys
or
hagrp -online nameSG1 -sys your_sys
haconf -makerw
3 Take the nameProcess1 resource offline.
hares -offline nameProcess1 -sys your_sys
4 Disable the nameProcess1 resource.
hares -modify nameProcess1 Enabled 0

5 Unlink the nameProcess1 resource from nameIP1 and nameMount1.
hares -unlink nameProcess1 nameIP1

hares -unlink nameProcess1 nameMount1
hares -delete nameProcess1
haconf -dump
Script location:
_________________________________________________________
loopy_dir
Copy these scripts:
startloopy
stoploopy
loopy
ls -l /name1
used.
hares -add nameApp1 Application nameSG1
hares -modify nameApp1 Critical 0

needed.
hares -modify nameApp1 StartProgram "/name1/startloopy

name 1 /name1"
hares -modify nameApp1 StopProgram "/name1/stoploopy

/name1"
Note: Quotation marks are not required when you use the GUI to
configure the StartProgram and StopProgram attributes.
hares -modify nameApp1 PidFiles /name1/loopypid
C
hares -modify nameApp1 Enabled 1
13 Bring the resource online on your system.
hares -online nameApp1 -sys your_sys
14 Verify that the resource is online in VCS and at the operating system level.
hares -display nameApp1
Check for loopy output displayed on the console.
haconf -dump

Each student configures the dependencies for their own service group.

nameApp1 nameIP1
nameApp1 nameMount1
hares -link nameApp1 nameMount1

hares -link nameApp1 nameIP1
2 Verify that the resources are linked properly.
hares -dep nameApp1
haconf -dump

1 Test the service group by switching it to their_sys in the cluster.
hagrp -switch nameSG1 -to their_sys
2 Verify that the service group came online properly on the other system.
hastatus -sum
3 Test the service group by switching it back to your system.
C
4 Verify that the service group came online properly on the original system.
hastatus -sum
hares -modify nameApp1 Critical 1
6 Save the cluster configuration and view the configuration file to verify your
changes.
haconf -dump
view /etc/VRTSvcs/conf/config/main.cf
haconf -dump -makero
kill -9 loopy_pid

haconf -makerw
hares -unlink nameApp1 nameIP1
haconf -dump

Lab 2 Solutions: Creating Oracle
Resources
Lab 2 Solutions: Creating Oracle Resources C11

name name nameSG1

nameSG1
App1* Listener1
name
nameIP1
Process resource if
Mount1
1.
1. Add
Add an
an Oracle
Oracle
AppendixA:
A:Lab
LabSynopses
Synopses
resource.
Appendix
AppendixB:
B:Lab
LabDetails
Details
2.
2. Add
Add aa Netlsnr
Netlsnr Appendix
AppendixC:
C:Lab
LabSolutions
Solutions
resource.
resource.
nameDG1
See
Seethe
thenext
nextslide
slidefor
forclassroom
classroomvalues.
values.
Lab 2 Solutions: Creating Oracle Resources


Lab Assignments

group
Location of Oracle
data files
C

hastatus -sum
hagrp -online nameSG1 -sys your_sys
or
2 Create a link for your SID to the /name1 file system on your system. You are
assigned a SID based on your system, as shown in the design worksheet.
ln -s /name1 /orcl1
ln -s /name1 /orcl2
ln -s /name1 /orcl3
ln -s /name1 /orcl4

3 On the other system in the cluster, create a link for your SID to the /name1
file system as shown.
Note: Be sure to complete this step. If you do not create the link on both systems,
you cannot switch the service group later.
Solaris, AIX
rsh their_sys ln -s /name1 /orcl1
HP-UX
remsh their_sys ln -s /name1 /orcl1
Linux
ssh their_sys ln -s /name1 /orcl1
C
Solaris, AIX
HP-UX
Linux
Solaris, AIX
HP-UX
Linux
Solaris, AIX
HP-UX
Linux

4 Obtain the location of the scripts provided for extracting Oracle data files from
your instructor and record the directory.
Location of scripts:
___________________________________________________________
script_dir
5 Change to the script directory.
cd script_dir
6 Extract the Oracle datafiles by running the script for your SID, supplied in the
worksheet and recorded in the following space. The script runs tar to copy the
Oracle data files to the shared storage volume you have imported.
Script name:_________________________________
ORCL1_files.sh (for train1 train5 train9)

resource.
a Record the IP address value of your nameIP1 resource.
IP address:_________________________________
hares -display nameIP1 -attribute Address
b Edit the /etc/hosts files on all systems in the cluster to add your Oracle
SID as the host name (ORCL#) with this IP address. Coordinate edits with
your lab partner.
vi /etc/hosts
C
192.168.xxx.xxx ORCL#
192.168.xxx.xxx ORCLpartner#
Solaris/AIX/Linux
ifconfig -a
HP-UX
netstat -in

1 Use the following commands to start Oracle on your system.
su - oracle
startup
exit
same name as your SID. Use the following command (as user oracle):
lsnrctl start ORCL#
3 Stop the Listener using the following command (as user oracle):
lsnrctl stop ORCL#
4 Stop the Oracle instance using the following commands (as user oracle):

shutdown
exit
exit (back to root)
5 Switch your nameSG1 service group to their system.
6 Log in to their system and perform the same test for Oracle on their system.
rlogin their_sys
a Start Oracle.
su - oracle
startup
exit

b After the Oracle instance is started, start the Listener.
lsnrctl start ORCL#
c Stop the Listener.
lsnrctl stop ORCL#
d Stop the Oracle instance.

shutdown
exit
exit (back to root)
7 Switch your nameSG1 service group to your system.
C
8 Exit from their system.
exit

All lab participants install the Oracle enterprise agent on their own systems. If you
__________________________________________________________
install_dir
If you do not have a lab partner, install the agents on each system.
cp install_dir/VRTSvcsor.gz /tmp
3 Change to the /tmp directory.
cd /tmp
4 Unzip the compressed package file.
gunzip VRTSvcsor.tar.gz
5 Extract the compressed file from the tar file.
tar -xvf VRTSvcsor.tar
Solaris
pkgadd -d VRTSvcsor
AIX
installp -ac -d VRTSvcsor.rte.bff VRTSvcsor.rte
HP-UX
swinstall -s pkgs VRTSvcsor
Linux
rpm -ihv VRTSvcsor-2.0-Linux.i386.rpm
7 If you do not have a lab partner, install the agents on each system.

1 Use the GUI to open the cluster configuration, import the Oracle resource
types, and save the configuration.
Select File>Open configuration.

Select File>Import types.
Browse to the /etc/VRTSvcs/conf directory.
Select OracleTypes.cf.
Click Import.
Select File>Save configuration.
C
2 View the main.cf file to see the addition of the include directive for the
view main.cf
3 Check the configuration directory for VCS for the presence of the
ls /etc/VRTSvcs/conf/config

All students configure their own Oracle resources.

Required Attributes
Home /oracle
Owner oracle
Sid ORCL#
Optional Attributes
initORCL#.ora
Critical? No (0)
Enabled? Yes (1)
1 Add the nameOracle1 resource to the service group.
hares -add nameOracle1 Oracle nameSG1
hares -modify nameOracle1 Critical 0
needed.
hares -modify nameOracle1 Home /oracle

hares -modify nameOracle1 Owner oracle
hares -modify nameOracle1 Sid ORCLx
hares -modify nameOracle1 Pfile /orcl#/admin/ORCL#/
pfile/initORCL#.ora

hares -modify nameOracle1 Enabled 1
hares -online nameOracle1 -sys your_sys

hares -display nameOracle1
haconf -dump

Each student configures their own Netlsnr resources.


Required Attributes
Home /oracle
Owner oracle
Listener ORCL#
Critical? No (0)
Enabled? Yes (1)
hares -add nameListener1 Netlsnr nameSG1
hares -modify nameListener1 Critical 0

needed.
hares -modify nameListener1 Home /oracle

hares -modify nameListener1 Owner oracle
hares -modify nameListener1 TnsAdmin /oracle/network/
admin
hares -modify nameListener1 Listener ORCL#
hares -modify nameListener1 Enabled 1
hares -online nameListener1 -sys your_sys
C
hares -display nameListener1
haconf -dump

Note: Skip this section unless your class is using a version of VCS earlier than 4.0.

Required Attributes
Home /oracle
Owner oracle
Listener ORCLx
Critical? No (0)
Enabled? Yes (1)
hares -add nameListener1 Sqlnet nameSG1

needed.
hares -modify nameListener1 Home /oracle

hares -modify nameListener1 Owner oracle
hares -modify nameListener1 TnsAdmin /oracle/network/
admin
hares -modify nameListener1 Listener ORCLx
hares -online nameListener1 -sys your_sys
C
hares -display nameListener1
haconf -dump

Each student configures their own dependencies.

1 Link the Oracle and Listener resources according to the design worksheet.
hares -link nameListener1 nameOracle1

hares -link nameListener1 nameIP1
hares -link nameOracle1 nameMount1
haconf -dump

Test the service group by switching it between the systems. Then, set the Oracle
1 Switch the nameSG1 service group to their system and verify that it comes
online.
2 Switch the nameSG1 service group to your system and verify that it comes
online.
hagrp -switch nameSG1 -to your_system
C
3 Set the nameOracle1 and nameListener1 resources to critical.
hares -modify nameOracle1 Critical 1

haconf -dump
5 When all students working in the cluster are finished configuring their Oracle
service groups, close the cluster configuration.

In this lab section each student works on their own to remove the Oracle-related
haconf -makerw
2 Take the nameOracle1 and nameListener1 resources offline.
hares -offline nameListener1 -sys your_sys

hares -offline nameOracle1 -sys your_sys
3 Disable the nameOracle1 and nameListener1 resources.
hares -modify nameOracle1 Enabled 0

4 Unlink the nameOracle1 and nameListener1 resources.
hares -unlink nameOracle1 nameMount1

hares -unlink nameListener1 nameOracle1
hares -unlink nameListener1 nameIP1
5 Remove the nameOracle1 and nameListener1 resources.
hares -delete nameOracle1

hares -delete nameListener1
haconf -dump

ps -ef | grep smon

hagrp -display nameSG1
2 Log in as the oracle user on that system.
su - oracle
3 Start sqlplus as internal.
C
4 Create an Oracle user named vcs with the password vcs. Type the following
commands from sqlplus:
create user vcs

identified by vcs
default tablespace users
quota 500K on users;
5 Grant permissions to vcs while in sqlplus:
grant create session to vcs;
Oracle from VCS:
create table vcs.vcstest (tstamp date);

insert into vcs.vcstest (tstamp) values (SYSDATE);

7 Select the data in the vcs.vcstest table to verify that a row is present:
select TO_CHAR(tstamp, 'MON DD, YYYY HH:MI:SS

exit (from sqlplus)
exit (back to root)
haconf -makerw
hagrp -freeze nameSG1

Use the values in the table to configure the nameOracle1 resource to perform
detailed monitoring.

Required Attributes
Home /oracle
C
Owner oracle
SID ORCL#
Optional Attributes
initORCL#.ora
User vcs
Pword vcs
DetailMonitor 1
Table vcstest
Monscript ./bin/Oracle/SqlTest.pl
Critical? No (0)
Enabled? Yes (1)

/opt/VRTSvcs/bin/vcsencrypt -agent vcs
2 Modify the nameOracle1 resource for detail monitoring. Use the values shown
in the outlined portion of the worksheet.
Note: Use the encrypted string from the previous step when setting the Pword
attribute.
hares -modify nameOracle1 Pword drjPgrG

hares -modify nameOracle1 User vcs
hares -modify nameOracle1 DetailMonitor 1
hares -modify nameOracle1 Table vcstest
hares -modify nameOracle1 MonScript \
./bin/Oracle/SqlTest.pl
hagrp -unfreeze nameSG1
haconf -dump
interval.
su - oracle
sqlplus "/ as sysdba"
select TO_CHAR(tstamp,'MON DD, YYYY HH:MI:SS
exit (from sqlplus)
exit (back to root)

Lab 3 Solutions: Creating NFS Resources
Lab 3 Solutions: Creating NFS Resources C35

name nameSG1
nameSG1
IP1
name
Process1*
*This may be an
Application name
Mount1
completed that
lab exercise.
1. Add nameVol1
Add an
an NFS
NFS Appendix
A:Lab
LabSynopses
Synopses
resource. Appendix
AppendixB:
B:Lab
LabDetails
Details
AppendixC:
C:Lab
LabSolutions
Solutions
resource. nameDG1
resource.
Lab Solution: Creating NFS Resources


hares -display nameMount1
cluster.
grep vx /etc/name_to_major
rsh their_sys grep vx /etc/name_to_major
C
3 If the major numbers do not match:
a Select major numbers that are not in use by any other device by viewing
the /etc/name_to_major file on each system.
view /etc/name_to_major
rsh their_sys view /etc/name_to_major
b To make the major numbers the same on both systems, use the
haremajor command to change the major numbers for vxio and
vxspec.
haremajor -vx 90 91
Note: You may need to change the major numbers on more than one

Add an NFS resource to your nameSG1 service group.

Resource Type NFS
Required Attributes
Critical? No (0)
Enabled? Yes (1)
haconf -makerw
hares -add nameNFS1 NFS nameSG1
hares -modify nameNFS1 Critical 0
hares -modify nameNFS1 Enabled 1
5 Bring the resource online if it is not already online. If the NFS daemons were
started by the operating system and running, the resource is marked as online.
Note: NFS is a persistent resource type; however, it is of type on-only.
hares -display nameNFS1

hares -online nameNFS1 -sys your_sys
hares -display nameNFS1
haconf -dump

Add a Share resource to your nameSG1 service group.

Resource Type Share
Required Attributes
PathName /name1
Critical? No (0)
C
Enabled? Yes (1)
hares -add nameShare1 Share nameSG1
hares -modify nameShare1 Critical 0
3 Set the required attributes for this resource, and any optional attributes, if
needed.
hares -modify nameShare1 PathName /name1
hares -modify nameShare1 Enabled 1
hares -online nameShare1 -sys your_system

hares -display nameShare1
haconf -dump

connect.

nameIP1 nameShare1
nameShare1 nameNFS1
1 Unlink nameProcess1|App1 from nameIP1, if it is not already unlinked.
hares -unlink nameProcess1 nameIP1
hares -link nameIP1 nameShare1

hares -link nameShare1 nameMount1
hares -link nameShare1 nameNFS1
3 Test the service group by switching it between the systems.

hagrp -switch nameSG1 -to your_system
4 Verify NFS functionality from a client system using the trainer system in the
classroom, or another system outside of your cluster.
a Log in to the client system.
rlogin trainer
b Create a mount point on the client system using your name as the mount
point.
mkdir /name

c Mount the NFS file system on the trainer machine using the virtual IP
address of your nameSG1 service group.
mount 192.168.xxx.xxx:/name1 /name
d Verify that the file system mounted.
mount
5 In another window, perform a switch of the nameSG1 service group to the

other system (their_sys) and verify that the NFS mounted file system remains
mounted on the client machine.
6 Switch the nameSG1 service group back to your system and verify that the
C
NFS mounted file system remains mounted on the client machine.
7 Unmount the NFS file system on the client system.
umount /name
8 Log off of the client system.
9 Set the NFS and Share resources to critical.
hares -modify nameNFS1 Critical 1

hares -modify nameShare1 Critical 1
haconf -dump

1 Clear all faulted resources in the service group.
hagrp -clear nameSG1
haconf -makerw
3 Take the nameIP1 and nameShare1 resources offline.
hares -offline nameIP1 -sys your_sys

hares -offline nameShare1 -sys your_sys
4 Disable the nameShare1 and nameNFS1 resources.
hares -modify nameShare1 Enabled 0

hares -modify nameNFS1 Enabled 0
5 Unlink the nameIP1 and nameShare1 resources.
hares -unlink nameIP1 nameShare1
6 Unlink the nameShare1 and nameNFS1 resources.
hares -unlink nameShare1 nameNFS1
7 Unlink the nameShare1 and nameMount1 resources.
hares -unlink nameShare1 nameMount1
hares -delete nameShare1

hares -delete nameNFS1

(child).
hares -link nameProcess1 nameIP1
haconf -dump

Appendix D
Design Worksheet: Template
Cluster Interconnect Configuration
First system:
/etc/VRTSvcs/comms/llttab Sample Value Your Value

set-node
(host name)
set-cluster
(number in host name of odd
system)
link
link
/etc/VRTSvcs/comms/llthosts Sample Value Your Value
/etc/VRTSvcs/comms/sysname Sample Value Your Value
D2 VERITAS Cluster Server for UNIX, Example Application Configurations

Second system:
/etc/VRTSvcs/comms/llttab Sample Value Your Value

set-node
set-cluster
link
link
/etc/VRTSvcs/comms/llthosts Sample Value Your Value
D
/etc/VRTSvcs/comms/sysname Sample Value Your Value
Appendix D Design Worksheet: Template D3

Cluster Configuration (main.cf)
Types Definition Sample Value Your Value

Include types.cf
Cluster Definition Sample Value Your Value

Cluster
Required Attributes
UserNames
ClusterAddress
Administrators
Optional Attributes
CounterInterval
System Definition Sample Value Your Value

System
System

Service Group Definition Sample Value Your Value
Group
Required Attributes
FailoverPolicy
SystemList
Optional Attributes
AutoStartList
OnlineRetryLimit

Service Group
Resource Name
Resource Type
Required Attributes
Optional Attributes
Critical?
Enabled?

Service Group
Resource Name
Resource Type
Required Attributes
Optional Attributes
D
Critical?
Enabled?

Service Group
Resource Name
Resource Type
Required Attributes
Optional Attributes
Critical?
Enabled?

Service Group
Resource Name
Resource Type
Required Attributes
Optional Attributes
D
Critical?
Enabled?

Service Group


Ha Vcs 410 101a 2 10 srtpg3 130918134134 Phpapp02

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Ha Vcs 410 101a 2 10 srtpg3 130918134134 Phpapp02

Transféré par

Droits d'auteur :

Formats disponibles

VERITAS Cluster Server for

UNIX, Example Application

VERITAS Software Corporation

Example 1: Clustering Applications

Example 2: Clustering Databases

Example 3: Clustering NFS

Appendix A: Lab Synopses

Appendix B: Lab Details

Appendix C: Lab Solutions

Appendix D: Design Worksheet: Template

VERITAS Cluster Server Curriculum

VERITAS Cluster Server for UNIX Fundamentals

High Availability Fundamentals

VERITAS Cluster Server for UNIX, Example Application

Intro2 VERITAS Cluster Server for UNIX, Example Application Configurations

Intro4 VERITAS Cluster Server for UNIX, Example Application Configurations

Server or cluster system that has faulted

Wide area network (WAN) cloud

Intro6 VERITAS Cluster Server for UNIX, Example Application Configurations

VCS service group

Offline service group

Topic After completing this lesson, you

12 VERITAS Cluster Server for UNIX, Example Application Configurations

Application Service Overview

Example 1 Clustering Applications 13

14 VERITAS Cluster Server for UNIX, Example Application Configurations

Example 1 Clustering Applications 15

VCS Agents for Managing Applications

16 VERITAS Cluster Server for UNIX, Example Application Configurations

Key Properties of VCS Agents

Example 1 Clustering Applications 17

VCS Agent Functions

18 VERITAS Cluster Server for UNIX, Example Application Configurations

The VCS Agent Framework

Example 1 Clustering Applications 19

vxrecover -g AppDG bs AppVol

How Agents Work

110 VERITAS Cluster Server for UNIX, Example Application Configurations

The Application Agent

Example 1 Clustering Applications 111

Application Resource Definition

112 VERITAS Cluster Server for UNIX, Example Application Configurations

Application Resource Configuration

Example 1 Clustering Applications 113

114 VERITAS Cluster Server for UNIX, Example Application Configurations

Lab 1: Creating an Application Resource

Example 1 Clustering Applications 115

Topic After completing this lesson, you

22 VERITAS Cluster Server for UNIX, Example Application Configurations

Database Service Example

Example 2 Clustering Databases 23

24 VERITAS Cluster Server for UNIX, Example Application Configurations

You can also manage some databases using the

VCS Database Agents

Example 2 Clustering Databases 25

26 VERITAS Cluster Server for UNIX, Example Application Configurations

Example 2 Clustering Databases 27

Monitoring Listener Processes

28 VERITAS Cluster Server for UNIX, Example Application Configurations

Example 2 Clustering Databases 29

Database Program (Executable) Files

210 VERITAS Cluster Server for UNIX, Example Application Configurations

Example 2 Clustering Databases 211