Académique Documents
Professionnel Documents
Culture Documents
Client Connected through SCAN name of the cluster (remember all three IP addresses round robin
resolves to same Host name (SCAN Name), here in this case our scan name is cluster01-
scan.cluster01.example.com
The request reaches to DNS server in your corp and then resolves to one of the node out of three. a.
If GNS (Grid Naming service or domain is configured) that is a subdomain configured in the DNS
entry for to resolve cluster address the request will be handover to GNS (gnsd)
Here in our case assume there is no GNS, now the with the help of SCAN listeners where end points
are configured to database listener.
The following background processes are an integral part of Automatic Storage Management:
ARBn performs the actual rebalance data extent movements in an Automatic Storage Management
instance. There can be many of these processes running at a time, named ARB0, ARB1, and so on.
ASMB runs in a database instance that is using an ASM disk group. ASMB communicates with the
ASM instance, managing storage and providing statistics. ASMB can also run in the ASM instance.
ASMB runs in ASM instances when the ASMCMD cp command runs or when the database instance
first starts if the SPFILE is stored in ASM.
MARK marks ASM allocation units as stale following a missed write to an offline disk. This essentially
tracks which extents require resync for offline disks.
RBAL runs in both database and ASM instances. In the database instance, it does a global open of
ASM disks. In an ASM instance, it also coordinates rebalance activity for disk groups.
The processes described in the previous list are important for the ASM instance and should not be
modified. In addition to the processes listed in this section, there are additional processes that run in
both the ASM and database instances, such as database writer process (DBWn), log writer process
(LGWR), Process Monitor Process (PMON), and System Monitor Process (SMON).
3) The following list describes the SHUTDOWN modes and describes the behavior of the
ASM instance in each mode.
NORMAL Clause
ASM waits for any in-progress SQL to complete before performing an orderly
dismount of all of the disk groups and shutting down the ASM instance. Before the
instance is shut down, ASM waits for all of the currently connected users to
disconnect from the instance. If any database instances are connected to the ASM
instance, then the SHUTDOWN command returns an error and leaves the ASM
instance running. NORMAL is the default shutdown mode.
ASM waits for any in-progress SQL to complete before performing an orderly
dismount of all of the disk groups and shutting down the ASM instance. ASM does
not wait for users currently connected to the instance to disconnect. If any database
instances are connected to the ASM instance, then theSHUTDOWN command returns
an error and leaves the ASM instance running. Because the ASM instance does not
contain any transactions, theTRANSACTIONAL mode is the same as
the IMMEDIATE mode.
ABORT Clause
The ASM instance immediately shuts down without the orderly dismount of disk
groups. This causes recovery to occur upon the next ASM startup. If any database
instance is connected to the ASM instance, then the database instance aborts.
4) Accessing an ASM Instance
This section describes how to connect to an ASM instance. In the examples where you
provide a user name, you are prompted for a password.
Note:
The SYS user is created by default by DBCA during installation process with all three system
privileges.
Use the following statement to connect locally to an ASM instance using operating system
authentication:
sqlplus / AS SYSASM
Use the following statement to connect to an ASM instance with SYSDBA privilege:
sqlplus / AS SYSDBA
SQLPLUS /NOLOG
SQL> CONNECT SYS AS SYSASM
Enter password: sys_password
Connected to an idle instance.
SQL> STARTUP
ASM instance started
SQLPLUS /NOLOG
SQL> CONNECT SYS AS SYSASM
Enter password: sys_password
Connected.
SQL> SHUTDOWN NORMAL
Supported Default
Disk Group Type
MirroringLevels Mirroring Level
Two-wayThree-
Normal redundancy wayUnprotected Two-way
(None)
All datafiles, controlfiles, SPFIles, redo log files must reside on cluster-aware shred storage.
The Clusterware software allows nodes to communicate with each other and forms the cluster
that makes the nodes work as a single logical server.
The software is run by the Cluster Ready Services (CRS) using the Oracle Cluster Registry
(OCR) that records and maintains the cluster and node membership information and the
voting disk which acts as a tiebreaker during communication failures. Consistent heartbeat
information travels across the interconnect to the voting disk when the cluster is running.
Voting Disk
Oracle RAC uses the voting disk to manage cluster membership by way of a health check and
arbitrates cluster ownership among the instances in case of network failures. The voting disk
must reside on shared disk.
A node must be able to access more than half of the voting disks at any time.
For example, if you have 3 voting disks configured, then a node must be able to access at
least two of the voting disks at any time. If a node cannot access the minimum required
number of voting disks it is evicted, or removed, from the cluster.
11) . Can you add voting disk online? Do you need voting disk backup?
Yes, as per documentation, if you have multiple voting disk you can add online, but if you
have only one voting disk , by that cluster will be down as its lost you just need to start crs in
exclusive mode and add the votedisk using
We can verify the current backup of OCR using the following command :
ocrconfig -showbackup
USAGE:
[oracle@amrdc1dbrprod01 ~]$ ocrconfig -help
Name:
ocrconfig - Configuration tool for Oracle Cluster/Local Registry.
Synopsis:
ocrconfig [option]
option:
[-local] -export <filename>
- Export OCR/OLR contents to a file
[-local] -import <filename> - Import OCR/OLR contents from a file
[-local] -upgrade [<user> [<group>]]
- Upgrade OCR from previous version
-downgrade [-version <version string>]
- Downgrade OCR to the specified version
[-local] -backuploc { <dirname> | +<diskgroupname> }
- Configure OCR/OLR backup location
[-local] -showbackuploc - Show OCR/OLR backup location
[-local] -showbackup [auto|manual] - Show OCR/OLR backup information
[-local] -manualbackup - Perform OCR/OLR backup
[-local] -restore <filename> - Restore OCR/OLR from physical backup
-replace <current filename> -replacement <new filename>
- Replace an OCR device or file <current filename> with <new filename>
-add <filename> - Add a new OCR device/file
-delete <filename> - Remove a OCR device/file
-overwrite - Overwrite OCR configuration on disk
-repair -add <filename> | -delete <filename> | -replace <current filename> -replacement
<new filename>
- Repair OCR configuration on the local node
-copy <source_filename> <destination_filename>
- Copy OCR physical backup from source to destination
-help - Print out this help information
13) . You have lost OCR disk, what is your next step?
The cluster stack will be down due to the fact that cssd is unable to maintain the integrity, this
is true in 10g, From 11gR2 onwards, the crsd stack will be down, the hasd still up and
running. You can add the ocr back by restoring the automatic backup or import the manual
backup.
A scan listener is something that additional to node listener which listens the incoming db
connection requests from the client which got through the scan IP, it got end points
configured to node listener where it routes the db connection requests to particular node
listener.
SCAN IP can be disabled if not required. However SCAN IP is mandatory during the RAC
installation. Enabling/disabling SCAN IP is mostly used in oracle apps environment by the
concurrent manager (kind of job scheduler in oracle apps).
Steps to disable the SCAN IP,
i. Do not use SCAN IP at the client end.
ii. Stop scan listener
srvctl stop scan_listener
iii.Stop scan
srvctl stop scan (this will stop the scan vip's)
iv. Disable scan and disable scan listener
srvctl disable scan
18. How do we verify that an instance has been removed from OCR after deleting an
instance?
Issue the following srvctl command:
srvctl config database -d database_name
cd CRS_HOME/bin
./crs_stat
GRD stands for Global Resource Directory. The GES and GCS maintain records of the status
of each datafile and each cached block using global resource directory. This process is
referred to as cache fusion and helps in data integrity.
What are the background process that exists in 11gr2 and functionality?
crsd •The CRS daemon (crsd) manages cluster resources based on configuration
information that is stored in Oracle Cluster Registry (OCR) for each resource. This includes
start, stop, monitor, and failover operations. The crsd process generates events when the
status of a resource changes.(cluster ready service deamon)
occsd.bin => Manages cluster node membership runs as oragrid user.Failure of this process
results in node restart.
The cssdagent process monitors the cluster and provides input/output fencing. This service
formerly was provided by Oracle Process Monitor daemon (oprocd), also known as
OraFenceService on Windows. A cssdagent failure results in Oracle Clusterware restarting
the node.
cssdmonitor => Monitors node hangs(via oprocd functionality) and monitors OCCSD
process hangs (via oclsomon functionality) and monitors vendor clusterware(via vmon
functionality).This is the multi threaded process that runs with elavated priority.
diskmon •Disk Monitor daemon (diskmon): Monitors and performs input/output fencing
for Oracle Exadata Storage Server. As Exadata storage can be added to any Oracle RAC node
at any point in time, the diskmon daemon is always started when ocssd is started.
evmd •Event Manager (EVM): Is a background process that publishes Oracle Clusterware
events
mdnsd •Multicast domain name service (mDNS): Allows DNS requests. The mDNS
process is a background process on Linux and UNIX, and a service on Windows.
gnsd •Oracle Grid Naming Service (GNS): Is a gateway between the cluster mDNS and
external DNS servers. The GNS process performs name resolution within the cluster.
orarootagent •Oracle root agent (orarootagent): Is a specialized oraagent process that helps
CRSD manage resources owned by root, such as the network, and the Grid virtual IP address
oclskd •Cluster kill daemon (oclskd): Handles instance/node evictions requests that have
been escalated to CSS
gipcd •Grid IPC daemon (gipcd): Is a helper daemon for the communications infrastructure
RAC Processess
LMON (GESM)
This process monitors global enques and resources across the cluster and performs global
enqueue recovery operations. This is called as Global Enqueue Service Monitor.
LMD (GESD)
This process is called as global enqueue service daemon. This process manages incoming
remote resource requests within each instance.
LMS (GCS)
This process is called as Global Cache service process. This process maintains statuses of
datafiles and each cahed block by recording information in a Global Resource
Dectory(GRD).This process also controls the flow of messages to remote instances and
manages global data block access and transmits block images between the buffer caches of
different instances.This processing is a part of cache fusion feature.
LCK0(IEP)
This process is called as Instance enqueue process.This process manages non-cache fusion
resource requests such as libry and row cache requests.
The RMSn processes perform manageability tasks for RAC. Tasks accomplished by an RMSn
process include creation of resources related to Oracle RAC when new instances are added to the
clusters.
RSMN: Remote Slave Monitor manages background slave process creation and communication on
remote instances. These background slave processes perform tasks on behalf of a coordinating
process running in another instance.
In RAC ACMS per-instance process ensure a distributed SGA memory update is either globally
committed if success or globally aborted if a failure occurs.
22). Can I stop all nodes in one command? Meaning that stopping whole cluster ?
crsctl check cluster -all; will chk cluster services on all nodes
23). CRS is not starting automatically after a node reboot, what you do to make it
happen?
crsctl enable crs (as root)
to disable
Cluster commands
Stop an instance
Start an instance
Scan listener
http://oracledbabay.blogspot.de/2012/11/srvctl-commands-10g-rac-11g-r2-rac.html
http://oracledbabay.blogspot.de/2012/10/srvctl-commands-in-11gr2.html
Start/stop ASM
So you can not start/stop ASM. You need to use crsctl to start/stop cluster
Show OCR backup – the voting file (disk) is also backed up with the OCR backup
Version : 3
ID : 9093549
The Oracle Interface Configuration Tool (oifcfg) is used define and administer network
interfaces such as the public and private interfaces.
to start/stop/status of asm
To go to asmcd login with unix user(from which u hv installed cluster) & type asmcd
sqlplus / as sysasm
New Facts:
We can store OCR And Voting disk on ASM or certified cluster file system.
Voting disk and OCR can be keep in same disk-group or different disk-group
Voting disk and OCR automatic backup kept together in a single file.
Automatic backup of Voting disk and OCR happen after every four hours, end of the day, end
of the week
i) the cluster on all nodes .(it will start the ASM as well)
ii) Start the database on all nodes
In 11gR2 RAC , OCR & voting disk reside in ASM , so you can’t shutdown the
ASM directly by command
Network heartbeat:-
Disk Heartbeat:-
Each node in the cluster “pings” (r/w) the Voting Disk(s) every second
Nodes must receive a response in (long / short) diskTimeout time
– IF I/O errors indicate clear accessibility problems timeout is irrelevant
Disk heartbeat failures will lead to node evictions
CSSD-log: …
[CSSD] [1115699552] >TRACE: clssnmReadDskHeartbeat:node(2) is down.
rcfg(1) wrtcnt(1) LATS(63436584) Disk lastSeqNo(1)
Now, we know with above possibilities (network, disk heartbeat failures can lead to node
eviction, but sometime when the server/occsd/resource request also makes the node get
evicted which are extreme cases)
On node reboot/crash, the init process on the OS spawns init.ohasd (as mentioned in the
/etc/inittab file), which in turn commence Oracle High Availability Service Daemon (ohasd).
The ohasd daemon is then responsible of starting off the other critical cluster daemon
processes.
The new oraagent and oraclerootagent layers then brings up Cluster Synchronization Daemon
(cssd), Cluster Ready Services (crsd), Event Manager Daemon (evmd) and other rest of
Cluster stack in addition to ASM, RDBMS instances and other resources on the cluster at
various levels.
Do I still need to backup my Oracle Cluster Registry (OCR) and Voting Disks?
You no longer have to back up the voting disk. The voting disk data is automatically backed
up in OCR as part of any configuration change and is automatically restored to any voting
disk added. If all voting disks are corrupted, however, you can restore.
Oracle Clusterware automatically creates OCR backups every four hours. At any one time,
Oracle Database always retains the last three backup copies of OCR. The CRSD process that
creates the backups also creates and retains an OCR backup for each full day and at the end
of each week. You cannot customize the backup frequencies or the number of files that
Oracle Database retains.
Should I configure Transparent Application Failure (TAF) in my service definition if using
Fast Connection Failure (FCF)?
Do not configure Transparent Application Failover (TAF) with Fast Connection Failover
(FCF) for JDBC clients as TAF processing will interfere with FAN ONS processing.
Can I use Fast Connection Failover (FCF) and Transparent Application Failover (TAF)
together?
No. Only one of them should be used at a time.
What is the status of Fast Connection Failover (FCF) with Universal Connection Pool
(UCP)?
FCF is now deprecated along with the Implicit Connection Caching in favor of using the
Universal Connection Pool (UCP) for JDBC.
The IP change is as follows. The SCAN IP is set up in /etc/hosts (not supported by Oracle).
Name Old IP New IP
rhel6m1 192.168.0.85 192.168.0.93
rhel6m2 192.168.0.86 192.168.0.94
rhel6m-scan 192.168.0.91 192.168.0.92
The steps could be done parallel (on all nodes) or in a rolling fashion on each node. Steps executed on
one node is shown here and changes were done parallel on all nodes. Since the information modified
is stored in OCR it is better to take a manual backup of the OCR before proceeding.
3. Verify VIP resource is down by using ifconfig on the OS and using crsctl
crsctl stat res -t
ora.rhel6m1.vip
OFFLINE OFFLINE
ora.rhel6m2.vip
OFFLINE OFFLINE
4. Add the new IP information to /etc/hosts (and DNS) and proceed to modifying the VIP. The modify
command must be run as root user
[root@rhel6m1 grid]#
srvctl modify nodeapps -n rhel6m1 –A 192.168.0.97/255.255.255.0/eth0
If the VIP is now on a different network then do not start VIP until public IP and SCAN IP is also
changed into the same network.
6. Stop the SCAN IP and the scan lister and verify it is stopped.
[root@rhel6m1 grid]# srvctl stop scan_listener
9. Updating public IP requires no cluster related work. Shutdown the cluster stack
[root@rhel6m1 grid]# crsctl stop crs
10. Change IP address on OS (eg. editing ifcfg-eth*) files and restart the network for the changes to
take effect. Once the IP changes are made it maybe worthwhile to do ssh between nodes using the new
IP so it is added to known host list and no prompts are shown during the ssh between the nodes. As
the final step restart the cluster stack with
# crsctl start crs
Rename SCAN
$ nslookup ol6-112-scan2
Server: 192.168.0.4
Address: 192.168.0.4#53
Name: ol6-112-scan2.localdomain
Address: 192.168.0.117
Name: ol6-112-scan2.localdomain
Address: 192.168.0.116
Name: ol6-112-scan2.localdomain
Address: 192.168.0.115
Display the current configuration using the following command as the oracle/grid
user.
$ export GRID_HOME=/u01/app/11.2.0.3/grid
$ $GRID_HOME/bin/srvctl config scan
SCAN name: ol6-112-scan, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /ol6-112-scan/192.168.0.115
SCAN VIP name: scan2, IP: /ol6-112-scan/192.168.0.116
SCAN VIP name: scan3, IP: /ol6-112-scan/192.168.0.117
$
Turn off the SCAN and SCAN listeners using the following commands as the
oracle/grid user.
$ export GRID_HOME=/u01/app/11.2.0.3/grid
$ $GRID_HOME/bin/srvctl stop scan_listener
$ $GRID_HOME/bin/srvctl stop scan
Modify the SCAN name using the following commands as the root user.
# export GRID_HOME=/u01/app/11.2.0.3/grid
# $GRID_HOME/bin/ srvctl modify scan -n ol6-112-scan2
# $GRID_HOME/bin/ crsctl modify type ora.scan_vip.type -attr
"ATTRIBUTE=SCAN_NAME, DEFAULT_VALUE=ol6-112-scan2"
Turn on the SCAN and SCAN listeners again using the following commands as
the oracle/grid user.
My db version : 11.2.0.3
My Operating System : AIX 7.1
My servers hostname : node1-node2
My database name : TEST01
My instance name : TEST011-TEST012
Here is the some basic commands, for commands details&options please review Reference docs:
I will not paste result because output is not clear in that page
You can use below command which is not recommended for 11g and which is depreciated
[oracle@node1]</home/oracle> crs_stat -t
Name Type Target State Host
————————————————————
ora….DATA.dg ora….up.type ONLINE ONLINE node1
ora….ER.lsnr ora….er.type ONLINE ONLINE node1
ora….N1.lsnr ora….er.type ONLINE ONLINE node1
ora….N2.lsnr ora….er.type ONLINE ONLINE node2
ora.ORADATA.dg ora….up.type ONLINE ONLINE node1
ora.asm ora.asm.type ONLINE ONLINE node1
ora.cvu ora.cvu.type ONLINE ONLINE node2
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora….SM1.asm application ONLINE ONLINE node1
ora….11.lsnr application ONLINE ONLINE node1
ora….b11.gsd application OFFLINE OFFLINE
ora….b11.ons application ONLINE ONLINE node1
ora….b11.vip ora….t1.type ONLINE ONLINE node1
ora….SM2.asm application ONLINE ONLINE node2
ora….12.lsnr application ONLINE ONLINE node2
ora….b12.gsd application OFFLINE OFFLINE
ora….b12.ons application ONLINE ONLINE node2
ora….b12.vip ora….t1.type ONLINE ONLINE node2
ora….network ora….rk.type ONLINE ONLINE node1
ora.oc4j ora.oc4j.type ONLINE ONLINE node2
ora.ons ora.ons.type ONLINE ONLINE node1
ora.test01.db ora….se.type ONLINE ONLINE node1
ora….int.svc ora….ce.type ONLINE ONLINE node2
ora….int.svc ora….ce.type ONLINE ONLINE node2
ora….kis.svc ora….ce.type ONLINE ONLINE node2
ora….est.svc ora….ce.type ONLINE ONLINE node1
ora….ry.acfs ora….fs.type ONLINE ONLINE node1
ora.scan1.vip ora….ip.type ONLINE ONLINE node1
ora.scan2.vip ora….ip.type ONLINE ONLINE node2
To can see current settings for Oracle High Availability Services stack when the server boots up,
follow:
or
[root@node1]cat /etc/oracle/scls_scr/node1/root/ohasdstr
enable
So as you can see my current setting is enable.If your system shown disable than :
For Disable:
[root@node1]crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.
# cat /etc/oracle/scls_scr/node1/root/ohasdstr
disable
For Enable:
[root@node1]crsctl enable has
CRS-4621: Oracle High Availability Services autostart is enabled.
PS: db_name is the name of the databasethis command is starting all the instances
PS: db_name is the name of the databasethis command is starting all the instances
Stop ASM
To start
cd $GRID_HOME/bin
./crsctl stop cluster -all
When an instance crashes in a single node database on startup a crash recovery takes place. In a
RAC enviornment the same recovery for an instance is performed by the surviving nodes called
Instance recovery.
How do you determine what protocol is being used for Interconnect traffic?
One of the ways is to look at the database alert log for the time period when the database was started
up.
What methods are available to keep the time synchronized on all nodes in the cluster?
Either the Network Time Protocol(NTP) can be configured or in 11gr2, Cluster Time Synchronization
Where does the Clusterware write when there is a network or Storage missed heartbeat?
You can query the V$ACTIVE_INSTANCES view to determine the member instances of the RAC
cluster.
Is there an easy way to verify the inventory for all remote nodes
You can run the opatch lsinventory -all_nodes command from a single node to look at the inventory
Q What are the modes of deleting instances from ORacle Real Application cluster
Databases?
We can delete instances using silent mode or interactive mode using DBCA(Database Configuration
Assistant).
OLR
In Oracle Clusterware 11g Release 2 an additional component related to the OCR called the Oracle
Local Registry (OLR) is installed on each node in the cluster. The OLR is a local registry for node
specific resources. THe OLR is not shared by other nodes in the cluster. It is installed and configured
when Oracle clusterware is installed.
Purpose of OLR
————–
It is the very first file that is accessed to startup clusterware when OCR is stored on ASM. OCR
should be accessible to find out the resources which need to be started on a node. If OCR is on ASM,
it can’t be read until ASM (which itself is a resource for the node and this information is stored in
OCR) is up. To resolve this problem, information about the resources which need to be started on a
node is stored in an operating system file which is called Oracle Local Registry or OLR. Since OLR is
a file an operating system file, it can be accessed by various processes on the node for read/write
irrespective of the status of the clusterware (up/down). Hence, when a node joins the cluster, OLR
on that node is read, various resources ,including ASM are started on the node . Once ASM is up ,
OCR is accessible and is used henceforth to manage all the clusterware resources. If OLR is
missing or corrupted, clusterware can’t be started on that node!
To restore OLR:
# crsctl stop crs
# ocrconfig -local -restore file_name
# ocrcheck -local
# crsctl start crs
$ cluvfy comp olr
You can view the status of the OLR file on each node by using the ocrcheck command with the –local
parameter as seen here:
#ocrcheck -local
With Oracle grid infrastructure 11g release 2 (11.2), Oracle Automatic Storage Management (Oracle ASM)
and Oracle Clusterware are installed into a single home directory, which is referred to as the Grid
Infrastructure home
In 11.2 grid infrastructure, the Oracle Clusterware Component Log Files are all
situated in the $GRID_HOME/log/<hostname>
For instance, if my host name is tzdev1rac, all my Oracle Clusterware Component Log
Files (for 11.2) are situated in $GRID_HOME/log/tzdev1rac:
pwd
/oracle/grid/11.2/log/tzdev1rac ($GRID_HOME = /oracle/grid/11.2)
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit
Production
With the Partitioning, OLAP, Data Mining and Real Application Testing
options
STATUS
------------
STARTED
SQL>
Also check you have available free space your disk groups?