Académique Documents
Professionnel Documents
Culture Documents
Flash Freeze
As the Oracle RDBMS continues to grow with new and complex functionality, and high
availability requirements are mandatory, efficient debugging and diagnostics becomes crucial.
As systems continue to scale and require 99.999% uptime, it is imperative that diagnosability
tools are secure, fast, and provide all the data necessary to determine root cause the first time a
fault occurs.
In previous versions, a flash freeze could only be initiated manually using ORADEBUG. This
required that someone be available at the time an error occurred to freeze the database
immediately. In an event-driven flash freeze, the freeze is performed automatically when a
predefined condition occurs.
The events triggering an automatic flash freeze are defined by a privileged user, either with
EVENT specifications in the initialization parameter file or with ALTER SYSTEM SET
EVENT commands. The triggering events include
• Internal errors, that is, any ORA-00600 error
• External events, typically driven from operating system scripts which would also
initiate a manual freeze
• External errors, that is, any ORA-xxxxx error, where xxxxx is any error number other
than 00600
SQL>
SQL> ALTER
ALTER SYSTEM
SYSTEM SET
SET EVENT=
EVENT=
22 "600
"600 flashfreeze
flashfreeze on
on error
error 1301,proc=PMON";
1301,proc=PMON";
System
System altered.
altered.
SQL>
SQL> ALTER
ALTER SYSTEM
SYSTEM SET
SET EVENT=
EVENT=
22 "10295
"10295 flashfreeze,proc=BGS,instance=single";
flashfreeze,proc=BGS,instance=single";
System
System altered.
altered.
Examples
The examples show events being set with the ALTER SYSTEM command. The same results
would occur if the event were set in the initialization parameter file. The following command
would trigger a flash freeze whenever internal error 1301 occurs in PMON:
SQL> ALTER SYSTEM SET EVENT =
2 "600 flashfreeze on error 1301, proc=PMON";
The following command would trigger a flash freeze in any background processes whenever
event 10295 occurs in a background process:
SQL> ALTER SYSTEM SET EVENT = "10295 flashfreeze, proc=BGS";
The full syntax of the EVENT clause is as follows:
<event name><action>{:<event name><action>}
where:
• <event name> is a symbolic name for the event, or an optional event number
• <action> is <action keyword><action qualifiers>
and
<action keyword> is "trace"|"debugger"|"crash"|"flashfreeze"
Examples (continued)
The action qualifier for the action "flashfreeze" is ""|"off"|"on <error tag>"
|"after <N> times"|"proc = all | fgs | bgs | <pid> | <pname>"
where
• "" does nothing because flash freeze requires a qualifier
• "off" disables flash freeze action for this event
• "on <error tag>" where <error tag> is "error <first argument
of internal error>"
• "after <N> times" causes flash freeze after <n> occurrences of this event
• "proc = all | fgs | bgs | <pid> | <pname>" where all means
all processes; bgs means all background processes; fgs means all foreground
processes; <pid> is an Oracle process ID; and <pname> is an Oracle process name
such as LMON, PMON, LMD0, and so on.
• Debugger operations
– Requires flash-frozen instance
– All Oradebug functionality availability
– SQL queries may be issued without impacting the
SGA, for example, the Shared Pool
– SQL queries for debug can be executed without
latches
• COW features
– Copies on write
– Modifications to the SGA as a result of any debug
operation do not affect the frozen SGA
– Implemented by making a process-private copy of
the modified page with the debugger process
Command-line Tools
GET_SGA_SNAPSHOT: for the online debugging of flash-frozen instances.
Online debugging of flash-frozen SGA requires that the current SGA contents be mapped into
a file before the high-level SQL commands are issued. This is needed to enable the Copy-On-
Write debugging, which guarantees that the current SGA contents is not destroyed. Therefore,
in order to set up the COW debugging on the frozen system’s SGA, GET_SGA_SNAPSHOT
command must be used first inside an Oradebug session:
SQL> oradebug
SQL> GET_SGA_SNAPSHOT 'filename'
where filename is a name of the file to which SGA is mapped. This command writes the
SGA to the specified filename using full access path. It then unmaps the SGA and remaps the
SGA in a copy-on-write mode. If the instance is not flash frozen, the command will return an
error message with status to indicate that a flash freeze must be in effect prior to issuing this
command. If the filename already exists or the disk is out of space, the appropriate messages
are flagged. The start of the SGA file contains special header information with a checksum to
identify it as the SGA. Non-SGA files (with incorrect headers) will not be mapped. Similarly,
files without access permission will not be mapped.
GET_SGA_OFFLINE: for the off-line debugging of SGA snapshots written to a file.
Off-line debugging of the SGA snapshot is performed by first saving SGA contents into a file,
and then reading it into a memory segment. In order to set up off-line debugging using this file,
GET_SGA_OFFLINE command must be used first inside the Mapsga utility:
%MAPSGA GET_SGA_OFFLINE 'filename'
where filename is a name of the file to which SGA has been saved.
%sqlplus
%sqlplus "/
"/ AS
AS sysdba"
sysdba"
SQL>
SQL> REM
REM Invoke
Invoke flash
flash freeze
freeze
SQL>
SQL> oradebug
oradebug ffbegin;
ffbegin;
SQL>
SQL> REM
REM Create
Create view
view private
private SGA
SGA
SQL>
SQL> oradebug dmpcowsga "/dbs/memdump";
oradebug dmpcowsga "/dbs/memdump";
SQL>
SQL> REM
REM Now
Now select
select from
from fixed
fixed tables
tables
SQL>
SQL> SELECT
SELECT ** FROM
FROM v$bh;
v$bh;
%mapsga
%mapsga get_sga_offline
get_sga_offline /dbs/memdump;
/dbs/memdump;
%sqlplus
%sqlplus "/
"/ AS
AS sysdba"
sysdba"
SQL>
SQL> REM
REM Now
Now select
select from
from fixed
fixed tables
tables
SQL>
SQL> SELECT
SELECT ** FROM
FROM v$bh;
v$bh;
Usage Rules
Not all SQL commands are possible in the debugging mode. Generally speaking, the user can
issue the same queries as are possible today when a database is not opened. More precisely, the
user will be able to do the following:
• Queries on fixed tables/views with already mentioned limitations.
• Dump individual data blocks using ALTER SYSTEM DUMP DATAFILEcommand.
• Dump control file information using ALTER SYSTEM DUMP CONTROLFILE
command.
• The entire set of Oradebug and flash freeze commands will be available.
Current Limitations
Some of the limitations of the COW debugger come from the basic server infrastructure, others
from the current implementation. The intrinsic limitation is that not all SGA structures are
viewable through the fixed tables/views, and even those which are may only be partially
represented.
The implementation-specific limitations include:
• Swap space can be exhausted during cursor compilation, if, for example, all objects in
the shared pool are pinned at the time of freeze and a large number of SGA pages are
accessed. In general, the copy-on write implementation uses swap only for the pages
that are modified, so there are only a few cases when swap space might run out.
• Compiling cursors into the shared pool changes the shared pool contents. It is quite
possible that the new state of the shared pool will not contain the clues that were
present in the original state. However, since the COW debugger only creates the local
view, which can be discarded and restarted.
• Some queries containing ORDER BY/GROUP BY clauses will not work. Memory is
used as much as possible, even beyond SORT_AREA_SIZE, but there may be cases
when it is not be possible to avoid the use of temporary space. In these cases, the
queries will not work.
• Some of the structures which are traversed either when compiling SQL or when
executing it may be in a corrupted state, because the SGA was in an arbitrary state
when it was frozen/dumped. This may cause compilation/execution to fail. Latch
recovery callbacks may be invoked at some appropriate points of execution, but it may
not always be enough.
• HP and Windows implementation of this functionality is not currently available.
On-line SGA
toolset Read and
Trace buffer B[i] Process
modify
P[i]
commands
Trace file
for P[i]
Off-line
review
Trace file
for P[j]
Copyright © Oracle Corporation, 2001. All rights reserved.
Tracing Tracing
DIAG DIAG
process process
SGA SGA
Copyright © Oracle Corporation, 2001. All rights reserved.
• Message ordering
• Communication model
– General features
– Protocols
• Process-adaptive timeout mechanism
• Master node election
Message Ordering
When a node wants to send a message to the multiple nodes in a broadcast or multicast
manner, it sends the messages to the master node which forwards the message to the
appropriate destination nodes. If multiple messages are received by the master, it puts the
messages in a local first-in, first-out (FIFO) queue.
Communication Model
General Features
When the first Oracle9i Real Application Cluster instance starts up, a DIAG process group is
created. Each time an new instance starts up, its DIAG process joins the DIAG process group.
A DIAG group reconfiguration is initiated by the group member who discovers the change in
membership so that all the existing group members can update with the latest membership
information.
A separate port, in additional to those used by LMD and LMON, is created for each DIAG and
its information is published to other remote DIAG processes.
Protocols
There are two types of communication protocols, namely port-based IPC and memory-mapped
copying, used for inter-instance communication between DIAGs. The port-based IPC
mechanism is targeted for message passing between the DIAG processes in a broadcast or
multicast manner. Memory-mapped operation is used for passing large amounts of data which
would tax the IPC mechanism.
Process-Adaptive Timeout Mechanism
Communication problems or hanging processes may delay message passing between nodes.
Since message processing times may vary among nodes, the largest delay will be chosen as a
final timeout value to determine whether a node is hanging.
Oracle9i
Oracle9i Enterprise
Enterprise Edition
Edition Release
Release 9.0.0.0.0
9.0.0.0.0 –– Beta
Beta
With
With the
the Partitioning
Partitioning and
and Real
Real Application
Application Clusters
Clusters options
options
JServer
JServer Release
Release 9.0.0.0.0
9.0.0.0.0 –– Beta
Beta
ORACLE_HOME
ORACLE_HOME == /ade/ilam_rdbms_lrg/oracle
/ade/ilam_rdbms_lrg/oracle
System
System name:
name: SunOS
SunOS
Node
Node name:
name: dlsun1932
dlsun1932
Release:
Release: 5.6
5.6
Version:
Version: Generic_105181-14
Generic_105181-14
Machine:
Machine: sun4u
sun4u
Instance
Instance name:
name: lrg
lrg
Oracle
Oracle process
process number:
number: 44
Unix
Unix process
process pid:
pid: 24026,
24026, image:
image: oracle@dlsun1932
oracle@dlsun1932 (LMON)
(LMON)
• Oradebug
• Requirements
– Non-intrusive
– Should not hang or crash the database
instance while in use
– Must continue to work even if one or more
nodes join or leave the cluster
– Use a well-defined user interface
• _trace_enabled
• _trace_archive
• _trace_buffers
_trace_buffers
_trace_buffers == "LMD0:100;LMON:75;FGS:50"
"LMD0:100;LMON:75;FGS:50"
_trace_enabled
Specifies whether the always-on tracing feature is enabled for the system or not. This is a
Boolean parameter with a default value of TRUE.
_trace_archive
Specifies whether the trace archive mode (logging to file) is on or not. This is a Boolean
parameter with a default value of FALSE.
_trace_buffers
Specifies the number of in-memory trace buffers to be configured for each process in the
system. The syntax of this string parameter is :
_trace_buffers = "proc-spec:size;proc-spec:size;..."
where,
proc-spec = proc | proc, proc-spec
proc = ALL | FGS | BGS | pid | pid - pid | pname
ALL = all the processes
BGS = all background processes
FGS = all foreground processes
pid = oracle process ID
pname = well-known Oracle process names such as LMON, PMON, and so on
size is an integer specifying the number of trace data records that should be allocated for the
process specified by the corresponding proc-spec parameter.
For example:
_trace_buffers = "LMD0:100;LMON:75;FGS:50"
The default value is "ALL:256", corresponding to all processes with 256 trace records.
• _trace_flush_processes
• _trace_file_size
• _trace_events
• _trace_options
_trace_flush_processes
Specifies the processes where flushing is enabled. Their trace buffers are logged periodically
to their respective trace files, by default, when the buffer is half full. The argument for this
parameter, proc-spec, was defined previously and has a default value of "ALL".
_trace_file_size
Specifies the size (in bytes) which the per process trace file can assume on the disk before
wrapping occurs. This parameter takes any positive integer for a value; 65536 is the default.
_trace_events
Specifies the trace events with levels and the processes for which these must be enabled for
trace collection. The syntax is
_trace_events = "event-spec:level:proc-spec"
where
event-spec = event | event, event-spec
event = ALL | event-id | event-id - event-id
event-id = 10000 – 10999
level = 0 – 255. All trace events with a level less than or equal to this number will be logged
in the trace buffer.
proc-spec is as described earlier
The default value is an empty string (" ") indicating that no events are enabled.
_trace_options
Specifies the trace dump file format (binary or text) and whether multiple trace files (per
process) will be generated. The syntax is
_trace_options = "text | binary, single | multiple"
where the first option specifies whether the dump file contains text or binary data, and the
second option determines if there is one trace file generated per process (multiple) or one per
instance (single). The default value is "text,multiple".
ALTER
ALTER TRACING
TRACING OFF;
OFF;
ALTER
ALTER TRACING
TRACING ENABLE
ENABLE "ALL:4:BGS";
"ALL:4:BGS";
ALTER
ALTER TRACING
TRACING ON;
ON;
ALTER
ALTER TRACING
TRACING DISABLE
DISABLE "10000-10500";
"10000-10500";
ALTER
ALTER TRACING
TRACING FLUSH
FLUSH "ALL";
"ALL";
Dynamic Views
Two fixed tables, X$TRACE and X$TRACE_EVENTS were added to Oracle9i to support the
new Real Application Clusters diagnosability features.
X$TRACE
This view shows all the traces collected in the trace buffers till the time of query. It can be
used to view the trace snapshots for a process or opcode or any other combination of trace
attributes. The columns in this view are as follows:
• event— event ID
• op— op code
• time— time at which the event was recorded
• seq#— system-wide unique identifier for this recorded event
• sid— session id of this event
• pid— Oracle process number
• data— related data for this event (implementation related)
X$TRACE_EVENTS
This view lists the currently enabled trace events and their attributes. It has the following
columns:
• event— event ID
• trclevel— tracing level at which this event is enabled
• status— 0 (disable) or 1 (enabled) for this event
• procs— which Oracle process for this event is enabled
Features
The Oracle Cluster Analysis Utility is designed for use by Windows system administrators or
database administrators who want to install and run Oracle9i Real Application Clusters
software for the first time. It performs a discovery of all the Windows cluster components
needed to support Real Application Clusters to determine if they are available.
During the analysis of the cluster, the utility will collect other data that can be of value when
planning and designing a database to run on the cluster. This includes information about the
available cluster interconnects and the size and names of any raw partitions (raw partitions are
needed for the database files in a Real Application Clusters database).
The utility will identify any missing components necessary for the support of a Real
Application Clusters database. However, if your analysis reports a properly configured cluster,
you are not guaranteed a successful installation of Real Application Clusters software and
configuration of a database. This is because other factors not analyzed by the Oracle Cluster
Analysis Utility, such as available disk and memory space, are also pertinent.
You can install the Oracle Cluster Analysis Utility independently of any other Oracle9i
software, so you can perform your analysis before running the Oracle Universal Installer to
install and configure your database software.
Usage Model
The utility can be run from the command line on any of the nodes in a cluster, by running the
clustercheck.exe file from the preinstall area. This executable, in turn, uses the
ocvfile.exe file to collect information from each node in the cluster.
The Oracle Cluster Analysis Utility provides the following:
• Complete cluster analysis from single node in cluster.
• Quick identification of working cluster. The utility analyzes the basic components and
a successful run identifies the cluster as functioning properly.
• If the program encounters an error, the severity and possible fixes are reported.
Limitations
The cluster check utility works only on Windows. It generates reports only on storage devices
connected to SCSI or Fiber Channel. Also, it only reports on TCP and VIA based networks.
The nodes in the cluster have to be typed in initially for the utility. The utility cannot
automatically discover the nodes in the cluster.
Dynamics
The node from where the utility is run acts as the coordinator node. The coordinator node
starts the Oracle Cluster Verifier service on all the nodes in the cluster (including itself). This
service collects information related to the storage devices and network interconnects present
on the node. The information collected by the Oracle Cluster Verifier service is stored inside
the opsm directory under a Temp directory of the node on which the service is running.
After populating the information file, the the Oracle Cluster Verifier service reports to the
coordinator node whether it was successful or not in collecting the information. When the
coordinator node hears back from all the nodes in the cluster, it decides whether the cluster is
in a state to support Oracle Real Application Clusters and displays its decision.
The services and the clustercheck utility use log files while doing their job. These logs files are
stored in the same location as the information files, which is the opsm subdirectory under
Temp directory.
Machine--------->INTELOPS1
Machine--------->INTELOPS1
IP-------------->144.25.190.128
IP-------------->144.25.190.128
Num
Num ofof Net
Net Crds->2
Crds->2
.. .. ..
Number
Number ofof Drives------>4
Drives------>4
Disk
Disk Number--->0
Number--->0
Bad
Bad Disk------>0
Disk------>0
Drive
Drive Signtr-->254434909
Signtr-->254434909
Total
Total Size------>8183MB
Size------>8183MB
Num
Num Part------>33
Part------>33
Partition
Partition Number---->1
Number---->1
Partition
Partition Letter---->None
Letter---->None
Partition
Partition Format---->6
Format---->6
Total
Total Size------------>274MB
Size------------>274MB
Free
Free Space---------->0Kbytes
Space---------->0Kbytes
Machine--------->INTELOPS1
IP-------------->144.25.190.128
Num of Net Crds->2
Number of Drives------>4
Disk Number--->0
Bad Disk------>0
Drive Signtr-->254434909
Total Size------>8183MB
Num Part------>33
Partition Number---->1
Partition Letter---->None
Partition Format---->6
Total Size------------>274MB
Free Space---------->0Kbytes
Partition Number---->2
. . .
Disk Number--->1
. . .