Spotlight On Oracle Getting Started

Spotlight
on Oracle 9.5
Getting Started Guide
2013 Quest Software, Inc.
ALL RIGHTS RESERVED.
This guide contains proprietary information protected by copyright. The software described in
this guide is furnished under a software license or nondisclosure agreement. This software may be
used or copied only in accordance with the terms of the applicable agreement. No part of this
guide may be reproduced or transmitted in any form or by any means, electronic or mechanical,
including photocopying and recording for any purpose other than the purchasers personal use
without the written permission of Quest Software, Inc.
The information in this document is provided in connection with Quest products. No license,
express or implied, by estoppel or otherwise, to any intellectual property right is granted by this
document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN
QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR
THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS
ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-
INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT,
INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES
(INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS
INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR
INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect
to the accuracy or completeness of the contents of this document and reserves the right to make
changes to specifications and product descriptions at any time without notice. Quest does not
make any commitment to update the information contained in this document.
If you have any questions regarding your potential use of this material, contact:
Quest Software World Headquarters
LEGAL Dept
5 Polaris Way
Aliso Viejo, CA 92656
email: legal@quest.com
Refer to our Web site (www.quest.com) for regional and international office information.
Patents
Protected by U.S. Patents #6,901,582, 7,784,027; 8,181,154; 8,015,454 and 8,347,148.
Additional patents pending.
Trademarks
Quest, Quest Software, the Quest Software logo and Spotlight are trademarks and registered
trademarks of Quest Software, Inc in the United States of America and other countries. For a
complete list of Quest Softwares trademarks, please see
http://www.quest.com/legal/trademark-information.aspx. Other trademarks and registered
trademarks are property of their respective owners.
Spotlight 9.5
Getting Started Guide
Friday, 31 May 2013
Table of Contents
Welcome 9
Oracle Architecture Overview 9
Spotlight on Oracle Overview 10
Home Page 10
Alarms 11
Drilldowns 11
Features of Spotlight 12
Features of Spotlight on Oracle 12
Predictive Diagnostics 12
DTrace Analysis (Unix Solaris 10 And Later) 13
Install Spotlight 14
Requirements 14
Steps To Install 15
Multiple Concurrent Installations 16
MSI Installers for Spotlight on Oracle 16
Uninstall Spotlight 16
Start Spotlight 17
Project Lucy 17
User Experience Feedback 18
Authorize Spotlight 18
Spotlight Users 19
Oracle User Wizard 19
Spotlight on Oracle 22
Connect to Oracle Instance 22
Add A New Connection 22
Spotlight on Oracle Getting Started Guide
Table of Contents
5
Configuring Oracle User for Spotlight Dialog 26
Toad DBA Suite for Oracle 27
Install And Configure DTrace Scripts 28
Spotlight on Oracle Home Page 29
Sessions Panel 30
Host Panel 30
Server Processes Panel 31
SGAPanel 32
Background Processes Panel 33
Disk Storage Panel 34
Spotlight on Oracle Alarms 34
Spotlight on Oracle Drilldowns 41
Spotlight Tools 45
Tools |Explain Plan 45
Tools |SQLOptimizer 45
Tools |Space Manager 46
Tune Your Database 46
Troubleshooting 48
Connection Problems 48
Dead Sessions in Top Sessions Drilldown 49
Data That Does Not Appear in The History Browser 49
Predict Database Performance 50
Using Predictive Diagnostics 50
Predictive Diagnostics And Spotlight 51
Requirements for Predictive Diagnostics 52
Troubleshooting 53
Predictive Diagnostics Unavailable 53
Table of Contents
6
Spotlight on Oracle RAC 54
What is Spotlight on Oracle RAC? 54
What Is Oracle RAC? 54
Connect to Spotlight on Oracle RAC 55
Spotlight on Oracle RAC Home Page 60
Interconnect and Global Status Panel 61
Instance Panels 62
IO Subsystem Panel 63
Exadata Storage Panel 63
Spotlight on Oracle RAC Alarms 63
Alarms on A Database Instance 64
Alarms on The Oracle RAC 64
Spotlight on Oracle RAC Drilldowns 67
Troubleshooting 69
Average Ping Time Chart Not Populated 69
Spotlight on Windows 71
Connect to Windows Systems 71
Add A New Connection 71
Spotlight on Windows Home Page 73
System Panel 73
Network Panel 73
Event Log Panel 74
CPUPanel 74
Memory Panel 75
Paging Files Panel 75
Disks Panel 76
Files Panel 76
Table of Contents
7
Spotlight on Windows Alarms 76
Spotlight on Windows Drilldowns 77
Troubleshooting Spotlight on Windows 78
Issues Connecting 78
Issues Monitoring a Windows Machine 81
Spotlight on Unix 84
Configure the Unix System 84
Connect to Unix Systems 86
Add a new connection 86
Spotlight on Unix Home Page 88
System Panel 88
Network Panel 88
CPUPanel 89
Memory Panel 90
Disk Activity Panel 90
Swap Space Panel 91
Spotlight on Unix Alarms 91
Spotlight on Unix Drilldowns 93
Spotlight on MySQL 95
Connect to a MySQLDatabase 95
Spotlight on MySQL Home Page 97
Sessions Panel 97
MySQL Panel 98
InnoDB Panel 99
Storage Panel 100
Host Panel 100
Spotlight on MySQL Alarms 101
Table of Contents
8
Spotlight on MySQL Drilldowns 102
Troubleshooting 103
Monitoring 103
Disabled Controls 103
Alarms Not Raised 104
Delay in Reporting Alarms 105
Alarm Help Is Displayed Instead of The Drilldown 105
Spotlight Home Page Hard To Read 105
NoChange to Data Flows 106
Operating System Connection / Statistics 106
I Cannot Open a Migrated Connection 108
Spotlight Consumes Too Many Resources on Either The Client Or The Server 108
Support 109
Using Trace To Add Details To Support Bundle 109
Build the Support Bundle 109
Appendix: Contact Quest 111
Contact Quest Support 111
Spotlight Support 111
Contact Quest Software 112
About Quest Software 112
Index 113
Welcome
Spotlight on Oracle is Quest Softwares powerful diagnostic and problem-resolution tool for
Oracle databases. Its unique user interface provides you with an intuitive, visual representation of
the activity on your database and operating system.
Oracle Architecture Overview
The following diagram illustrates some of the basic components of an Oracle server at the
memory, database and disk layers. This architecture has been used to design the Spotlight on
Oracle home page.
The numbers indicate the order of the flow of information.
The numbered labels in the Oracle architecture diagram correspond to the following activities:
1. The client program (for example, SQL*PLUS, Oracle Power Objects, or some other tool)
sends a SELECT statement to the server process.
2. The server process looks in the shared pool for a matching SQL statement. If none is
found, the server process parses the SQL and inserts the SQL statement into the
shared pool.
3. The server process looks in the buffer cache for the data blocks required. If found, the
data block must be moved on to the most recently used end of the Least Recently
Used (LRU) list.
1
Welcome
10
4. If the block cannot be found in the buffer cache the server process must fetch it from the
disk file. This requires a disk I/O.
5. The server process returns the rows retrieved to the client process. This may involve some
network or communications delay.
6. When the client issues the UPDATE statement the process of parsing the SQL and
retrieving the rows to be updated must occur. The update statement then changes the
relevant blocks in shared memory and updates entries in the rollback segment buffers.
7. The update statement also makes an entry in the redo log buffer that records the
transaction details.
8. The database-writer background process copies modified blocks from the buffer cache to
the database files. The Oracle session performing the update does not have to wait for
this to occur.
9. When the COMMIT statement is issued the log writer process must copy the contents of
the redo log buffer to the redo log file. The COMMIT statement does not return control to
the Oracle session issuing the commit until this write is complete.
10. If running in ARCHIVELOG mode, the archiver process copies full redo logs to the
archive destination. A redo log is not eligible for re-use until it has been archived.
11. At regular intervals, or when a redo log switch occurs, Oracle performs a checkpoint. A
checkpoint requires all modified blocks in the buffer cache to be written to disk. A redo
log file cannot be re-used until the checkpoint completes.
Recommended Reading
The best information to help you tune and configure Oracle databases can be found online. Use
as your first point of reference web sites such as:
l The Quest Software Oracle Solutions Web site.
l The Oracle home page.
Spotlight on Oracle Overview
Spotlight on Oracle makes it possible for you to view your database at different levels of detail.
Home Page
The Spotlight home page is a visual representation of the status of the major components of the
database. The detail at this level is designed to help you locate and identify bottlenecks as they
develop. Additionally, Spotlight provides tools that allow you to identify and rectify
inefficiency in your databases configuration.
If Spotlight detects that some component of the Oracle database constitutes a performance
bottleneck, the corresponding component generates a visual or auditory alarm. The exact
appearance of the alarm depends on how you have configured alarm severities. When an alarm is
current you can press F1 to display help for the alarm. Clicking the component (on the home
page ) displays the relevant drilldown.
Welcome
11
See "Spotlight on Oracle Home Page" (page 29) for more information.
The home page display is based on the Oracle database architecture. See "Oracle Architecture
Overview" (page 9) for more information.
Alarms
Spotlight alarms indicate when the Oracle database exceeds acceptable performance thresholds.
See "Spotlight on Oracle Alarms" (page 34) for more information.
Drilldowns
Once a problem is isolated you can display a drilldown page with charts and tables that provide
a detailed breakdown of the underlying statistics.
For example, use the Activity drilldown to display overall efficiency and resource usage. In
particular, the Event waits chart shows the amount of time database sessions spend waiting for
various resources, as well as the amount of CPU being utilized. Efforts at tuning the database are
most likely to succeed when they concentrate on the resources being used most heavily and the
Wait activity topic contains a discussion of the meaning and implications of various wait events.
Other pages help you examine detailed activity. For instance, the Top Sessions drilldown
identifies the users who contribute most to database activity. You can display various details for
these sessions, including the SQL statements, locks and resource usage.
See "Spotlight on Oracle Drilldowns" (page 41) for more information.
Welcome
12
Features of Spotlight
Features of Spotlight on Oracle
Spotlight is designed for ease of installation and use, and offers a range of tools for
problem diagnosis.
Some of the main features and benefits of Spotlight on Oracle are that it:
l Presents a visual representation of process flows within an Oracle instance, allowing you
to observe actual database activity in real time.
l Visually identifies bottlenecks and provides extensive drilldown capabilities.
l Displays the details of problem areas, including most active sessions, inefficient SQL,
locks & latches, and disk I/O, for rapid problem resolution.
l Provides visual and audible warnings to alert you when performance metrics exceed
acceptable thresholds.
l Provides detailed information about a specific component allowing you to pinpoint the
source of problems.
l Learns the normal range of values for your database.
l Has a quick and simple installation process.
l Can be launched from Quest Softwares Foglight for 24x7, unattended monitoring of
systems, databases, and applications.
l Has UTF8 support.
Contact Quest Software's support team if you need more information than is supplied in this
help system.
Predictive Diagnostics
Predictive Diagnostics allows Spotlight to collect and analyze performance metrics for:
l SQL statements as they are executed on the database instance.
l System bottlenecks as they affect database throughput and response time.
l Database resources as they reach their full capacity and affect database throughput and
concurrency.
When enough performance data is collected, Spotlight may be able to predict the future
performance for individual SQL statements on that database, for potential bottlenecks in the
system, and for database CPU, memory, and disk I/O resources.
Note: When you start using Predictive Diagnostics, Spotlight will take about 10 days to collect
enough data to make valid predictions on the future performance of the database.
Welcome
13
DTrace Analysis (Unix Solaris 10 And Later)
DTrace is a dynamic analysis tool for Sun Microsystems operating systems. It was first
introduced in the Solaris 10 operating system, and can be used ONLY on Sun platforms from
Solaris 10 onwards.
Spotlight and DTrace
Spotlight on Oracle uses DTrace to capture details of the interaction between Oracle and the
operating system.
When Oracle sessions use CPU time, most of that elapsed time is spent executing Oracle code.
However, Oracle sometimes uses system calls to perform functions such as:
l Allocating or deallocating private (PGA) memory.
l Synchronizing activities with other processes through semaphores or other IPC
mechanisms.
l Allocating shared memory (SGA).
l Performing I/O operations on datafiles, redo logs, raw devices, and so on.
l Capturing CPU or elapsed timing to support the time model and wait interfaces.
Spotlight uses DTrace to investigate these system calls to calculate where that elapsed time is
being spent.
Install and Configure DTrace scripts
Before you can capture the details of the interaction between Oracle and the Solaris operating
system, you need to Install And Configure DTrace Scripts (page 28).
Install Spotlight
Requirements
Windows 8 / Windows 7 / Windows Vista / Windows Server 2012 and 2008 :
Section Requirement
User Privileges Ensure the Windows user installing Spotlight has
Windows Administrator privileges.
Folder Permissions During installation, Spotlight requires write access to
the following:
l The Spotlight installation folder (by default,
Program Files\Quest Software\Spotlight)
l Program Files\Common Files\Quest Shared
l ProgramData\Quest Software\Spotlight
l Users\<current user>\AppData\Roaming\Quest
Software\Spotlight
Note:For 64-bit operating systems the Program Files
directory is Program Files (x86).
Permissions to Windows
registry keys and sub-keys
During installation, Spotlight requires write access to
the following:
l HKEY_LOCAL_
MACHINE\SOFTWARE\Wow6432Node\Quest
Software\Products
l HKEY_LOCAL_
MACHINE\SOFTWARE\Wow6432Node\Quest
Software\Spotlight
l HKEY_CLASSES_ROOT
2
Install Spotlight
15
Windows XP / Windows Server 2003:
Section Requirement
User Privileges Ensure the Windows user installing Spotlight has
Windows Administrator privileges.
Folder Permissions During installation, Spotlight requires write access to
the following:
l The Spotlight installation folder (by default,
Program Files\Quest Software\Spotlight)
l Program Files\Common Files\Quest Shared
l Documents and Settings\All Users\Application
Data\Quest Software\Spotlight
l Documents and Settings\<current
user>\Application Data\Quest
Software\Spotlight
Permissions to Windows registry
keys and sub-keys
During installation, Spotlight requires write access to
the following:
l HKEY_LOCAL_MACHINE\SOFTWARE\Quest
Software\Products
l HKEY_LOCAL_MACHINE\SOFTWARE\Quest
Software\Spotlight
l HKEY_CLASSES_ROOT
Steps To Install
To install Spotlight from CD:
1. Close all open applications on your PC.
2. Insert the Spotlight Installer CD into your CD-ROM drive.
If the installation program does not start select Windows Start | Run
l For Spotlight delivered as part of the Toad DBA Suite, enter the command
d:\TOAD DBA Suite for Oracle vvv.exe (where d: is the CD-ROM identifier and
vvv is the Spotlight version).
l For Spotlight delivered as a standalone installer, enter the command
d:\SpotlightonOracle_vvv.exe (where d: is the CD-ROM identifier and vvv is the
Spotlight version).
3. Follow the instructions on the screen to complete the installation.
Install Spotlight
16
To install Spotlight from the downloaded Quest Spotlight installer from the Quest Web site:
1. Close all open applications on your PC.
2. Execute the downloaded installer.
3. Follow the instructions on the screen to complete the installation.
Note:To cancel installation at any time, press Esc or click Cancel.
Multiple Concurrent Installations
There has been a major shift in the architecture from Spotlight on Oracle 9.5. You can install
major and minor versions of Spotlight on Oracle side by side (or upgrade). Patch and progressive
builds will upgrade.
Note:If Spotlight is delivered as part of the Toad DBA Suite and you require multiple Spotlight
installations on your computer then contact Quest Software for more information.
MSI Installers for Spotlight on Oracle
MSI installers are available for Spotlight on Oracle. These installers can be run silently, and
produce an installation that can be captured and re-used via automatic deployment techniques.
Contact Quest Software for more information.
Uninstall Spotlight
To remove the application and all associated files from your machine, if required.
To uninstall Spotlight
1. Open Windows Control Panel.
2. Select Programs and Features. (On Windows XP select Add/Remove Programs.)
3. Click Spotlight | Remove | Yes.
Start Spotlight
17
Start Spotlight
To start Spotlight:
Double click on the Spotlight desktop icon
or
On the Windows task bar, click Start | Programs |Quest Software | Spotlight |
Spotlight.
During operation
Spotlight requires write access to the following:
Windows Server 2012, Windows Server 2008,
Windows Server 2003, Windows 8, Windows 7,
Windows Vista
Users\<current
user>\AppData\Roaming\Quest
Software\Shared
Users\<current
user>\AppData\Roaming\Quest
Software\Spotlight
Windows XP Documents and Settings\<current
Software\Shared
Documents and Settings\<current
Software\Spotlight
You should also verify that you can run the required Oracle client on the Spotlight machine.
Project Lucy
The first time you run Spotlight following install you will be prompted to upload data to Project
Lucy. Project Lucy is a free service provided by Quest Software that analyzes your performance
data, gives you reports and insight into your performance and allows you to compare your
experience to everyone else that uses the service. Performance data is uploaded to Project Lucy
once a day.
Start Spotlight
18
Actions Description
To open the Project Lucy home page, in the Spotlight
Browser click Welcome to Spotlight | Spotlight on the
Web | Project Lucy.
l Learn more about
Project Lucy and how
it can benefit you.
l Login to your Project
Lucy account and use
Project Lucy services.
l Create a Project Lucy
account.
To configure uploading data from Spotlight on Oracle to
Project Lucy, click View | Options | Project Lucy.
See "Project Lucy" in the
online help for more
information.
User Experience Feedback
You can influence the design of future versions of Spotlight and help us improve its quality,
reliability and performance. To participate or change your mind select Help | User
Experience Feedback.
Authorize Spotlight
Spotlight is supplied with a time-limited trial key so you may test the product. When the trial
key expires Spotlight reverts to a preview mode, which limits access to the Spotlight home page
and one connection at a time. If you select to open a drilldown in preview mode the following
message is displayed:
The current license does not allow access to this functionality.
Contact your Quest Software representative to obtain the necessary authorization keys to ensure
Spotlight remains fully functional when the trial key expires.
To enter a new authorization key:
While Spotlight is running
1. Click Help | About Spotlight.
2. Ensure Installed Products is to the front.
3. Select View / change product license.
4. Click Change this license.
5. Enter the Authorization key.
Notes:
Start Spotlight
19
l The Windows user that updates the authorization key must have administrator access to
the Windows registry and write access to the Console\Licenses sub-folder of the
Spotlight installation folder.
l Make sure you enter the authorization key exactly as stated on your Product
Authorization sheet or as provided by your Quest Software distributor. The site message
is case sensitive. If you enter the authorization key incorrectly, the following message is
displayed:Invalid authorization key
l If Spotlight is in preview mode and the trial period has not expired, check your computer
system date is set correctly.
Spotlight Users
Spotlight on Oracle connects to the Oracle database system through the Spotlight user.
l You should not use any system accounts. We recommend creating a separate user for
Spotlight or using a personal account.
l The Spotlight user must have sufficient privileges, views, and tables to diagnose the
Oracle database. You can view the privileges, views and tables required by Spotlight
users in the Database Objects and Privileges document in the Spotlight installation
folder: \Plug-ins\SoO\.
The Oracle User Wizard configures Spotlight users.
Oracle User Wizard
The Oracle User Wizard configures Spotlight users. Use the Oracle User Wizard to:
l Add a new user to the Oracle database
l Convert the account of an existing user on the Oracle database to use Spotlight
l Reconfigure an existing Spotlight user to work with new versions of Spotlight on
Oracle and Spotlight on Oracle RAC. Use this procedure if you have upgraded your
version of Spotlight.
Notes:
l Spotlight users must be configured for each database they are to diagnose.
l You cannot configure Spotlight users for a database while Spotlight is diagnosing
that database.
l You must have the user ID and password of the DBA that manages the database to which
you are adding the Spotlight user.
l Select Don't show this page in future to hide the first page of the Oracle User Wizard.
To fill in the Oracle User Wizard
The Oracle User Wizard runs automatically when required or to initiate it:
Start Spotlight
20
1. Click File | Oracle User Wizard...
2. At the DBA connection credentials page:
Field Description
Enter the
SQL*Net
connection
string
Enter the database instance on which you wish to create the user.
Enter the user
name of a
DBA
account
Enter a DBA user name for that database instance (so that the
Spotlight user and the relevant objects can be created). The user
name requires SYSDBA access.
Enter the
password
Enter the DBA user password.
3. Select Create a new user
4. At the New oracle user page:
Field Description
User name Enter a name for the new user.
Password Enter a password for the new user.
Confirm password Confirm the password.
Select any table Select to allow the user to generate explain plans.
Alter system Select to allow the user to trace and terminate sessions.
5. At the Select tablespace page:
Field Description
Spotlight tables The default tablespace is USERS, if available.
temporary segments The default temporary tablespace is TEMP, if available.
6. At the Predictive Diagnostic Settings page:
Select Create/Upgrade Predictive Diagnostics schema to employ Predictive Diagnostics
on this database instance.
7. Select to use an existing Predictive Diagnostics schema (recommended) or install
a new schema.
Start Spotlight
21
8. Select to install a set of sample data (which you can use to experiment with the Predictive
Diagnostics feature).
9. Select the settings Spotlight will use to schedule the Oracle jobs that collect data on SQL
performance and impending bottlenecks.
Spotlight on Oracle
Connect to Oracle Instance
1. From the Spotlight Browser select All Connections |Spotlight on Oracle
2. Select the database. If the database is not on the list then add a new connection.
Notes: (Now the database is listed in the Spotlight Browser |Live connections. .)
l Click Set timed statistics now if you are prompted to do so. You see this prompt
when for the Spotlight user the timed_statistics parameter is set to false in the
init.ora file. Spotlight requires this parameter be true for the Spotlight user to
accurately monitor the database. Clicking Set timed statistics now corrects the
problem only if the Spotlight user has Alter System privileges. Ensure the
Spotlight user has sufficient privileges. See "Spotlight Users" (page 19) for more
information.
l Spotlight operates at reduced functionality once the initial trial period has expired.
See "Authorize Spotlight" (page 18) for more information.
Add A New Connection
Notes: (Before you add a new connection)
l Ensure you create a Spotlight user with sufficient privileges to diagnose the Oracle
database. See "Spotlight Users" (page 19) for more information.
l Ensure the Unix or Windows server on which the Oracle database is installed is
accessible to Spotlight on Oracle. See the Spotlight on Unix or Spotlight on Windows
Getting Started Guide.
3
Spotlight on Oracle
23
1. Click File |Connect
2. Select Spotlight on Oracle on the Connections menu.
3. Double-click Add new connection.
4. Ensure Details is selected. Fill in the Properties window.
Oracle connection details
Field Description
Connection
String
Select the name of the database that you wish to connect to.
Note: When selected, the Connection name field resets to this value.
You can then edit the Connection name field if required.
Oracle
Username
Enter the Spotlight user name. See "Spotlight Users" (page 19) for
more information.
Tip: If you enter a Username that does not exist, Spotlight will
prompt you when you complete this dialog to create a new Spotlight
user.
Oracle
Password
Enter the Spotlight user password. See "Spotlight Users" (page 19) for
more information.
Oracle
Home
Select an appropriate Oracle Home. Select a location where Oracle
client software is installed.
Note:Choose a home that contains a version of the Oracle client that
is compatible with the Oracle server.
ASM connection details
ASM is built into the Oracle database kernel. It simplifies the creation and space
management in Oracle databases.
Note: Available only in Oracle 10g and later.
Spotlight on Oracle
24
Field Description
Monitor
ASM
Select to monitor the performance of Oracle's Automated Storage
Management (ASM) technology.
ASM
Connection
String
Enter the connection string required to link to the ASM database
instance that is used for storage management for this Oracle database.
Note:The TNSNAMES entry for the ASM instance must include the
option (UR = A) for the Spotlight connection to work properly.
ASM
Password
Enter the user password for Spotlight to login to the ASM database
instance.
The user that Spotlight uses to login to the ASM database is SYS.
This is the only user name that can provide access to ASM-specific
information.
Providing SYS access to an ASM instance does NOT create a security
issue for the Oracle database, but you should limit access to the SYS
account to trusted individuals.
Because the SYS password is stored in encrypted format, non-
privileged users can still access Spotlight on Oracle. To do so:
l Use a pre-configured or shared Spotlight on Oracle
connection, OR
l Use a Spotlight connection without ASM access. Spotlight
will display Oracle-related data, but the ASM drilldown will
contain no data.
Server connection details
Field Description
Monitor OS Select to monitor the operating system on the database server.
OS Server Enter the name of the server.
Tip: Use a period . to monitor the local database.
Spotlight on Oracle
25
Field Description
OS Type Select the type of operating system.
l Windows
l Unix(REXEC)
l Unix(SSH)
l Unix(SSH using Public/Private Keys)
If you select one of the Unix systems, make sure the SSH or REXEC
daemon is running on the database server and is configured to
receive remote connections. Spotlight supports both SSH1 and SSH2
protocols. Public-key encryption is supported under SSH2 only.
SSH Port Enter the port number for SSH (secure shell) connection to the
database server. The default value is 22.
Note: Applicable when OSType is Unix(SSH) or Unix(SSH using
Public/Private Keys).
OS
Username
Type the user name to login to the database server.
Note: When OSType is Windows:
l Not required if your login details have been assigned remote
access to the registry of the database server.
l Ensure the user has access to the registry of the server.
SSH Key
Type
Select the key type from the options:RSA and DSA.
Note: Applicable when OSType is Unix(SSH using Public/Private
Keys).
SSH Private
Key
Filename
Locate the filename that contains the private key for the Spotlight
connection.
Hint: Use the "..." button.
Keys).
SSH
Passphrase
Type the passphrase used to decrypt the private key.
Keys).
OS
Password
Type the user password to login to the database server (if required).
Note: Not applicable when OSType is Unix(SSH using
Connect to
OS
Spotlight
Select to have the database server connection open automatically
when you open the connection to the Oracle database.
Spotlight on Oracle
26
Field Description
when
starting this
connection
ESX connection details
Note: Requires Server connection details | Monitor OS selected.
Field Description
Monitor
ESX
Select to monitor the virtual machine that hosts the database.
ESX
Web
Service
Type or select the URL of the ESX Web Service that manages the
virtual machine.
ESX
Username
Type the username required to connect to the ESX Web Service.
Note: The username specified must have read only permissions for both
the ESX virtual machine and the ESX host that the virtual machine
resides on, or may reside on.
ESX
Password
Type the password required to connect the specified user to the ESX
Web Service.
Virtual
Machine
Name
Click ... to select the virtual machine.
5. Click Test to verify Spotlight can connect successfully with the details provided.
Note:For connections to virtual machines, testing the connection validates that Spotlight
can connect to the Web Service using the specified username and password combination,
and that the virtual machine exists. It does not check permissions, that the virtual machine
is turned on, or that it is the same machine as the one specified in the Server Connection
Details section.
6. Click Save password details (for this connection) to save all the entered
password details.
Configuring Oracle User for Spotlight Dialog
Note:This dialog is applicable to Oracle 11g Release 2 and above.
Spotlight on Oracle
27
This dialog will open from Spotlight Browser | All connections or the Connection Details
dialog when selecting to monitor an Oracle or Oracle RAC connection, and the setup of the
Spotlight User through the Oracle User Wizard is incomplete.
Field Description
Install Objects in
my schema
Click to install Spotlight objects in your schema. This automates the
running of the Oracle User Wizard (page 19) up to Predictive
Diagnostic Settings if you have DBAprivileges. If you do not have
DBAprivileges you will be prompted for a DBAlogin.
Configure another
user
Click to configure another user to connect to this Oracle instance. To
complete these steps you require SYSDBAprivileges.
Show connection
details
Click to view or edit the connection details. Open the Connection
Details dialog. See "Connect to Oracle Instance" (page 22) for more
information.
View information
about objects
installed and
privileges required
by spotlight
Click to open the Spotlight on Oracle Database Objects and
Privileges document.
Toad DBA Suite for Oracle
Note: This section is relevant for systems with the following software installed: Toad DBASuite
for Oracle with Spotlight on Oracle.
To connect to an Oracle database from within Toad
1. Click the Spotlight button where it appears in Toad.
2. Spotlight on Oracle opens at the Oracle database connection associated with the Toad for
Oracle connection.
The first time you open the Oracle database connection from within Toad
You are required to fill in the Details page with the following additions. See "Connect to Oracle
Instance" (page 22) for more information.
l The Oracle database and username fields are pre filled in as per the Toad connection.
Security requires that you re-enter your Oracle password.
l The default connection name is Toad followed by the name of the Oracle database. You
may choose to replace this with any connection name of your choice. Note that a
limitation of the integration of Toad with Spotlight on Oracle is that this is a new
Spotlight connection, ignoring all other pre-existing Spotlight connections.
Spotlight on Oracle
28
l You may choose a different Oracle username to connect to Spotlight. For example, in an
environment with multiple database administrators there may be a communal username for
Spotlight connections. Simply replace the filled in Oracle username with the appropriate
Oracle username and password details.
l You can enter additional parameters to monitor ASM, the operating system and for
virtualization.
Using Spotlight on Oracle
l Spotlight opens at the Home Page or Drilldown relevant to the location in Toad from
where Spotlight was opened.
l You are prompted to create a new Spotlight user if the Oracle username has not been
created in Spotlight. See "Spotlight Users" (page 19) for more information.
l This Spotlight connection can be found in the Spotlight Connection Manager. It can be
managed the same as any other Spotlight connection. For example, you can close the
connection, edit the connection details and re-open the connection in the Spotlight
Connection Manager.
Caution:When you click on the Spotlight button in Toad to connect to the Oracle database,
a new Spotlight connection to the Oracle database is created, regardless of any pre-existing
Spotlight connections to the Oracle database. To avoid the annoyance of setting up Spotlight
connections in Spotlight that will be set up identically through Toad later, you may choose
to create all connections through Toad if Toad integration is desired.
Install And Configure DTrace Scripts
DTrace is a dynamic analysis tool for Sun platforms. Spotlight on Oracle uses DTrace to capture
details of the interaction between Oracle and the operating system (Solaris 10 onwards).
To use DTrace within Spotlight, you need to install and configure a number of DTrace scripts on
the server. Once installed, use the DTrace drilldown to view the DTrace analysis. See "Spotlight
on Oracle Drilldowns" (page 41) for more information.
To install and configure DTrace scripts
1. Locate in the Spotlight installation folder: Plug-ins\SoO\SoO_DTrace_scripts.tar
2. Unpack SoO_DTrace_scripts.tar to a directory on the database server.
3. Ensure the unpacked files have execute permission.
4. For the account that will be used to run operating system commands from Spotlight:
l Edit .profile to define a QUEST_SOO_HOME variable that points to the directory on
the database server where the unpacked files are.
l Provide DTrace authority. Add this line to /etc/user_attr:
<<username>>::::defaultpriv=basic,dtrace_proc,dtrace_kernel
Spotlight on Oracle
29
Spotlight on Oracle Home Page
The Spotlight home page shows the flow of information and commands between various sub-
components and the size and status of internal resources such as processes, disk files and memory
structures.
Related statistics are grouped together on panels that are connected by a series of graphical flows
and icons. Spotlight updates these flows in real time so you can see how quickly data is moving
through the system. The icons change color as Spotlight alarms are raised, upgraded, downgraded
and canceled.
The panels on the Spotlight on Oracle home page are described in more detail next. For a full
discussion of the panels, see the Spotlight online help.
Tip: Hover the mouse pointer over a panel component for more information.
Spotlight on Oracle
30
Sessions Panel
The Sessions panel indicates the status of the Oracle communications
between client and server. Information displayed here includes:
l The time (in milliseconds) that it takes for the query select user from
dual to be submitted, executed and returned.
l The total number of users connected to the database.
l The number of users who are currently active.
l The sum of DB time over all sessions, divided by elapsed time.
Note: The Average Active Sessions gauge is available for (Oracle 10g and
later).
The flows between the Sessions panel and the Server Processes panel represent the rate of data
transfer in both directions across the SQL*NET interface.
Host Panel
The Host panel shows metrics for the operating system:
l The total amount of CPU being used.
l The number of tasks waiting for CPU resources.
l The percentage of CPU that is unavailable to the virtual machine (if
applicable).
l The available physical RAM.
l The amount of memory that has been reclaimed from the virtual
machine by ESX (if applicable).
Spotlight on Oracle
31
Server Processes Panel
The Server Processes panel indicates the status of Oracle server processes
that perform database activities on behalf of end users, and mediate database
connections:
l Statistics for the PGA memory allocated by server processes.
l Dedicated server processes that perform work on behalf of a single
client process.
l Shared or multi-threaded servers (MTS) that perform work on behalf
of more than one client process.
l MTS dispatchers that coordinate the allocation of shared servers to
client tasks.
l Server processes that support parallel execution of queries.
l The current length of the Parallel Execution Queue.
Note: The Parallel Exec Queue gauge applies only to (Oracle 11g
Release 2).
l Job queue server processes that run PL/SQL commands submitted to
the Oracle job queue via the DBMS_JOB package.
l The state of the EM agent connection between Oracle Enterprise
Manager (if installed) and the database.
l The number of server processes that are currently active.
The flow to the Server Processes panel from the Disk Storage panel represents the rate of direct
physical reads from disk.
The flows between the Server Processes panel and the SGA panel represent the rates of:
l Blocks reads from the SGA by all server processes.
l Block changes in the SGA by all server processes.
l Redo buffer entries made by all server processes.
l SQL parse requests per second by all server processes.
l SQL execution requests per second by all server processes.
Spotlight on Oracle
32
SGAPanel
The SGA panel shows details of specific memory areas within the system
global area (SGA). The SGA is an area of shared or common process memory
that is used to cache frequently used data, SQL statements, procedures and
other structures:
l The size of the buffer cache and its components. These hold data that
is accessed frequently.
l The current size of the SGA.
l The size of the Java Pool, which caches class definitions, Java
methods and Java objects.
l The size of the Large Pool allocation heap, which is used in MTS for
session memory. It can be used by parallel execution and backup
processes.
l The size of the redo buffer. This contains redo entries that are
eventually written to the redo log.
l The size of the Shared Pool, which caches SQL statements, PL/SQL
programs, object definitions and session memory for MTS sessions.
l The size and activity of the result cache (Oracle 11g and later).
l Whether the database is in manual or auto memory management
mode (Oracle 10g and later).
The flow between the SGA panel and the Disk Storage panel represents the rate of block
reads from disk.
The flows between the SGA panel and the Background Processes panel represent the rates of:
l Block writes by the Database writer (DBWR) process.
l I/O writes by the Recovery writer (DBWR) process.
l Redo log block writes by the Redo Log Writer (LGWR) process.
Spotlight on Oracle
33
Background Processes Panel
The Background Processes panel displays the following key Oracle
background processes:
l Database Writer (DBWRn).
l The Recovery Writer (RVWRn).
l The Log Writer (LGWRn).
l The Archiver (ARCHn).
l Data Guard Overhead and Apply Lag
l Predictive Diagnostics.
Note: The n value in each process icon indicates the number of processes
there are of that type.
The flows between the Background Processes panel and the Disk Storage
panel represent:
l The rate at which modified blocks are written from the SGA to disk
by the DBWR processes.
l The rate of I/O write operations to the flash recovery area by the
RVWR processes.
l The rate at which redo log entries are written to the redo log files by
the LGWR processes.
l The rate at which the archiver copies from redo log files to archived
logs.
The Background Processes panel also contains a status indicator for
Predictive Diagnostics (if installed). See "Using Predictive Diagnostics"
(page 50) for more information.
Spotlight on Oracle
34
Disk Storage Panel
The Disk Storage panel represents Oracle database and log files on disk,
including:
l Oracle database files.
l Size of the flash cache.
l Oracle flashback recovery files.
l Oracle redo logs.
l The archive log destinations.
Right-click the Disk Storage icon to open Space Manager. See "Tools
|Space Manager" (page 46) for more information.
Spotlight on Oracle Alarms
Spotlight alerts you to problems with your system by issuing an alarm. You can configure
Spotlight in the level of severity that constitutes an alarm, to disable an alarm, and the actions
Spotlight takes on raising the alarm.
When an alarm is raised Spotlight performs one or more of the following actions:
l Changes the color or intensity of relevant components.
l Gives audible warnings.
l Sends email notifications.
Actions you can take on an alarm being raised include:
l View details of the statistics that caused the alarm in a Spotlight drilldown page. See
"Spotlight on Oracle Drilldowns" (page 41) for more information.
l View the Spotlight online help.
l View details of the alarm in the Spotlight Alarm Log drilldown.
l Snooze the alarm.
Spotlight on Oracle
35
l Save the alarm details.
l Filter the list of alarms.
Following are the alarms specific to Spotlight on Oracle. For possible solutions to the problems
indicated by these alarms or for information on how to diagnose problems further, see Spotlight
on Oracle alarms in the online help.
Alarm Description
Agent Status alarm Spotlight raises the Agent Status alarm when the status of the EM
agent is down.
OEM is a Web-based database administration and monitoring tool
that manages Oracle components and administers Oracle databases.
When installed and active, OEM communicates with the databases it
administers via a set of EM agent processes under the automated
DBSNMP user account.
Archive Critical
Failure alarm
An Archive Critical Failure alarm indicates that the number of
working archive destinations for the database instance is about to fall
below the minimum number needed to archive Oracle redo logs
successfully. When you see this alarm, you should make space in the
archive log destination by purging or by backing up to an offline
medium any archived files that are not immediately needed.
Archive
Destination Failed
alarm
Spotlight raises an Archive Destination Failed alarm when at least
one of the archive destinations specified for redo log files has become
full. See Archive Destination Failure alarm below.
Archive
Destination Failure
alarm
An Archive Destination Failure alarm when at least one of the
archive destinations specified for redo log files is about to become
full. When you see this alarm, you should make space in the archive
log destination by purging it, or by backing it up to an offline
medium.
Array Fetch Size
alarm
When more than one row is retrieved from an Oracle database in a
single SELECT (an array fetch), the amount of network traffic
associated with SELECT operations is radically reduced. The Array
Fetch Size alarm is raised when the array fetch size for the database
may be too low.
ASM Balance
alarm
This alarm indicates that there is unbalanced access to one or more of
the disks in the ASM (Automatic Storage Management) disk group
that has been allocated to the Oracle instance.
ASM Service Time
alarm
This alarm indicates that the average time taken to perform I/O on the
ASM instance has exceeded a specified threshold value. (Service time
represents the response time of a disk plus any wait time experienced
during disk operations.)
Spotlight on Oracle
36
Alarm Description
ASM Status alarm When a Spotlight connection is lost, Spotlight automatically tries to
re-establish the connection. The ASM Status alarm indicates that
Spotlight is re-establishing its connection to the ASM diskgroup for
the Oracle instance.
Average Time to
Sync a Redo Log
Entry alarm
The Average Time to Sync a Redo Log Entry alarm (formerly the
Average Redo Write Time alarm) is activated when the time taken to
write redo log entries exceeds a threshold. Unlike most other Oracle
write I/Os, Oracle sessions must wait for redo log writes to complete
before they can continue processing.
Buffer Busy Wait
alarm
The Buffer Busy Wait alarm occurs when a session cannot access a
block because it is in use by another session. The two most common
causes are insufficient free lists for a table or insufficient rollback
segments.
Cache Buffer
Chains Latch Wait
alarm
The Cache Buffer Chains Latch Wait alarm occurs because
contention for the cache buffer chains latch exceeds a threshold.
Cache Buffer LRU
Chains Latch Wait
alarm
The Cache Buffer LRU Chains Latch Wait alarm occurs because
contention for the cache buffer chains latch exceeds a threshold.
Collection Status
alarm
Spotlight raises the Collection Status alarm when collection of data
for the OEM (Oracle Enterprise Manager) stops. Perhaps the server is
running low on disk space.
The Collection Status alarm often occurs with the Files Pending
Upload Alarm.
Continued Fetch
Rate alarm
Oracle sometimes needs to migrate a row in a database from its
original block to a new block, or needs to chain an updated row
across multiple blocks. These migrated and chained rows cause
additional I/O, and the Continued Fetch alarm occurs when the
number of these "continued row" fetches exceeds a threshold.
CPU Busy alarm The CPU Busy alarm occurs when the total CPU utilization of the
system exceeds a threshold.
CPU Virtualization
Overhead alarm
The CPU Virtualization Overhead alarm is activated when
Virtualization Overhead is greater than a threshold and CPU Usage is
95% or higher.
Datafile Read
Time alarm
The Datafile Read Time alarm occurs if the average time for a random
datafile read exceeds a given threshold.
Data Guard The Data Guard Configuration Alarm occurs on a Primary database in
Spotlight on Oracle
37
Alarm Description
Configuration
Alarm
an Oracle Data Guard environment. Please supply details of Standby
connections to Spotlight so Spotlight can monitor the Oracle Data
Guard environment. In monitoring the Oracle Data Guard
environment Spotlight can alert you if key elements of the Oracle
Data Guard environment are not working, or if Standby instances are
lagging behind.
Dispatchers Busy
alarm
The Dispatchers Busy alarm occurs when all or most dispatcher
processes are busy. In an MTS environment, multiple sessions share a
smaller number of shared server processes. The servers are usually
allocated by a smaller number of dispatcher processes.
Estimated Shared
Pool Size alarm
This alarm occurs when the Shared Pool is set too small. A possible
cause is the use of literals rather than bind variables in SQL
statements. You can also increase the size of the Shared Pool.
Excessive RBS
Activities alarm
The Excessive RBS Activities alarm occurs when there are high
rollback extent/shrink/wrap activities.
Files Pending
Upload alarm
Spotlight raises the Files Pending Upload alarm when a significant
number of XML files are pending upload indicating a problem with
the upload process. The strategy for dealing with this alarm is
dependent on the OEM control. The OEMcontrol is defined in the
Spotlight configuration for OEM.
This alarm may occur in conjunction with the Collection Status
Alarm.
Flash Cache Insert
Skip: DBWR
Overloaded alarm
The Flash Cache Insert Skip: DBWR Overloaded alarm is raised
when the ratio of insert skips from the DBWR being overloaded
exceeds a threshold.
Flash Cache Read
Latency
The Flash Cache Read Latency alarm is raised when the time taken to
read a single block from flash cache exceeds a threshold. This may be
caused by a low performance flash device.
Flash Cache Write
Complete Waits
The Flash Cache Write Complete Waits alarm is raised when the ratio
of 'write complete waits: flash cache' to DB time exceeds a threshold.
Flashback Buffer
Wait alarm
The Flashback Buffer Wait alarm indicates that the percentage of
available time spent waiting for the Recovery Writer (RVWR) to
write data to the flashback logs has exceeded a specified threshold.
Flashback Space
Usage alarm
This alarm is raised when the amount of used space in the Flashback
Recovery Area exceeds a threshold.
Free Buffer Waits
alarm
The Free Buffer Waits alarm occurs when free buffer waits (as a
proportion of total waits) exceeds a threshold.
Spotlight on Oracle
38
Alarm Description
In Memory Undo
Latch Wait alarm
The in memory undo latch is associated with Oracle's relatively new
in memory undo (IMU) structures. These structures hold in memory
information formerly maintained in rollback (undo) segments.
Instance Failed
alarm
Spotlight raises an Instance Failed alarm when a specified instance
has failed because of an Archiver error. See Archive Critical Failure
alarm above.
Insufficient
Flashback Logs
alarm
This alarm fires when the flashback logs in the flashback area do not
cover the amount of time specified in the parameter DB_
FLASHBACK_RETENTION_TARGET.
Job Processes Busy
alarm
The Job Processes Busy alarm occurs when all or most job queue
processes are busy.
Large Pool Full
alarm
This alarm fires when the Large Pool allocation heap is full. The
Large Pool is used in MTS for session memory, and can be used by
parallel execution and backup processes.
Latch Free Waits
alarm
If a session needs to acquire a latch that is held by another session, a
latch free wait may occur. The presence of latch free waits may
indicate a bottleneck within the SGA. (The specific action depends
on the latch.)
Library Cache
Latches alarms
These alarm fire when the time spent waiting on library cache latches
or library cache pin latches exceeds a threshold. Library cache latches
(or mutexes in 10g R2 and above) protect the cached SQL statements
and object definitions held in the library cache within the Shared
Pool.
Library Cache
Miss Rate alarm
The Library Cache Miss Rate alarm occurs when the library cache hit
ratio falls below a given threshold.
Lock Wait alarm The Lock Wait alarm occurs when the proportion of time sessions
spend waiting for locks exceeds a threshold.
Log Buffer Space
Wait alarm
The Log Buffer Space Wait alarm occurs if the amount of time spent
waiting for space in the redo log buffer exceeds a threshold.
Log Switch Time
alarm
The Log Switch Time alarm occurs when the amount of time spent
waiting for log switch events exceeds a threshold.
Low Free Physical
RAM alarm
The Low Free Physical RAM alarm occurs when the servers
available RAM is low.
Maximum Queued
Time alarm
The Maximum Queued Time alarm becomes active when a session
has been in the parallel execution queue for longer than the specified
threshold.
Spotlight on Oracle
39
Alarm Description
Note: Applies only to Oracle 11g Release 2.
Multi-threaded
Server alarm
The Multi-threaded Server alarm occurs when all or most multi-
threaded server (MTS) or dispatcher processes are busy.
Non-reclaimable
Flashback Space
alarm
This alarm is raised when the amount of non-reclaimable used space
in the Recovery Area (flashback area) exceeds a threshold. Non-
reclaimable files include archived redo logs, which must be backed
up before being deleted.
Oracle Memory on
Disk alarm
The Oracle Memory on Disk alarm is activated when the amount of
guest physical memory mapped to machine memory is less than the
current combined size of Oracle's SGA and PGA.
Oracle Parameters
for Predictive
Diagnostics alarm
The Predictive Diagnostics feature cannot predict the behavior of
Oracle databases without the access to Oracle data that is granted by
several Oracle parameters.
Oracle Status alarm When a Spotlight connection is lost, Spotlight automatically tries to
re-establish the connection. The Oracle Status alarm indicates that
Spotlight is re-establishing its connection to the Oracle instance.
Parallel Query
Server alarms
The Parallel Query Server alarm occurs when all or most parallel
servers are busy. Parallel server processes perform work to resolve
SQL statements that are being executed in parallel. If all parallel
server processes are busy, parallel SQL may be downgraded, or even
executed serially, thus increasing its execution time. Spotlight can
raise these Parallel Query Server alarms: Parallel Query Servers Busy,
Parallel Queries Downgraded, and Parallel Queries Serialized.
Parse Ratio alarm The Parse Ratio alarm is raised when the ratio of parse calls to
execute calls exceeds a threshold.
Note This alarm is initially disabled; see Total Parse Wait Time Ratio
alarm.
Predictive
Diagnosticsalarms
Predictive Diagnostics alarm (SQL Collector, Bottleneck Collector or
Performance Modeling Collector alarm) is raised when the
appropriate collection of predictive data has not been scheduled.
Predictive
DiagnosticsCustom
SQL Error alarm
This alarm indicates that the customized SQL query use to represent
database concurrency within Predictive Diagnostics has returned an
error.
Predictive
DiagnosticsOracle
Parameters alarm
The Predictive Diagnostics feature cannot predict the behavior of
Oracle databases without the access to Oracle data that is granted by
several Oracle parameters.
Spotlight on Oracle
40
Alarm Description
Redo Allocation
and Redo Copy
Latch Wait alarms
The Redo Allocation and Redo Copy Latch Wait alarms are raised
when there is contention for the redo allocation latch and redo copy
latch.
Result Cache
Find/Create Ratio
alarm
When Oracle executes SQL queries, the result cache stores the result
sets of specified queries in shared memory for later use. The
Find/Create Ratio alarm occurs when the rate of find operations on
the result cache drops below the rate of create operations, and
indicates that the contents of the result cache are under-used.
Result Cache
Latch Sleep Ratio
alarm
Spotlight raises the Result Cache Latch Sleep Ratio alarm when the
rate of latch sleeps for the Result Cache latch indicates that there is
contention on the Result Cache latch, where multiple processes are
competing to use the latch and are causing a significant degradation
in response time.
Row Cache objects
Latch Wait
The row cache objects latch protects the contents of the row cache;
the section of the shared pool to cache dictionary information.
Sequential Read
Time alarm
This alarm fires when the average time that Oracle takes to read a
single block of data typically an index-based lookup is
significantly higher than the average time recorded during
calibration.
Sessions Queued
alarm
The Sessions Queued alarm becomes active when the number of
sessions in the parallel execution queue exceeds a threshold.
Note: Applies only to Oracle 11g Release 2.
Shared Pool Latch
Wait alarm
Shared pool latches control access to the shared pool memory map.
Shared Pool Lock
Percentage alarm
This alarm is raised when one user session holds a lock on object(s)
in the Shared Pool for a prolonged period, and causes other sessions
to wait on the same resource. This "blocking lock" happens rarely.
When it does occur, use a SQL query to find the affected sessions.
Shared Server
Converted into
Dedicated Server
alarm
Spotlight raises an alarm when many shared servers are converted,
leaving too few shared servers to meet the demand. Converted shared
servers will have a status of WAIT(RECEIVE) in v$shared_server.
Status Error alarm This alarm is raised when there is a problem with the way the OEM
configuration has been defined in Spotlight. Correct the values
entered in the dialog View | Options | OEM Monitoring.
Temporary IO
Wait Ratio alarm
Spotlight raises the Temporary IO Wait Ratio alarm when the time
spent in temporary IO waits is too high as a percentage of the time
Spotlight on Oracle
41
Alarm Description
spent performing user calls on the Oracle database.
Total Parse Wait
Time Ratio alarm
Spotlight raises this alarm when the amount of time spent parsing
SQL statements as a percentage of the active database time exceeds a
level that may cause latch contention on the shared pool and library
cache latches.
Note: The Total Parse Wait Time Ratio alarm is available for (Oracle
10.2 and later). When monitoring earlier Oracle systems, consider
reactivating the (disabled) Parse Ratio alarm.
Unarchived Logs
alarm
The Unarchived Logs alarm occurs when the number of unarchived
redo logs exceeds a threshold. If all online logs need archiving,
database update activity can be suspended while the redo log writer
waits for the next log to be archived.
Unix Status alarm When a Spotlight connection is lost, Spotlight automatically attempts
to re-establish the connection. Where the Oracle database is hosted
on a Unix server, this alarm indicates that Spotlight is re-establishing
its connection to the Unix server.
WMI Status alarm When a Spotlight connection is lost, Spotlight automatically attempts
to re-establish the connection. Where the Oracle database is hosted
on a Windows server, this alarm indicates that Spotlight is re-
establishing its WMI connection to the Windows server.
Write Complete
Wait alarm
Spotlight raises the a Write Complete Wait alarm when write
complete waits (as a proportion of total waits) exceed a threshold.
Write complete waits occur when a session tries to modify a block
that is currently being written to disk by the database writer process.
Note: Spotlight on Oracle Data Guard alarms occur in the Alarm Log of the Primary Database
in the Oracle Data Guard environment. See Spotlight on Oracle Data Guard Alarms in the online
help for more information.
Spotlight on Oracle Drilldowns
When you have isolated a problem, you can display a drilldown page, whose charts and tables
provide a detailed breakdown of the underlying statistics.
Following are Spotlight on Oracle drilldowns.
Spotlight on Oracle
42
Drilldown Click
to
open
Description
Top Sessions The Sessions drilldown displays information about the
users connected to the Oracle database. The pages and
sub-pages in the drilldown display information on:
l All users connected to the Oracle database.
l Session details for those users, including waits,
SQL statements executed, locks, resource usage,
and Oracle statistics.
l Parallel queries executed for all users connected to
the Oracle database.
l Oracle jobs currently scheduled for the database.
SQL&Application
Workload
The SQL & Application Workload drilldown allows you
to select, sort and display SQL statements executed
according to specified criteria. The drilldown includes:
l The Top SQL page, which displays the SQL
statements that consume the most system resources
for the current connection.
l The Array Fetch page, which displays information
on array processing for the Oracle database under
investigation.
l The Parse Activity page, which displays
information on SQL statements associated with
high parse rates (Oracle 10.2 and later).
l The Sort Activity page, which displays
information on SQL statements associated with
high sort activity (Oracle 10.2 and later).
l The Analyze Trace page (formerly a separate
drilldown), which displays the contents of trace
files that store information on how Oracle
processes the execution of SQL statements, and on
the resources it uses to do so (Oracle 10.2 and
later).
l The Result Cache page (Oracle 11g and later)
shows information about the result cache, which
stores the result sets of specified queries so they
can be re-used.
Activity The Activity drilldown displays the activity on the
database. The pages and sub-pages in the drilldown
display information on waits, waiting events, blocking
Spotlight on Oracle
43
Drilldown Click
to
open
Description
locks, lock activity, latches, query servers, transactions,
rollbacks, and buffer busy waits.
I/O The I/O drilldown displays the input and output activity
for the database. The pages and sub-pages in the
drilldown display information on I/O by category, session,
SQL, tablespaces, datafiles, segments and row chaining.
Configuration &
Memory
The Configuration and Memory drilldown shows the
memory utilization for the Shared Pool and the SGA, as
well as shared pool and buffer cache hit ratios, Oracle
alert log and Oracle parameters.
Operating System The Operating System drilldown provides information
about the performance, processes, and storage on the
database machine under investigation.
Disk Storage The Disk Storage drilldown displays information about
online redo logs, tablespace utilization, archive
destinations, and Flashback recovery.
Right-click the tables in the Disk Storage drilldown to
open Space Manager. See "Tools |Space Manager" (page
46) for more information.
Tuning You can use the Tuning drilldown to view information
about Oracle memory resources in the Oracle SGA and
PGA (System and Program Global Areas), to tune memory
resources where feasible, and to view and tune latch
activity for systems with multiple CPUs.
ASM If your system uses Automatic Storage Management
(ASM) to manage disk storage, you can use the ASM
drilldown within Spotlight on Oracle to diagnose ASM
performance. ASM is a storage management solution from
Oracle Corporation that automatically reorganizes data
based on statistics collected from usage and user access
patterns over time.
Analyze Trace To understand how well SQL is being processed by the
system under investigation, and how to improve that
processing, you must trace the execution of SQL
statements for:
l How Oracle is processing the statements.
Spotlight on Oracle
44
Drilldown Click
to
open
Description
l The resources that are required to execute those
statements.
Oracle stores that information in trace files. The pages in
Spotlight's Analyze Trace page allow you to view the
contents of those trace files and save the contents to the
client's computer.
DTrace DTrace is a dynamic analysis tool for Sun platforms.
Spotlight on Oracle uses DTrace to capture details of the
interaction between Oracle and the operating system
(Solaris 10 onwards). See "Install And Configure DTrace
Scripts" (page 28) for more information.
Predictive
Diagnostics (if
enabled)
Predictive Diagnostics (if available) allows Spotlight to
collect and analyze performance metrics for an Oracle
database in order to predict its future behavior. In doing
so, it identifies:
l SQL statements ("degrading SQL") whose
performance may not scale adequately in the future
as data volumes and SQL execution rates increase.
l Waiting events ("bottlenecks") that may in the
future affect database throughput and response
time.
l Database resources (CPU, memory, and disk I/O)
whose limitations may in the future affect database
performance.
See "Using Predictive Diagnostics" (page 50) for more
information.
Virtualization Use the Virtualization drilldown to view detailed
information about the CPU and memory use of a virtual
machine.
Data Guard Monitor an Oracle Data Guard environment with
Spotlight on Oracle Data Guard.
Note: The Data Guard icon is enabled for Primary
databases configured in the Oracle Data Guard
environment.
Reports The Report wizard generates printable reports of four
types of monitoring data: alarms by percentage of time,
waits, Top SQL and IO. The reports are designed and
Spotlight on Oracle
45
Drilldown Click
to
open
Description
delivered through SAP Crystal Reports
.
Alarm Log The Alarm Log drilldown displays information on the
alarms associated with the Oracle database.
You can filter the Alarm Log to display only the alarms
that meet specified criteria.
Spotlight Tools
Tools |Explain Plan
Tools |Explain Plan allows you to determine the execution plan that Oracle applies to a
particular SQL statement. Spotlight on Oracle allows you to view graphical representations of
the execution plan for SQL statements being executed by a user, or for those identified by the
Top SQL Page.
To open the Explain Plan
1. Open a connection to an Oracle Database.
See "Connect to Oracle Instance" (page 22) for more information.
2. Click Tools |Explain Plan.
Tools |SQLOptimizer
SQL Optimizer provides context-sensitive tuning advice for SQL statements based on the Oracle
execution plan and the database structure.
Before you open SQL Optimizer
l Ensure SQL Optimizer is installed on your machine.
l Ensure you have DBA privileges for the database you are diagnosing.
To open SQLOptimizer
2. Click Tools |SQLOptimizer.
Spotlight on Oracle
46
Tools |Space Manager
Space Manager provides a comprehensive solution for space management and reorganization in
Oracle databases. Space Manager offers preventive maintenance, problem detection and
resolution, and capacity planning across any number of databases.
Before you open Space Manager
l Ensure Space Manager (version 4.5 or later) is installed on your machine.
l Ensure you have DBA privileges for the database you are diagnosing.
To open the Space Manager
2. Click Tools |Space Manager.
Tune Your Database
Spotlight supports an approach to Oracle performance tuning that could be described as "tuning
by bottleneck". Spotlight alerts you if any component of the Oracle architecture is forming a
bottleneck. Additionally, Spotlight provides tools that allow you to identify and rectify any
inefficiency in your databases configuration.
In general, you tune your Oracle instance by reiteratively identifying bottlenecks, contention and
critical resources, and by using Spotlight's online advice to remove the bottleneck or contention
to improve the performance of the resource.
The following is a summary of steps that you can undertake to tune Oracle using Spotlight.
l Use the Spotlight home page to alert you to any obvious bottlenecks. If Spotlight detects
that some component of Oracle constitutes a performance bottleneck, the corresponding
component generates a visual or auditory alarm. The exact appearance of the alarm
depends on how you have configured alarm severities. When an alarm is current you can
press F1 to display help for the alarm. Clicking the object (in the main window) displays
the relevant drilldown.
See "Spotlight on Oracle Home Page" (page 29) for more information.
l Use the Activity drilldown to display overall efficiency and resource usage. In particular,
the Event waits chart shows the amount of time database sessions are spending waiting
for various resources, as well as the amount of CPU being utilized. Tuning efforts are
most likely to succeed if they are concentrated on the resources being used most heavily.
The Wait activity topic contains a discussion of the meaning and implications of various
wait events.
See "Spotlight on Oracle Drilldowns" (page 41) for more information.
Spotlight on Oracle
47
l Other pages help you examine detailed activity. For instance, the Top Sessions drilldown
shows the users who contribute most to database activity. You can display various details
for these sessions, including the SQL statements, locks and resource usage.
For further information on tuning your database, see the online help.
Troubleshooting
Connection Problems
Cannot Connect to Database
Possible cause Solution
Oracle is not running. Verify that the Oracle instance is running and
can be accessed from a host or client
workstation.
The connect string and alias of the
database are incorrect or have changed.
Check your SQL* Net configuration and
verify the connect string and alias you are
using are correct.
The user ID you are using to log into
Spotlight does not exist in the database.
Run the User Wizard to establish the user ID
in the database.
The Spotlight user has the wrong
permissions in the database.
Contact your DBA to add relevant
permissions to the user.
Spotlight takes too long to open the
Oracle connection so it appears to be
unable to connect.
Turn on Fast Initialize.
More information on Fast Initialize is
available in the online help.
1. Click View | Options.
2. Select Data Collection |Oracle
Initialization | Turn Fast Initialize
on.
3. Restart Spotlight.
Cannot Connect to ASM
When installed and available, Automatic Storage Management (ASM) stores Oracle data in a
separate ASM database instance. Spotlight on Oracle displays performance metrics for ASM on
the home page and in the ASM drilldown.
If the database under investigation uses ASM but Spotlight cannot connect to it to retrieve the
required information, you may need to edit the TNSNAMES entry for the ASM instance.
1. Locate the tnsnames.ora file in Oracles network/admin sub-folder of the Oracle
installation folder on your Spotlight client.
4
Spotlight on Oracle Getting Started Guide 49
2. Edit tnsnames.ora so that the TNSNAMES entry for the target ASM instance contains the
option (UR = A).
Dead Sessions in Top Sessions Drilldown
The solution is to reset the Top Sessions drilldown.
To reset the Top Sessions drilldown
From the Top Sessions drilldown
1. Click Filter.
2. Select Reset
3. Select the appropriate frequency.
Data That Does Not Appear in The History Browser
Certain features in Spotlight on Oracle cannot be captured for display in the Spotlight History
browser, even when the relevant Spotlight options are chosen. The features that cannot be
collected in historical snapshots are:
l The SQL & Application Workload Drilldown.
l The Activity Drilldown | Waits Page | Grid.
l The Activity Drilldown | ( Waiting Events Page (Oracle 10g and later) | Waiting Events
Page (Oracle 9i))
l The I/O Drilldown | Summary Page | Bottom chart.
l The Configuration & Memory Drilldown | Buffer Cache Page | lower grid.
l The Configuration & Memory Drilldown | Oracle Parameters Page.
l The Operating System Drilldown | OS Storage Page.
Predict Database Performance
50
Using Predictive Diagnostics
Use Predictive Diagnostics to predict the future performance of a database (and identify how to
improve that performance):
l Identify SQL statements whose performance may not scale adequately as data volumes
and SQL execution rates increase.
l Identify bottlenecks that may in the future affect database throughput and response time.
l Predict when database resources such as CPU, memory and disk I/O are expected to reach
their full capacity, and recommending strategies to enable the database to handle high
loads at those times.
Notes:
l To ensure that Predictive Diagnostics has access to all the Oracle information it needs,
check that the Oracle Parameters for Predictive Diagnostics are set to the required values.
l It takes about 10 days for Spotlight to collect enough data to make valid predictions on
the future performance of the database.
How to use Predictive Diagnostics
1. Select the Spotlight on Oracle connection in the Spotlight Browser.
2. Click Predictive Diagnostics.
The page contains several sections Configuration, Find Impending Problems,
Performance Modeling, and What To Do Next and a set of images that indicate a basic
workflow through the predictive process.
3. Click a hyperlink in the Predictive Diagnostics home page to open the corresponding
diagnostic page.
First time users
Note: Although you can use any feature in Predictive Diagnostics at any time, we recommend
the following procedure for first-time users.
51
1. Follow the suggestions in What To Do Next.
The quality of predictions depends on the quality of the data on which they are based.
What To Do Next tracks the current status of the data and indicates when that data is
incomplete or outdated.
2. Wait until enough real data is available.
This usually occurs about 10 days after you begin monitoring the database. Before that
time, become familiar with how Predictive Diagnostics works by using the supplied
sample data.
3. Improve how the database is used now by using Identify Degrading SQL and Identify
Database Bottlenecks to optimize data processing on the host system.
Note: The changes you make as a result of using these features MAY cause the data you
collected earlier to become outdated. If so, you may need to set Trend Changes and
collect new data for another 10 days.
4. Establish the resources available to the database.
Use Configuration to set realistic parameters for the CPU, memory and disk I/O resources
available to the database. (The default settings we provide for these parameters may not
be appropriate for your database.)
In particular, use Peak Processing Periods to identify WHEN database resources are in
greatest demand. This will improve the precision of your predictions.
5. Use Performance Modeling to predict when and why database resources will run out,
what will cause performance to suffer, and what you need to do to counter those
Predictive Diagnostics And Spotlight
For detailed information on Predictive Diagnostics, see the Spotlight online help.
When... Do this...
identifies a SQL statement that
may cause problems
Open the Top SQL drilldown to view further details
about the SQL statement.
Open Explain Plan to view further information about
the statement's execution plan.
Open SQL Optimizer to view context-sensitive tuning
advice for the SQL statement based on the Oracle
execution plan and the structure of the database.
52
When... Do this...
identifies a bottleneck that may
cause problems
Open the Activity |Waiting Events drilldown to view
further details about the current behavior of potential
bottlenecks.
identifies a database resource
whose limitations may cause
problems
Open the I/O, Configuration & Memory, and Tuning
drilldowns to view further details about the current
behavior of CPU, memory, and disk I/O resources.
Requirements for Predictive Diagnostics
The disk storage requirements for Predictive Diagnostics schemas comprise:
Situation Requirement
Sample data Approximately 70 MB of disk space.
SQL
statements
A fixed overhead of approximately 10 MB per schema.
An additional requirement of approximately 1 KB per snapshot for each
SQL statement.
The number of snapshots collected depends on the application workload and
size of the shared pool, the collection interval for snapshots, and the length
of time each snapshot is retained.
Example
If there are 500 eligible SQL statements per snapshot, and snapshots are
collected twice a day, and each snapshot is kept for one year, the disk
storage requirement for SQL statements is:
10MB + (500 x .001MB x 2 x 365) = 375 MB
Impending
bottlenecks
Disk storage requirements depend on how frequently data is collected, and
the length of time that snapshots are retained.
Examples
If data is collected every 10 minutes and retained for six months:
approximately 200 MB.
Performance
modeling
Disk storage requirements depend on how frequently data is collected, and
the length of time that performance data is retained.
Example
53
Situation Requirement
If data is collected every 10 minutes and retained for one year:
Note: Parameters set in View | Options | Predictive Diagnostics and the SQL Statement Trend
Analysis wizard affect the storage requirements for Predictive Diagnostics data. See the online
help more for more information.
Troubleshooting
Predictive Diagnostics Unavailable
You can use Predictive Diagnostics to collect and analyze performance metrics for SQL
statements as they are executed on the database instance, system bottlenecks as they affect
database throughput and response time, and database resources whose limitations affect database
performance.
If Predictive Diagnostics is unavailable for your Spotlight connections, re-run the Oracle user
wizard to install Predictive Diagnostics.
1. Close and re-open Spotlight on Oracle.
2. Click File | Oracle User Wizard...
3. Create a spotlight user. Make sure that you install Predictive Diagnostics as part of the
procedure. See "Spotlight Users" (page 19) for more information.
Note: When you start using Predictive Diagnostics, Spotlight will take about 10 days to collect
enough data to make valid predictions on the future performance of the database. During that
time, you can use the supplied sample data to view Predictive Diagnostics at work.
Spotlight on Oracle RAC
54
What is Spotlight on Oracle RAC?
Note: Spotlight monitoring of Oracle RAC systems requires additional licensing.
Use Spotlight on Oracle RAC to diagnose Oracle databases on multiple host machines under the
Real Application Clusters (RAC) architecture.
When monitoring the behavior of an Oracle RAC cluster, a database administrator needs
to know:
1. How is each of the instances in my cluster performing?
2. How is the database as a whole performing?
3. Is my database experiencing cluster-related contention?
To further investigate the behavior of individual database instances within the cluster, the
database administrator can use Spotlight on Oracle RAC to launch Spotlight on Oracle on any
particular instance within a cluster. See "Instance Panels" (page 62) for more information.
What Is Oracle RAC?
Real Application Clusters (RAC) is a clustering technology introduced by Oracle in release 9i. It
supersedes Oracles previous clustering technology Oracle Parallel Server (OPS).
Oracle RAC is a shared disk clustering solution that is significantly different from the
architectures offered by Microsoft SQL Server and IBM DB2. Each node in an Oracle RAC
cluster has equal access to all of the database data through a shared disk subsystem. Data is
neither partitioned to specific nodes, nor replicated across nodes. A high-speed network
interconnect allows each node to keep its in-memory view of data consistent.
An Oracle RAC configuration has the following architectural characteristics:
1. Many instances of Oracle run across several nodes.
2. Many instances share a single physical copy of a shared Oracle database.
3. All instances have common data and control files.
4. Every instance has individual redo logs and undo segments.
5. Every instance can simultaneously execute transactions against the same database.
The following diagram illustrates the basic components of an Oracle RAC cluster.
55
A user should find an Oracle RAC database operationally identical to a database hosted on a
single server. From top to bottom, the important features represented are:
l Cluster interconnect - A high-speed, high-bandwidth communication facility that
connects the nodes in an Oracle RAC cluster.
l Multiple nodes/instances - Oracle instances run on the nodes (host machines) in the
Oracle RAC cluster. Each instance comprises an Oracle System Global Area (SGA)
plus the corresponding Oracle background processes that retain and process Oracle
database requests.
l Shared disk subsystem - Database files in Oracle RAC systems are stored on multiple
disks that are shared by all the nodes in the cluster, and all nodes must be able read and
write to those disks.
Connect to Spotlight on Oracle RAC
Use this procedure to connect to an Oracle instance running on an Oracle RAC cluster.
1. From the Spotlight Browser select All Connections |Spotlight on Oracle RAC
2. Select the RAC to connect to. If the RAC is not on the list then Add a new connection
56
(page 56).
3. The Verify Cluster Connection Details dialog is displayed if Spotlight encounters an
error during the connection process. See "Verify Cluster Connection Details Dialog" in
the online help for more information.
Note:You can request Spotlight displays this dialog every time you connect to the
Oracle RAC. On the connection Properties | Details dialog select Display Connection
Details Verification Dialog. Alternatively, on the Verify Cluster Connection Details
dialog select Display this dialog for the current connection.
Add a new connection
1. Ensure you create a Spotlight user with sufficient privileges to diagnose the Oracle RAC.
See "Spotlight Users" in the online help for more information.
3. Select Spotlight on Oracle RAC on the Connections menu.
5. Ensure Details is selected. Fill in the Properties window.
Oracle Connection Details
Field Description
Connection
String
Select the name (the SQL*Net alias) of the Oracle RAC instance that
you wish to connect to.
Notes:
57
Field Description
l The database name is defined within SQL*Net configuration
utilities.
l While this can be the name of any instance in the Oracle RAC
cluster, we recommend using the name of the load-balanced
instance the instance whose TNSNAMES entry includes the
option (LOAD_BALANCE = yes)
l When selected, the Connection name field resets to this value.
You can then edit the Connection name field if required.
Oracle
Username
Enter the Spotlight user name.
Oracle
Password
Enter the Spotlight password.
Oracle
Home
Select an appropriate Oracle Home a location where Oracle client
software is installed.
Notes:
l Choose a home that contains a version of the Oracle client
that is compatible with the Oracle RAC server.
l You CANNOT change this value while an active Spotlight
connection is already using an Oracle Home.
l When you specify an Oracle Home for a particular Spotlight
connection, the connection-based Oracle Home overrides any
default values.
l The first active Spotlight connection in a session forces all
subsequent connections to use its specified Oracle Home.
ASM connection details
ASM is built into the Oracle database kernel. It simplifies the creation and space
management in Oracle databases.
Note: Available for Oracle 10g and later.
Field Description
Monitor
ASM
Select to monitor the performance of Oracle's Automated Storage
Management (ASM) technology.
ASM
Connection
String
Enter the connection string required to link to the ASM database
instance that is used for storage management by this Oracle database.
Note:The TNSNAMES entry for the ASM instance must include the
option (UR = A) for the Spotlight connection to work properly.
58
Field Description
ASM
Password
Enter the password to login to the ASM database instance.
Note: There is no ASM Username field. The ASM Password is to the
ASM user name of SYS. SYS is the only username that can provide
access to ASM-specific information.
The user that Spotlight uses to login to the ASM database is SYS.
This is the only user name that can provide access to ASM-specific
information.
Providing SYS access to an ASM instance does NOT create a security
issue for the Oracle RAC, but you should limit access to the SYS
account to trusted individuals.
Because the SYS password is stored in encrypted format, non-
privileged users can still access Spotlight on Oracle RAC. To do so:
l Use a pre-configured or shared Spotlight on Oracle
connection, OR
l Use a Spotlight connection without ASM access. Spotlight
will display Oracle-related data, but the ASM drilldown will
contain no data.
Server connection details
Field Description
Monitor
OS
Select to monitor the operating system on the database server.
Note: Clear this option if the database is running on an operating
system NOT supported by Spotlight on Oracle RAC. If you leave this
option selected and Spotlight is unable to collect statistics from the
server operating system, you may experience errors.
OS Server Enter the name of the server.
Tip: Use a period . to monitor the local database.
l Windows
l Unix(REXEC)
l Unix(SSH)
l Unix(SSH using Public/Private Keys)
daemon is running on the database server and is configured to receive
remote connections.
Note:Spotlight supports both SSH1 and SSH2 protocols; public-key
59
Field Description
encryption is supported under SSH2 only.
SSH Port Enter the port number for SSH (secure shell) connection to the
database servercluster. The default value is 22.
Note: Applicable when OSType is Unix (SSH) or Unix(SSH using
OS
Username
Type the user name to login to the database servercluster.
Notes:
l Spotlight will connect successfully to all nodes in the cluster
only if every node employs the SAME username and password.
l If the username and password if required are not
completed, you will not be able to view the operating system
performance statistics for the database servercluster.
l When OSType is Windows:
l Not required if your login details have been assigned
remote access to the registry of the nodes in the database
servercluster.
l If your login details have NOT been set up on the
database .servercluster, you must enter a username and
password that has access to the registry of each node.
SSH Key
Type
Choose the type of key to use when making the Spotlight
connection:RSA or DSA.
Note: Applicable when OSType is Unix (SSH using Public/Private
Keys).
SSH
Private
Key
Filename
Locate the filename that contains the private key for the Spotlight
connection.
Hint: Use the "..." button.
Keys).
SSH
Passphrase
Keys).
OS
Password
Type the user password to login to the database servercluster (if
required).
Note: Not applicable when OSType is Unix (SSH using
6. Select Launch Spotlight on Oracle to open a Spotlight on Oracle connection to the
60
database when you open the Spotlight on Oracle RAC connection.
7. Select Display Connection Details Verification Dialog to display the Connection
Details Verification dialog when you connect to a specified Oracle RAC cluster. The
dialog allows you to specify and test connections to ALL instances in the cluster.
8. Select Save password details to save all the password details that have been entered.
Spotlight on Oracle RAC Home Page
The Spotlight home page shows an overview of the activity in an Oracle RAC database cluster.
Related operating system statistics are grouped together on panels that are connected by a series
of graphical flows and icons. Spotlight updates these flows in real time so you can see how
quickly data is moving through the system. The icons change color as Spotlight alarms are
raised, upgraded, downgraded and canceled.
The panels on the Spotlight on Oracle RAC home page are described in more detail next. For a
full discussion of the panels, see the Spotlight online help.
Tip: Hover the mouse pointer over a panel component for more information.
Where an Oracle Exadata System is installed and licensed the Spotlight on Oracle RAC Home
Page looks slightly different. An additional Exadata Storage Panel (page 63) monitors the
Exadata Storage System (Cells).
61
Interconnect and Global Status Panel
This panel shows a combination of Interconnect and Global Cache Service information, and
information about the Oracle RAC database overall. It consists of information aggregated from
queries sent out to the individual instances. This provides information on how well data is being
shared and transferred between nodes in the cluster.
Some information displayed here is easily calculated: for example, the Total Logical Reads
spinner is the sum of the individual Logical Reads values. Other components, such as Cluster
Latency and Cluster Overhead, have a more complex calculation.
Spotlight uses the Global Database label to indicate alarms in one or more Oracle instances that
may affect the performance of the Oracle RAC database as a whole.
The flows between the Interconnect panel and the Instances panel represent the flow of data
from one instance to another across the interconnect.
62
Instance Panels
This panels show the participating instances of a cluster. Together with separate panels for all
participating instances, this panel provides information on the performance of individual
instances in the cluster.
Spotlight on Oracle RAC supports a theoretical maximum of 36 instances, with up to 6 instances
shown on the home page. When there are more than 6 instances (not shown here), you can use a
navigation control in the Instances panel to navigate between groups of instances.
The color of the border and label on each instance represents the Spotlight on Oracle severity of
the instance. You can move the mouse over a named instance to display a list of all the alarms
currently raised against that instance.
Note: You can launch Spotlight on Oracle from any Instance panel on the Spotlight on Oracle
RAC home page.
The overall status of the cluster does not necessarily reflect the status of individual instances,
as problems with individual instances do not necessarily translate to a problem with the
cluster as a whole.
Two further metrics Balance and Availability are shown in the instances panel. Balance
indicates the comparative load on the instances. Availability shows the percentage of instances
for which a SQL*Net connection is currently open and working.
63
IO Subsystem Panel
This panel shows information relating to the shared storage subsystem for the Oracle
RAC cluster.
Some of this information is aggregated from queries sent to the individual instances; other
information is obtained from a single query to one of the instances.
The ASM spinner is populated only when ASM (Automatic Storage Management) is being used.
See "Connect to Spotlight on Oracle RAC" (page 55) for more information.
The flows between the I/O Subsystem panel and the individual instances represent the flow of
physical reads and writes between the instances and the physical database.
Exadata Storage Panel
Note: Spotlight on Oracle RAC may include monitoring of Exadata systems depending on the
license purchased.
This panel shows information relating to the Exadata Storage Servers (Cells).
l Cell Averages - Measure how much the Exadata Storage Servers are used.
l Smart Scans - Exadata's Smart Scans optimize SQL statements executed in Exadata's
Storage Servers. Smart Scans are typically for scan operations like Full Table Scans.
l Smart Flash Cache - Speed up I/O performance for frequently accessed data. Exadata
Smart Flash Cache acts like a large, database-aware disk cache for the storage cell.
l Cell Summary - Summary statistics for Exadata's Storage Servers.
Spotlight on Oracle RAC Alarms
64
"Spotlight on Oracle RAC Drilldowns" (page 67) for more information.
l Snooze the alarm.
Alarms on A Database Instance
Problems in a single instance may not always affect the operation of the cluster as a whole. In
such cases, Spotlight on Oracle RAC can display an alarm in the relevant instance panel without
reporting a specific Spotlight on Oracle RAC alarm.
However, if a problem in a single instance DOES affect the operation of the whole cluster or is
repeated in multiple instances so that the combined effect is significant Spotlight on Oracle
RAC reports the problem as an aggregated alarm.
These alarms are displayed on the Spotlight on Oracle RAC home page usually in the Global
Database heading of the Interconnect and Global Status panel, or on the individual components
of the IO Subsystem panel.
Notes:
l Aggregated alarms within Spotlight on Oracle RAC are the same as the corresponding
alarms in Spotlight on Oracle.
l Not all Spotlight on Oracle alarms are aggregated in Spotlight on Oracle RAC.
Alarms on The Oracle RAC
Following are the alarms specific to Spotlight on Oracle RAC. For possible solutions to the
problems indicated by these alarms or for information on how to diagnose problems further, see
Spotlight on Oracle RAC alarms in the online help.
Alarm Description
Archive
Critical
Failure alarm
An Archive Critical Failure alarm indicates that the number of working
archive destinations for the database cluster is about to fall below the
minimum number needed to archive Oracle redo logs successfully. When
you see this alarm, you should make space in the archive log destination
65
Alarm Description
by purging or by backing up to an offline medium any archived files
that are not immediately needed.
Archive
Destination
Failed alarm
Spotlight raises an Archive Destination Failed alarm when at least one of
the archive destinations specified for redo log files has become full. See
Archive Destination Failure alarm below.
Archive
Destination
Failure alarm
An Archive Destination Failure alarm when at least one of the archive
destinations specified for redo log files is about to become full. When you
see this alarm, you should make space in the archive log destination by
purging it, or by backing it up to an offline medium.
ASM Status
alarm
When a Spotlight connection is lost, Spotlight automatically tries to re-
establish the connection. The ASM Status alarm indicates that Spotlight is
re-establishing its connection to the ASM diskgroup for the Oracle RAC
cluster.
Availability
alarm
This alarm indicates that the instances participating in the cluster are NOT
all currently available for access. The database administrator should
determine why the instances are unavailable, and restart them after taking
the required actions.
(Global)
Cache Miss
Rate alarm
See Miss Rate alarm below.
Cluster
Latency
alarm
Spotlight raises the Cluster Latency alarm when cluster latency from all
causes (interconnect, GES, LMS overhead) exceeds a specified threshold.
Cluster latency is defined as the average elapsed time from when a
requesting instance requests a block of data to the time when the
requesting instance receives the block.
Cluster
Overhead
alarm
Spotlight raises the Cluster Overhead alarm when the percentage of time
dedicated to cluster co-ordination activities exceeds a specified threshold.
Cluster overhead is the amount of time spent waiting for cluster related
activities as a percentage of the total time spent waiting for all activities.
Corrupt or
Lost Blocks
alarms
Spotlight raises separate alarms for corrupt and lost blocks when a
specified percentage of the blocks transferred across the interconnect are
corrupted or lost during global cache consistent read requests.
High levels of lost or corrupt blocks should not happen in normal
circumstances. If these alarms fire at all, it indicates either very serious
overloading of the Oracle RAC cluster, or software or hardware faults in
the cluster.
Data Guard
Configuration
The Data Guard Configuration Alarm occurs on a Primary database in an
Oracle Data Guard environment. Please supply details of Standby
66
Alarm Description
Alarm connections to Spotlight so Spotlight can monitor the Oracle Data Guard
environment. In monitoring the Oracle Data Guard environment Spotlight
can alert you if key elements of the Oracle Data Guard environment are
not working, or if Standby instances are lagging behind.
Instance
Failed alarm
Spotlight raises an Instance Failed alarm when a specified instance has
failed because of an Archiver error. See Archive Critical Failure alarm
above.
Load
Imbalance
alarm
Spotlight raises the Load Imbalance alarm when:
l The load across instances in the cluster is significantly unbalanced,
and
l The imbalance is proven to be statistically significant, beyond a
95% probability.
Cluster balance is a fundamental indication of the health and productivity
of an Oracle RAC environment. If load is not reasonably evenly
distributed across the instances in a cluster, then the DBA is likely to
regret having chosen a clustered environment.
Spotlight on Oracle RAC calculates balance by recording a user-definable
metric of load over time on each instance (the default is Logical Reads,
but you can switch this to CPU usage or Physical Writes via the
Spotlight on Oracle RAC Options window) and calculating the relative
balance of load across the cluster.
In the normal course of events, there are always minor variations in load
between instances within the cluster, so Spotlight uses a statistical
technique to determine if a variation is significant.
Note:The activation of specific Oracle Services can unbalance the
workload across the Oracle RAC cluster. In these cases, Spotlight triggers
the Load Imbalance alarm only when a more significant threshold is
reached.
Miss Rate
alarm
This alarm, more specifically known as the Cache Miss Rate alarm or
Global Cache Miss Rate alarm, is raised when the global cache local miss
rate exceeds a specified threshold. This rate is the percentage of time that
a block required for a logical read is located in an inconsistent mode on
ANOTHER instance in the cluster and must be transferred.
67
Alarm Description
Average
Percentage
CPU Load
Across The
Exadata
Storage Cells
alarm
The Average Percentage CPU Load Across The Exadata Storage Cells
alarm occurs when the total CPU utilization of the Exadata Storage Servers
exceeds a threshold. One or more of the Server CPU may encounter a large
number of requests, or you may have un-tuned SQL, which uses excessive
amounts of CPU.
Maximum
Percentage
CPU Load on
The Exadata
Storage Cells
alarm
The Maximum Percentage CPU Load on The Exadata Storage Cells alarm
occurs when the CPU utilization of one or more Exadata Storage Servers
(Cells) exceeds a threshold.. One or more of the Server CPU may encounter
a large number of requests, or you may have un-tuned SQL, which uses
excessive amounts of CPU.
Single Block
Read alarm
The Single Block Read alarm is raised if the average time taken to
complete a request for a single block from the Exadata Storage Servers
exceeds a given threshold. The source of the latency may be any database
using the Exadata Storage Servers.
Cell Balance
alarm
The Cell Balance alarm is raised when there is an unequal distribution of
work-load between the Exadata Storage Servers.
Exadata
Storage
Server (Cell)
Not
Connected
Spotlight has lost its connection to at least one Exadata Storage Server,
and is unable to re-connect. Check the status of your network connection.
Check the status of the hosts of each Exadata Storage Server.
No Nodes
Available
alarm
This alarm indicates that Spotlight has lost its connection to the Oracle
RAC cluster, and cannot re-connect to any of the nodes in the cluster.
Check the status of your network connection, your host machines, and
your database instances.
Note: Spotlight on Oracle Data Guard alarms occur in the Alarm Log of the Primary Database
in the Oracle Data Guard environment. See Spotlight on Oracle Data Guard Alarms in the online
help for more information.
Spotlight on Oracle RAC Drilldowns
Following are Spotlight on Oracle RAC drilldowns.
68
Drilldown Click
to
open
Description
Top Sessions As per Spotlight on Oracle.
SQL&Application
Workload
As per Spotlight on Oracle. The SQL & Application
Workload drilldown in Spotlight on Oracle RAC does
NOT contain the following pages:
l Analyze Trace
l Result Cache
Activity As per Spotlight on Oracle. The Activity drilldown in
Spotlight on Oracle RAC does NOT contain the following
pages:
l Lock Activity
l Server
l Rollback
I/O As per Spotlight on Oracle.
Configuration &
Memory
As per Spotlight on Oracle. The Configuration & Memory
drilldown in Spotlight on Oracle RAC does NOT contain
the following pages:
l Shared Pool Utilization
l Buffer Cache
Disk Storage As per Spotlight on Oracle. The Disk Storage drilldown in
Spotlight on Oracle RAC does NOT contain the following
pages:
l Online Redo
l Recovery Area
Cluster The pages in the Cluster drilldown display information
about the performance of the Oracle RAC cluster, including
details on cluster latency, balance, overhead, miss ratios,
and corrupt or lost blocks. For further information see
Cluster drilldown in the online help.
ASM As per Spotlight on Oracle.
Services The Services drilldown displays details of the services
defined and deployed for the Oracle RAC cluster under
investigation.
Note: Available for Oracle 10.2 and later.
69
Drilldown Click
to
open
Description
Predictive
Diagnostics (if
enabled)
As per Spotlight on Oracle.
Data Guard If the system is a Primary database in the Oracle Data
Guard environment, you can use Spotlight on Oracle Data
Guard to monitor the system's Standby databases. Standby
databases are alternative/supplementary repositories to the
production Primary database.
Note:The Data Guard icon is enabled once the system is
configured as a Primary database in the Oracle Data Guard
environment.
Exadata Detect and help resolve performance issues on Oracle's
Exadata machine.
Note: Spotlight on Oracle RAC may include monitoring of
Exadata systems depending on the license purchased. When
licensed, the Exadata icon is enabled on connection to an
Oracle Exadata system.
Alarm Log As per Spotlight on Oracle.
Troubleshooting
Average Ping Time Chart Not Populated
Problem
There is no data in the Average Ping Time chart on the Cluster drilldown | Latency.
Solution
First, check the Oracle version for the instances running on the Oracle RAC cluster the
Average Ping Time chart is populated only for Oracle 10g and later RAC clusters running on
Linux Servers. You must also have an OS connection to the host machine.
Second, check that the ping command package has been installed on the nodes in the cluster
(installations of the Linux 2.4 kernel may not install the ping command by default). If ping is
not installed:
70
1. Install the iputils package on ALL nodes in the cluster.
2. Create a new Spotlight on Oracle RAC connection that connects to the operating system
(select Monitor OS in the Connection properties dialog: Connect to Spotlight on Oracle
RAC (page 55)).
Spotlight on Windows
71
Connect to Windows Systems
1. From the Spotlight Browser select All Connections |Spotlight on Windows
2. Select the Windows system. If the Windows system is not listed then follow the
instructions to add a new connection.
Add A New Connection
You will
need
Description
Administrator
access
Ensure your login to the Windows machine has Administrator access
privileges.
To add a new connection to the Windows machine you are currently
using, ensure you are logged in to the machine with Administrator access
privileges. If necessary, logout and login again.
Privileges Spotlight on Windows retrieves its data from Windows performance
counters and the windows registry of the monitored system. An
administrator login to the Windows machine has such access.
Remote
connectivity
To add a new connection to a Windows machine other than the one you
are currently using, ensure the machine is accessible to the Spotlight client.
Spotlight uses NetBIOS traffic to retrieve perfmon and registry
information, so any firewalls between the Spotlight on Windows client
and the machine being monitored must allow this traffic to pass through.
See "Troubleshooting Spotlight on Windows" (page 78) for more
information.
72
2. Select Spotlight on Windows on the Connections menu.
4. Fill in the Details page of the Properties window as follows:
Field Description
Connection
name
Enter the preferred display name for the Windows machine.
Note: If left blank, the Connection name field resets to the value of
the Address field.
Local
Machine
Select to diagnose the machine you are now using.
Address Enter the hostname, IP address or URLto the Windows machine.
Note:The value changes to "." and cannot be edited when Local
Machine is selected.
Logon Details
Note:Not applicable when Local Machine is selected.
Field Description
Domain The name of the domain the user belongs to (to login to the Windows
machine).
Note: If you intend to leave User blank then Domain should also be
left blank.
User Enter the user name to login to the Windows machine.
Note: Ensure this user has administrator rights.
Password Enter the password to login to the Windows machine.
Note: Applicable when Save password details (for this connection) is
selected.
Select Save password details (for this connection)to save the password.
73
Note:Not applicable when Local Machine is selected.
5. Close the dialog.
Click Description
Connect Save changes and open the connection in Spotlight.
OK Save changes. Do not open the connection in Spotlight.
Cancel Do not save changes.
Spotlight on Windows Home Page
System Panel
The System panel indicates:
l The type and version of operating system.
l The name of the service pack installed on the system (if
any).
Network Panel
The Network panel shows the total number of connected users, and the rate
at which packets are being sent from and received by the system:
l The number of clients connected to this system.
l The current number of Windows Networking sessions open on this
machine.
l The level of network traffic graphed against a "theoretical" maximum
bandwidth for the network card specified.
The flows between the Network panel and the CPU panel represent the rate of data transfer
between the Windows host and the connected network.
74
Event Log Panel
The Event Logs panel provides a link to the Event Log drilldown.
l It alerts you to items that have reached alarm status as per
your configuration of Event Log alarms.
l The button icon shows the number of Event Log alarms that
have been raised but not yet cleared. You can acknowledge
and clear alarms via the Event Log drilldown.
Note: This button is enabled on Enable Event Logs selected. See
CPUPanel
The CPU panel shows processor and load information for the Windows
system:
l The number of processors installed in the system.
l The speed of the processors installed in the system.
l The type of processors installed in the system: Intel Pentium for
example.
l The length of time since Windows was last booted (started).
l The amount of CPU being used on the system.
l The average number of threads (program execution units) that are
waiting to run on each processor.
l The number of applications and services that are currently active on
the system.
l The number of threads currently active in all processes.
The flows between the CPU panel and the Memory panel represent paging information between
CPU and memory on the Windows system. The flows include the number of memory pages
written out and read in to memory per second.
75
Memory Panel
The Memory panel shows detailed information on the Windows system.:
l The total amount of physical memory in RAM.
l The amount of physical memory available to applications.
l The percentage of physical memory that is available to applications.
l The amount of physical memory (RAM) Windows is using.
l The maximum amount of virtual memory available without
expanding the paging file.
l The amount of virtual memory available.
l The percentage of virtual memory available to applications.
l The amount of Virtual Memory available without expanding the
paging file
l How much Virtual Memory is in use.
l The percentage of soft pages versus hard pages found in RAM.
l The percentage of disk requests that are found in cache memory.
The flows between the Memory panel and the Paging Files panel represent the rate at which
virtual memory pages are written to disk from memory, and swapped from disk into memory.
The flows between the Memory panel and the Disks panel represent the rate at which memory is
written to disk and read from disk.
Paging Files Panel
The Paging Files panel shows information on paging space for all
paging files active on the system.
l The total size (MB) of all paging files on all disks in the
system.
l The current amount of space (MB) being used by all paging
files on all disks in the system.
76
Disks Panel
The Disks panel shows information for each logical disk on the
system.:
l The drive letter.
l The size of the disk (MB).
l The space used on the disk (MB).
Files Panel
The Files panel shows the number of files whose file size is
currently being tracked by Spotlight on Windows.
Note: This panel is hidden while no files are being tracked. To add
/ remove files, right click on the Files panel and select Files
Options. See the online help for more information.
Spotlight on Windows Alarms
77
"Spotlight on Windows Drilldowns" (page 77) for more information.
l Snooze the alarm.
Following are the alarms specific to Spotlight on Windows. For possible solutions to the
problems indicated by these alarms or for information on how to diagnose problems further, see
Spotlight on Windows alarms in the online help.
This alarm Alerts you
Cache alarms When the level of cache activity exceeds a threshold.
Disk alarms As the logical disk is filling up, and also when disk activity exceeds a
threshold.
Event Log alarm When an Event Log entry is found which matches an active Event Log
severity rule.
Memory alarms When the available memory (both physical and virtual) drops below
specified thresholds.
Paging alarms When there is excessive paging activity on the target system.
Process alarms When processes sustain a high CPU load.
Windows CPU
alarm
When the servers CPU usage exceeds a specified percentage.
Windows status
alarms
Spotlight is re-establishing its connection to the Windows system.
Spotlight on Windows Drilldowns
Following are Spotlight on Windows drilldowns.
78
Drilldown Click
to
open
Description
Processes Detailed information about all processes and services currently
running on the machine you are diagnosing.
CPUs Details of recent processor activity as measured by Windows.
Memory Details about recent physical and virtual memory usage.
Disks Information about the logical and physical disks on the system you
are diagnosing.
Network Detailed information about the network activity to and from the
system being diagnosed.
Activity
Summary
Summaries of recent Windows activity including, paging and
packet rates, CPU usage across all processors, the number of
threads waiting to be run, and the number of I/O requests that were
queued for each logical disk.
Event Log Recent event log items that have occurred on the target machine.
Single
Application
Detailed information about specified individual processes (or a
group of processes) on a Windows system.
Choose the processes you want Spotlight to analyze by View
|Options | Data collection | Windows applications metrics.
Alarm Log Information on alarms, including the name of the component that
issued the alarm, the date and time at which the alarm was logged,
and the severity of the alarm.
Troubleshooting Spotlight on Windows
This section identifies general problems that you may encounter when using Spotlight on
Windows, and details how to address those problems. If you do not find a solution in this
section then check the Release Notes.
Issues Connecting
The Registry ... is not accessible.
On connecting to a Windows machine, if Spotlight returns the following error:
The registry on "<<MachineName>>" is not accessible. You need to
79
connect to this machine with an account with privileges to
retrieve server information, query the registry, and access
performance monitor objects.
Solution
Login to the Windows machine using an account with administrative rights.
A login with administrative rights satisfies the requirements of Spotlight on Windows, with the
privileges to retrieve server information, query the registry, and access WMI and performance
monitor objects.
Network Name (Path) Not Found
On connecting to a Windows 2000 machine, if Spotlight returns the following error:
Network name not found
or
Network path not found
Solution
Ensure the Remote Registry service is running on the Windows machine.
Spotlight on Windows will return this error if you attempt to connect to a Windows 2000
machine that has the Remote Registry service stopped.
Connecting As The Logged-in User
To connect to a Windows machine as a user who is already logged in, do NOT enter any user
credentials (Domain, User or Password) in the Connection Properties window.
Notes:
l Spotlight on Windows requires Administrator access on all the Windows machines under
diagnosis. If the current user is not an administrator on a machine, that Spotlight
connection will fail.
l Windows connections can only have one set of credentials in use at a time. If a user
connects to a machine by using NO user name and password, and then disconnects and
reconnects with an Administrator password, Windows returns an ERROR_SESSION_
CREDENTIAL_CONFLICT error.
For example, if you have mapped a drive to the machine \\serverA\share while logged in
as serverA\administrator, you CANNOT map another drive to the same machine while
logged in as serverA\user.
You can, however, connect to the same machine with different credentials if you map to
the machine via its IP address.
80
In the example above, if serverA has the IP address 192.168.1.100, you CAN employ the
net use command to connect to the machine \\192.168.1.100\share as serverA\user.
WMI Connection Problems
If Spotlight on Windows is displaying a WMI access denied error, and the user specified in the
connection profile is an administrator on that machine then it may be necessary to adjust your
DCOM or Firewall settings.
Complete the following steps to ensure that your DCOM and Firewall settings are configured to
allow WMI connections.
To test WMI is working on the Spotlight client:
1. Click Start | Run.
2. Enter WBEMTest to run the WMI Tester Utility.
3. Click Connect | Connect.
4. If an Access is denied error is displayed you do not have rights to make WMI
connections. In this case, check (and adjust if necessary) your local DCOM setting
(see below).
To check the DCOM access permissions for the monitored machine:
1. Login to the monitored machine.
2. Click Start | Run.
3. Enter DCOMCnfg to run the DCOM configuration utility.
4. Select Component Services.
5. Click .
6. Click COM Security.
7. Under Access Permissions, click Edit Permission.
Ensure Remote Access is set to Allow for the user specified in the connection profile (or
the group containing this user).
8. Under Launch and Activation Permissions, click Edit Permission.
Ensure Remote Launch and Remote Activation are set to Allow for the user.
For more information, see Securing a Remote WMI Connection on the MSDN site.
To check if Windows Firewall is active on the Spotlight client:
1. Open the Control Panel | Windows Firewall.
2. If Windows Firewall is on, see Connecting Through Windows Firewall on the MSDN site
for information on how to configure the firewall to allow WMI connections.
81
For further information, see How to troubleshoot WMI-related issues in Windows XP SP2 on the
Microsoft site.
Issues Monitoring a Windows Machine
Display Problems on The Home Page Disks Panel And Disk Drilldown
The Disks panel on the Spotlight on Windows home page and the Disks drilldown are not
displaying data. Ensure disk performance counters are enabled on the Windows machine
being monitored.
Solution:Enable collection of disk data:
On the Windows system being monitored
1. Start a command line window
2. At the command prompt, type the following
diskperf -y
3. Restart the system.
Display Problems on The Home Page Network Panel, Flows And
Network Drilldown
Spotlight on Windows cannot display data in the Network panel, flows, and drilldowns. The
likely cause is that the appropriate performance counters have not been enabled on the Windows
machine being diagnosed.
Solution:Enable the collection of network data:
l Check that the PerfNet counters are active on the system. To do so, use the Exctrlst
utility, which you can download from the Microsoft Web site.
l Check that at least one network device is using NBT (NetBIOS over TCP/IP). To do so,
check the properties of all network connections (in particular, Advanced TCP/IP Settings
| WINS) to ensure that the NetBIOS setting is not disabled.
l Check that Service Pack 4 is installed on Windows 2000 systems.
The Processes Page And Process Affinity
In Windows systems where there are multiple processors, an activity (thread) in a process can
migrate from processor to processor and every migration reloads the processor cache.
Process affinity, which you can set on the target Windows machine via the Windows Task
Manager, allows you to establish an association that permits specified processes to run ONLY on
nominated processors. This limits the number of cache reloads, which may be desirable on
systems operating under heavy load.
82
You CANNOT set process affinity via Spotlight on Windows.
What you CAN do is:
Use the Processes drilldown to indicate the CPUs where the specified process is permitted
to run. This enables Spotlight to report the CPU usage for that process with accuracy.
Example: Running a process on an eight-CPU system
Spotlight uses the Windows Performance Monitor (PerfMon) to retrieve important data from
target Windows systems, including the total percentage of CPU usage by specified processes.
On a single-CPU system, PerfMon can report a maximum CPU usage of 100%.
On a multiple-CPU system, that maximum value is multiplied by the number of CPUs. For an
eight-CPU system, PerfMon can report a maximum CPU usage of 800%. Spotlight on Windows
compensates by dividing the reported PerfMon value by the number of CPUs on the system.
However, when the affinity for a process is set so that it can run only on two nominated CPUs of
the eight available, the default Spotlight calculation fails. For example, if PerfMon reports a CPU
usage of 120% for the process (corresponding to a 60% usage on each permitted CPU), Spotlight
initially calculates a value of 15%. Use Spotlight's Configure Affinity function to adjust the
Spotlight calculation.
How to set process affinity
Process affinity can be set via the Windows Task Manager on Windows machines that have
multiple CPUs.
1. Log in to the monitored Windows machine.
2. Press CTRL+ALT+DEL.
3. Click Task Manager | Processes.
4. Right-click the name of the process whose affinity you want to set and select Set Affinity.
Note:This option is unavailable when the system has one CPU.
5. Select which of the available CPUs you want to use for the process.
Note:You can also set process affinity for Microsoft SQL Server through the SQL Server
Enterprise Manager.
Installing The SNMP Protocol
Spotlight is unable to display Open Ports information if the SNMP protocol has not been
installed on:
l The Windows machine under diagnosis, AND
l The machine running the Spotlight client.
83
To install the SNMP protocol (Windows XP)
1. Open the Control Panel | Add or Remove Programs | Add/Remove Windows
components.
2. Select Management and Monitoring Tools from the list of components.
3. Click Details.
4. Select Simple Network Management Protocol.
5. Follow the installation instructions.
Spotlight on Unix
84
Spotlight on Unix
Configure the Unix System
To monitor a Unix system with Spotlight, ensure the Unix system is configured as follows.
Unix servers and versions.
Spotlight on Unix supports the following server operating systems:
l Sun Solaris 9, 10, 11
l HP-UX 11i, 11i v2, 11i v3
l IBM AIX 5.2, 5.3, 6.1
l Red Hat and SUSE operating systems running Linux 2.4 and 2.6 kernels
Unix programs.
Ensure the following Unix programs are accessible to the Unix login (through Spotlight):
l Perl 5.x
l awk
l cat
l date
l df
l grep
l ifconfig
l iostat (not HP-UX)
l netstat
l ps
l sar
l sed
l tr
l uname
l uptime
l vmstat
l wc
l who
For HP-UX, additionally
l bdf
l cstm
l getconf
l swapinfo
l /usr/sbin/ioscan
For AIX, additionally
l lsattr
l lsdev
l lsps
l prtconf
Spotlight on Unix
85
For Solaris, additionally
l /etc/swap
l mpstat
l nawk
l prtconf
Unix User Permissions and Installation Settings
l The Unix user should have no special processing on log-on. In particular there must be no
input required from the user, and nor should any special login banners be displayed.
l On AIX, the user must be a member of the "adm" group to be able to run the
sar command.
l On Linux, the sysstat package must be installed to enable the user to get detailed disk
I/O information.
l On Linux the /proc filesystem must be present.
l For connection using SSH, the sshd daemon must be installed and running.
Remote Connectivity:SSH or REXEC
Spotlight on Unix will require you to select the connection type: SSH or REXEC. Information
on SSH and REXEC is freely available on the Internet. We recommend SSH as password data is
transmitted encrypted. REXEC does not encrypt password data.
Notes:
l Make sure the relevant SSH or REXEC daemon is running on the Unix machine and is
configured to receive remote connections.
l Commands to observe system activity (for example, netstat, vmstat, iostat, sar) must be
accessible to REXEC / SSH sessions for Spotlight to observe Unix activity. Ensure these
commands are located in the search path for REXEC / SSH sessions. If not, Spotlight will
display an error.
Notes (Specific to SSH):
l Spotlight supports both SSH1 and SSH2 protocols.
l To allow Spotlight to make SSH connections to any Unix or Linux hosts that permit SSH
connections, you may need to alter the PasswordAuthentication configuration item in the
sshd_config file. Set the value of PasswordAuthentication to yes. Once you have
modified the sshd_config file you must restart the sshd process to apply the new setting.
l Public-key encryption is supported under SSH2 only. DSA and RSAare supported.
Note (Specific to REXEC): When Spotlight is monitoring a Unix operating system via REXEC
with a valid user ID and password, remote commands may not work on the Unix host unless that
user ID is added to the /etc/hosts.equiv file on the host.
Spotlight on Unix
86
Connect to Unix Systems
1. From the Spotlight Browser select All Connections |Spotlight on Unix
2. Select the machine to connect to. If the Unix machine is not listed then follow the
instructions to add a new connection.
Note: Ensure the Unix system is configured before you connect to it. See "Configure the Unix
System" (page 84) for more information.
1. Click File | Connect
2. Select Spotlight on Unix on the Connections menu.
4. Fill in the Details page of the Properties window:
Field Description
Connection
name
Enter the preferred display name for the Unix machine.
Note: If left blank, the Connection name field resets to the value of
the Address field.
Server Connection Details
Spotlight on Unix
87
Field Description
Address Enter the hostname or IP address to the Unix machine.
User Enter the user name to login to the Unix machine.
Note: The root user is not allowed. Remote login as "root" has been
disallowed for security reasons.
Password Enter the password to login to the Unix machine.
Note:Not applicable when Use SSH Public / Private Keys is
selected. Applicable when Save password details (for this
connection)is selected.
Connection
Type
Select according to the Unix system's configuration. The options are:
REXEC and SSH. See "Configure the Unix System" (page 84) for
more information.
Port
Number
Enter the port number for SSH (secure shell) connection to the Unix
machine. The default value is 22.
Note:Applicable when the Connection Type is SSH.
SSH Key Authentication
Note:Applicable when the Connection Type is SSH.
Field Description
Use SSH
Public /
Private
Keys
Select according to the Unix machine's configuration.
SSH Key
Type
Choose the type of key to use when making the Spotlight
connection: RSAor DSA. See "Configure the Unix System" (page
84) for more information.
SSH Private
key
Filename
Locate the file that contains the private key for the Spotlight
connection.
Either click
or type the filename and location.
SSH
Passphrase
Note:Applicable when Save password details (for this
connection)is selected.
Select Save password details (for this connection)to save all entered password details.
Spotlight on Unix
88
5. Close the dialog.
Click Description
Connect Save changes and open the connection in Spotlight.
Note:If you experience problems connecting there may be an issue with
the Unix configuration. See "Configure the Unix System" (page 84) for
more information.
OK Save changes. Do not open the connection in Spotlight.
Cancel Do not save changes.
Spotlight on Unix Home Page
System Panel
The System panel indicates the status of the operating system:
l The type and version of operating system.
l The length of time since the machine was last booted.
l The date of the last data collection. (system date)
l The time of the last data collection. (system time)
Network Panel
The Network panel shows the logins, connections and movement of data on
the Unix system:
l The number of users currently logged on to the machine.
l The number of external programs with which the machine is currently
communicating. The three types of connections shown are
Established, Time_Wait, and Close_Wait.
l How fast is data moving through the NICs (network interface cards).
The statistics refer to the rate at which network packets and errors are
being received and sent through the NICs.
The flows between the Network panel and the CPU panel represent the rate of data transfer
between the Unix host and the connected network. The flows include the incoming and
Spotlight on Unix
89
outgoing rates of data to and from the Unix host, and the rates at which bad packets are received
from, or sent to, the network.
CPUPanel
The CPU panel shows processor and load information for the Unix system:
l The number of CPUs in the machine.
l The CPU load (as a percentage) across all CPUs on the machine.
l The percentage of time the CPU is in the User state across all CPUs.
l The percentage of time the CPU is in the System state across all
CPUs.
l The percentage of time the CPU is in the Wait state across all CPUs.
Wait is where the machine can only wait for I/O, or something
similar, to complete.
l The length of the run queue in which processes are waiting to be
executed. Processes in this queue will be run when the CPU becomes
available.
l The total number of processes that are running on the machine.
l The number of "zombie" processes: child processes whose termination
has not been acknowledged by their parent process.
l The number of processes waiting for some event or condition before
they can continue execution.
The flows between the CPU panel and the Memory panel represent paging information between
CPU and memory on the Unix host. The flows indicate the rate data is written out of and read in
to memory. The unit of measure is dependent on the Operating System. See "CPUPanel" in the
online help for more information.
Spotlight on Unix
90
Memory Panel
The Memory panel displays detailed information about the physical and
virtual memory on the Unix system. It shows statistics such as:
l The total amount of physical memory in RAM.
l The amount of physical memory that is free.
l The total amount of virtual memory for the system.
l The amount of virtual memory that is free or unused.
l The amount of virtual memory consumed, expressed as a percentage
of its maximum size.
l Where applicable, the number of processes whose pages have all
been moved into swap space.
The flows between the Memory panel and the Swap Space panel represent the rate at which
processes are being swapped out to disk and swapped in from disk.
Disk Activity Panel
The Disk Activity panel shows information on disk usage:
l Activity gauges that display the percentage activity of the disks or
partitions with the highest and second-highest read/write rates.
l The name of the mount point of the disk with the least amount of
free space that is not already 100% full.
Note: You can configure this control to display the total and used
space on ANY available disk on the system. Click View | Options.
For more information see the online help.
l The total size of this disk.
l The amount and percentage of disk space that is currently being used
by this disk.
The flows between the Disk Activity panel and the Memory panel represent the number of
writes the system makes to the disks per second, and the number of reads the system makes from
the disks per second.
Spotlight on Unix
91
Swap Space Panel
The Swap Space panel shows the configured swap space on
the Unix system:
l The total amount of configured swap space on the
system.
l The amount of swap space currently in use.
Spotlight on Unix Alarms
"Spotlight on Unix Drilldowns" (page 93) for more information.
l Snooze the alarm.
Following are the alarms specific to Spotlight on Unix. For possible solutions to the problems
on Unix alarms in the online help.
Alarm Description
Blocked
Processes
alarm
Blocked processes are processes that are waiting on information to become
available in memory, or that are waiting for a shared resource, or for the
completion of disk I/O or network I/O. A high level of blocked processes may
indicate an under-configured machine.
Spotlight on Unix
92
Alarm Description
Input
Error
Packets
Flow
alarm
The Input Error flow represents the number of erroneous network packets
coming into the machine per second. The Input Error Packets Flow alarm
may warrant investigation as even a low error rate can indicate network
problems.
Output
Error
Packets
Flow
alarm
The Output Error flow represents the number of erroneous network packets
sent by the machine per second. The Output Error Packets Flow alarm may
warrant investigation as even a low error rate can indicate network problems.
Page In
alarm
A large number of page-ins may be a symptom of a large number of recent
page-outs (see Page Out alarm below). To find out what process is reading
these pages in, check the resident size of the processes you are running, and
see which ones are increasing in size. Alternatively, a large number of page-
ins may occur when a large process has just stopped, and the kernel decides
to fill that space with memory pages owned by other processes. Check if an
important process has ended unexpectedly.
Page Out
alarm
If an active process asks the kernel for more memory than there is immediately
available, the kernel will write old memory pages out to swap space. This is
known as paging. To stop paging, make sure that there is enough RAM
available to support the size of the processes you want to run.
Percentage
Busy
alarm
The Most Active Disk gauge shows the disk with the highest read/write rates.
The Percentage Busy alarm is triggered when the read/write rate of the most
active disk exceeds a specified threshold. The type of alarm that is activated
is determined by the percentage of read/write activity experienced by the
disk.
Swap
Space
alarm
If the total amount of swap space allocated to a Unix host becomes full, the
machine may halt all processes, or critical actions may be prevented from
occurring. The Swap Space alarm may indicate a runaway process or an
under-configured machine.
Swap In
Flow
alarm
The Swap In flow represents the number of processes swapped from disk per
second. The Swap In Flow alarm may indicate a machine that is under-
configured for its workload.
Swap Out
Flow
alarm
The Swap Out flow represents the number of processes swapped to disk per
second. The Swap Out Flow alarm may indicate a machine that is under-
configured for its workload.
Unix
Status
Spotlight is re-establishing its connection to the Unix system.
Spotlight on Unix
93
Alarm Description
alarm
Used
Percentage
alarm
The Least Space / Disk Space container displays the percentage of used disk
space for a specified filesystem. The Used Percentage alarm indicates that the
specified filesystem is full or filling up. If the filesystem fills up completely,
no more data can be written to it.
Zombie
Processes
alarm
The Zombie button in the CPU panel shows the number of terminated child
processes that have not been acknowledged by their parent process. The
Zombie Processes alarm may indicate the presence of an inferior program, or
an entry in /etc/inittab (or equivalent) that prevents init from completing its
startup sequence.
Spotlight on Unix Drilldowns
Following are Spotlight on Unix drilldowns.
Drilldown Description
Processes
and
Services
The pages in the Processes and Services drilldown list all the processes
running on the Unix machine (including "zombie" processes), and all the
services found in the /etc/services file (running or not). Useful features in this
drilldown include:
l Processes page: View processes in a standard table or tree structure.
In the tree structure secondary processes are shown as "children" of
the processes that run them. Click on a process for more detailed
information on that process including CPU and Memory utilization.
From the Memory Usage chart it is possible to detect memory leaks of
the process over the display period.
l Zombies page: View child processes whose termination has not been
acknowledged by their parent process.
l Services page: Use a shortcut menu option to test whether a specified
service is available.
Activity
Summary
The pages in the Activity Summary drilldown Activity, Disk, Memory,
CPU, Network, and Logins
show details of the different types of activity on the Unix system.
Spotlight on Unix
94
Drilldown Description
Filesystems The Filesystems drilldown displays detailed information about the
filesystems on the Unix machine. This information can be viewed in chart or
table form: right click over the drilldown page.
Alarm Log The Alarm Log drilldown displays information on the alarms associated
with the Unix machine.
Spotlight on MySQL
95
Spotlight on MySQL
Connect to a MySQLDatabase
1. From the Spotlight Browser select All Connections |Spotlight on MySQL
2. Select the database to connect to.
If the database is not on the list of MySQL connections then follow these instructions:
Note: Before you add a new connection ensure the Unix or Windows server on which the
MySQLdatabase is installed is accessible to Spotlight on MySQL.
1. Click File | Connect
2. Select Spotlight on MySQL on the Connections menu.
4. Fill in the Details page of the Properties window:
Spotlight on MySQL
96
MySQLConnection Details
Field Description
Host Enter the name of the host machine for the MySQL database.
Username Enter the user name to login to the MySQLdatabase.
Password Enter the password to login to the MySQLdatabase.
Port Enter the port used for the Spotlight on MySQL connection. The
default value is 3306.
Database Select the name of the database that you wish to connect to.
OSConnection Details
Field Description
Monitor
OS
Select to monitor the operating system on the database server.
Tip: Clear this option if the database is running on an operating system
that is NOT supported by Spotlight on MySQL. If you leave this
option checked and Spotlight on MySQL is unable to collect statistics
from the server operating system, you may experience errors.
Address Enter the address of the host machine for the MySQL database.
l Windows
l Unix(REXEC)
l Unix(SSH)
daemon is running on the database server and is configured to receive
remote connections.
SSH Port Enter the port number for SSH (secure shell) connection to the database
server. The default value is 22.
Note: Applicable when OSType is Unix(SSH).
Username Type the user name to login to the database server.
Note: When OSType is Windows:
l Not required if your login details have been assigned remote
access to the registry of the database server.
l Ensure the user has access to the registry of the server.
Password Type the user password to login to the database server (if required).
Spotlight on MySQL
97
5. Click Save password details (for this connection) to save all the entered
password details.
Note: Repeat the above procedure to add a connection to another MySQLdatabase.
Spotlight on MySQL Home Page
Sessions Panel
The System panel displays information about user sessions on
the MySQL database under investigation. Information
displayed here includes:
l The number of user sessions currently connected to the
MySQL server.
l The number of sessions connected to the MySQL
server that are currently executing an SQL statement or
other database request.
l The percentage of the currently connected sessions that
are currently executing a SQL statement or other
database request.
The flows between the Sessions Panel and the MySQL panel
represent the flow of data between the MySQL server and all
clients. The flows include:
l The rate at which client connections are sending SQL
statements or other database requests to the server.
l The rate at which data (including command packets) is
being sent to the server from all clients.
l The rate at which data is being sent from the MySQL
server to all clients. Most of the data transmitted from
the server to clients is generally in the form of result
sets from SELECT statements.
Spotlight on MySQL
98
MySQL Panel
The MySQL panel displays information on the performance of queries on
the MySQL database. Information displayed here includes:
l The size of the sort buffer and the number of rows sorted per second.
l The current activity in the MySQL query cache, including the rate at
which queries are being served directly from the query cache.
l The number of connection threads currently pooled for re-use.
The flows between the MySQL and the InnoDB panel to its right represent
the rate of operations between the database and the InnoDB storage engine
(if used). The flows include:
l The rate at which write operations are performed on the InnoDB
buffer pool.
l The rate of logical read requests performed by InnoDB.
l The rate at which data is being written to the InnoDB log buffer.
Spotlight on MySQL
99
InnoDB Panel
The InnoDB panel provides an overview of the activity within the
transaction-capable InnoDB storage engine, if that storage engine is used by
the MySQL database under investigation. Information displayed here
includes:
l The size, availability and hit rate of the InnoDB buffer pool.
l The rate of insert, update, delete and select operations expressed in
rows processed per second.
l The size of the InnoDB log buffer.
l The number of threads performing I/O for InnoDB.
The flows between the InnoDB panel and the Storage panel represent the
rate of I/O operations between the InnoDB storage engine and tablespace
files on disk. The flows include:
l The rate at which InnoDB writes data to tablespace files on disk.
l The rate of of physical I/O operations from the InnoDB tablespace
files into the InnoDB buffer pool.
l The I/O rate to the InnoDB transaction log.
Spotlight on MySQL
100
Storage Panel
The Storage panel shows details of the physical storage of tablespace files
and transaction logs for the database under investigation. Information
displayed here includes:
l The total size of the InnoDB tablespace files and the total amount of
free space in those files.
l Whether tables in the InnoDB tablespace can be set to autoextend
when they become full.
l The number and size of the transaction logs used by the InnoDB
storage engine.
Host Panel
The Host panel contains relevant information about the
machine that hosts the MySQL server. Information displayed
here includes:
l The total amount of CPU being used. Sustained high
CPU utilization can adversely affect the performance of
the system.
l The number of tasks waiting for CPU resources.
l The available physical RAM on the host.
Spotlight on MySQL
101
Spotlight on MySQL Alarms
"Spotlight on MySQL Drilldowns" (page 102) for more information.
l Snooze the alarm.
Following are the alarms specific to Spotlight on MySQL. For possible solutions to the problems
on MySQL alarms in the online help.
Alarm Description
Buffer
Pool Hit
Rate
alarm
The Buffer Pool Hit Rate alarm fires when the proportion of table data that
can be found in the InnoDB buffer pool drops below a specified threshold.
CPU Busy
alarm (%
CPU
usage)
The CPU Busy alarm occurs when the total CPU utilization of the system
exceeds a threshold. The CPU may encounter a large number of requests, or
you may have un-tuned SQL, which unnecessarily uses excessive amounts of
CPU.
Inefficient
Sort alarm
(Merge
per Sort)
The Inefficient Sort alarm fires when the number of temporary files created for
sorts is high in comparison to the number of sorts performed.
Low Free
Physical
RAM
The Low Free Physical RAM alarm occurs when the servers available RAM
is low.
Spotlight on MySQL
102
Alarm Description
alarm
Query
Cache Hit
Rate
alarm
The Query Cache Hit Rate alarm fires when the proportion of SQL queries
that can be found in the MySQL query cache drops below a specified
threshold.
Spotlight on MySQL Drilldowns
Following are Spotlight on MySQL drilldowns.
Drilldown Click
to
open
Description
Top
Sessions
The pages in the Top Sessions drilldown display information about
the users connected to the MySQL database.
Activity The pages in the Activity drilldown display details of the activity
on the database you are diagnosing.
Operating
System
The pages in the Operating System drilldown display information
related to the performance of, and processes on, the operating system
where the MySQL server is running.
Alarm
Log
The Alarm Log drilldown displays information on the alarms
associated with the MySQLdatabase.
Troubleshooting
103
Troubleshooting
Monitoring
Disabled Controls
Problem
I have a number of disabled controls on a Spotlight page. Every disabled control has one of these
icons associated with it:
Solution
To solve the problem, view the Overhead Management page in the Component Properties editor
for one of the disabled controls. You should find that the refresh setting for that component has
been set to Do not refresh this control. See the online help for more information.
Background to The Problem
When Spotlight collects data from a system under investigation, it places a load on that
system's resources.
Usually, the Spotlight load does not have a significant effect on the performance of the system,
but when it does you can use Spotlight's overhead management tools to reduce or eliminate the
load. Two factors influence overhead management:
l The categories of data collected.
l The refresh rate assigned to each category.
The categories of data collected are defined by the current Spotlight application, and are set by
criteria that may include:
l The kind of data being collected.
l The load that collecting the data will put on the system.
l Where the data in the category is to be displayed.
l The importance of the data.
l How often the data is needed.
By default, every category is assigned one of the following refresh rates:
Troubleshooting
104
l Always refresh at the foreground rate.
l Refresh at the default rate.
l Refresh only while data is displayed on the screen.
l Do not refresh data in this category.
When you change the refresh rate for the data in a Spotlight control (a component, table or
chart), you also change the refresh rate for ALL data in the same category. When you disable
the refresh rate for the data in a control, you disable the refresh rate for ALL data in the
same category.
A control whose data collection is disabled displays one of the indicated icons:
Icon Description
Data for this control is not available. (Alarms associated with the control are
disabled.)
Data for this control is not available. (There are no alarms associated with this
control.)
Alarms Not Raised
Problem
A metric displayed on the Spotlight home page contains a value that should have raised an
alarm, but there is no alarm.
Solution
l Check there is a realistic threshold value for the metric. See the online help for more
information.
l Has Spotlight had time to raise a new alarm? See "Delay in Reporting Alarms" (page 105)
for more information.
Extra information for Spotlight on Oracle RAC
Within Spotlight on Oracle RAC the rules that govern alarms are complicated by the fact that
Spotlight is diagnosing the operation of a cluster of database instances as well as the overall
behavior of the cluster itself.
Perhaps Spotlight has inhibited the alarm because it has not met a secondary criterion. One such
criterion is that of "low load", where a cluster element is essentially idle.
This is an important consideration, for example, when the Balance gauge on the Instance Panels
(page 62) shows an imbalance without raising an alarm. Even though one database instance has a
higher workload than others in the cluster, Spotlight has determined that the overall workload on
the cluster is too low for the imbalance to be a problem.
For more information on this, see Cluster | Balance Page.
Troubleshooting
105
Delay in Reporting Alarms
Problem
The current displayed value of a Spotlight metric should be raising an alarm, but that alarm has
not yet been reported.
Solution
There are two parts to this problem:
l Refresh rates
Spotlight refreshes the metric data it collects at a rate specified in Spotlight Options. In
most cases (but not all) this is the foreground rate when the control is visible, or the
background rate when the control is not visible. These refresh rates (which you can set)
are usually set to different values, with foreground data being collected more often. See
l Metric smoothing
Spotlight uses metric smoothing (averaging) to prevent the generation of spurious alarms.
It does this by averaging the data that it collects over a specified period of time. See the
online help for more information.
The combination of these two features means that an alarm is raised faster for metrics in the page
you are viewing than it is for those in a page that is hidden.
Alarm Help Is Displayed Instead of The Drilldown
Spotlight can be configured to show the help or jump directly to the related drilldown on
clicking a home-page control.
To show the drilldown
1. Click View |Options.
2. Select Alarms and notifications | Balloon help.
3. Select Always jump directly to the related drilldown.
Spotlight Home Page Hard To Read
The screen resolution is too low.
Increase the number of colors your computer is using. Spotlight requires a monitor that supports
65536 colors or higher (or Hi Color/16-bit) so that it can properly shade its graphs.
1. Open the Windows Control panel and double-click Display.
2. Click the Settings tab.
Troubleshooting
106
3. Increase the number of colors in the palette. If this option is not available you should
upgrade your hardware.
NoChange to Data Flows
Either the refresh rate is too slow or Spotlight is paused. Do the following:
l Update the screen. Click View | Refresh.
l Check the foreground refresh rate setting (try a rate of approximately 30 seconds).
To view / change the rate click View | Options and select Data collection | Data
refresh rates
l Resume Spotlight diagnostics. Click View | Resume. (If Resume is not available,
Spotlight is not paused.)
Operating System Connection / Statistics
Differences between versions of operating systems mean that Spotlight is unable to collect all
O/S statistics in all environments. Refer to the Spotlight on Oracle Release Notes for more
information.
Connecting to Windows operating systems
The following error message is displayed:
Connection has been refused by the server
This means that Spotlight cannot access the operating statistics of the (Windows) database
machine. Users must have access to the registry so that Spotlight can retrieve the operating
system statistics. A user account with administrative rights to the target machine will allow this.
Connecting to Unix operating systems
Spotlight on Unix works with full functionality only when the iostat command is installed on
the Unix system. This is NOT true for (HP-UX).
If iostat is NOT installed, Spotlight may display an error message such as iostat: command not
found. If you encounter this error, you can:
l Choose to ignore the error. When you do so, the Spotlight error will no longer appear,
but neither will a number of important disk statistics.
l Install the iostat command on the server. (On Red Hat Linux, this command is included
in the sysstat package.)
Note: While diagnosing a MySQL installation on a Linux server, Spotlight on MySQL works
with full functionality only when the iostat command is installed on the Unix system.
Troubleshooting
107
When monitoring a database, what happens to the Operating System drilldown when
connection is lost to the server?
The Operating System drilldown may report inaccurate data if Spotlight looses access to the
database server.
1. Reconnect to the database. When the Connect dialog is displayed ensure that the OS
server and OS type fields (and if necessary the OS Username and OS Password) are
completed correctly.
2. Ensure the Unix or Windows server on which the Oracle database is installed is
accessible to Spotlight on Oracle. See Spotlight on Unix | Configure the Unix System
(page 84) or for Spotlight on Windows there could be problems with Remote
Connectivity, see Troubleshooting Spotlight on Windows (page 78).
Extra information for Spotlight on Oracle connections and Unix operating systems
If the database is running on Unix, Spotlight on Oracle may be using the REXEC or SSH
protocols to submit various Unix commands (for example, netstat, vmstat, iostat, sar) to analyze
system activity.
Example
If Spotlight uses REXEC, these commands must be accessible to an REXEC session for
Spotlight on Oracle to be able to analyze Unix activity. If these commands are not located in the
standard search path available to REXEC, Spotlight on Oracle displays an error.
Ensure that all Unix commands are located in the standard search path available to REXEC, or
one of the following paths:
/usr/bin
/usr/sbin
/usr/ucb
If the command is located elsewhere in the file system, you must create a symbolic link in one of
the areas listed above.
When observing a Unix operating system with a user ID/password that exists on the machine, it
is possible that remote commands will not work on the machine unless the user ID is added to
the /etc/hosts.equiv file on that Unix machine.
Extra information for Spotlight on Oracle RAC connections and Windows operating systems
My Spotlight on Oracle RAC connection is configured to collect operating-system (OS) data for
the cluster under investigation, but no OS data has been collected. Where the Monitor OS
option is selected (and the corresponding OS Username and OS Password details entered) in the
Connection Properties dialog, make sure that the specified OS user account is valid for ALL
nodes in the cluster. If not, add the relevant user account to all nodes where it is not valid. See
also Connect to Spotlight on Oracle RAC (page 55).
Troubleshooting
108
Note: While the correct Oracle connection details for an Oracle RAC cluster will automatically
be valid for all the Oracle instances in the cluster, the same is not necessarily true for the OS
connection details of each node in the cluster.
I Cannot Open a Migrated Connection
Problem
Spotlight is unable to open a migrated connection.
Solution
When you open a Spotlight connection, you use connection details that are stored in files
somewhere on your local computer or (in the case of migrated connections) somewhere on
your network.
When Spotlight cannot access those connection details (for example, when the relevant network
machine is unavailable), it displays a Spotlight message box that you can use to change the
location where connection details are stored. The choices you have are:
Option Description
Yes Click to open the Spotlight Migration wizard to change the location where
connection details are stored.
No Click to abandon the attempt to open the desired Spotlight connection.
Retry Sometimes the network location that you have been using is only temporarily
unavailable. Click to reuse the existing connection details to open a Spotlight
connection.
Note: You should not migrate connections to locations that may be offline frequently or for
extended periods. You cannot migrate connections that are currently in use or inaccessible.
Spotlight Consumes Too Many Resources on Either The
Client Or The Server
To reduce Spotlight's consumption of client and server resources, try one of these solutions:
Action Description
Reduce
refresh rates.
Reduce the refresh rates at which Spotlight collects data. For example,
instead of the default periods of 15 seconds (foreground) and 60 seconds
(background), try periods of 30 seconds and 120 seconds respectively. See
Use
Overhead
Management
Use Overhead Management to decrease the load that Spotlight puts on the
database under diagnosis. See the online help for more information.
Troubleshooting
109
Support
Using Trace To Add Details To Support Bundle
When you contact Quest support to seek assistance on a Spotlight issue, you can use Spotlight
tracing to include additional information in the support bundle that you send to Quest.
To turn on Spotlight tracing
1. Close all open connections.
2. Click View | Options | Tracing
3. Select Turn client tracing on to turn on tracing for the computer Spotlight is running on.
4. Select Turn server tracing on to turn on tracing for the computer systems to be
monitored.
5. Click OK.
6. Open the connection whose performance you want to trace.
With tracing turned on, Spotlight can collect additional information (such as the output from
operating-system commands) for the rest of the current session.
When you create a support bundle with tracing turned on, the support bundle contains the
additional information that Quest can use to investigate your support issue.
See "Contact Quest Support" (page 111) for more information.
Note: All tracing information gathered is from the Spotlight client. There is no server tracing
capability.
Build the Support Bundle
In emailing a request for assistance to Quest Software (support@quest.com) be sure to include a
support bundle.
To create the support bundle
1. Click Help |Support Bundle.
2. Select the Spotlight module(s) whose information you want to collect.
3. Click Collect.
A file is created containing a snapshot of your Spotlight installation. The file is called
SpotlightSupport.zip
Note: You can find SpotlightSupport.zip in the
folder:C:\Users\<user>\AppData\Roaming\Quest
Software\Spotlight\Default\Support
Troubleshooting
110
For Windows XP / Windows Server 2003 the folder is C:\Documents and
Settings\<user>\Application Data\Quest Software\Spotlight\Default\Support
Appendix: Contact Quest
Contact Quest Support
Quest Support is available to customers who have a trial version of a Quest product or who have
purchased a Quest product and have a valid maintenance contract. Quest Support provides
unlimited 24x7 access to our Support Portal at www.quest.com/support.
From our Support Portal, you can do the following:
l Retrieve thousands of solutions from our Knowledge Base
l Download the latest releases and service packs
l Create, update and review Support cases
View the Global Support Guide for a detailed explanation of support programs, online services,
contact information, policies and procedures. The guide is available at: www.quest.com/support.
Spotlight Support
Whenever you have a question about Spotlight, click Support Bundle or Contact Support on the
Help menu. This creates a file called spotlightsupport.zip in
C:\Users\<user>\AppData\Roaming\Quest Software\Spotlight\Default\Support
For Windows XP / Windows Server 2003 the file is created in C:\Documents and
Settings\<user>\Application Data\Quest Software\Spotlight\Default\Support
This file contains a snapshot of your Spotlight installation. Send this file and your request for
assistance to Quest Software (support@quest.com).
Notes:
l Support bundles contain information for the active instance only. If you are using
Spotlight to diagnose multiple instances ensure that the correct instance is displayed
before creating the support bundle.
l Support bundles cannot be created when the History Browser is open.
Appendix: Contact Quest
112
Contact Quest Software
Email: info@quest.com
Mail: Quest Software, Inc.
World Headquarters
5 Polaris Way
Aliso Viejo, CA 92656
USA
Web site: www.quest.com
Refer to our Web site for regional and international office information.
About Quest Software
Quest Software simplifies and reduces the cost of managing IT for more than 100,000 customers
worldwide. Our innovative solutions make solving the toughest IT management problems easier,
enabling customers to save time and money across physical, virtual and cloud environments. For
more information about Quest go to www.quest.com.
Index
A
about Quest Software 112
activity drilldown 11, 42
aggregated alarms 64
alarm log 78
alarm log drilldown 34, 45, 64, 91, 94,
101-102
alarms 11, 34, 63, 76, 101
event log 74
analyze trace 42
architecture diagram, Oracle 9
architecture diagram, Oracle RAC 55
array fetch 42
ASM 23, 35, 43, 48, 57, 63, 65
ASM drilldown
no data 58
authorize Spotlight 18
Automated Storage Management 23, 35, 43,
48, 57, 63,
65
B
background processes panel flows 32-33
C
Close_Wait 88
configuration and memory drilldown 43,
52
connect
Spotlight on MySQL 95
connect to Oracle database 22
connect to Oracle RAC 55
connect to Windows 71
Contact Quest Software 112
cpu 77-78
cpu panel 74, 89
cpu panel flows 73-74, 88-89
Customer support 111
D
data flows
troubleshooting 106
disk activity panel 90
disk activity panel flows 90
disk performance counters 81
disk requests 75
disk storage drilldown 43
disk storage panel 34
disk storage panel flows 31-33
disks drilldown 81
disks panel 76, 81
disks panel flows 75
drilldowns 11, 41, 67, 77, 102, 105
event log 74
Spotlight on Windows 46
DSA 25
Index
114
DTrace 13, 28, 44
E
ESX 26, 30
event log 77-78
event log panel 74
event waits 11
Exadata Storage 63
explain plan 51
F
fast initialize 48
file size tracking 76
files panel 76
H
help
support bundle 109
home page 10, 29
Spotlight on Oracle RAC 60
troubleshooting 106
host panel 30, 100
I
I/O drilldown 43, 52
InnoDB panel 99
install 14
instance panels 62
instance panels flows 61
interconnect and global status panel 61
interconnect panel flows 61
IO subsystem panel 63
iostat 106
M
memory 77-78
memory panel 75, 90
memory panel flows 74-75, 89-90
MySQL panel 98
N
network 78-79
network panel 73, 88
network panel flows 73, 88
O
operating system drilldown 43
Oracle architecture 9-10
Oracle Enterprise Manager 31
Oracle RAC architecture 54
Oracle user wizard 19, 53
P
paging files panel 75
paging files panel flows 75
parse activity 42
permissions 14-15, 17, 19
predictive diagnostics 12, 20, 33, 39, 44,
53
preview mode 18
privileges 14-15, 72, 78
Spotlight user 19
processes 78, 89-91
processors 74
Project Lucy 17
Index
115
Q
Quest Software 112
Quest support 111
R
result cache 42
REXEC 25, 58, 86, 96
RSA 25
S
server processes panel 31
server processes panel flows 30-31
sessions drilldown 42
sessions panel 30, 97
sessions panel flows 30
Set timed statistics now 22
sga panel 32
sga panel flows 31-32
sort activity 42
space manager 34, 46
Spotlight on Unix
alarms 91
drilldowns 46
Spotlight users 19
SQL & application workload drilldown 42
SQL optimizer 45, 51
SSH 25, 58, 86, 96
start Spotlight 17
storage panel 100
Support 111
support bundle 109
swap space 92
swap space panel 91
system panel 73, 88
T
Technical support 111
Time_Wait 88
Toad DBA Suite for Oracle 15-16, 27
top sessions drilldown 11, 49
top SQL drilldown 42, 51
trial key 18
tuning drilldown 43, 52
U
uninstall 16
user, Spotlight 19
V
virtual system 26, 30, 44
W
wait activity 11
waiting events 52
Windows 25, 58, 96
Z
zombie processes 89, 93

Spotlight On Oracle Getting Started

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Spotlight On Oracle Getting Started

Transféré par

Droits d'auteur :

Formats disponibles

Spotlight

Vous aimerez peut-être aussi