Spotlight on SQLServer 8.

0
Getting Started Guide

 2011 Quest Software, Inc. ALL RIGHTS RESERVED. This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchasers personal use without the written permission of Quest Software, Inc. The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document. If you have any questions regarding your potential use of this material, contact: Quest Software World Headquarters LEGAL Dept 5 Polaris Way Aliso Viejo, CA 92656 email: legal@quest.com Refer to our Web site (www.quest.com) for regional and international office information. Patents Protected by U.S. Patent #6,901,582. Additional patents pending. Trademarks Quest, Quest Software, the Quest Software logo and Spotlight are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. For a complete list of Quest Softwares trademarks, please see http://www.quest.com/legal/trademark-information.aspx. Other trademarks and registered trademarks are property of their respective owners. Spotlight on SQL Server 8.0 Getting Started Guide Monday, 16 May 2011

Table of Contents
Use Spotlight on SQL Server Diagnose CPU Diagnose IO Diagnose Blocking View Historical Data
Understand Playback

5 5 6 8 10
11

Identify Expensive SQL

Turn on and configure SQL Analysis Real-Time Data and Expensive SQL Data Over a Selected Time Frame and Expensive SQL Filter Displayed and Collected Data

12
12 14 15 16

Report on Database Growth Analyze Wait Statistics Find Long Running SQL Configure the Virtualization Overhead Guage
Connections to VMware Servers

18 20 21 22
23

Troubleshoot Spotlight on SQL Server Installation Issues

Remote installation of the diagnostic server fails

24 24
24

Diagnostic Server Issues Connection Issues

Problems connecting to a SQL Server Instance

24 25
26

WMI Issues
Error 0x800706BA Error 0x80070005 Error 0x80041010

26
26 27 27

Spotlight on SQL Server Getting Started Guide Table of Contents

Error 0x80041017

29

Miscellaneous Issues
Monitor Multiple SQL Server Instances Unable to perform certain operations Missing SQL Server Performance Counters Session Trace Errors Playback

29
29 29 30 30 31

Connections to SQL Servers Appendix: Contact Quest Contact Quest Support

Spotlight Support

32 35 35
35

Contact Quest Software About Quest Software Index

36 36 37

1
Use Spotlight on SQL Server
Diagnose CPU
Spotlight continually monitors CPU usage on the target server and alarms if it crosses a threshold. The thresholds are: CPU Usage 0 67% 67 80% 80 93% 93 100% Alarm severity Normal Low Medium High

A CPU alarm indicates that a bottleneck on CPU is likely. This bottleneck could exist for a number of reasons expensive queries could be running, excessive compilation or recompilation could be occurring, or a process on the server, not even related to SQL Server, is hogging the CPU. With Spotlight, you can investigate the causes of a bottleneck both in real-time and in the immediate past. When Spotlight detects that there is a bottleneck on the CPU, an alarm is raised. You can see alarms on the Spotlight home page, and in Spotlight Today.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

Our first order of business is to find out what process is using all of the CPU. From the Spotlight Home Page Click Diagnostics. From Spotlight Today Select the alarm and click Diagnose.

This opens the Diagnostics drilldown. The Diagnostics drilldown helps you answer questions such as:
l l l

Why is CPU usage higher than usual? Is the SQL Server causing high CPU usage? If it is the SQL Server, is it optimizing queries or executing them?

The first grid on the CPU page shows possible causes of CPU pressure and the likelihood that each is contributing to high CPU usage on the server being monitored. Click an item in the grid to display an explanation of the diagnosis and supporting information.

Diagnose IO
Sustained high I/O wait time rates are a good indicator that of a disk subsystem bottleneck and that I/O device service times will be degraded. When Spotlight detects unusually high disk activity on the SQL Server machine it raises an alarm. Spotlight looks at SQL Server I/O wait times to determine if there is an IO bottleneck. A value larger than 15 ms/IO indicates a possible bottleneck. The thresholds for the I/O wait time alarm are:

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

I/O Wait Time 0 - 15 15 - 50 50 - infinity

Alarm severity Normal Medium High

After detecting a potential disk bottleneck, Spotlight displays an alarm on the home page and in Spotlight Today.

The first thing we need to do is identify the process that is generating high amounts of I/O activity. From the Spotlight Home Page Click Diagnostics |IO. From Spotlight Today Select the alarm and click Diagnose.

This opens the I/O Diagnostics drilldown. The I/O Diagnostics drilldown helps you answer questions such as:
l l l l

Why is the I/O higher than usual? Is the high I/O due to paging activity? Is the SQL Server causing the high I/O? If the SQL Server is causing the high I/O, is it through query executions, scan operators, PAGE_IO_LATCH or LOG waits?

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

The first grid on the I/O page shows the possible causes of high I/O and the likelihood of each contributing to this instance of the alarm. Click an item in the grid to display an explanation of the diagnosis and supporting information.

Diagnose Blocking
Blocking in SQL Server occurs when a session that was performing some task is unable to progress because it must wait on a resource that is currently being used by another session. The resource that is being waited on can be either a physical structure like a lock on a table or an internal SQL Server resource such as a latch. Excessive blocking can be a major cause of poor application performance since it reduces the throughput of the system. Often a user of an application does not realize that they are waiting on a resource held by another user. From their point of view, it often seems like their application has stopped responding. When diagnosing blocking, you want to start your investigation by answering the following questions:
l l l

Who is waiting on what? How long have they been waiting? What SQL was running while they were waiting?

Spotlight not only alerts you to blocking issues in your database but also helps you answer the above questions so you can quickly resolve the problem causing the blocking.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

When Spotlight detects a blocking issue, an alarm is displayed on the home page.

The first thing we want to do is look at the Blocking drilldown. The Blocking drilldown shows details about the current blocks and in particular, answers the questions stated above. On the Spotlight homepage, click the Blocked Processes button, and then click the Blocking drilldown.

The Blocking grid shows the sessions that are either blocked or are causing other sessions to be blocked. The hierarchy in the tree diagram makes it easy to see which session is causing the blocking. In addition, the resource on which the sessions are waiting is displayed. You can ask Spotlight to resolve these to actual object names by clicking on the ? button. The time spent waiting and the SQL last executed are also displayed.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

10

From here, you can view session details. 1. Select the entry in the grid. 2. Click View Session Details.

View Historical Data

Note: This feature has limited support in the Desktop Edition. A maximum of four hours playback data is available. Scheduling cannot be configured. In Spotlight, you can view events and data collected in the recent past as though they were happening in real-time. You can do this via the Playback ribbon The Spotlight home page and page drilldowns show details in playback identical in format to that displayed in "live" state, so you can view historical data in the same way as you view a live connection. To playback time 1. Select a connection from the Spotlight browser. 2. Select the Monitor ribbon tab. 3. Click to show: Alarms Description Select an alarm or moment in time in the past to return to and view in the Spotlight home page and page drilldowns. Playback

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

11

Alarms

Description Return to present time the Spotlight home page and page drilldowns.

Real Time Rewind

Go back in time on the Spotlight home page or page drilldown. Click the associate arrow to define how far back to travel: 5 minutes, 10 minutes, 1 hour or 1 day. Starting from the past, skip forward in time on the Spotlight home page or page drilldown. Click the associate arrow to define how far forward to skip: 5 minutes, 10 minutes, 1 hour or 1 day. Starting from the past, step forward in time through the alarms in sequence. Show on the Spotlight home page or page drilldown. Click the associate arrow to define the speed of play.

Skip

Play

Understand Playback
Note: This feature has limited support in the Desktop Edition. A maximum of four hours playback data is available. Scheduling cannot be configured. Click Playback and a new control opens.

Control

Description Select the date. Use the back and forward buttons to change the date.

Click the time to playback to. The alarms list scrolls to show those alarms occurring at the selected time. Note:The time scale is colored according to the most severe alarm for the connection.

The alarms occurring on the selected date. Click an alarm to playback to the time the alarm occurred. Note:The color indicates the severity of the alarm.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

12

Identify Expensive SQL

Identifying expensive SQL can be a time consuming process. Use the SQL Analysis feature in Spotlight to identify SQL that is consuming resources on your server. With SQL Analysis, you can quickly answer questions such as:
l l l l

What SQL has executed the most? What SQL consumed the most CPU or I/O in total? What SQL consumed the most Average CPU or I/O? What SQL took to longest time to execute?

Answering these questions will help you to direct your tuning efforts more productively. SQL Analysis uses either ROWSET or Server Side SQL Server trace to collect raw data on SQL statement execution. It then aggregates this data by similar SQL statements so that potentially thousands or even millions of rows of raw data are distilled down into an aggregate picture of workload. You can set filters at point of collection so that only the events that you are interested in are collected and when viewing the data, you can also filter the data to narrow the focus down even further.

Turn on and configure SQL Analysis

1. Click Configure | SQLAnalysis Filters

2. Choose one of the following from the connection list:

l l

To change the setting for an individual connection, select the connection name. To change the setting for all connections, select Default Settings.

3. Select Override the default settings. 4. Select Enable SQL Analysis. 5. Click Data Collection. 6. Configure the following fields on the Data Collection page: Option SQL Analysis is scheduled to collect every n minutes Description Note: This feature is not available in the Desktop Edition. How often Spotlight collects SQL Analysis data. The default is six hours.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

13

Option

Description To change how often Spotlight collects SQL Analysis data a. Click the link. b. Select Override the default settings for the collection 'SQL Analysis'. c. Under Store reporting data in the Spotlight Statistics Repository, click the Store data link. d. Set the collection schedule. Data can be collected at regular intervals or at a specific time on one or more days.

Use continuous Rowset trace

Select to use Rowset trace to collect SQL Analysis data. Rowset trace is convenient though may affect throughput on busy servers. It should be used with care. It is the default collection method. Select to use sampling Server Side trace to collect SQL Analysis data. Server Side trace is more suitable for busy servers than Rowset trace. Server Side trace requires additional configuration.

Use sampling Server Side trace

Run trace for or until trace file reaches

Server Side trace will run until one of these conditions is met. Specify the maximum amount of time Server Side trace should run for. Specify the maximum size of the trace file.

Server trace file location (on the SQL Server host) Retrieve data through SQ Server

Type a location for the server trace file. The path is relative to the SQL Server instance. For example, C:\Trace Files. Select to process trace files on the SQL Server instance. This option has the potential to affect throughput on busy servers. Use this option when there are permission or firewall settings preventing the Diagnostic Server machine from retrieving files from the SQL Server machine. This is the default option.

Retrieve data from the file system

Select to process trace files on the Diagnostic Server. The Diagnostic Server machine must have the appropriate

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

14

Option

Description permissions required to connect to the SQL Server instance. This option is not available when configuring default settings.

Trace file location (from Diagnostic Server host)

Type the location of the server trace files on the SQL Server instance. The location should be a shared folder the Diagnostic Server can connect to. For example, \\computername\Trace Files. This option is not available when configuring default settings.

Real-Time Data and Expensive SQL

You can use the SQL Analysis grid when load testing in a testing environment. It shows SQL Analysis data in real time. If you are trying to answer the question I am simulating application workload in my test environment and I want to look at a breakdown of SQL statements in real-time. I am interested in what SQL statements are consuming the most average CPU right now., you would want to look at an aggregate of SQL statement executions (workload) in real-time. The SQL Analysis grid in the SQL Activity drilldown allows you to do exactly that. 1. Select a SQLServer connection from the Spotlight browser. 2. Click Monitor |SQLActivity.

3. Select SQLAnalysis. 4. Do the following: To identify SQL that consumed the most CPU The average CPU consumed by a particular statement SQL that was executed the most SQL with the highest logical I/O Tips: Sort the grid by Total CPU Average CPU

Execution Count Average Reads

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

15

Use filters to refine the data shown in the grid. See "Filter Displayed and Collected Data" (page 16) for more information. Some columns are hidden by default. To view hidden columns, right-click the grid header row and select Organize Columns. Results are cleared when you leave the SQL Analysis grid. Click Clear Grid to clear the grid manually.

To wrap a column onto multiple lines, right-click the data content of the grid and select Properties | Options. Note: Word wrap may degrade the performance of Spotlight.

If Quest SQL Optimizer is installed, you can use it to tune the non-conforming SQL that has been identified. To do this, click Optimize SQL.

Data Over a Selected Time Frame and Expensive SQL

Note: This feature is not available in the Desktop Edition. The SQLAnalysis - Workload view shows the aggregate workload over a selected time period. Use the SQL Analysis - Workload view when diagnosing slow response times. If you are trying to answer the question What SQL statement consumed the most CPU over month end processing?, you would want to look at an aggregate of SQL statement executions (workload) over a time range. The SQL Analysis Workload view allows you to do exactly that. 1. Select Reporting and Trending |Views | SQLServer Performance | SQLAnalysis Workload in the Spotlight Browser.

2. Click Report | Select Connections and select at least one connection.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

16

3. Select a Time Range.

4. Click Refresh.

5. Do the following: To identify SQL that consumed the most CPU The average CPU consumed by a particular statement SQL that was executed the most SQL with the highest logical I/O Sort the grid by Total CPU Avg CPU

Execution Count Avg Reads

Tip: Use filters to refine the data shown in the grid. See "Filter Displayed and Collected Data" (page 16) for more information.See "Filter Displayed and Collected Data" (page 16) for more information.

Filter Displayed and Collected Data

To filter the data displayed Start from the data displayed in the SQLAnalysis grid or the SQLAnalysis Workload view SQLAnalysis grid 1. Click Change Filter. 2. Select Filter results. To add a filter to the list 1. Click Add. 2. From the Column list, select the column you want to base your filter on. 3. From the Condition list, select a condition to apply to the column. 4. In the Value field, type a value to filter events by. You can use the percent sign (%) wildcard with the is like/is not like condition. SQLAnalysis Workload view 1. Click Filter Data. 2. Select Filter results.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

17

Tips:
l

You cannot use wildcard characters for the Database name when connected to a SQL Server 2000 instance. To turn off filtering, clear the Filter results checkbox. To edit or delete a filter select it in the filter list and click the appropriate button.

l l

To filter the data collected 1. Click Configure | SQL Analysis Filters.

2. Choose one of the following from the connection list:

l l

To change the setting for an individual connection, select the connection name. To change the setting for all connections, select Default Settings.

3. Select Override the default settings. 4. Ensure the Filters page is selected.
l

Click Add. Tips:

l l

You can use the percent sign (%) and underscore (_) as wildcards. You cannot use wildcard characters in the Database name field when connected to a SQL Server 2000 instance. When filtering on a string field such as DatabaseName or ApplicationName, use only the "LIKE" and "NOTLIKE" operators. The other operators apply only to numeric values.

Tips:
l l

Use the Arrow buttons to change the order of the filters. Edit or delete a filter by selecting it in the filter list and clicking the appropriate button. Filtering affects the average, minimum, maximum, and last values displayed in the grid, as calculations are performed only on data retrieved from the SQL Server instance. To turn off filtering, on the Filters page, click Clear. This removes all filters.

5. You can specify the number and type of SQL Executions retrieved from the SQL Server instance

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

18

a. Select the Advanced page. b. In the Aggregate the top n SQL statements field, enter the maximum number of SQL executions to be retrieved from the SQL Server instance. c. From the Sorting by list, specify the criterion you want Spotlight to use to determine what the "top" SQL statements are. Choose from Average Duration, Average CPU, or Average IO. Spotlight aggregates the data collected and then stores only the aggregated data.

Report on Database Growth

Note: This feature is not available in the Desktop Edition. To report on database growth across multiple SQL servers, use the Large Databases view in Spotlight Reporting and Trending. The Large Databases view shows the largest databases on the selected SQL Servers. With Spotlight, you can view this data in a number of different ways. 1. Select Reporting and Trending |Views | SQLServer Environment |Large Databases in the Spotlight Browser.

2. Click Report | Select Connections and select at least one connection.

3. Select a Time Range.

4. Click Refresh.

By default, the data is shown as a bar chart, listing each database on the selected servers in order of size. You can change the way the data is displayed using the Chart View button, located above the chart.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

19

For example, if you wanted to view the growth of your largest databases over time you can change the chart to a line graph. To do this: 1. Click the Chart View button and then select Line from the list.

2. Click Refresh.

To look at the space consumption proportionally you could use a pie chart. Click Chart View |Pie. You can use the data and chart from the view in your own reports by exporting the view. To do this, click Export and choose a location to save the files. Spotlight saves the chart as a JPG file and the grid as a CSV file.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

20

Analyze Wait Statistics

When troubleshooting SQL Server performance issues, one area you can focus your investigation on is waits. Long wait times can be an indicator of performance bottlenecks so identifying where waits are occurring may assist you in your tuning efforts. Spotlight not only provides an easy way to see if your SQL server has a problem with waits but also identifies on which particular resource waits are occurring. Spotlight takes the raw data from SQL Server and automatically calculates the rate of wait over time so you have a more current perspective. To start your investigation into waits, go to the SQL Activity drilldown and click the Wait Statistics tab.

Look at the Wait Time Breakdown chart. Spotlight breaks waits down into categories based on the type of wait. The chart shows the amount of time used for each wait category as a proportion of the total CPU time available. If the chart is showing a large proportion of waits or signal waits then your SQL server is spending time waiting and not doing useful work. Ideally, the Wait Time Breakdown chart will show a high proportion of either CPU time or unused CPU time, meaning the server is either doing useful work or is idle.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

21

To see the data from the Wait Time Breakdown chart shown over time, look at the Wait Time Trend chart.

The Waits Detail grid shows detailed waits statistics for the SQL Server instance being monitored. It shows all wait types in SQL Server and breaks down the statistics into signal time and resource wait time. (SQL Server provides only wait time and signal wait time.) By default, the Waits Detail grid is sorted by Wait Time Rate (ms/s) which allows you to immediately see which wait types are being waited on right now.

Find Long Running SQL

The SQL - Long Running SQL alarm alerts you to SQL that has been running longer than a specified time frame. Because this time frame is unique to each environment, the SQL - Long Running SQL alarm, by default, is not configured.

Scenario
You work in an OLTP environment and often get complaints about delays in server processing. You want to be notified when SQL on a particular server is running longer than expected so you can investigate what is causing the delay. You configure Spotlight to raise a medium severity alarm when it detects SQL running for longer than five minutes. When the SQL - Long Running SQL alarm is raised, you use the Sessions drilldown to view the long running SQL. To configure the Alarm: SQL - Long Running SQL 1. Click Configure |Alarms.

2. Select a connection to configure. Select Default Settings to apply the configuration to all connections. 3. Select the alarm SQL - Long Running SQL. 4. Select Override the default settings for the alarm "SQL - Long Running SQL". 5. Click Add Severity and select a severity.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

22

Scenario: Click Add Severity and select Medium. 6. Select the check box of the new severity. 7. Copy and paste the text in the Description cell from the Normal severity to the new severity. 8. In the Start cell, type the duration in seconds, of how long the SQL should run for before the alarm is raised. Scenario: In the Start cell, type 300 . Tip: You can also configure different alarm severities using keys. In the SQL - Long Running SQL alarm, the key is the SPID. See "Configure Keyed Alarms" in the online help for more information.

Configure the Virtualization Overhead Guage

Note: This feature is not available in the Desktop Edition. The Virtualization Overhead gauge shows the percentage of CPU that is unavailable to this virtual machine because it is being consumed either by other virtual machines or by VMware itself. In a virtual environment, the physical CPU of a host is shared by virtual machines and the VMware Hypervisor. This means there can be multiple processes wanting to use the physical CPU of the host at the same time. The result is that virtual machines may be ready to run but have to wait to be scheduled on a CPU. Vmware call this ready time. The Virtualization Overhead gauge shows the amount of ready time as a percentage of the theoretical maximum CPU available to the virtual machine. To see virtualization data on Spotlight Home Pages 1. There needs to be a connection to the VMware Server. See "Connections to VMware Servers" (page 23) for more information. 2. In the Spotlight Browser, right-click the Windows connection and select Properties. 3. In the Host field, select the connection to the VMware Server. 4. In the VM Name field, select the name of the virtual machine. The Virtualization Overhead gauge now displays data on the corresponding Windows Home Page and SQL Server Home Page. Tip: SQL Server connections automatically use the virtualization data from their corresponding Windows connections. You do not need to configure the SQL Server connection once the Windows connection is configured.

Spotlight on SQL Server Getting Started Guide Use Spotlight on SQL Server

23

Connections to VMware Servers

Note: This feature is not available in the Desktop Edition. To create a connection: Note: Use the following procedure to create a VMware connection to the ESX or VirtualCenter server that hosts the Windows machine. 1. Click Configure | Connections.

2. In the Connection Bar on the left, click VMware. 3. Double-click Add new connection. 4. Complete the connection details: Field Address Description The IP address or hostname of the ESX server or VirtualCenter server you want to connect to. A valid user account on the target system. Note:For Spotlight to monitor a VMware server, the username specified must have at least a read-only role for the ESX server or VirtualCenter server that you want to connect to. Password The password required to connect the specified user to the server.

User

2
Troubleshoot Spotlight on SQL Server
Installation Issues
Remote installation of the diagnostic server fails
Note: This feature is not available in the Desktop Edition. Remote installation of the Diagnostic Server fails if the user has both a domain account and a local account with the same user name. Additionally, if the user uses a domain account that does not have permission to access the remote machine (that is, the account does not have administrator privileges) then the remote installation will not complete.

Diagnostic Server Issues

A Diagnostic Server on a separate machine continuously performs the collection of Spotlight data. The collected data can then be displayed whenever a Spotlight client is started on a machine that has access to the specified Diagnostic Server. You may encounter the following situation(s): Unable to establish a Spotlight connection to a SQL Server instance When the Diagnostic Server is unable to establish a Spotlight connection to a specified SQL Server, you may need to upgrade the current installation of Microsoft's Data Access Components (MDAC) on:
l l

The machine where the Spotlight client is installed AND The machine where the Diagnostic Server is installed

For more information on MDAC and MDAC installation, go to the Microsoft Download Center at http://www.microsoft.com/downloads/en/default.aspx and search for "MDAC". Unable to connect to the Diagnostic Server Note:This section is not relevant to the Desktop Edition. If you are unable to create a connection between the Spotlight client and Diagnostic Server, check the following:

Spotlight on SQL Server Getting Started Guide Troubleshoot Spotlight on SQL Server

25

If a user is not a member of either the Quest Diagnostic Users or Quest Diagnostic Administrators groups on the Diagnostic Server they will be unable to connect to the Diagnostic Server. When this happens a "Diagnostic Server Unavailable" page is displayed on the Spotlight on SQL Server client with the message "User authentication failed". See "Update the Quest Diagnostic User Groups" in the online help for more information on how to add a user to the Quest Diagnostic user groups.

Ensure the Spotlight client and Diagnostic Server are running the same version of Spotlight on SQL Server. When upgrading Spotlight on SQL Server you must upgrade both the Spotlight client and Diagnostic Server. For more information, see "Upgrade Spotlight on SQL Server" in the Spotlight on SQL Server Deployment Guide.

Unable to collect data If, after making a connection, alarms are raised related to inability to collect data, check the following:
l

The domain is included in the user name when specifying Windows OS (not SQL Server) connection details. For example, domain\johnsmith, instead of johnsmith. If the monitored host is the same as the machine running the Diagnostic Server, select the Windows authentication option (under OS Connection Details) on the Details page, which is displayed when you create a new connection. If the monitored host is in a different domain to the machine running the Diagnostic Server, specify the host name as a fully qualified address (for example, machine1.domain.company.corp). You can specify the host name on the Details page when you create a new connection. Tip: If the host name is not fully qualified, the connection may work but only intermittently. It is recommended that the host name be fully qualified

If you get errors executing "LogicalDisk" or "PagingFile" queries, start a command prompt on the monitored machine and execute the command "diskperf -y" then reboot. Do not use Windows authentication for connections remote to the Diagnostic Server machine unless you have configured the Diagnostic Server process to run under a user with access to the remote machine. (By default the Diagnostic Server runs under the "Local System" account, which will not have privileges on remote machines).

If you have done all of the above and there are still connection issues, reboot the Diagnostic Server machine (this is sometimes needed due to credentials conflicts in cached Windows connections). This must be a full machine reboot, not just a restart of the Diagnostic Server.

Connection Issues
Spotlight on SQL Server retrieves its data from the SQL Server via the Diagnostic Server. The Diagnostic Server also uses WMI queries to retrieve performance counter information from the

Spotlight on SQL Server Getting Started Guide Troubleshoot Spotlight on SQL Server

26

Windows server on which SQL Server is running. The Diagnostic Server cannot connect to a server unless it can access all of this information. This means you need to have the appropriate permissions on the server. Also, the data must be retrievable over your network. Spotlight on SQL Server requires more than just a specific port TCP/IP connection to SQL Server to do this. The port on the Diagnostic Server must be open. For more information, see the Spotlight on SQL Server Deployment Guide. This document is available in PDF from Start | Quest Software | Spotlight.

Problems connecting to a SQL Server Instance

If you have trouble connecting to a SQL Server instance, check that you can:
l

Connect to the SQL Server using another tool such as Microsoft's SQL Server Management Studio, or sqlcmd. Connect to WMI using another tool such as Microsoft's WMI CIM Studio, or wmic.

If you have problems with any of these checks, you will need to investigate and resolve these issues before you can connect to that server using Spotlight on SQL Server.

WMI Issues
The Diagnostic Server uses Microsoft WMI technology to collect performance data. The following is a guide to troubleshooting WMI errors.

Error 0x800706BA
The RPC server is unavailable. Exception from HRESULT: 0x800706BA Cause The Diagnostic Server is not able to establish a connection with the target computer. Solution Check the following:
l l

The address of the server is entered correctly via the Spotlight Connection Manager. The Remote Procedure Call (RPC) service is running on the target computer. TCP Port 135 is open both to internal traffic on the Diagnostic Server and the target computer. WMI opens an undetermined port in addition to port 135. This can be troublesome in a firewalled environment. Specifying a port range for WMI is recommended for this type of environment. See "WMI connectivity" in the Spotlight on SQLServer Deployment Guide

Spotlight on SQL Server Getting Started Guide Troubleshoot Spotlight on SQL Server

27

for instructions on how to specify a port range. The use of port scanning software is sometimes required to locate the ports needed to be opened.

Error 0x80070005
Access is denied. Exception from HRESULT: 0x80070005 (E_ACCESSDENIED) Cause The Windows user specified to monitor the target computer is unknown to the target or doesnt have administrator rights. Solution Check the following:
l

If the connection is configured to use Windows Authentication, check that the user running the Diagnostic Server Service has administrator rights. If the connection is configured to use named credentials, check that the user has administrator rights.

Error 0x80041010
WMI query failed: Invalid class. 0x80041010 Cause The WMI class does not exist on the target machine Solution Try any of the following to recreate the WMIclasses: Windows Connections
l

Recreate the WMI classes by opening a command prompt window and running the following command:
wmiadap /f

Use the Microsoft Extensible Counter List (Exctrlst) utility to enable the following counters: Win32_PerfDisk, Win32_PerfNet, Win32_PerfOS and Win32_PerfProc. See http://www.microsoft.com/downloads/details.aspx?familyid=7ff99683-b7ec-4da692ab-793193604ba4&displaylang=en for information. Note: A reboot of the target machine is required.

SQL Server Analysis Services Connections

Spotlight on SQL Server Getting Started Guide Troubleshoot Spotlight on SQL Server

28

Recreate the WMI classes by opening a command prompt window on the machine hosting the SQL Server Analysis Services instance. Run the following command:
wmiadap /f

Unregister and re-register the WMI classes. To do so: 1. On the problematic machine with SQLServer Analysis Services installed, find out the SQL installation path. See below for the default installation path. Your installation path may be different. SQL Server Analysis Services 2008 C:\Program Files\Microsoft SQL Server\MSAS10.MSSQLSERVER\OLAP\bin\Counters SQL Server Analysis Services 2005 C:\Program Files\Microsoft SQL Server\MSSQL.2\OLAP\bin Note: You are looking for the directory that contains the following files: For unamed instances: MSSQLServerOLAPService perf-MSSQLServerOLAPServicemsmdctr.ini For named instances: MSOLAP$Your_SSAS_NamedInstanceName perf-Your_SSAS_NamedInstanceNamemsmdctr.ini If you need to check the named instance name, use services.msc. 2. At the command prompt, change directory to the installation path. For example, cd C:\Program Files\Microsoft SQL
Server\MSAS10.MSSQLSERVER\OLAP\bin\Counters

3. Run the following commands to unload and load counters: For unamed instances, run:
unlodctr MSSQLServerOLAPService

then
lodctr perf-MSSQLServerOLAPServicemsmdctr.ini

For named instances, run:

unlodctr MSOLAP$Your_SSAS_NamedInstanceName

Spotlight on SQL Server Getting Started Guide Troubleshoot Spotlight on SQL Server

29

then
lodctr perf-Your_SSAS_NamedInstanceNamemsmdctr.ini

Tip: Replace Your_SSAS_NamedInstanceName with the SQLServer Analysis Services named instance name. If you need to check the named instance name, use services.msc. 4. Run the following command to parse all the performance libraries on the system and refresh the performance counter classes on the problematic machine:
wmiadap /f

Error 0x80041017
WMI query failed: Invalid query. 0x80041017 Cause The columns returned by the WMI class are incomplete or unexpected. Solution Recreate the WMI classes using:
wmiadap.exe /f

See http://msdn.microsoft.com/en-us/library/aa394528(VS.85).aspx for more information.

Miscellaneous Issues
Monitor Multiple SQL Server Instances
This version of Spotlight on SQL Server is designed to monitor a maximum of 100 SQL Server or Analysis Services or Replication instances and 100 Windows servers. Monitoring more than this recommended limit may result in poor performance or product instability.

Unable to perform certain operations

If, whilst using the Spotlight on SQL Server client, a message is displayed stating: You are not connected to the Diagnostic Server as a member of the Quest Diagnostic Administrators group. Only members of that group can perform the requested operation. and the user needs to be able to perform the operation that led to this message, the Spotlight on SQL Server client user must be added to the Quest Diagnostic Administrators group on the Diagnostic Server.

Spotlight on SQL Server Getting Started Guide Troubleshoot Spotlight on SQL Server

30

See Update the Quest Diagnostic user groups in the Spotlight on SQL Server Deployment Guide, for more information.

Missing SQL Server Performance Counters

Spotlight on SQL Server uses the SQL Server virtual table sysperfinfo (SQL Server 2000) and sys.dm_os_performance_counters (SQL Server 2005 and 2008) to retrieve data for many of its displays. In some rare cases, this table may not contain information. When this is the case, Spotlight on SQL Server will not be able to collect the data it requires, and will display "0" for many of its components. Most obvious will be the Memory icons on the home page, which will show 0 MB of memory used by SQL Server. Also, many of the flows on the home page will show no activity, and many drilldowns will show incomplete information. Spotlight on SQL Server raises an informational alarm shortly after connecting if it detects that the sysperfinfo or sys.dm_os_performance_counters table contains no data. You can confirm the existence of this problem by running the following SQL in Query Analyzer for SQL Server 2000:
select * from master..sysperfinfo

or by running the following SQL in SQL Server Management Studio for SQL Server 2005 and 2008:
select * from sys.dm_os_performance_counters

If this query returns no records, then your SQL Server performance counters are missing and Spotlight on SQL Server will not be able to operate correctly. Sometimes, for a variety of reasons, the SQL Server Performance Monitor counters will not show up as they should. Often, but not always, this problem can be fixed by following these steps: 1. At the command prompt, type the following: unlodctr.exe MSSQLServer 2. Then type: lodctr.exe <SQL Server path>\binn\sqlctr.ini 3. Reboot the server. More information can be found in the SQL Server Knowledge Base at msdn.microsoft.com.

Session Trace Errors

If errors are displayed on the Session Trace page (SQL Activity drilldown | Sessions page), install the following on the Diagnostic Server:
l

SQL Server Management Tools When diagnosing SQL Server 2000 or SQL Server 2005 instances, the SQL Server 2005 Management Tools are required to use the Session Trace functionality in Spotlight. When diagnosing SQL Server 2008, the SQL Server 2008 Management Tools are required.

Spotlight on SQL Server Getting Started Guide Troubleshoot Spotlight on SQL Server

31

Windows XP Service Pack 2 Applies only if the Diagnostic Server is running on Windows XP.

Playback
This feature has limited support in the Desktop Edition as a maximum of four hours playback data is available. Playback data is collected at scheduled intervals, in response to some alarms, and also while viewing a drilldown using the Spotlight client. This means that history data may not be available for a drilldown in Playback mode. You can change the rate at which data is collected. See "Configure Scheduling" in the online help for more information.

3
Connections to SQL Servers
To view an existing connection: Select the connection from the Recent connections or Connections |SQLServer menu in the Spotlight Browser. This opens the Spotlight Home Page. To add multiple SQLServer instances via discovery or by importing them from a file 1. Click Configure |Connections.

2. In the Connection Bar on the left, click SQLServer. 3. In the Connection list, double-click Discover. 4. Complete the steps in the wizard. Action Discovery Information Note: The Discovery Wizard locates and displays the SQL Servers on the network that you may want the Diagnostic Server to monitor. If the wizard discovers no instances of SQL Servers on the network, one reason may be that the SQL Server components that enable Spotlight to discover those servers have not been installed. See "Install SQL Server Client Tools" in the online help for more information. To import SQLServer instances from a file, each line in the file must contain a reference to a single SQLServer instance with optional user name and password details. The format is:
Instance name[,User[,Password]]

Import from file

If no user name is specified, Windows authentication is used. Limitations This version of Spotlight on SQL Server is designed to monitor a maximum of 100 SQL Server or Analysis Services or Replication instances and 100 Windows servers. Monitoring more than this recommended limit may result in poor performance or product instability.

Spotlight on SQL Server Getting Started Guide Connections to SQL Servers

33

To add an individual connection 1. Click Configure |Connections.

2. In the Connection Bar on the left, click SQLServer. 3. In the Connection list, double-click Add new connection. 4. Complete the connection details: Field Description

Target SQL Server Instance Address Enter the connect string used to link to a server (that is, the Server Name, Server Instance Name, or IP address). Select this option to use the user name and password configured on the Diagnostic Server to connect to the SQL Server instance.

Windows Authentication (using Diagnostic Server credentials) Database User Password

A valid user that enables Spotlight to connect to the target system. The password required to connect the specified user to the target system.

Linked OS Connection Connection The IP address, hostname, or URL of a Windows machine you want to connect to. If you do not want to monitor an associated Windows machine, select Do not monitor. Create Click to create a new connection to a Windows machine. You will need to complete the following: Address The IP address, hostname, or URL of the Windows machine you want to connect to. Use Diagnostic Server credentials Select this option to use the user name and password configured on the Diagnostic Server to connect to the Windows machine. User

Spotlight on SQL Server Getting Started Guide Connections to SQL Servers

34

Field

Description A valid user account on the target system. Note: Spotlight on Windows needs to use an account with the privileges required to retrieve server information, query the registry, and access WMI and performance monitor objects. An account with administrative rights to the target machine allows this. Password The password required to connect the specified user to the Windows machine.

Appendix: Contact Quest

Contact Quest Support
Quest Support is available to customers who have a trial version of a Quest product or who have purchased a Quest product and have a valid maintenance contract. Quest Support provides unlimited 24x7 access to SupportLink, our self-service portal. Visit SupportLink at http://support.quest.com. From SupportLink, you can do the following:
l l l

Retrieve thousands of solutions from our online Knowledgebase Download the latest releases and service packs Create, update and review Support cases

View the Global Support Guide for a detailed explanation of support programs, online services, contact information, policies and procedures. The guide is available at: http://support.quest.com.

Spotlight Support
When contacting Quest Support about a Spotlight product, please have the version and build number handy. You can obtain both by choosing the About option from the Help menu in the relevant Spotlight application. If your question is about an error message, write the message down and have it available for the technical support representative. Whenever you have a question about Spotlight, click Support Bundle or Contact Support on the Help menu. This creates a file called SpotlightSupport.zip. This file contains a snapshot of your Spotlight installation. Send this file and your request for assistance to Quest Software (support@quest.com). Note: You can find SpotlightSupport.zip in different locations under different operating systems.
l

Windows XP: C:\Documents and Settings\<user>\Application Data\Quest Software\Spotlight\Default\Support Vista: C:\Users\<user>\AppData\Roaming\Quest Software\Spotlight\Default\Support

If you find that information in the Help system or printed user guides is incorrect, unclear, or incomplete, please report the problem to Quest Support.

Spotlight on SQL Server Getting Started Guide Appendix: Contact Quest

36

Contact Quest Software

Email info@quest.com Quest Software, Inc. World Headquarters Mail 5 Polaris Way Aliso Viejo, CA 92656 USA Web site www.quest.com

See our web site for regional and international office information.

About Quest Software

Now more than ever, organizations need to work smart and improve efficiency. Quest Software creates and supports smart systems management productshelping our customers solve everyday IT challenges faster and easier. Visit www.quest.com more information.

Index
installation issues A about Quest Software B blocking C connect connection limit connections issues troubleshoot Contact Quest Software counters CPU usage Customer support D database growth Diagnostic Server issues E error codes, WMI expensive SQL H historical data 10 26 12 24 18 25 25 36 30 5 35 server side trace session trace errors SQL expensive long running SQL Analysis configure filter data SQL Server connection connection limit rowset trace Quest Software Quest support 32 29 8 36 MDAC monitor IO bottleneck

I 24 6 M 24 32 P performance counters missing Playback Q 36 35 R 13 S 13 30 30 10

12 21

12 16

32 29

Spotlight on SQL Server Getting Started Guide Index

38

performance counters Support T Technical support troubleshoot connections Diagnostic Server installation performance counters Playback session trace unable to perform certain operations user groups WMI V Virtualization Overhead gauge VMware connection W wait statistics WMI issues

30 35

35

25 24 24 30 31 30 29 29 26

22

23

20 26