HP SiteScope

for the Windows, Solaris and Linux operating systems Software Version: 10.00

SiteScope WebSphere Application Server Solution Template

Document Release Date: January 2009 Software Release Date: January 2009

Legal Notices
Warranty The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. The information contained herein is subject to change without notice. Restricted Rights Legend Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. Third-Party Web Sites HP provides links to external third-party Web sites to help you find supplemental information. Site content and availability may change without notice. HP makes no representations or warranties whatsoever as to site content or availability. Copyright Notices Copyright 2005 - 2009 Mercury Interactive (Israel) Ltd. Trademark Notices Adobe and Acrobat are trademarks of Adobe Systems Incorporated. Intel, Pentium, and Intel XeonTM are trademarks of Intel Corporation in the U.S. and other countries. JavaTM is a US trademark of Sun Microsystems, Inc. Microsoft, Windows, Windows NT, and Windows XP are U.S registered trademarks of Microsoft Corporation. Oracle is a registered US trademark of Oracle Corporation, Redwood City, California. Unix is a registered trademark of The Open Group.

Documentation Updates
This guides title page contains the following identifying information: Software Version number, which indicates the software version. Document Release Date, which changes each time the document is updated. Software Release Date, which indicates the release date of this version of the software.

To check for recent updates, or to verify that you are using the most recent edition of a document, go to: http://h20230.www2.hp.com/selfsolve/manuals This site requires that you register for an HP Passport and sign-in. To register for an HP Passport ID, go to: http://h20229.www2.hp.com/passport-registration.html Or click the New users - please register link on the HP Passport login page. You will also receive updated or new editions if you subscribe to the appropriate product support service. Contact your HP sales representative for details.

Support
You can visit the HP Software Support web site at: http://www.hp.com/go/hpsoftwaresupport This web site provides contact information and details about the products, services, and support that HP Software offers. HP Software Support Online provides customer self-solve capabilities. It provides a fast and efficient way to access interactive technical support tools needed to manage your business. As a valued support customer, you can benefit by using the HP Software Support web site to: Search for knowledge documents of interest Submit and track support cases and enhancement requests Download software patches Manage support contracts Look up HP support contacts Review information about available services Enter into discussions with other software customers Research and register for software training

Most of the support areas require that you register as an HP Passport user and sign in. Many also require a support contract. To find more information about access levels, go to: http://h20230.www2.hp.com/new_access_levels.jsp To register for an HP Passport ID, go to: http://h20229.www2.hp.com/passport-registration.html

HP SiteScope WebSphere Application Server Solution Template

SiteScopes WebSphere Application Server Solution Template allows you to monitor and troubleshoot WebSphere Application Servers 5.x and 6.x with SiteScope. This chapter includes:

Introduction on page 5 WebSphere Application Server Performance Overview on page 7 SiteScope WebSphere Application Server Solution Template on page 8 Default Metrics on page 9 Key Performance Indicators and Troubleshooting on page 18 Appendix - Sources on page 26

Introduction
The proliferation of Web-based applications is well-known. Today, many companies have Web-enabled significant numbers of business processes and the trend continues. This proliferation has driven the need for solid Web infrastructure based on open standards. One standard that has proven itself over again is the Java 2 Platform Enterprise Edition or J2EE.

HP SiteScope WebSphere Application Server Solution Template

J2EE has provided a standard for developing component-based, multi-tier Web applications that many companies have embraced and adopted. A key work horse of many J2EE implementations is the application server. Application servers have stepped forward as the workhorse of any Web infrastructure, primarily because they make it fairly simple and easy to implement complex business logic using open interfaces and standardized components. Because application servers house business logic, which is really a companys business processes translated into Web-based technologies, it is a critical system in the infrastructure. In an application server is not functioning, it generally means that critical business processes, which can often be related directly to revenue generation, are also not functioning. In additions, because application servers assume the "middleman" role in communication with the presentation layer (Web servers) and the back-end layer (databases, mainframes, ERP, CRM, etc.), ensuring application server performance is paramount. One application server that has proven its worth is IBMs WebSphere Application Server. This document describes how HPs SiteScope product helps organizations ensure the performance of IBM WebSphere Application Servers through the SiteScope WebSphere Application Server Solution Template. This document contains the following information:

An introduction to general WebSphere server performance concepts. An introduction to the HP SiteScope WebSphere Application Server Solution Template. A detailed list of the default metrics included in the SiteScope solution template, including why each metric is important. A "key indicators" section detailing which metrics are most important for uncovering WebSphere performance problems and basic troubleshooting steps that can be taken.

HP SiteScope WebSphere Application Server Solution Template

WebSphere Application Server Performance Overview

The WebSphere Application Server, like other J2EE-compliant application servers, provides all of the standard J2EE services. Therefore, while WebSphere performance depends on many factors, those factors related to J2EE services are the most critical. Below are important WebSphere performance categories:

Enterprise Java Bean (EJB) performance EJBs are the main workhorses of any J2EE application server. They are the components that implement business logic and represent persistent application entities. EJB response time is the critical performance metric.

Servlets Servlets act as server-side "applets" and perform many standard Web application tasks. Response time is also important for servlets as are aspects such as number of requests and servlet session manager performance.

Java Virtual Machine (JVM) Runtime The JVM is abstract computing entity. Similar to a hardware computing machine, it has instruction sets and controls memory. Memory usage and general memory statistics are key performance indicators for the JVM.

ORB container An object request broker enables communication between disparate objects by taking care of all interoperability issues. Thread metrics and pool sizes are key aspects or ORB performance.

Transaction manager Transaction managers are responsible for supporting transparent, safe and reliable transactions between J2EE components. Key performance aspects for the transaction manager include transaction duration, concurrent active transactions and transaction timeouts.

Database connections Database connections are perhaps the hardest working portion of an application server. Access to all back-end data takes place via these connections. Important performance metrics include connection pool sizes and time executing and waiting for transactions.

HP SiteScope WebSphere Application Server Solution Template

SiteScope WebSphere Application Server Solution Template

To address the needs of WebSphere Application Server performance monitoring, HP has built a WebSphere Application Server solution template. A solution template is a pre-built collection of monitors that consists of default metrics, default thresholds (where applicable) and proactive tests (where applicable) all of which have been researched using best practice data and expertise from various sources including HPs administrators, HPs professional services organization, technical documentation, books from industry experts and IBMs own collateral and documentation. The purpose of a solution template is to provide easy-to-use, comprehensive monitoring without requiring the SiteScope user of the iT organization to be experts on the application. Benefits of the WebSphere solution template include:

Reduces the need for WebSphere server performance domain expertise Reduces the time to configure and deploy WebSphere Application Server monitors Helps identify both real-time performance bottlenecks and longer term trends Adds no overhead to production systems Supports WebSphere Application Servers 5.x and 6.x (there is a separate solution template for each version)

HP SiteScope WebSphere Application Server Solution Template

Default Metrics
Perhaps the greatest value in the SiteScope WebSphere Application Server solution template is the built-in-list of default metrics and thresholds. These default metrics are heavily based on IBMs monitoring recommendations which can be found in various locations including the IBM corporate Web site. Below are tables listing the most critical WebSphere metrics that are enabled as defaults in the solution template.

Note: In addition to the metrics listed below, WebSphere utilizes several log files that should be monitored. These log files can be monitored via a standard SiteScope log file monitor. Important log files include:

Systemount.log (located in installation_directory/logs/applicationServerName) Systemerr.log (located in installation_directory/logs/applicationServerName) Activity.log (located in installation_directory/logs/applicationServerName)

WebSphere Application Server Metrics

Num

Counter/Metric EJB General Num Instantiates Num Destroys Concurrent Lives Total Method Calls Avg. Method Rt (ms)

Description

The number of times that bean objects were instantiated. The number of times that bean objects were freed. The number of concurrently live beans. The total number of method calls. The average time in milliseconds on the bean methods (home, remote, local).

HP SiteScope WebSphere Application Server Solution Template

Counter/Metric Active Methods

Description The number of concurrently active methods - the number of methods called at the same time.

Entity EJB Performance Concurrent Lives Drains from Pool The number of concurrently live beans. The number of times that the daemon found the pool was idle and attempted to clean it (entity and stateless). The number of times that beans were removed. The number of times that the returning object was discarded because the pool was full (entity and stateless). The number of times that a retrieve found an object available in the pool (entity and stateless). The average response time in milliseconds on the bean methods (home, remote, local). The number of times that bean data was stored in persistent storage (entity). The number of times that bean objects were freed. The number of calls retrieving an object from the pool (entity and stateless). The number of time that beans were created. The number of times that beans were passivated (entity and stateful). The number of calls returning an object to the pool (entity and stateless). The number of times that beans were activated (entity and stateful). The total number of method calls. The number of times that bean data was loaded from persistent storage (entity).

Num Removes Returns Discarded

Gets Found Avg Method Rt Num Stores Num Destroys Gets From Pool Num Creates Num Passivates Returns To Pool Num Activates Total Method Calls Num Loads

10

HP SiteScope WebSphere Application Server Solution Template

Counter/Metric Num Instantiates Active Methods Avg Drain Size Pool Size

Description The number of times that bean objects were instantiated. The number of concurrently active methods - the number of methods called at the same time. The average number of objects discarded in each drain (entity and stateless). The number of objects in the pool (entity and stateless).

Stateful Session EJB Performance Concurrent Lives Num Removes Avg Method Rt Num Destroys Num Creates Num Passivates Num Activates Total Method Calls Num Instantiates Active Methods The number of concurrently live beans. The number of times that beans were removed. The average response time in milliseconds on the bean methods (home, remote, local). The number of times that bean objects were freed. The number of times that beans were created. The number of times that beans were passivated (entity and stateful). The number of times that beans were activated (entity and stateful). The total number of method calls. The number of times that bean objects were instantiated. The number of concurrently active methods - the number of methods called at the same time.

Stateless Session EJB Performance Concurrent Lives The number of concurrently live beans.

11

HP SiteScope WebSphere Application Server Solution Template

Counter/Metric Drains From Pool

Description The number of times that the daemon found the pool was idle and attempted to clean it (entity and stateless). The number of times that the returning object was discarded because the pool was full (entity and stateless). The number of times that a retrieve found an object available in the pool (entity and stateless). The average response time in milliseconds on the bean methods (home, remote, local). The number of times that bean objects were freed. The number of calls retrieving an object from the pool (entity and stateless). The number of calls returning an object to the pool (entity and stateless). The total number of method calls. The number of times that bean objects were instantiated. The number of concurrently active methods - the number of methods called at the same time. The average number of objects discarded in each drain (entity and stateless). The number of objects in the pool (entity and stateless).

Returns Discarded

Gets Found Avg Method Rt Num Destroys Gets From Pool Returns To Pool Total Method Calls Num Instantiates Active Methods Avg Drain Size Pool Size

Message Driven EJB Performance Concurrent Lives Drains From Pool The number of concurrently live beans. The number of times that the daemon found the pool was idle and attempted to clean it (entity and stateless).

12

HP SiteScope WebSphere Application Server Solution Template

Counter/Metric Returns Discarded

Description The number of times that the returning object was discarded because the pool was full (entity and stateless). The number of times that a retrieve found an object available in the pool (entity and stateless). The average response time in milliseconds on the bean methods (home, remote, local). The number of times that bean objects were freed. The number of calls retrieving an object from the pool (entity and stateless). The number of calls returning an object to the pool (entity and stateless). The total number of method calls. The number of times that bean objects were instantiated. The number of concurrently active methods - the number of methods called at the same time. The average number of objects discarded in each drain (entity and stateless). The number of objects in the pool (entity and stateless).

Gets Found Avg Method Rt Num Destroys Gets From Pool Returns To Pool Total Method Calls Num Instantiates Active Methods Avg Drain Size Pool Size

Database Connections Num Creates Num Destroys Num Returns Num Allocates Pool Size Concurrent Waiters Total number of connections created. Total number of connections destroyed. Total number of connections returned. Total number of connections allocated. The size of the connection pool. The number of threads that are currently waiting for connections.

13

HP SiteScope WebSphere Application Server Solution Template

Counter/Metric Avg. Wait Time (ms) Faults Percent Used Percent Maxed. PrepStmt Cache Discards JVM Runtime Total Memory (bytes) Free Memory (bytes) Used Memory (bytes) Servlet Session Manager Created Sessions Invalidated Sessions Session Lifetime Active Session

Description The avg. waiting time in milliseconds until a connection is granted. The total number of faults, such as timeouts, in the connection pool. The average percent of the pool that is in use. The average percent of the time that all connections are in use. The total number of statements discarded by the least recently used (LRU) algorithm of the statement cache.

The total memory in the JVM run time. The free memory in the JVM run time. The used memory in the JVM run time.

The number of sessions that were created. The number of sessions that were invalidated. The average session life time in milliseconds (time invalidated - time created). The number of concurrently active sessions. A session is active if the WebSphere Application Server is currently processing a request that uses that session. The number of sessions that are currently cached in memory.

Live Sessions

ORB Container Thread Pool Thread Creates Thread Destroys The total number of threads created. The total number of threads destroyed.

14

HP SiteScope WebSphere Application Server Solution Template

Counter/Metric Active Threads Pool Size Percent Maxed

Description The number of concurrently active threads. The average number of threads in pool. The average percent of the time that all threads are in use.

Web Container Thread Pool Thread Creates Thread Destroys Active Threads Pool Size Percent Maxed The total number of threads created. The total number of threads destroyed. The number of concurrently active threads. The average number of threads in pool. The average percent of the time that all threads are in use.

Transaction Manager Global Trans Begun Global Trans Involved Local Trans Begun Active Global Trans Active Local Trans Global Trans Duration Local Trans Duration Global Before Completion Duration Global Prepare Duration The total number of global transactions started on the server. The total number of global transactions involved on the server (for example, begun and imported). The total number of local transactions started on the server. The number of concurrently active global transactions. The number of concurrently active local transactions. The average duration of global transactions. The average duration of local transactions. The average duration of before_completion for global transactions. The average duration of prepare for global transactions.

15

HP SiteScope WebSphere Application Server Solution Template

Counter/Metric Global Commit Duration Local Before Completion Duration Local Commit Duration Num Optimizations Global Trans Committed Local Trans Committed Global Trans RolledBack Local Trans RolledBack Global Trans Timeout Local Trans Timeout Web Applications Num Loaded Servlets Number Reloads Total Requests Concurrent Requests Response Time (ms) Num Errors

Description The average duration of commit for global transactions. The average duration of before_completion for local transactions. The average duration of commit for local transactions. The number of global transactions converted to single phase for optimization. The total number of global transactions committed. The number of local transactions committed. The total number of global transactions rolled back. The number of local transactions rolled back. The number of global transactions timed out. The number of local transactions timed out.

The number of loaded servlets. The number of reloaded servlets. Total number of requests that a servlet processed. The number of requests that are concurrently processed. The response time, in milliseconds, of a servlet request. Total number of error in a servlet or JavaServer Page (JSP).

Servlets Total Requests Total number of requests that a servlet processed.

16

HP SiteScope WebSphere Application Server Solution Template

Counter/Metric Num Errors Concurrent Requests Response Time

Description Total number of errors in a servlet or JavaServer Page (JSP). The number of requests that are concurrently processed. The response time, in milliseconds, of a servlet request.

Note: Depending on the version and configuration of WebSphere being monitored, some of the metrics may differ or be unavailable.

17

HP SiteScope WebSphere Application Server Solution Template

Key Performance Indicators and Troubleshooting

The goal of any monitoring solution is to help administrators avoid or at the very least identify performance problems before they affect large numbers of users. SiteScope solution templates include key metrics that can act as early indicators of performance problems. This section provides a list of key performance indicators of WebSphere Performance and describes a basic process for troubleshooting performance issues. Apart from checking standard system performance metrics such as CPU and memory utilization and Web server threads, the process of troubleshooting WebSphere performance problems relates to four key areas:

EJBs Servlets Thread pools Database connection pools JVM heap

Note: This section does not include all of the detail necessary to uncover the root cause of WebSphere performance problems. Information is provided in these sections to help you take the first steps in troubleshooting. For more information on troubleshooting or tuning situations, it is recommended to visit the IBM WebSphere information center at: http://publib.boulder.ibm.com/infocenter/ws51help/index.jsp.

18

HP SiteScope WebSphere Application Server Solution Template

EJBs
Since EJBs are a critical workhorse component of any Web application, their performance is a direct indicator of overall Web application performance. The SiteScope WebSphere Application Server solution template contains many metrics related to EJB performance. The following table specifies the most important and useful metrics. It should be noted that most Web applications contain hundreds if not thousands of EJBs. It is therefore important to have some knowledge of the Web application to be monitored in order to identify which EJBs are most important to be monitored. Once the critical EJBs have been identified, use the following metrics for ongoing performance monitoring.
Metric Entity and Stateful EJBs
Num Passivates Num Activates

What to Look For If these metrics are exhibiting high values, then the EJB container is changing between a beanss active (in use) statue and uts passive (stored in the disk) state. This metric should be baselined for critical methods within EJBs and compared over time. Start with methods within EJBs that access back-end systems such as databases and mainframes.

Troubleshooting Consider increasing the EJB cache size.

All EJBs
Avg. Method RT

High avg. method response times usually indicates inefficient code. Contact the Web application developers.

19

HP SiteScope WebSphere Application Server Solution Template

Metric Timeout Ratio

Lock Manager Timeout

What to Look For By dividing the Lock Manager Timeout Total Count by the Lock Manager Access Count, a ratio is provided that gives an indication of the number of times a thread timed out while waiting for a lock on an EJB.

Troubleshooting A high ratio indicated a high occurrence of timeouts that can cause exceptions and transaction rollbacks, Check the "transaction timeout" value and consider increasing it. Increasing this metric too high can cause server threads to wait too long and may increase request times. A high timeout ratio generally means the "transaction timeout" setting is too low. Be careful to adjust this in increments because too high of a setting can cause other bottlenecks.

Total Count
Lock Manager Access

Count (applies to entity and stateful\session beans)

Transaction Timeout Ratio

Transaction Timed Out

Total Count
Transactions

Committed Total Count

By dividing the Transactions Timed Out Total Count by the Transactions Committed Total Count, a ratio is provided that indicates how often threads are timing out waiting for a transaction to complete.

20

HP SiteScope WebSphere Application Server Solution Template

Servlets
Another important component of Web-based applications is servlets. Servlet performance often affects presentation layer responsiveness and can also affect overall application response time. The following table specifies key servlet metrics.
Metric Num Errors What to Look For Servlet errors can indicate anything from a problem serving graphics or text to problems with plug-ins and security. Ideally, this metric should be 0. Troubleshooting 1 View the error code that is generated in the browser. 2 Search on "servlet errors" at the following URL: http://publib.boulder.ibm.co m/infocenter/ws51help/inde x.jsp Small drops in total requests are good indicators of potential future performance problems. Large drops indicate current problems. Evaluating load on the application server in terms of numbers of users is a good first troubleshooting step. Investigating dynamic caching is also a good next step. Many times, servlet response time degrades because of problems accessing back-end systems. Check database connection pool performance as a first step. Also, check to make sure that EJB performance is ok since many times, servlets rely on EJBs.

Total Requests

This metric indicates how many requests a servlet processes. By baselining its value and comparing it to current performance, it is possible to track whether or not a servlet capacity is staying high. A drop in total requests indicates that the servlet is struggling to process requests. This metric also needs to be baselined but a good rule of thumb is that response time shouldnt be greater than 8 seconds.

Response Time

21

HP SiteScope WebSphere Application Server Solution Template

Thread Pools
Key containers such as the Object Request Broker and Web Application containers contain pools or resources such as threads and connections for processing requests. Creating and destroying threads is an expensive process and therefore can greatly affect performance. Monitoring thread pool activity in order to ensure that thread settings are optimal is therefore an important aspect to ensuring optimal performance. The following table outlines important metrics for both the Web and ORB container thread pools.
Metric
Thread Creates Thread Destroys

What to Look For A significant number of threads creates and destroys indicates that the thread pool does not have enough allocated threads. Creation and destruction of threads can include a significant amount of overhead and negatively affect performance. Since this metric specifies the percentage of time that all threads are in use, it is a good indicator of how much head room exists in thread pools. If this metric is consistently in double digits, the application server may be approaching a performance slowdown scenario.

Troubleshooting In conjunction with evaluation the Percent Maxed metric below, consider increasing the thread pool size by increasing the "maximum number if threads to allow".

Percent Maxed

In conjunction with evaluating the Thread Creates and Destroys, consider increasing the thread pool size by increasing the "maximum number of threads to allow".

22

HP SiteScope WebSphere Application Server Solution Template

Database Connection Pools

Database connections are perhaps the single most likely contributor to applications server performance problems. Application server performance can and does slow down due to time waiting on database processing or inefficient connections to data stores. This section provides metrics that help identify if these types of situations are occurring and what to do about them.
Pool

Metric Concurrent Waiters

What to Look For Ideally, this counter should never be greater than 0. If it is, it means that Web applications are waiting for a connection to a back-end data store. If this percentage is low under normal application usage, this indicates that the connection pool may be too big.

Troubleshooting Consider increasing the connection pool "maximum connections" value if CPU load is not close to 100%. If CPU load is high, this is most likely the problem. Consider decreasing the connection pool "maximum connections" value.

Percent Used

23

HP SiteScope WebSphere Application Server Solution Template

Metric Pool Size

What to Look For This metric can help administrators determine if their back-end database is configured for an appropriate number of connections. A situation can occur where the pool size has been increased past what the database was configured to handle. It may not always be possible to obtain a balance between "maximum connections" and "used connections" such that "concurrent waiters" equals 0. If this is the case, there will be some wait time for a connection. Tracking/baselining "avg. wait time" can help identify if wait time is getting abnormal.

Troubleshooting Use this value to ensure that number of connections that the back-end database is configured to support is at least equal to the WebSphere database connection pool size. Of these are not equal, applications will fail and issue SQL exception errors. Baseline this metric and then determine if current values are out of ordinary. If wait times are growing, it is a good indication that the pool size should be increased. If wait times are within norms but significant "Connection Wait Timeout Exceptions" are being thrown, then an opportunity exists to tune the "connection timeout" value to wait a little longer.

Avg. Wait Time

24

HP SiteScope WebSphere Application Server Solution Template

JVM Heap
Runtime memory usage is a critical aspect of WebSphere Application Server health because object creation and destruction are memory intensive processes and require sufficient memory allocation. By monitoring a couple of critical JVM heap (memory) counters, it is possible to keep a close eye on this critical resource. The following table specifies important JVM heap metrics.
Metric
Free Memory

What to Look For There are three main aspects to check regarding these to metrics:
If memory is increasing

Troubleshooting Memory leak situations require an in-depth analysis of the Web application code. It may be necessary to utilize tools such as HPs Deep Diagnostics for J2EE to troubleshoot at the method and call levels. 85% free memory indicates a maximum heap size value that is too large which usually means the application server is under utilizing the memory allocated for the heap. Longer-term memory usage patterns that coincide with slow EJB or Servlet response times can indicate inefficient code ot component communication issues.

(bytes)
Used Memory

(bytes)

at a much higher rate than free memory, this is a likely indication of a memory leak.
Check to see if free

memory is 85% or more.

Compare memory

consumption patterns with the performance of critical EJBs and Servlets over time.

25

HP SiteScope WebSphere Application Server Solution Template

Appendix - Sources
The following sources were used in developing this best practice guide. It is highly recommended that these sources be consulted in the process of troubleshooting and tuning WebSphere application servers.

IBM Web Site

http://www306.ibm.com/software/webservers/appserv/was/performance.html http://publib.boulder.ibm.com/infocenter/ws51help/index.jsp http://www.redbooks.ibm.com/abstracts/sg246198.html?Open

26