Java Memory Diagnostic Run Book

Java Memory Diagnostic Run Book
A Best Practices Approach to Solving Memory Issues in the JVM
With a garbage memory collector, are memory leaks a thing of the past with
Java?
Not exactly. Resource leaks, memory misuse, aggressive caches, and
improperly scoped user data can lead to down servers, poor performance, and
major headaches for any application team.
In this session, well discuss how to diagnose memory leaks in user loaded
production applications and review a series of best practices you can use when
first approaching JVM heap issues.
January 26, 2005
Copyright 2005 Wily Technology, Inc. All Rights Reserved.

CONFIDENTIAL
Java Memory 101

Heap (Java)
All Java objects are born here and live here. The JVM Garbage Collector
manages this space for you. Includes multiple generations and subspaces.
Heap (Native)
Native stack for JNI code (including some type III JDBC Drivers) so that JNI
code can work correctly. Native code allocates and manages memory on its
own, completely independent of the JVM.
Virtual Machine
The JVM process itself uses memory, too. For example, the JIT compiles Java
bytecode into native machine code, and keeps pages of these instructions
around in memory. Memory not constrained by heap settings!
The underlying OS will allocate at least as much as the initial heap minimum,
possibly up to the maximum heap setting, and perhaps another multiple beyond
due to the JVM process memory utilization.
Performing Java memory diagnosis is confined to the first area.
CONFIDENTIAL
Possible Diagnostic Tools

HProf: Available in all 1.2+ JVMs. Provides detailed trace file of ALL referenced
objects in the heap. Requires profiling flag to enable. Upon execution of the
dump, JVM is unstable and should be restarted. Tracefiles are very large,
cumbersome, and complex to analyze.
Heap Dump: IBM tool which supersedes HProf. Adds mapping of references
and allocation of objects. IBM provides better tools to analyze such as
HeapRoots. JVM is still somewhat unstable after execution of the heap
dump.
OptimizeIt: User-friendly debugging tool.
Wily Introscope: Two options are available for memory diagnosis with
production load. LeakHunter extension provides visibility into aggressively
growing data structures and their allocation. Introscope tracer Instance
Counter provides running count of any Java object in the heap.

CONFIDENTIAL
Potential Memory Problems

1. Resource Leaks
API misuse of Java objects with resource-style lifecycle. API requests

that objects are taken through a cycle of
Create Use Destroy
The lifecycle is not enforced. If the Destroy case is not called, the
object will remain in heap longer.
Examples:
JDBC Connections -- Create open() Destroy close()
I/O -- Sockets, File
Detection:
Examine growth of specific objects. Examine source code for API
misuse (look for lack of Destroy method call in finally blocks, or
improperly placed Destroy in try blocks).
CONFIDENTIAL

2. Growing Data Structures
Objects added to data structures while never being removed. Scope of

data itself is different compared to scope of overall structure.
Examples:
HTTP session-bound data added to web container-bound collection.
Session expires but data never removed from collection.
Special API misuse:
Collection has addListener and removeListener methods.
Leak caused because removeListener rarely or never called.
Detection:
Examine aggressively growing data structures, their allocation, and
insertions. For instance, monitor inserts, removes, and total size of
HashMap, Hashtable, ArrayList, Vector, etc.
CONFIDENTIAL

3. Leaked Resource Pools
Most pools require API utilization such as:

CheckOut Use Return
API use is not enforced, and objects may never be returned.
When pools are depleted, new objects are created until max pool size is
reached. Until pool determines the objects will never be returned, the
objects remain in memory.
This is a special case with qualities similar to both memory problem
types 1 and 2. Note, these leaks rarely exhibit critical memory
concerns, while they may introduce performance issues due to
overactive pool re-creating resources.
Examples: JDBC Connection Pools, Servlet Thread Pools
Detection: Compare reported pool sizes with actual instance counts of
underlying objects.
Look for API misuse in source code.
CONFIDENTIAL

4. Excessive Object Thrash
Too many objects (or objects too large in size) are spawned per
transaction.
If too many transactions are called in a small time period, the load on
the system will cause many Java objects to be created and then quickly
destroyed, placing extra burden on the garbage collector.
Examples:
Transactions that dynamically create or pull into memory bloated objects
such as
Images; Database CLOB, LOB, or BLOB; PDF
Detection:
Look at aggregate Java heap for fluctuating saw-tooth in each time
period. Monitor instance counts of major objects tied to transactions,
such as database related objects like ResultSets.
CONFIDENTIAL

4. Excessive Object Thrash
Object thrash approximately

40 Messages.
Heap thrash approximately
600 MB.
Quantify number of requests.
CONFIDENTIAL

5. Overly Aggressive Cache
A cache designed to improve performance either has no maximum size

or a constraint that is too large.
Note this is a special case of memory problem type 2.
Examples:
Any cache. Beware of caches that are automatically enabled by
application server (you may not even know they are there).
Detection:
Monitor cache collection object.

CONFIDENTIAL
Management by Restart
As problems arise in production, typically it is hard to diagnose because the
problem is not reproducible in load testing. Then you get stuck in the dreaded
App is Down
Again!!!
Server Ran Out

of Memory
Management
By Forced
Restarts
Hours Later
Restart App
This may be acceptable for short periods

until more resources can be acquired
(more physical RAM) or the leak can be
determined. However there are
significant downsides:
Personnel always in chaos mode.
Support preoccupied with restarting.

Restarting may take very long time for some
JVMs. Need to migrate sessions prior to
restart. Sometimes entire day is spent in
constant restart processes within cluster.
App performance degrades as GC continually

bumps into memory constraints (overactive
GC).

CONFIDENTIAL
10
Wily Rules for Garbage Tuning Flags

-X XX variety
Rule #1: Keep it simple
-Xms
Minimum heap size - typically one half or equal to max heap size.
-Xmx
Maximum heap size.
Rule #2: Custom Size Only Where Appropriate

If you load a significant amount of classes (lots of JSP directories) or use a
significant amount of JNI, you might consider increasing the permanent heap
generation:
-XMaxPermSize
If your web application serves up a significant amount of short lived objects and
relatively few objects live longer than a few minutes (HTTP request bound), you
might consider increasing the new generation:
-XNewSize
-Xmn
-XMaxNewSize
CONFIDENTIAL
11

-X XX variety
Rule #3: Playing With Numbers
Most of your numbers will be confined to Rule #2. In some cases, other spaces,
thresholds, ratios, generations, etc. can be tuned via numeric values. Only do
so after exhaustive analysis of GC data.
Rule #4: Stop!
Dont be overly tempted by alternative flags. Especially with web applications. A
variety of flags enable specific types of garbage collector activity. Unless your
application has a well known AND constant memory usage pattern, these flags
can introduce problems. In fact, many are experimental. All are narrowly based
on a specific memory behavior. Web applications do not exhibit any one specific
type of behavior as user load/behavior constantly fluctuates.
You may see immediate gains when using these flags but GC activity may
suffer hours down the line. Beware!
CONFIDENTIAL
12

-X XX variety
Analysis of GC data can lead to areas which can be tweaked via numeric values.
But first understand the application memory usage in detail.
Examples: Ratios, Size, Max, Min, Thresholds

CONFIDENTIAL
13

So youve detected a memory problem... How were they discovered?
1. OOM: OutOfMemory in logs. Leads to eventual system crash.
2. Watch Heap grow eventually
to maximum setting. Need
to restart.
Forced to restart every hour!

CONFIDENTIAL
14

What are the potential reasons for any memory problem?
1. Memory constraints are too low. Adjust maximum heap size.
2. There is a memory leak. You may be able to mask it by throwing more

memory at the JVM, but eventually the leak will return.

CONFIDENTIAL
15

Are memory constraints too low?
To determine if maximum heap setting is too low, you must do the following:
1. Determine base line memory usage of an idle application.

2. Ramp a defined typical user load. Measure object thrash per transaction
per user.
3. Determine the maximum number of users per JVM (think worst case).
4. Do arithmetic:
(Xactions Per User Per Thrash)*(Thrash)*(Max Users)+Baseline
5. Is your maximum heap size sufficient?
Difficulties: Caches grow as load increases. Session data may persist

beyond thrash period. Moving baseline may need to be accounted.
CONFIDENTIAL
16
Constraints are sufficient. So there must be a leak.

You must monitor BOTH aggregate heap growth and allocation of
aggressively growing objects.
Difficulty: Where to start looking? When watching a system with heavy

load, you simply cant watch everything. Good starts are
Known caches
Known user transactional objects
Database driver objects
Monitor with adjustable user load (QA/testing). If the problem cannot be

replicated, you must troubleshoot under real user load (Production).
Monitoring requires watching at regular time intervals, not just dumping heap
state. The more granular the time interval the better.
CONFIDENTIAL
17

Finding the Leak
Look at growth of database objects during user load.

CONFIDENTIAL
18

Finding the Leak
Look at growth of database objects during user load. Frequently there is direct
correlation with growth in heap.

CONFIDENTIAL
19
Database resources. Putting it all on the same page.

CONFIDENTIAL
20

Sessions and load

CONFIDENTIAL
An effective means of
determining
relationship between
memory and load is
watching memory as
sessions are
invalidated.
21

Sessions and load
When sessions are

invalidated, the session
objects die and all data
referenced to them
should die. There should
be a full GC that frees up
memory.

CONFIDENTIAL
22

Sessions and load
When sessions are

invalidated, the session
objects die and all data
referenced to them
should die. However,
here are object types
that remain.

CONFIDENTIAL
23

Still dont know what youre looking for?
LeakHunter automatically finds the fastest growing data structures in the entire JVM.
Real leaks
possess growth
slopes that
match GC Heap
growth.

CONFIDENTIAL
24
Q&A

CONFIDENTIAL

Java Memory Diagnostic Run Book

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Java Memory Diagnostic Run Book

Transféré par

Droits d'auteur :

Formats disponibles

Java Memory Diagnostic Run Book

A Best Practices Approach to Solving Memory Issues in the JVM

January 26, 2005

Copyright 2005 Wily Technology, Inc. All Rights Reserved.

Java Memory 101

Possible Diagnostic Tools

Copyright 2005 Wily Technology, Inc. All Rights Reserved.

Potential Memory Problems

API misuse of Java objects with resource-style lifecycle. API requests

Potential Memory Problems

Objects added to data structures while never being removed. Scope of

Potential Memory Problems

Most pools require API utilization such as:

Potential Memory Problems

Potential Memory Problems

Object thrash approximately

Potential Memory Problems

A cache designed to improve performance either has no maximum size

Copyright 2005 Wily Technology, Inc. All Rights Reserved.

Server Ran Out

This may be acceptable for short periods

Personnel always in chaos mode.

Support preoccupied with restarting.

App performance degrades as GC continually

Copyright 2005 Wily Technology, Inc. All Rights Reserved.

Wily Rules for Garbage Tuning Flags

Rule #2: Custom Size Only Where Appropriate

Wily Rules for Garbage Tuning Flags

Wily Rules for Garbage Tuning Flags

Copyright 2005 Wily Technology, Inc. All Rights Reserved.