Common causes of Java memory leak and diagnosis using Java SE 6 tools

Xinfeng.Liu@sun.com Diagnosing Java memory leak is not an easy job. It is good news that Java SE 6 platform provide effective, free and built-in tools to help diagnose Java memory leak problems. This article introduces common Java memory leak causes from a Java developer's perspective and introduces practical ways to analyze memory leaks in Java heap using Java SE 6 tools. Contents What is Java memory leak and common causes of memory leak Using Java SE 6 tools (jmap, jhat, hprof) to diagnose memory leak in Java heap Summary References What is Java memory leak and common causes of Java memory leak The JVM is responsible for automatic memory management, which reclaims the unused memory for the application. However, if an application keeps a reference to an object that it no longer needs, the object cannot be garbage collected and will occupy space in the heap until the object is removed. Such unintentional object retention is referred to as a memory leak. From application developers' perspective, memory leak has a little different and wider scope of definition. The developers often regard such situations as memory leaks :

Meet with OutOfMemoryError when running applications for some period. Observe the heap memory occupation is unexpectedly high. This result in application's performance degrading. Observe garbage collection occurs frequently but it cannot collect enough objects timely and the application does not response to end users.

However, above situations may not mean true memory leak, the reasons could be:

The heap size setting (-Xms and -Xmx) is too small. -Xmx setting is too large in 32-bit JVM. The 32-bit JVM has 4GB address space, which includes not only Java heap but also stack, JVM libraries and native heap. The permanent generation (XX:PermSize and -XX:MaxPermSize) setting is too small to accommodate class, method objects and interned strings. This problem could happen in large scale Java EE and web services applications. The physical memory and swap space setting on the operating system are too small or other processes in the system consume all memory resources. The application or 3rd party libraries create huge amounts of temporary objects and

consume lots of heap memory. These live objects can not be collected until some processing logics are completed. This could happen particularly in xml processing, report generating, image processing, large file processing, etc. Redesigning the software or algorithm to use memory more efficiently is needed to solve this kind of problem.

The application process enters an unlimited cycle (dead cycle) eating up CPU cycles and creating objects very fast, causing GC thread could not kick in and collect objects timely. Memory mapping a huge-size file in 32-bit JVM. The 32-bit JVM address space is 4GB, mapping too big file will cause problems. A lot of objects pending on finalization. There is no guarantee when a finalizer will be run or that it will be run at all. An object that has a finalizer will not be garbage collected until its finalizer is run. This can cause lots of objects cannot be collected timely. The finalize method is used for making sure to release external resources in the end. In Java SE 6, using jmap -finalizerinfo command or Jconsole GUI utility can print information on objects awaiting finalization.

For unintentional object retention memory leak, the reasons could be:

Forgetting to release resources explicitly in the codes, such as connections, JDBC statements, etc. The method name is not always close() though. For example, forgetting to call end() method of java.util.zip.Defaltor after the compressor is no long used will cause the native heap space cannot be released timely. The finalize() method of java.util.zip.Defaltor also calls end() method automatically, but as above stated, you should not reply on finalizers to clean up resources timely. An exception throws before releasing resources. As a good practice, the codes releasing resources should be in finally{} clause. Some applications or Java libraries use hash maps, arrays or other similar Java collection objects to cache or track objects but the codes forgets to remove them from the hash map when those objects are not needed any more. This problem often occurs on some complicated server-side Java applications or some heavy-weight Java GUI applications. Permanent generation leak. In most Java EE environments, each web module or EJB module has a separate classloader. Incorrect use of class references will cause the application classes and its classloaders can not be collected even after the application is undeployed. Frank Kieviet gives a good example in his blog: Any reference from outside the application to an object in the application of which the class is loaded by the application's classloader will cause a classloader leak. Memory leaks in native codes. It is the leak in native heap, probably because of the bug of JNI codes or JVM. In this case you will not see leak problems in Java heap. Diagnosing this problem varies on different operating systems. On Solaris, prstat, pmap, dbx, mdb can help detect and diagnose the native heap memory leak. For more information, please refer to 3.4.3 Tracking Memory Allocation With OS Support in Troubleshooting Guide for Java SE 6 with HotSpot VM

Using Java SE 6 tools to diagnose memory leak in Java heap There are many tools for diagnosing Java memory leak, for example, commercial tools such as JProbe from Quest Software, and OptimizeIt from Borland ; free tools such as Netbeans profiler. However, Java SE 6 provides built-in tools that provide free and simple ways to help diagnose memory leaks in Java heap.

jhat jhat is a heap analysis tool, it reads a heap dump and starts an HTTP server on a specified port. jhat allows you to instrument any class and any object in the Java heap, providing deep knowledge about instance variable values, class loaders, full reference maps and so on. In addition, jhat also provides a built-in Object Query Language (OQL) interface to query the Java heap. jmap jmap can print Java heap summary and histogram of Java object heap. It can also dump Java heap that can be analyzed by jhat utility. (Note: jmap is also available in Java SE 5, but with less features) hprof hprof is a profiling tool built in JVM, it can provide heap information and other profiling information. Use java -agentlib:hprof=help command to get help information. (Note: hprof is also available in earlier Java SE versions, but it is not stable in JDK1.3 and 1.4.)

For more details about above utilities, please refer to Chapter 3. Troubleshooting Memory Leaks in Troubleshooting Guide for Java SE 6 with HotSpot VM Examples: First, set PATH environment variable to include <Java SE 6 Home Directory>/bin Example 1: When you suspect memory leak occurs, look at the heap histogram information first. Use jmap -histo:live <Java process ID> to look at histogram information of the Java heap. You need to run jmap multiple times during the typical workload. If some object types' instances or size keeps growing, then they're the candidates to drill down. -bash-3.00# jmap -histo:live 763 num #instances #bytes class name -------------------------------------1: 24001 960040 MyData 2: 24015 576360 Java.util.HashMap$Entry 3: 24132 386112 java.lang.Integer 4: 17 132368 [LJava.util.HashMap$Entry; 5: 23 87496 [I 6: 168 44016 [C 7: 11 25256 [B 8: 313 13048 [Ljava.lang.Object; 9: 35 11200 <objArrayKlassKlass> ..............

In this example, the leaking class (MyData) is specific, if it is used in a few places in the application, you can easily to locate the problem code. Example 2: However, in many cases, the histogram's output is not very instructive. You need other techniques to drill down. num #instances 1: 2: 3: 4: 5: 6: 7: 8: ........... 60169 60166 60015 60132 17 24 10 313 #bytes class name

-------------------------------------2092400 [C 1443984 java.lang.String 1440360 Java.util.HashMap$Entry 962112 java.lang.Integer 525584 [LJava.util.HashMap$Entry; 269304 [I 25232 [B 13048 [Ljava.lang.Object;

In this example, char arrays occupy the most space. ([C means char array). Since the number of [C and java.lang.String are nearly equal, and java.lang.String is backed by char arrays, you can drill down from java.lang.String first. Get the heap dump of a running Java process, and run jhat utility. -bash-3.00# jmap -dump:live,file=dump.bin 1031 Dumping heap to /export/home/lxf/work/Java-memleak/LeakDemo/dump.bin ... Heap dump file created -bash-3.00# jhat dump.bin Reading from dump.bin... Dump file created Fri Apr 13 17:13:36 CST 2007 Snapshot read, resolving... Resolving 121710 objects... Chasing references, expect 24 dots........................ Eliminating duplicate references........................ Snapshot resolved. Started HTTP server on port 7000 Server is ready.

By default, jhat start a http server listen on port 7000. In a browser, access http://localhost:7000/oql/ to run OQL query. Tips:

you can access http://localhost:7000/oqlhelp/ to get the help and samples for OQL. you can add JVM options when invoking jhat because jhat consumes much memory for large heap analysis, for example: jhat -J-mx512m

You need check first if there exists a few very long strings, so type below query statement in OQL query window then click Execute button: select s.count from java.lang.String s where s.count > 100 The output shows:

Object Query Language (OQL) query

All Classes (excluding platform) OQL Help

Execute

select s.count from java.lang.String s where s.count > 100 307.0 171.0

Apparently, the size of strings are very average, only 2 strings' length exceeds 100. It can not help you identify the cause of leak because of too many instances of strings. In this case, you can use hprof utility for further checking. Re-run the application using JVM options: java -agentlib:hprof=heap=sites,depth=20,file=dump.txt <your Java program> Press ctrl-\ (on Unix or linux) or ctrl-break (on Windows) to get the dump file immediately,

or wait the program finish running, the dump file will be generated automatically. Open the dump.txt file, and search rank . The output looks like: SITES BEGIN (ordered by live bytes) Fri Apr 13 12:35:28 2007 percent rank self accum 1 27.90% 27.90% 2 19.61% 47.51% 3 19.61% 67.12% 4 13.01% 80.13% .......... l ive bytes objs 856888 25100 602376 25099 602376 25099 399536 24971 alloc'ed bytes objs stack trace class name

856888 25100 300313 char[] 602376 25099 300318 Java.util.HashMap$Entry 602376 25099 300314 java.lang.String 399536 24971 300320 java.lang.Integer

Then search by trace ID(300313 in this example) in the dump.txt file to get the stack trace responsible for creating the objects that occupy the most space. TRACE 300313: java.lang.Integer.toString(Integer.Java:306) java.lang.Integer.toString(Integer.Java:116) java.lang.String.valueOf(String.Java:2932) LeakDemo2.main(LeakDemo2.Java:16) .......... So you know which codes created those objects. Although it does not locate the problem codes directly, the information can give you some hints and then you can check related source codes to help find the problem codes. (Note: Using hprof to do heap profiling has performance overhead, use it with caution on production systems) Example 3: In some other cases, the histogram of the Java heap shows there are some arrays occupying the most space. In this example, http://localhost:7000/histo shows int arrays occupy the most space :

Heap Histogram
All Classes (excluding platform) Class class [I class [C class java.lang.Class class [B Instance Count Total Size 8 196 359 10 401372 43704 27284 25129 10748

class [Ljava.lang.Object; 313

You can use the OQL to check if a specific array occupying most of space. Sort and print array sizes using the following statement: select map(sort(heap.objects('[I'), 'sizeof(lhs)< sizeof(rhs)'), '{size: sizeof(it), obj: it} ')

The output (part) shows in the browser window:

Object Query Language (OQL) query

All Classes (excluding platform) OQL Help

Execute

select map(sort(heap.objects('[I'), 'sizeof(lhs)< sizeof(rhs)'), '{size: sizeof(it), obj: it} ') [ { obj:[I@0xd0b2a1f8, size:400008, }, { obj:[I@0xd0b273f0, size:1032, }, { obj: [I@0xd4402fd8, size:88, }, { obj:[I@0xd4402d60, size:88, }, { obj:[I@0xd0b20d28, size:48, }, { obj:[I@0xd0b21348, size:48, }, { obj:[I@0xd4405948, size:36, }, { obj: [I@0xd44054b8, size:24, } ]

You can see the int array with object ID 0xd0b2a1f8 occupies most of space. Click the object link [I@0xd0b2a1f8 to view the detailed information of this array. The output shows in the browser window:

Object at 0xd0b2a1f8
{-1335162313, 2016429573, 1794306261, -1213265203, -144683428, -410774308, -535947466, 712627042, 2086951882, -1201434550, -1816979687, -2000985945, -1924206285, -651593694, -81680044, -1039918152, -1542015828, -1470009314, -20432353, 1995504241, 1511243903, -32215515, 965735436, -279748268, 35498304, -1233696560, -1293285836, 249216943, 1614500126, -1995101654, 1672386994, ..................... 147110089, 1122353943, 1096605074, -452385356, -580033509, 2052876183, -117949302, -508044649, -249325233, 1547761677, 1185364867, -1375544305, 1204321580, 436558131, 1630351469, 1000710972, -1107487252, -1692731055, 2030529016, -1990013767, -598137040, 299609896, ... }

References to this object:

class LeakDemo (84 bytes) : static field cachetable

Other Queries
Reference Chains from Rootset

Exclude weak refs Include weak refs

Objects reachable from here

Then you can click the link below Reference to this object or the link below Reference Chains from Rootset to get the object reference maps. These information link the memory leak to Java classes in your application to help find the problem codes. Summary Diagnosing Java memory leak is a difficult job. Currently no single tool can diagnose all memory leak problems. Understanding common causes of Java memory leaks and using the new Java SE 6 tools can help prevent and solve the Java memory leak problem. References Monitoring and Managing Java SE 6 Platform Applications http://Java.sun.com/developer/technicalArticles/J2SE/monitoring/ Troubleshooting JavaTM SE

http://Java.sun.com/Javase/6/webnotes/trouble/ Troubleshooting Guide for Java SE 6 with HotSpot VM http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/toc.html How to fix the dreaded "java.lang.OutOfMemoryError: PermGen space" exception (classloader leaks) http://blogs.sun.com/fkieviet/entry/how_to_fix_the_dreaded http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_Java