Troubleshooting

Thread dump OR Java core or Java dump:

Thread dump needs to be taken if you have hung threads or thread related issues on
application servers.
Hung threads might be b/w web server and web container or WAS and DB.
Hung threads might be occurred be’se of bad application code (if opened connections are
not closed properly) or any system behavior ( WAS, DB etc.)
You see hung thread error messages in the JVM SystemOut.log file.
Eg: There are 10 hung threads on ---JVM-----
Note: Hung threads will get cleared by system itself. You need to monitor and make sure the
hung thread messages in SysOut file before to restart the JVM.
You will get a notification from monitoring tool or some operations or Help Desk regarding
System/application slowness then you will look into JVM logs.
Thread dump will be generated by command, kill -3 <JVM PID>
The thread dump info will be logged to native_stdout.log file.
Thread dump is in txt format.
Thread can also be analyzed by using IBM Thread Analyzer.
 Troubleshooting
• Download thread analyzer from http://www.alphaworks.ibm.com/tech/jca/download
• Open thread analyzer by double click on JAR or
• Java –Xmx512m –jar <jar file name> ( run this in a dir where your JAR is located)
• Click Open > load the thread dump. It will give you the thread activity on a JVM.

• Thread dump will be generated using wsadmin script in windows.

• wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
• wsadmin>$AdminControl invoke $jvm dumpThreads

• Thread can also be generated by configuring the JVM using DOS prompt. Need to press
ctrl+Break buttons simultaneously.

• You will see hung threads exception in logs be’se app or any system behaviour.
 Garbage Collection (GC)

An object is garbage when it can no longer be reached from any pointer in the running
program.
The JVM performance can be improved by reducing GC duration. The response time will
get reduced if we reduce the GC duration.
The GC will be taken place in different generations, young generation (minor collections)
and old generation (major collections). The GC activity takes place in different generations
as some objects need to be live longer and some of the objects can be collected after their
work is done.
The minor collections are also called as scavenges. The GC duration of scavenges are much
lower than the major collections. The scavenges duration is <1 sec. The major collection
GC duration is b/w 5 and 13 seconds.
GC occurs in each generation when it fills up.
Young generation contains different regions like eden and survivor spaces. The objects
created in eden. The objects die there be’se infant mortality. When eden fills up it causes
minor collection, in which some surviving objects are moved to an older generation.
JVM heap reaches 80% (by default) of heap size then major collection occurs.
The JVM will Stop The World (STW) during the major collection.
 GC
• Young generation contains Eden, and two survivor spaces.
• The different JVM arguments which decide the memory allocation for young and old
generations.
• Xmn and survivor ratio values are defined in Generic JVM arguments list of a JVM.
JVM takes default values if you do not specify generic args.
• Xmn – Young generation value/parameter.
• The space is allocated to eden and survivor spaces based on the survivor ratio.
• Suvivor Ratio = Eden space : Survior space
• Eg: Xmn = 180m (if you do not specify Xmn, JVM calculates and assign the Xmn value
in Heap size), Survivor ratio = 4 (Default value is 16). Survivor Ratio = Eden space:
Survivor space
• Survivor Ratio = 4 => Eden = 180*4/6 = 120m. Survivor is 30m.
• Two parts are allocated to survivor space.
• JVM heap size as 768m, Xmn = 180, Old generation space = 768 – 180 = 588m.
• The JVM heap is allocated based on your requirements like how many applications are
going to run in a JVM and how big and how busy they are.
 GC
• Permanent Generation : (perm size) : In which JVM and application classes will be
stored. The perm Gen size is different from heap size, In other words perm Gen space is
not included in heap size.
• Note: If you make many deploys to one JVM/cluster, the permGen will go OOM. The
JVM needs to be restarted when permGen goes to OOM.
• PermGen is defined in Generic arguments list, -xx:permGen. There will be minimum
permGen and max permGen spaces.
• JVM arguments : Servers > Application Servers > <serverName> > Java Process and
Management > Process definition > JVM, Generic JVM arguments
• Initial/Max heap size are set in Initial and Max heap size bizes respectively.
• Note: usually initial and max heap sizes are allocated equally to avoid the memory
leakages.
• Note: The saw tooth wave ( sin wave) in Tivoli represents there are no memory
leakages.

How to tune the JVM :

 Monitor the GC activity by generating the verbose GC file. Select the verbose GC
collection option under JVM args list. You need to specify the verboseGC file location
in the generic JVM args list. –xx:verboseGC:/opt/Webspehere/AppServer/profiles/<prof
name>/<logs>/<JVM-name>/native_stdout.log.
 Admin Console
 Analyze the GC using the tools like HPjtune, GC viewer etc.
 Monitor GC activity or JVM runtime activity using Tivoli (TPV).
 Tune the JVM arguments to reduce the GC duration. Collect the more objects in young
generation, this will improve the performance.
 The GC duration of scavanges/young generation is less compared to major
collections/old generation.

 To target JVMs/clusters for a app : console > Applications > enterprise applications >
app-name > Target mappings
 To Know application context root, Click on view deployment descriptor
 To Map or the application from one cluster to another cluster, Click “Map modules to
Servers” > select Modules, target servers and click “apply”. Save it and Synch it.
 Start the application.
 Disable app to not to start with JVM start/restart, App-name > Target mappings > click
in server/cluster > uncheck “enabled”. Save and Synch.
 You can view security roles and datasources or JNDIs for a app using console.
• JVM Heap size=768m 128m
S
Eden= S U
180*4/6= U R
Old
120m R V
V generatio Perm
I
I n/ Major anent
V
V colletion gener
O
O R 588m ation
R 3
30 0
m
Young generation/minor colletions
Scavanges = 180m