Oracle BPM Engine Troubleshooting: Purpose

Oracle BPM Engine Troubleshooting

Purpose
This document is intended for Customers and Service Professionals that have encountered a problem with the
Oracle BPM Engine and would like to perform some troubleshooting before contacting Technical Support.
Additionally, the document describes the information that Customer Support will generally require and how to obtain
it. While the document has been written based on BPM 5.7 it has also been revised for versions 5.5, 6.0 and 10.3.
Differences between versions are pointed out when they exist.
Introduction
This document provides general information to perform a basic or medium level of Engine troubleshooting. Among
the provided information you will find: description and details of Engine logs, tools for gathering troubleshooting
data, several sources of information depending on the nature of the problem, troubleshooting configurations,
debugging settings, sets of information that should be analyzed or collected in case of an incident, and more.
The type of issue within the BPM Engine can vary widely. Depending on each case, different sets of information
might be required. That is why this document aims at the most common required pieces, which are enumerated
below:
Basic Engine Info:

o Engine version.
o Database and Database driver full version.
o Directory full version (when applicable).
o Application Server full version (when applicable).
o Deployment topology (when applicable).
o Operating system version.
Engine Logs in Debug level.
Affected Project/s
Instance Information from the Engine DB:
o Entries in the tables of Pending Execution Items.
o Affected entries in the table of Instances and Instance Events.
Java Full Thread Dumps of the JVM running the Engine (when dealing with performance issues).
Java Heap Dumps (when dealing with out of memory or memory leak problems).
This document describes these different sets of information and provides instructions on how to obtain them.
Additionally, it contains a brief section on how to tune some engine settings to address typical performance issues
and a section on debugging flags.
Oracle Proprietary and Confidential
Table of Contents
1. Collecting Basic Engine Information ....................................................................................4
1.1. Engine version ...................................................................................................................4
1.2. Database and Database driver full version ........................................................................5
1.2.1. Database version.........................................................................................................5
1.2.2. Database driver full version .........................................................................................5
1.3. Directory full version...........................................................................................................5
1.4. Application Server full version ............................................................................................5
1.4.1. WebLogic Server full version .......................................................................................5
1.4.2. WebSphere Server full version ....................................................................................6
1.5. Deployment topology .........................................................................................................6
1.6. Operating system version...................................................................................................6
1.7. Basic Info Recommendations.............................................................................................7
2. Engine Logs ............................................................................................................................7
2.1. Description of the Engine Logs ..........................................................................................7
2.1.1. A Log entry ..................................................................................................................7
2.1.2. To-Do Item messages .................................................................................................8
2.1.3. Log files location..........................................................................................................8
2.1.4. Log files description.....................................................................................................9
2.1.5. Specifying an alternative Log location .........................................................................9
2.2. Configuring Log size and Severity level ...........................................................................10
2.2.1. Configuring the Log settings in the Process Administrator ........................................10
2.2.2. Enabling Asserts in the JVM that runs the BPM Engine ............................................10
2.2.2.1. Enabling Asserts in the Standalone Engine............................................................10
2.2.2.2. Enabling Asserts in the WebLogic Server...............................................................11
2.2.2.3. Enabling Asserts in the WebSphere Server............................................................12
2.2.3. Restarting the BPM Engine .......................................................................................13
2.2.3.1. Start output of the WebLogic Server.......................................................................13
2.3. Using LogViewer (in graphical environments) ..................................................................14
2.4. Using LogReader (in character terminals)........................................................................14
2.4.1. Windows logReader.bat script ...................................................................................15
2.4.2. Unix logReader.sh script............................................................................................15
2.5. Additional Logs related to the Engine...............................................................................15
2.5.1. WebLogic Server Logs ..............................................................................................15
2.5.2. WebSphere Server Logs ...........................................................................................16
3. Project Debugging................................................................................................................16
3.1. logMessage method.........................................................................................................16
3.2. Meaningful logging ...........................................................................................................16
4. Obtaining Affected Projects.................................................................................................16
4.1. Obtaining a Project Export with all libraries included........................................................17
4.2. Recovering Process Versions from the Process Administrator ........................................17
5. Obtaining Information from the Engine DB ........................................................................19
5.1. Obtaining all entries in the table of Pending Execution Items ..........................................19
5.1.1. To-Do Item Types......................................................................................................19
5.2. Obtaining affected entries in the table of Instances and Instance Events ........................19
5.2.1. Instance States..........................................................................................................20
5.2.2. Instance Event Types ................................................................................................20

5.3. Other relevant queries to the Engine Schema..................................................................21
5.4. Relevant queries to the Directory Schema .......................................................................21
6. Obtaining Full Thread Dumps..............................................................................................22
6.1. Obtaining FTDs using the Process Administrator.............................................................22
6.2. Obtaining FTDs in Windows OSs.....................................................................................22
6.3. Obtaining FTDs in Unix-like OSs......................................................................................23
6.4. Obtaining FTDs using other tools.....................................................................................23
6.4.1. Sun jstack ..................................................................................................................23
6.4.2. AdaptJ StackTrace ....................................................................................................23
6.5. Java Core Dumps ............................................................................................................23
6.5.1. Sun JVM Fatal Error Log ...........................................................................................23
6.5.2. IBM Thread and Monitor Dump Analyzer...................................................................24
7. Obtaining Java Heap Dumps ...............................................................................................24
7.1. How to generate a Heap Dump........................................................................................24
7.1.1. HeapDumpOnOutOfMemoryError JVM option ..........................................................24
7.2. Tools to analyze Heap Dumps .........................................................................................24
7.2.1. Sun jhat .....................................................................................................................24
7.2.2. IBM HeapAnalyzer.....................................................................................................25
8. Java Profiling Tools..............................................................................................................25
8.1. Sun Tools and Options.....................................................................................................25
8.2. WebLogic JRockit Profiling Tools.....................................................................................25
8.3. Other Profiling Tools ........................................................................................................25
9. Basic Performance Tuning ..................................................................................................26
9.1. Performance deterioration due to insufficient space in the PAPI Instance cache.............26
9.2. Performance deterioration due to lack of Instance disposing...........................................26
9.3. Performance deterioration in J2EE environments due to frequent SQL timeouts ............27
9.4. Performance deterioration in J2EE environments due to Disposer timeouts ...................27
9.5. Performance deterioration in J2EE environments due to Scheduler timeouts .................28
10. Most commonly used debugging flags ............................................................................28
10.1. PAPI News Debug .........................................................................................................28
10.2. Directory Polling Debug .................................................................................................29
10.3. Hybrid LDAP Debug.......................................................................................................29
11. BPM Documentation On-line .............................................................................................29
12. Contacting Oracle BPM Support .......................................................................................30
1. Collecting Basic Engine Information

1.1. Engine version
The most accurate way of obtaining the Engine version is finding the engine log entry (of severity level Info) that is
generated at startup and contains several pieces of environment information. This entry looks like this:
Environment information: Version 5.7 GA, protocol 3.1
Build #76599 - Jun 08,2007 12:59:31
awt.toolkit=sun.awt.windows.WToolkit
Another way of obtaining the Engine version is opening the Admin Center About page which is shown below. Note
that while this is the version of the BPM installation, the Engine version could be different due to incorrectly installed
upgrades or fixes. So, when in doubt, it is a good idea to check if both build numbers match.
1.2. Database and Database driver full version

1.2.1. Database version
To find out the version of your Database you should probably contact your DB administrator. There are a few
exceptions where the full version is required, thus knowing the major version number will be enough (for instance:
Oracle 10g, IBM DB2 version 8.1, MS SQL Server 2000, etc).
1.2.2. Database driver full version

The JDBC database driver version is generally of greater importance due to the different defects or improvements
that might be present in each release.
The general way of obtaining the drivers version is from the manifest file included in the drivers jar or zip. The
driver might be found at different locations depending on your environment (Standalone, J2EE, etc.) and it is
important to obtain the manifest from the driver that the Engine is using. For instance, in standalone the driver is
found at $BPM_HOME/ext/<db-driver jar or zip>. Please, refer to the BPM Administrators Guide for details on the
location of the driver in J2EE environments:
http://download.oracle.com/docs/cd/E13165_01/albsi/docs57/adminguide/index.html.
Below is an example of a Manifest.mf file (this one was obtained from the file C:\bea\albpm5.7\j2eewl\ext\
ojdbc14.jar, which corresponds to the driver that the Admin Center uses to connect to FDI):
Manifest-Version: 1.0
Specification-Title:
Oracle JDBC driver classes for use with JDK14
Sealed: true
Created-By: 1.4.2_08 (Sun Microsystems Inc.)
Implementation-Title:
ojdbc14.jar
Specification-Vendor:
Oracle Corporation
Specification-Version: Oracle JDBC Driver version - "10.2.0.2.0"
Implementation-Version: Oracle JDBC Driver version - "10.2.0.2.0"
Implementation-Vendor: Oracle Corporation
Implementation-Time:
Tue Jan 24 08:55:21 2006
1.3. Directory full version

If the BPM Directory (or FDI) is supported by an LDAP, Active Directory or a Database version other that the one
used for the Engine Database you should also provide this Directory provider version information.
1.4. Application Server full version

If the Engine is deployed in a J2EE environment the Application Server version and the deployment topology are
required to troubleshoot any Engine issue.
The Application Server full version can be obtained in different ways depending on the Vendor. In the next subsections we present the most common examples:
1.4.1. WebLogic Server full version

In the WebLogic Server, the version (and the applied patches, if any) can be found in the start log:
<Jan 18, 2008 4:38:39 PM CST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server

with Java HotSpot(TM) Client VM Version 1.5.0_04-b05 from Sun Microsystems Inc.>
<Jan 18, 2008 4:38:40 PM CST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 9.2
Fri Jun 23 20:47:26 EDT 2006 783464 >
<Jan 18, 2008 4:38:42 PM CST> <Info> <WebLogicServer> <BEA-000215> <Loaded License :
C:\bea\license.bea>
<Jan 18, 2008 4:38:42 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to
STARTING>
<Jan 18, 2008 4:38:42 PM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning
thread pool>
1.4.2. WebSphere Server full version

And in WebSphere the version can be found in the Welcome page:
1.5. Deployment topology

The Engine can be deployed in a cluster configuration in a Web Application server or it can have a Backup location
in the Standalone version.
When running in a Web Application, the Support analyst will be interested in knowing:
The number of nodes in the cluster where the Engine has been deployed.
The physical locations of the nodes
The number of nodes running at the time of the incident
When running Standalone, the Support analyst will like to know if there is a Backup location and if it was active at
the time of the incident.
1.6. Operating system version

The Operating System version is also relevant for troubleshooting the Engine. Minor version details or build
numbers might not be required, though. As general rule, just include in the report Vendor and mayor version (for
instance: Windows XP Sp2, Red Hat Linux ES 4 update 3, Solaris 10 (SPARC), HP-UX 11i V2, etc.)
1.7. Basic Info Recommendations

As you can see, the list of Basic Info generally required is quite long. Thus, we will like you to consider having this
gathered in a document to reduce the time spent at the moment you face an issue.
Have in mind that each support case can be handled by a different analyst who might not be familiar with your
environment. And even in the cases where youve been in contact with that same analyst in the past, he might not
be aware of upgrades or all the different environment configurations that you might have.
Therefore, if you can provide an up-to-date document detailing this Basic Info for the environment where you
detected the problem you will accelerate the case resolution and your Support analyst will appreciate it.
2. Engine Logs
2.1. Description of the Engine Logs
The Engine logs are composed of a fixed number of files with a given maximum size. In order to maintain the log
size within this constrains, when all log files have been written the new log entries will be written to the oldest file in
a round-robin fashion. This limit on the amount of log stored translates into a limited time frame of log info. We will
address how to change this amount in section 2.2.1. Configuring the Log settings in the Process
Administrator.
While the content of the files is text, they are not easily readable without the help of the LogViewer or LogReader.
We will briefly cover them in sections 2.3. Using LogViewer and 2.4. Using LogReader.
2.1.1. A Log entry

A log entry is composed of the following fields:
Severity
Message
Time
Date
Application
Module
Thread
Indicates the level of severity of the message. The possible values, in decreasing order of severity,
are: FATAL, SEVERE, WARNING, INFO, DEBUG and TRACE. Note that TRACE is an internal
value used in special occasions to dump stack traces of an Engine thread (this value is not
available to the PBL logMessage method).
Text of the entry.
Time when the entry was generated. Note that the definition of the value is in seconds. Also note
that the date is stored in GMT-0 format and by default it is converted to the local time zone of the
computer where the LogViewer or LogReader is running. The LogViewer time zone can be
configured in the Preferences dialog.
Date when the entry was generated.
The application that sent the message. The possible values are:
* Default: corresponds to internal entries generated by the Engine and
* CIL: corresponds to entries generated by the project code (see section 3. Project
Debugging for more info).
Module that sent the message. Possible values are:
* Main: it is the most frequent value.
* MailDispat: corresponds to entries generated by the Mail Dispatcher module.
Name of the Thread that generated the entry. Note that some thread names might be truncated
(for instance: <8>...tenerThreadPool : 2), but no information is lost because of that.
2.1.2. To-Do Item messages

This particular kind of messages generated by the Engine is very useful when troubleshooting. These messages
have always a severity level of Info, their application is Default, and their module is Main. The following is an
example in text format for version 5.5 and 5.7:
<I>, "Oct 1, 2009 5:13:35 AM", Default, Main, <47...spatcher (63372763), "Executing item:
RUN_AUTOMATIC Inst (214,19093433,0) Act [ApplyConfig] Proc [/ResolveOperations#Default-1.4]
Due [2009-10-01 02:05:28-04 1254377128000000] TimeStamp [2009-10-01 02:05:28-04] id[57671033]
later [false]"
The values shown in this type of messages follow the following pattern:
<Severity>, "Data_&_Time", Application, Module, Thread, "Executing item: ToDo_Item_Type Inst
(ProcessId,InstanceId,ThreadId) Act [ActivityId] Proc [OU/ProcessName#Variation-Version] Due
[Time_the_Item_is_due]
TimeStamp
[Time_the_Item_was_created]
id[Todo_ItemId]
later
[isLater?]"
As you can see, this message is telling us what type of execution (ToDo_Item_Type), process, activity, and
instance the engine is about to run. For a listing and description of the To-Do Types, refer to section 5.1.1. To-Do
Item Types.
The following is an example in text format for version 6.0 and 10.3:
<D>, "Aug 25, 2009 4:30:45 PM", Engine, Main, <5> ET(3), "Executing item: RUN_AUTOMATIC Inst
(1,21,0) Act [StartExecution[End]] Proc [/LeaveProcess#Default-1.0] Due [2009-08-25 17:30:4504:00 1251235845000000] TimeStamp [2009-08-25 17:30:45-04:00] id[63150014] later [false]"
The only difference in the pattern is the addition of the Micro-Activity Id, as shown below:
<Severity>, "Data_&_Time", Application, Module, Thread, "Executing item: ToDo_Item_Type Inst
(ProcessId,InstanceId,ThreadId)
Act
[MicroActivityId[ActivityId]]
Proc
[OU/ProcessName#
Variation-Version]
Due
[Time_the_Item_is_due]
TimeStamp
[Time_the_Item_was_created]
id[Todo_ItemId] later [isLater?]"
The component elements of the message listed here are not going to be described in this document. If you have
questions about them, please, contact Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
2.1.3. Log files location

The default location is $BPM_HOME/log and the value can be modified in the following Process Administrator page
(Engines <engine-name> Basic Configuration [Tab]):
2.1.4. Log files description

The files follow the following name convention: <engine-name>.log.<index>. Where index takes the values from 0
to the maximum number of log files and tz. For instance, for the engine used in this example j2eewl57engine
we will have the following set of files:
C:\bea\albpm5.7\j2eewl\log\j2eewl57engine.log.0
C:\bea\albpm5.7\j2eewl\log\
C:\bea\albpm5.7\j2eewl\log\j2eewl57engine.log.tz
Important: All files listed above are part of the Engine Log. This means that none of the BPM Log applications will
be able to read the logs if any of these files are missing. Therefore, please, include all <engine-name>.log.* files
when sending the Engine Log to a Support analyst. An easy way to make sure all files were gathered is to
compress the whole log directory (for instance, C:\bea\albpm5.7\j2eewl\log).
2.1.5. Specifying an alternative Log location

In cluster environments with more than one Node running on the same Host, you will need to specify a different log
location for each Engine node. This can be achieved by setting the following JVM additional parameter to the Java
Virtual Machine that runs the Engine Node:
-Dfuego.server.log.file=<file-path>/<log-filename>.log
Note that log-filename (that by default corresponds to the engine name) here can be set to any other value. Like,
for instance, the name of the Application Server Node.
Please, have in mind that this parameter will override the value specified by the Process Administrator. Therefore,
any change to the log directory in the Process Administrator will be ignored.
2.2. Configuring Log size and Severity level

2.2.1. Configuring the Log settings in the Process Administrator
In the following Process Administrator snapshot (Engines <engine-name> Log [Tab]) three settings have
been changed from the default:
1. Messages Logged from Server: these are the internal BPM Engine messages. The value has been set to
Debug level to capture full execution details.
2. Messages Logged from BP-Methods: these are the messages logged by the project code (see section 3.
Project Debugging for more info). The value has been set to Debug level.
3. The Maximum Number of Log Files has been increased to 20 (being this the minimum recommended for
this configuration). This is important since having log levels set in Debug means hundreds of messages
more than in the default scenario and, in turn, this means that the log time frame will be reduced (given that
the number of log files is fixed and that old items are overwritten once the maximum number of files has
been reached).
2.2.2. Enabling Asserts in the JVM that runs the BPM Engine
In BPM versions 5.5 and 5.7 (but not in 6.0 and 10.3), Java Asserts must be enabled in order for Debug messages
from the Engine to be generated. This configuration involves adding an additional JVM argument (-ea) to the start
script. The place where to add this argument depends on what type of engine you are running. The following
sections describe how to enable asserts in the Standalone, WebLogic and WebSphere versions.
2.2.2.1. Enabling Asserts in the Standalone Engine

For the Standalone Engine (and not for the J2EE versions), the additional JVM argument can be added in the
Execution tab of the corresponding engine within the Process Administrator. An example is shown below:
10
2.2.2.2. Enabling Asserts in the WebLogic Server

The following snapshot corresponds to WebLogic Server version 9.2.0. The argument -ea has been added to the
Server Start page of the Server where the BPM Engine is deployed.
11
If, for any reason, this configuration change does not affect the actual java command executed when WLS starts
(see section 2.2.3.1. Start output of the WebLogic Server), then change the startWebLogic.cmd or .sh
(depending on the OS) adding the -ea option as follows:
In Windows OS (startWebLogic.cmd):
set JAVA_OPTIONS=%SAVE_JAVA_OPTIONS% -ea
In Unix-like OS (startWebLogic.sh):
JAVA_OPTIONS=$SAVE_JAVA_OPTIONS ea
2.2.2.3. Enabling Asserts in the WebSphere Server

The following snapshot corresponds to WebSphere Application Server version 6.0.2. The argument -ea has to be
set to the Generic JVM arguments field on the JVM page of the server running the BPM Engine (e.g. Application
servers server1 Process Definition Java Virtual Machine).
12
2.2.3. Restarting the BPM Engine

Once the changes are completed, you need to restart the Engine. In a Standalone environment this can be done
from the Process Administrator while in a J2EE environment this implies restarting the EAR, the node or the cluster
depending on the deployment topology.
2.2.3.1. Start output of the WebLogic Server

The start log for WLS should look like the following (Note that the JAVA Options shown at the beginning is not
printed by the default script):
JAVA Memory arguments: -Xms256m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=48m -XX:MaxPermSize=128m
.
JAVA Options= -Xverify:none
-da -Dplatform.home=C:\bea\WEBLOG~1 -Dwls.home=C:\bea\WEBLOG~1\server Dwli.home=C:\bea\WEBLOG~1\integration
-Dweblogic.management.discover=true
-Dwlw.iterativeDev=
Dwlw.testConsole=
-Dwlw.logErrorsToConsole=
Dweblogic.ext.dirs=C:\bea\patch_weblogic920\profiles\default\sysext_manifest_classpath -ea
.
WLS Start Mode=Development
.
CLASSPATH=;C:\bea\patch_weblogic920\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\bea\JDK1
50~1\lib\tools.jar;C:\bea\WEBLOG~1\server\lib\weblogic_sp.jar;C:\bea\WEBLOG~1\server\lib\weblogic.jar;C:\b
ea\WEBLOG~1\server\lib\webservices.jar;;C:\bea\WEBLOG~1\common\eval\pointbase\lib\pbclient51.jar;C:\bea\WE
BLOG~1\server\lib\xqrl.jar;;
.
PATH=C:\bea\patch_weblogic920\profiles\default\native;C:\bea\WEBLOG~1\server\native\win\32;C:\bea\WEBLOG~1
\server\bin;C:\bea\JDK150~1\jre\bin;C:\bea\JDK150~1\bin;D:\oracle\product\10.2.0\db_1\bin;C:\WINDOWS\syste
m32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Microsoft SQL Server\80\Tools\BINN;C:\Program
Files\ZipGenius
6\;C:\Documents
and
Settings\gcavalli\bin;C:\Program
Files\QuickTime\QTSystem\;C:\bea\WEBLOG~1\server\native\win\32\oci920_8
.
13

***************************************************
* To start WebLogic Server, use a username and
*
* password assigned to an admin-level user. For *
* server administration, use the WebLogic Server *
* console at http:\\hostname:port\console
*
***************************************************
starting weblogic with Java version:
java version "1.5.0_04"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode)
Starting WLS with line:
C:\bea\JDK150~1\bin\java -client
-Xms256m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=48m
XX:MaxPermSize=128m -Xverify:none -da -Dplatform.home=C:\bea\WEBLOG~1 -Dwls.home=C:\bea\WEBLOG~1\server
-Dwli.home=C:\bea\WEBLOG~1\integration
-Dweblogic.management.discover=true
-Dwlw.iterativeDev=
Dwlw.testConsole=-Dwlw.logErrorsToConsole=
Dweblogic.ext.dirs=C:\bea\patch_weblogic920\profiles\default\sysext_manifest_classpath
-ea
Dweblogic.Name=AdminServer
-Djava.security.policy=C:\bea\WEBLOG~1\server\lib\weblogic.policy
weblogic.Server
Finally open the LogViewer and verify that Debug Messages coming from the server are being generated
(Application is Default for messages coming from the Engine and CIL for messages coming from BPM Methods
or custom code).
2.3. Using LogViewer (in graphical environments)

The BPM LogViewer is a graphical application to read and perform different type of searches of log entries. Do not
confuse it with the simple html Log Viewer, available at the Process Administrator (Engines <engine-name>
Basic Configuration [Tab] Log Viewer). The latter only allows to browse the entries and does not provide the
search functionality that the former does.
If BPM Enterprise is installed on a box with a graphical environment, the LogViewer can be started from the
following script: $BPM_HOME/bin/albpmlogviewer.sh (an example in windows will be: C:\bea\albpm5.7\j2eewl\
bin\albpmlogviewer.exe). The application can read log files located locally or remotely.
For more information on this application, please, refer to the LogViewer documentation at:
http://download.oracle.com/docs/cd/E13165_01/albsi/docs57/olh_logviewer/index.html
2.4. Using LogReader (in character terminals)

The BPM LogReader is a command line application that parses the contents of the Engine log files and writes the
log items in a readable format to the Standard Output. This application is useful in environments where a graphic
terminal might not be available. The recommended use is to redirect the output to a file and then parse it with a tool
like grep or simply open it with a text editor.
The script syntax is the following:
$BPM_HOME/bin/logreader.sh <engine log file location>.log > /tmp/engine_log.txt
Note that the log filename <engine log file location>.log does not require an index after the .log.
This is an example of a grep command to find all items regarding a particular instance (in this case, instance 1234):
egrep 1234,|1234\/ /tmp/engine_log.txt > /tmp/Instance1234_log.txt
14
2.4.1. Windows logReader.bat script

The followings are copies of the Windows script, in case your BPM distribution does not contain it:
BPM Versions 5.5 and 5.7:
set BPM_HOME=C:\bea\albpm5.7\j2eewl
set JAVA_HOME=%BPM_HOME%\jre
%JAVA_HOME%\bin\java
%JAVA_OPTIONS%
fuego.log.LogReader %1
-ea
-classpath
%BPM_HOME%\lib\ftlib.jar

set BPM_HOME=C:\bea\albpm6.0\j2eewl
set JAVA_HOME=%BPM_HOME%\jre
%JAVA_HOME%\bin\java
%JAVA_OPTIONS%
fuego.log.LogReader %1
-ea
-classpath
%BPM_HOME%\lib\fuego.lib.jar
2.4.2. Unix logReader.sh script

These are the Unix versions of the same script:
BPM_HOME=/bea/albpm5.7/j2ee
JAVA_HOME=$BPM_HOME/jre
$JAVA_HOME/bin/java $JAVA_OPTIONS -ea -classpath $BPM_HOME/lib/ftlib.jar fuego.log.LogReader
$1

BPM_HOME=/bea/albpm6.0/j2ee
JAVA_HOME=$BPM_HOME/jre
$JAVA_HOME/bin/java
$JAVA_OPTIONS
fuego.log.LogReader $1
-ea
-classpath
$BPM_HOME/lib/fuego.lib.jar
2.5. Additional Logs related to the Engine

In a J2EE environment it is also recommended to gather the corresponding Application Server Logs. This is so
because some error messages from the underlying mechanisms that the J2EE Engine uses are not be logged in
the Engine logs.
The following sub-sessions describe the default location for these logs. When in doubt, please contact your
Application Server Administrator.
2.5.1. WebLogic Server Logs

The default location of the logs for the WebLogic Server is: $BEA_HOME/user_projects/domains/<domain-name>/
servers/AdminServer/logs (for instance, in a single node Windows installation where the domain name is ALBPM
the logs can be found at: C:\bea\user_projects\domains\ALBPM\servers\AdminServer\logs).
You can find more WebLogic troubleshooting information at:
http://download.oracle.com/docs/cd/E13222_01/wls/docs81/cluster/trouble.html
For more info, refer to WebLogic Server documentation at:
http://download.oracle.com/docs/cd/E13222_01/wls/docs81/index.html
15
2.5.2. WebSphere Server Logs

The default location of the logs for the WebSphere Application Server is: $WEBSPHERE_HOME/AppServer/
profiles/default/logs/<server-name> (for instance, in a single node Windows installation where the server name is
server1 the logs can be found at: C:\Program Files\IBM\WebSphere\AppServer\profiles\default\logs\server1).
You can find more WebSphere information at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp
3. Project Debugging
Whenever you experience a problem with the BPM Engine there is chance that the issue is triggered by:
- an aspect of your project design,
- a defect in the code of the project,
- a defect in one of the resources utilized by the Engine (and/or the Project),
- or a defect in the Engine.
In order to rule out the first two causes it is always of great help to include sufficient log messages in the BPM
Project code. In this section we are going to indicate how to do it in a meaningful way.
3.1. logMessage method

BPM has a predefined method to add entries to the Engine logs. The method is called logMessage and below is
shown a basic example of its use:
logMessage("message", Severity.DEBUG);
3.2. Meaningful logging

As you might have seen in section 2.1.1. A log entry the list of fields of a log entry does not include Process,
Instance of Method information. Thus, in order to provide meaningful log information we strongly recommend to
include in all your entries as much info about the Instance or Process as possible. Bellow youll find two best
practice examples for the most common cases:
3.2.1. Log message for Instance Methods (this is, that have access to instance information. This includes methods
in most activities except from Global Activites -that do not have instance access-):
logMessage(id.id
<severity_level>);
"::"
activity.name
"::<method_name>|<your_message>",
3.2.2. Log messages for Static Methods (this is, that do not have access to instance information. This includes
methods in Global Activites -that do not have instance access-):
logMessage(process.name
<severity_level>);
"::<activity
name>::<method_name>|<your_message>",
4. Obtaining Affected Projects

The Project/s the Engine is running are as important as the Engine logs for troubleshooting purposes. In this
section we will give you a few hints on how to collect relevant Projects files.
16
4.1. Obtaining a Project Export with all libraries included

When sending a Project to Support you should Export the relevant Projects from BPM Studio including all libraries.
This way we will have access to versionable and non-versionable libraries that your project might use.
Below is a snapshot of the option in Studio to perform, what we can call, a full export:
4.2. Recovering Process Versions from the Process Administrator

In the case that you are uncertain of what Process version is actually deployed in the environment where the
problem has being found, the way to retrieve it from the Process Administrator is accessing the Project History by
following the link shown below:
17
Then you can either select the link on Processes or Revisions in the following page:
Until you get to the relevant Revision (depending on the case, it could be the latest one or not) and there youll find
the link to Download the process files for it (called Recover the project revision), as shown below:
18
5. Obtaining Information from the Engine DB

At this time we dont provide any export tool for information on the Engine DB, so you should check with your DB
Admin if you need assistance with the task. Alternatively, you can use the SQL client of your preference to generate
the export (for instance, DBVisualizer, DB Artisan, Toad, Erwin, etc).
5.1. Obtaining all entries in the table of Pending Execution Items

It is generally relevant to include in your report an export of the Pending Execution Items. There are two tables that
contain this information: one is PTODOITEMS and the other is PWKTODO and both can be found in the Engine
Database. The tables contain a number of rows in the order of hundreds (or less), so exporting them as comma
separated values and then compressing them should produce a file in the order of 100kb. Please, remember to
include all columns and all rows in the export.
5.1.1. To-Do Item Types
When navigating the Pending Execution Items table (PTODOITEMS), you might find useful to know what the
meaning of the different type ids is. The following is a list valid from version 5.5 onwards:
0 = RUN_AUTOMATIC
1 = INSTANCE_EXPIRATION
2 = ACTIVITY_EXPIRATION
3 = MAXRETRY_EXCEPTION
4 = GENERIC_EXCEPTION
5 = IPC_SEND
6 = IPC_PROCESS_TW
7 = IPC_PROCESS_NW
8 = WAKEUP_CADUCATOR
9 = IPC_INTERRUPTION
10 = GAUTOMATIC_POLLING
11 = DAILY
12 = REPLACEMENT
13 = WAIT_NOTIFICATION_TIMEOUT_EXCEPTION
14 = ITEM_RECOVERY
15 = GAUTOMATIC_SCHEDULED
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
ABORT_COPIES
COMPONENT_NOTIFICATION
INSTANCE_RESUME
UNSELECT
BAM_UPDATER
SHUTDOWN
POLLING_DIRECTORY
CADUCATOR
GAUTOMATIC_INIT_LISTENING
PROCESS_CHECKPOINT
RUN_IMMEDIATE
ABORT_INSTANCE_EXCEPTION
START_SERVICES
GAUTOMATIC_JMS_LISTENING
UNDEPLOY
GAUTOMATIC_JMX_LISTENING
The description of the To-Do item types exceeds the scope of this document. If you have questions, please, contact
Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
5.2. Obtaining affected entries in the table of Instances and Instance Events
In the case that you have found a problem with a specific instance it is always a good idea to collect the relevant
entries in the table of Instances (PPROCINSTANCE) and, optionally, the entries in the table of Instance Events
(PPROCINSTEVENT). Both of these tables are also located in the Engine Database. The following are examples of
how to obtain these entries:
5.2.a. Retrieves all copies of the Instance 1234 from the PPROCINSTANCE table:
select CREATIONTIME, NTHREADS, PROCESSID, HASPARTFORROLE, ACTIVATIONTIME, ITEMDEADLINE,
SOURCEACTIVITYNAME, PROCESSDEADLINE, NREMARKS, PENDINGITEMID, ACTIVITYNAME, NEXTPARTICIPANT,
PRIORITY, NAME, ACTIVITYDEADLINE, THREADID, PARENTTHREAD, CTHREADS, RECEIVETIME, LASTRESULT,
ROLEIN, INSTID, NCHANGES, NATTACHMENTS, STATE, INSTANCEDATA, AUTHOR, TOTALTHREADS, PARTICIPANT
from <Engine_Schema>.PPROCINSTANCE where INSTID = 1234;
19
5.2.b. Retrieves all events for the Instance 1234 from the PPROCINSTEVENT table:
select ACTIVATETIME, EVENTDATA, EVENTID, PROCESSID, THREADID, TYPE, ACTIVITYNAME, RECEIVETIME,
CATEGORY, EVENTTIME, INSTID, PARTICIPANT
from <Engine_Schema>.PPROCINSTEVENT where INSTID = 1234;
In the preceding examples we have included all BPM relevant columns excluding project sensitive information (i.e.
external variables: columns that start with V_). But you might as well run a select * if, for instance, the problem
was found on a test environment where external variables do not contain sensitive info for your business.
5.2.1. Instance States
When navigating the Process Instance table (PPROCINSTANCE), you might find useful to know what the meaning
of the different states is. The following is a list valid from version 5.5 onwards:
1 (2^0) = RUNNING
2 (2^1) = SUSPENDED
4 (2^2) = GRABBED
8 (2^3) = JOIN_WAITING
16 (2^4) = EXCEPTION
32 (2^5) = COMPLETED
64 (2^6) = ABORTED
128 (2^7) =
256 (2^8) =
512 (2^9) =
1024 (2^10)
2048 (2^11)
4096 (2^12)
8192 (2^13)
ACTIVITY_COMPLETED
ABORTING
AUTOMATIC_RESUME
= SHARING_VARIABLES
= STICKY_PARTICIPANT
= IN_RELAY
= SELECTED_FOR_EXECUTION
Note that several states can be present at the same time. When that is the case, the instance state is the addition
of the current states. For example, if the instance state is RUNNING and SHARING_VARIABLES, the
corresponding state code is 1 + 1024 = 1025. Conversely, if you find a state code of 1088, this means that the
instance state is 1088 = 1024 + 64, which means ABORTED and SHARING_VARIABLES. Another way of looking
at the state code is analyzing it as a binary word where each bit represents a different state. The individual states
th
are encoded with a bit in the n position of the word (being n the power of 2 shown in the list above).
The description of the states exceeds the scope of this document. If you have questions, please, contact Oracle
BPM Support (see section 12. Contacting Oracle BPM Support).
5.2.2. Instance Event Types
When navigating the Instance Event table (PPROCINSTEVENT), you might find useful to know what the meaning
of the different type ids is. The following is a list valid from version 5.5 onwards:
0 = INSTANCE_CREATED
1 = INSTANCE_END
2 = ACTIVITY_IN
3 = ACTIVITY_OUT
4 = GLOBAL_EXECUTED
5 = ITEM_SELECTED
6 = ITEM_UNSELECTED
7 = ITEM_EXECUTED
8 = INSTANCE_SUSPENDED
9 = INSTANCE_RESUMED
10 = INSTANCE_GRABBED
11 = INSTANCE_UNGRABBED
12 = INSTANCE_SELECTED
13
14
15
16
17
18
19
20
21
22
23
24
25
=
=
=
=
=
=
=
=
=
=
=
=
=
INSTANCE_UNSELECTED
SUBFLOW_INSTANCE_CREATED
INSTANCE_ABORTING
INSTANCE_TO_EXCEPTION
INSTANCE_ABORTED
ACTIVITY_COMPLETED
INSTANCE_UPDATED
USER_EVENT
INSTANCE_REASIGN
MEASUREMENT_START
MEASUREMENT_STOP
MEASUREMENT_START_STOP
VARIABLE_MEASUREMENT
The description of the event types exceeds the scope of this document. If you have questions, please, contact
Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
20
5.3. Other relevant queries to the Engine Schema

The following is a list of useful queries to the Engine Schema (using Oracle DBs functions and predefined
variables).
5.3.a. Number of active instances and copies per process:
select PROCESSID, Count(*)
from <Engine_Schema>.PPROCINSTANCE
where STATE not in (32, 64, 1056, 1088, 17440, 17472) group by PROCESSID;
5.3.b. Number of completed or aborted instances and copies per process:

from <Engine_Schema>.PPROCINSTANCE
where STATE in (32, 64, 1056, 1088, 17440, 17472) group by PROCESSID;
5.3.c. Number of To Do Items per process, activity and type:

select PROCESSID, ACTIVITYNAME, TYPE Count(*)
from <Engine_Schema>.PTODOITEMS
group by PROCESSID, ACTIVITYNAME, TYPE;
5.3.d. Number of expired To Do Items per process:

where DUETIME < sysdate
group by PROCESSID;
5.3.e. Number of expired To Do Items per instance:

select INSTID, Count(*)
group by INSTID;
5.3.f. Number of expired To Do Items per type:

select TYPE, Count(*)
group by TYPE;
5.3.g. Number of Events associated to completed or aborted instances per process:

select I.PROCESSID, Count(*)
from <Engine_Schema>.PPROCINSTEVENT E, <Engine_Schema>.PPROCINSTANCE I
where E.INSTID = I.INSTID and I.STATE in (32, 64, 1056, 1088, 17440, 17472)
group by I.PROCESSID;
5.4. Relevant queries to the Directory Schema

To understand in detail some of the information returned from these queries you will need to execute the following
queries against the Directory Schema (formerly known as FDI). These are some of the queries:
5.4.a. List of Deployed Processes:
select *
from <Directory_Schema>.FUEGO_DEPPROC;
5.4.b. List of Published Processes (or Process Definitions):

select *
from <Directory_Schema>.FUEGO_PROCDEF;
21
5.4.c. Where to find Participants:

select *
from <Directory_Schema>.FUEGO_PARTICIPANT
where FUEGO_ID = <Participant_Id: This value is used in the column PARTICIPANT from the table
PPROCINSTANCE>;
6. Obtaining Full Thread Dumps

A somehow rudimentary but useful tool when troubleshooting performance issues with the BPM Engine is to collect
several Full Thread Dumps (FTD) of the Java Virtual Machine (JVM) that runs the Engine during the moment when
the slow performance is experienced. A FTD is a text output of the stack traces of all the threads a JVM is running.
6.1. Obtaining FTDs using the Process Administrator

One way to perform this is by clicking the Thread Dump button in the Process Administrator Engine page, as
shown below:
This option might not work in all environments. In addition to that, it might be cumbersome to collect or copy the
results. In any case, there are other simply alternatives that will depend on the Operating System.
6.2. Obtaining FTDs in Windows OSs

For instance, in Windows if the command console that runs the Engine JVM is available (like it will be, for instance,
when running WebLogic Server) you can obtain a FTD by pressing the Control+Break keys in the command
console window.
If the command console is not available, you should use an alternative way like the one described in section 6.4.
Obtaining FTDs using other tools.
22
6.3. Obtaining FTDs in Unix-like OSs

In Unix-like OSs you can send a signal to the JVM process to generate the FTDs. You simply need to find out what
is the process id (or pid) and then send a SIGQUIT to it.
To find the pid you can run the following command:
ps -fea | grep java
And once you have the process id, the SIGQUIT can be sent with the following command:
kill -3 <pid>
And advantage of this command is that you can create a simple script that calls it iteratively and with that collect
several consecutive FTDs at a regular time interval. A simple example of such script for a Unix system is:
while true
do
kill -3 $1
sleep .5
done
Note that this script will run forever if it is not aborted.
6.4. Obtaining FTDs using other tools

Here we mention a couple of free tools to obtain Full Thread Dumps. Note that these tools require the process id of
the JVM running the Engine. Weve seen how to obtain it in Unix OSs in the previous section. For Windows OSs
you will find it in the Task Manager window.
6.4.1. Sun jstack

jstack is an application to obtain FTDs provided with Sun JVM 1.5.0 (and newer versions). It should be found in the
bin directory of the JDK or JRE Java installation. For more details about its use and syntax refer to:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html
6.4.2. AdaptJ StackTrace

Other useful tool to obtain FTDs is StackTrace by AdaptJ Systems (http://www.adaptj.com/main/stacktrace). The
tool can be launched online or it can be downloaded from the following site: http://www.adaptj.com/main/download.
6.5. Java Core Dumps

In some scenarios for instance, on a JVM crash- a javacore dump is generated. Javacores are a more
comprehensive text outputs than FTDs, but they are rarer and more difficult to obtain. Additionally, not all JVM
support them in the same way. A continuation there is some basic information about them.
6.5.1. Sun JVM Fatal Error Log

A Fatal Error Log is the equivalent of a javacore dump in the Sun Java SE 6 release. For more info refer to:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/felog.html
23
6.5.2. IBM Thread and Monitor Dump Analyzer

This is a tool that reads and analyzes IBM JVM javacores. More info on the tool can be found at:
http://www.alphaworks.ibm.com/tech/jca
7. Obtaining Java Heap Dumps

Heap Dumps are binary dumps of all objects in the applications memory space (the heap) of a JVM. These are
very useful to troubleshoot Out of Memory conditions. The only disadvantage they have is that they tend to be very
large (specially when the applications memory has been increased to try to prevent an Out of Memory)
7.1. How to generate a Heap Dump

There are several ways to generate a java heap dump:
A heap dump will be generated when OutOfMemoryError is thrown by specifying
HeapDumpOnOutOfMemoryError JVM option.
Use jmap -dump option to obtain a heap dump at runtime
(http://java.sun.com/javase/6/docs/technotes/tools/share/jmap.html).
Use jconsole option to obtain a heap dump via HotSpotDiagnosticMXBean (http://java.sun.com/javase/6/
docs/jre/api/management/extension/com/sun/management/HotSpotDiagnosticMXBean.html) at runtime
(http://java.sun.com/javase/6/docs/technotes/tools/share/jconsole.html).
Use hprof (http://java.sun.com/developer/technicalArticles/Programming/HPROF.html).
7.1.1. HeapDumpOnOutOfMemoryError JVM option

This is the best option for gathering information on an Out of Memory condition, since it does not affect
performance and it is easy to configure (compared with Java Profiling tools).
The option is available in Suns JVM 1.4.2 update 12; 5.0 update 9 and newer versions and on IBMs JVM. To
enable it, pass the additional argument to the JVM:
-XX:+HeapDumpOnOutOfMemoryError
For more details refer to:

Java HotSpot VM Options: http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
Getting Heap dumps on Solaris platform: http://www-1.ibm.com/support/docview.wss?uid=swg21242314
7.2. Tools to analyze Heap Dumps

7.2.1. Sun jhat
Sun jhat is the basic tool for Heap Dump analysis. A recommendation about its use is that the memory assigned to
the application should be as much as possible (you will see that Heap Dumps are quite big and opening them
requires a lot of memory in order to prevent OutOfMemoryExceptions). The following is an example where the
memory has been set to 1024 Mb:
jhat -J-mx1024m <heap_dump.file>
For more details about its use and syntax refer to: http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html
24
7.2.2. IBM HeapAnalyzer

IBM HeapAnalyzer is nice graphical tool. The only problem with it is that it requires a memory larger than the size of
Java heaps to be able to open them (it will result in an OutOfMemoryException otherwise). In fact, it requires more
memory than jhat. This tool is available at: http://www.alphaworks.ibm.com/tech/heapanalyzer
8. Java Profiling Tools

In addition to Full Thread Dumps and Heap Dumps there are other tools which, in spite of probably being more
complex to setup or having a moderate impact on performance, are more powerful in their analytic capabilities and
simplicity to use. Here we mention some sources of information.
8.1. Sun Tools and Options

The following document is of great help: Troubleshooting Guide for Java SE 6 with HotSpot VM. You can find it
at: http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/index.html
Note that even when the document is designed for Java SE 6 many recommendations and tools also apply to
earlier versions. For instance, Java SE 6 jhat can open heap dumps generated with Java 1.5.0.
These other pages are also of help:
Monitoring and Managing Java SE 6 Platform Applications:
http://java.sun.com/developer/technicalArticles/J2SE/monitoring/
Monitoring and Management for the Java Platform:
http://java.sun.com/javase/6/docs/technotes/guides/management/index.html
Java HotSpot VM Options http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
JDK Tools and Utilities: http://java.sun.com/javase/6/docs/technotes/tools/index.html
JConsole: http://java.sun.com/javase/6/docs/technotes/guides/management/jconsole.html
8.2. WebLogic JRockit Profiling Tools

JRockit JVM has a profiling API and the implementation comes with a set of tools: the JRockit Mission Control
tools suite. For details visit:
Profiling and Debugging with WebLogic JRockit:
http://download.oracle.com/docs/cd/E13188_01/jrockit/docs81/portblty/jvmtools.html
JRockit Mission Control: http://dev2dev.bea.com/jrockit/tools.html
JRockit Mission Control Documentation: http://e-docs.bea.com/jrockit/tools/
8.3. Other Profiling Tools

There is a list of them. We have used some of them (JProfiler, for instance) and they proved to be of great help in
identifying performance issues. Here are some examples:
JProfiler: http://www.ej-technologies.com/products/jprofiler/overview.html
YourKit: http://www.yourkit.com/
JProbe: http://www.quest.com/jprobe/
JIP: http://jiprof.sourceforge.net/
Extensible Java Profiler: http://ejp.sourceforge.net/
25
9. Basic Performance Tuning

In this section we will discuss a few of the issues that most commonly affect the Engines performance. You should
realize that these are general recommendations and that the particular characteristics of your projects might call for
different settings. When in doubt, please, contact Oracle BPM Support (see section 12. Contacting Oracle BPM
Support).
9.1. Performance deterioration due to insufficient space in the PAPI Instance cache
This is likely to affect projects with a considerable number of active instances per project and an elevated number
of users executing filters (or opening the Inbox). The PAPI cache is a cache of header information for instances. Its
main purpose is to eliminate unnecessary access to Instance tables by providing an abbreviated description of the
instance state and its main variables. The default size of the PAPI cache is 10,000 instances per process. In some
circumstances this number will fall short and you might find messages in the Engine log indicating the following:
Instance Cache: Opening the cache for process '/<ProcessName>#Default-<Version>'.
The maximum number of instances has been exceeded.
The best indicator for this type of problem is to execute the query 5.3.1. If there are more than 10,000 active
instances per process, you should increase the PAPI cache size. A good rule of thumb is to set the cache size to
double the number of current active instances per process. Values up to 1,000,000 can be used. The setting is
available in the WorkSpace (or Portal) configuration file (or in your custom PAPI application configuration). An
example of the location of this file in BPM 5.7 (which is similar to 5.5) is C:\bea\albpm5.7\enterprise\
webapps\portal\WEB-INF\portal.properties. The property within this file is the following:
fuego.portal.papi.instancesCacheSize=10000
An example in BPM 6.0 (which is similar to 10.3) is C:\bea\albpm6.0\enterprise\webapps\workspace\WEB-INF\
workspace.properties. The property within this file is the following:
fuego.workspace.papi.instancesCacheSize=10000
Please, remember that after changing this property in a WorkSpace running as a J2EE application you will need to
regenerate the WorkSpace EAR and redeploy it.
9.2. Performance deterioration due to lack of Instance disposing

The Engine Database is designed to handle hundreds of thousands of active instances. As the total number of rows
in the different engine tables grow, the engine overall performance will deteriorate. Therefore, to reach optimal
performance it is highly recommended to configure the Disposer. This is a mechanism that will either delete or
archive Completed or Aborted Instances. The following screenshot shows where the Disposer related settings can
be configured and the generally recommended values. For more details, please, refer to the BPM Administrators
guide: http://download.oracle.com/docs/cd/E13165_01/albsi/docs57/adminguide/index.html.
26
9.3. Performance deterioration in J2EE environments due to frequent SQL timeouts

In J2EE environments Engine transactions are regulated by the J2EE Server transaction manager. One of the
functionalities of this manager is to coordinate transactions across resources and makes sure that the total time
spent in a particular transaction does not exceed the global transaction timeout (or JTA timeout). Although the
default timeout might be sufficient for small projects in test environments, this value is not recommended for
production environments. In the latter case, the minimum recommended JTA timeout is 5 minutes (300 seconds).
For instructions on how to set this value, please, refer to your J2EE Server documentation.
A JTA Transaction timeout exception can be identified by the following message in the engine log:
Unexpected exception while enlisting XAConnection java.sql.SQLException: Transaction
rolled back: Transaction timed out after 30 seconds
Whenever you are experiencing this SQL Exception (no matter what is the full stack trace); the first step you need
to take is to verify that the JTA timeout is not smaller than 5 minutes. If it is, the value should be increased to 5 or
more minutes (a timeout of up to 8 minutes is reasonable for most environments). Note that if the JTA timeout will
affect other projects in the J2EE environment, you should check with the J2EE Administrator before making any
change.
9.4. Performance deterioration in J2EE environments due to Disposer timeouts

In J2EE environments with short global transaction timeouts (JTA timeout) you might find that the Disposer is
unable to process the default number of instances (1000) in a single transaction. That is why on the latest 5.7 build
(and newer versions) we have made available a JVM property to modify the number of instances deleted per
transaction. The property is the following:
27
-Dfuego.server.amountInstancesToCommit=200
The value shown above is the recommended setting to be used when the problem has been found on the
environment. The common indicator for this type of problem is a SQLException caused by a transaction rolled
back. The stack trace will contain invocations to these methods:
o fuego.server.persistence.Persistence.getInstancesToDispose()
o fuego.server.execution.Caducator.disposeInstances()
o fuego.server.execution.Caducator.execute()
You should note that the syntax of the method might vary depending on the version. Therefore, when looking at the
logs you should look for key words like: "getInstancesToDispose", "caducator", "dispose", etc
Additionally, you might find the following message:
o "Could not dispose instances from process '/<process_name>#Default-<version>'"
9.5. Performance deterioration in J2EE environments due to Scheduler timeouts

In J2EE environments with short global transaction timeouts (JTA timeout) you might find that the Scheduler is
unable to process the default number of later To Do Items (500) or non-later To Do Items (5000) in a single
transaction. That is why on the latest 5.7 build (and newer versions) we have made available two JVM properties to
modify the number of To Do Items processed per transaction. The properties are the following:
-Dfuego.ebjengine.todoqueue.maxitems.nonlater=1000
-Dfuego.ebjengine.todoqueue.maxitems.later=100
The values shown above are the recommended settings to be used when the problem has been found on the
environment. The common indicators for these types of problems are:
Non-later items timeout problem: You will find a SQLException caused by a transaction rolled back. The
stack trace will contain invocations to these methods:
o fuego.ejbengine.ejb.EngineStartupBean.recoverLostItems()
o fuego.ejbengine.servlet.SchedulerServlet$RecoverLostItemsTask.runImpl()
Later items timeout problem: You will find a SQLException caused by a transaction rolled back. The stack
trace will contain invocations to these methods:
o
fuego.ejbengine.ejb.EngineStartupBean.sendLaterItems()
o
fuego.ejbengine.servlet.SchedulerServlet$SendItemsTask.runImpl()
You should note that the syntax of the method might vary depending on the version. Therefore, when looking at the
logs you should look for key words like: "recoverLostItems", "sendLaterItems", "Scheduler", etc
10. Most commonly used debugging flags

In this section youll find some of the most commonly used debugging flags, their purpose, and instructions on how
to apply them.
10.1. PAPI News Debug

If instances in the WorkSpace are not being refreshed appropriately or if there is some other synchronization
problem between the Engine and the WorkSpace, then you should enable the PAPI News Debug property. This
property is a JVM system property that must be specified in the Java Optional Arguments of the JVM running the
Engine and in the JVM running the WorkSpace. Its syntax is show below:
28
-Dfuego.papi.news.debug=true
This property is available in all versions.
For more information on this property, please, refer to the Issue # 7729408 (in https://bug.oraclecorp.com/) or
contact Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
10.2. Directory Polling Debug

If participants and their entitlements are not being synchronized appropriately or if there is some other
synchronization problem between the Directory and the Engine, then you should enable the Directory Polling
Debug property. This property is a JVM system property that must be specified in the Java Optional Arguments of
the JVM running the Engine and in the JVM running the WorkSpace. Its syntax is show below:
-Dfuego.fdi.polling.debug=true
Note that this property is only available in version 6.0 and 10.3.
10.3. Hybrid LDAP Debug

If the Directory has a Hybrid configuration (LDAP + DB) and there are delays in the synchronization of the
Directory, then you should enable the Hybrid LDAP Debug property. This property is a JVM system property that
must be specified in the Java Optional Arguments of the JVM running the Engine and in the JVM running the
WorkSpace. Its syntax is show below:
-Dfuego.fdi.hybrid.ldap.debug=true
Note that this property is only available in version 10.3.
11. BPM Documentation On-line

For more information on BPM, please refer to the following sites:
Oracle BPM 5.7 documentation:

http://download.oracle.com/docs/cd/E13165_01/albsi/docs57/index.html
http://download-llnw.oracle.com/docs/cd/E13165_01/albsi/docs60/index.html
http://download.oracle.com/docs/cd/E13154_01/bpm/docs65/index.html
29
12. Contacting Oracle BPM Support

For any Oracle BPM product question or issue, please, contact Oracle BPM Support following the indications in this
page: http://www.oracle.com/support/contact.html
Finally, if you have any questions regarding this document feel free to email at: gustavo.cavallin@oracle.com
Last edition date: 10-23-09 15:00 (GMT-06 DST)
30

Oracle BPM Engine Troubleshooting: Purpose

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Oracle BPM Engine Troubleshooting: Purpose

Transféré par

Droits d'auteur :

Formats disponibles

Oracle BPM Engine Troubleshooting