Oracle BPM 10gR3 Clustering Configuration Tips

Oracle BPM 10.3 Enterprise

for WebLogic and
WebSphere Configuration
Tips

Author: Eduardo Chiocconi

Last Updated: April 14th, 2010

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

Table of Contents
Database and Transactions.....................................................................................................................4

Note on JDBC Drivers........................................................................................................................4

Data Sources..........................................................................................................................................4

Connection Pool Size..........................................................................................................................4

Timeouts .................................................................................................................................................5

Additional Properties .......................................................................................................................5

Commonly intensive queries ............................................................................................................6

Threading ....................................................................................................................................................7

Introduction............................................................................................................................................7

Automatic work threads .....................................................................................................................7

Threads for Global Automatics .....................................................................................................8

Messaging ...................................................................................................................................................9

Introduction............................................................................................................................................9

OBPM Engine Queue Tips................................................................................................................9

OBPM Engine Topic Tips.................................................................................................................9

Logging..................................................................................................................................................... 10

Introduction......................................................................................................................................... 10

Tips........................................................................................................................................................ 10

Notifications and Updates ................................................................................................................... 12

Introduction......................................................................................................................................... 12

JMX Configuration........................................................................................................................... 12

OBPM Engine Metadata Distributed Cache .................................................................................. 13

Introduction......................................................................................................................................... 13

Where is Apache Tribes configured in OBPM? .................................................................... 13

Important Note on Apache Tribes port selection............................................................... 14

IMPORTANT Best Practices ........................................................................................................ 14

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

PAPI-WS Tips ........................................................................................................................................ 15

Introduction......................................................................................................................................... 15

Session Pooling ................................................................................................................................. 15

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

Database and Transactions

The inherent nature of processes implemented in OBPM dictates that we

need to persist all steps when the instances are moving from one step to the next
one in the business process. When the OBPM Engine is deployed to a JEE container,
all the access to the Engine DB and the Directory Service DB is performed through
the defined Data Sources and completely managed by the JEE container transaction
manager. It is worth pointing out, that all transactions executed by the OBPM Engine
are distributed transactions using the XA protocol.

Note on JDBC Drivers

Oracle BPM has traditionally worked with the Oracle internal JDBC Drivers
(OEM from DataDirect JDBC Drivers). Recently (as of March 2010), Oracle BPM
certified the Oracle Thin JDBC Drivers for connecting to Oracle RAC instances when
deployed on WebLogic Server. When using the Oracle Thin JDBC Driver, the
recommendation is to use the one distributed with WLS. For all other
configurations, it is still recommended you use the Oracle Internal JDBC Drivers
distributed with the Oracle BPM distribution.

Data Sources

The most important aspect is the Connection Pool. Proper sizing is important
and there is a strong relationship with the amount of threads that can execute BPM
transactions. Additionally, it is important to properly set the transaction timeouts to
enforce proper execution.

Connection Pool Size

A general rule of thumb is that the connection pool size should not be too big
on each cluster node. We also need to consider that all cluster nodes will be
accessing a single Engine DB instance and as such if multiple cluster nodes exists
with large connection pools, a big stress will be generated on the Database side.
Some past performance analysis shows that a value between 30 and 50 is a good
number. Please check the “Threading” section for best practices and relationships
between Thread and DB Connections.

It is worth remarking the constraint and relationship between threads and

available connections to the Engine DB. For this, we need to ensure the Connection

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

Pool is properly dimensioned. This needs to be calculated as the sum of automatic

work + interactive work. The automatic work can be extrapolated from the MDB
settings (as outlined in the previous section – the suggested value of 20 should be
sufficient and recommended). The interactive work and the needed amount of
threads can be extrapolated from the activity generated by PAPI, PAPI-WS or PAPI-
Block opened sessions.

Then the Engine Database Data Source size should be estimated as follows:

Connection Pool Size = PAPI-WS/PAPI/PAPI-Block + MDB Threads + 1.

When configuring the Directory Service DB, it is important to consider that if

multiple BPM JEE applications (Engine, WorkSpace, PAPI-WS) are deployed to the
same container, this pool will be shared by all these applications and as such, it will
need to be configured in such a way that is provides connections without too much
contingency to all of them. In this situation, it is usually advisable to set the
Directory Service Data Source Connection Pool size to twice the value of the Engine
one.

General good rules of thumb for Connection Pool sizes on a per container
basis are:

Engine Data Source Connection Pool Size = 30

Directory Service Data Source Connection Pool Size = 60

This is taking into consideration a cluster of 3 to 4 nodes and assuming the

backend database can manage this much concurrency appropriately.

Timeouts

When threads start executing, they need a database connection. These XA

transactions are also rules by timeouts that control transactions do not run for a
longer time than they are supposed to.

DISTRIBUTED_LOCK_TIMEOUT >= XA Transaction Timeout >= WebLogic Server JTA

timeout

Additional Properties

Oracle BPM supports multi lingual capabilities. All schemas are prepared to
store double byte code to ensure data integrity regardless of the location and locale
where OBPM is used. When using the Oracle Internal JDBC Drivers (OEM from

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

DataDirect), everything is automatically prepared. However, when using the Oracle

Thin JDBC Driver, it is necessary to add the following property for proper data
conversions: "defaultNChar=true". The following picture depicts how this is done in
WLS:

Commonly intensive queries

When the OBPM Engine starts processing lot of load, the following 2 queries
will be shown as top in the list in any database analysis.

SELECT … FROM PPROCINSTANCE … FOR UPDATE

SELECT … FROM PTODITEMS … FOR UPDATE

All instance information is in the PPROCINSTANCE table and as such

contingency will take place here. One thing to observe is whether the existing and
default indexes are used and no full scans are performed mostly when needing to
implement queries executed from the OBPM WorkSpace or customer PAPI or PAPI-
WS clients filtering by project variables.

Usually the immediate action from a SELECT … FROM PTODOITEMS … FOR

UPDATE is a SELECT … FROM PPROCINSTANCE … FOR UPDATE. The automatic
dispatching mechanism, first needs to exercise a lock on the automatic task to
execute that contains context on the instance to be processed. After that another
lock is required on that PPROCINSTANCE table row to enforce that no other node in
the cluster attempts to process the same instance at the same time.

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

Threading

Introduction

One other important aspect of an optimal processing is the need to fine tune
the amount of threads that can process business process operations on the engine
side. There are 2 specific sets of threads that can be controlled and managed by an
OBPM Engine. The threads dedicated to automated work and the threads dedicated
to processing interactive requests.

Automatic work threads

The automated work is always dispatched through a MDB (Message Driven

Bean). This MDB can be configured to have a particular amount of threads dedicated
to processing requests coming into the MDB. For more information on how to
configure the MDB Thread Pool size, refer to the WLS official documentation
through this link: http://download-
llnw.oracle.com/docs/cd/E13222_01/wls/docs100/perform/mdbtuning.html#wp1
133034.

In WebLogic, the MDBs can be configured to use a parameter called

“dispatch-policy” that is contained in one of the XML descriptor files bundled in the
OBPM Engine EAR file.

In order to fine tune this parameter, the following file can be edited
$ENTERPRISE/j2ee/template/engine/default/ejb/META-INF/weblogic-ejb-jar.xml .
This is on the machine that runs the Oracle Process Administrator Web Application
and there the BPM JEE applications are assembled. After modifying this file, it will
ne necessary to re-assemble the OBPM Engine EAR file and redeploy to the JEE
container.

The editions should be the following to the weblogic-ejb-jar.xml (marked in

bold for convenience):
…

<weblogic-enterprise-bean>

<ejb-name>item-execution-$escape.escapeXML($engine)</ejb-name>
<message-driven-descriptor>

<destination-jndi-name>$escape.escapeXML($itemExecutionQueue)</destination-jndi-
name>

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

<ejb-reference-description>

<ejb-ref-name>ejb/EngineStartup</ejb-ref-name>

<jndi-name>local/engines/startup/$escape.escapeXML($engine)</jndi-name>

</ejb-reference-description>

<dispatch-policy>wm/albpmAutomaticWorkManager</dispatch-policy>
</weblogic-enterprise-bean>

Once that a Work Manager is specified in the Engine EAR descriptor file, it is
necessary to make sure the referenced Work Manager is created and that it is
assigned a Min and Max value. Otherwise, it will use up to 16 threads.

Threads for Global Automatics

When deploying projects that have Global Automatics on them, we can

specify how many concurrent threads would be executing the Global Automatic
activity implementation. By default, the configured amount of threads is 1, but this
number can be increased. For this, it is necessary to specify a a system property
with the name albpm.engine.j2ee.jmsworkersize=X, where X is the amount of
threads.

It may be also necessary to increase the amount of threads on the Work

Manager linked to process Global Automatics as shown in the Engine EAR xml
descriptors.

TOTAL # of THREADS = Amount of Global Automatic Activities * X

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

Messaging

Introduction

This section has as objective to describe general tips and best practices when
configuring the messaging subsystem needed by the Oracle BPM Engine to process
automated work exercised completely by the Oracle BPM Engine.

OBPM Engine Queue Tips

The OBPM Engine Queue is used to connect the dispatcher when the actual
thread executing the automatic activity implementation defined in the business
process. The Queue connects with the Oracle BPM Engine MDB as explained in the
previous section.

It is very important that the queue is configured to be NON-PERSISTENT.

This will improve performance and will avoid unnecessary redundancy in the work
that needs to be processed. The Oracle BPM Engine database has a table named
PTODOITEMS that host in the form of a queue (rows with timestamps) what is the
work that needs to be processes automatically. It is the realm of the OBPM Engine’s
responsibility to dispatch the work in the Oracle BPM Engine PTODOITEMS table
into the queue on the right moment and time as well as executing recovery tasks in
the event certain nodes in the cluster dies and messages get trapped in that instance
of the OBPM Engine Queue.

OBPM Engine Topic Tips

The OBPM Engine Topic is used by the Engine to broadcast updates on

instances that are between the Begin and End nodes. This subsystem is very
important as it is the communication bridge between the OBPM Engine and its
clients and how to keep instance information consistent to its latest state.

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

Logging

Introduction

Properly managing the information that is logged into the Oracle BPM Engine log
files and the accuracy of this data is very important.

The following are some general-purpose tips of how to tune the Oracle BPM Engine
logging subsystem.

Tips

• The Oracle BPM Engine logging can be configured through the Oracle BPM
Process Administrator on a per engine basis on the “Log” Tab for each
defined Oracle BPM Engine showing in the “Engines” list

• Debugging information should be used in Development, Test where it is

necessary to capture detailed information. Another situation is when support
is instructing to collect more data. Other than these situations, it is
recommended that the logs are set to a maximum “Info” severity to minimize
the overhead on the Engine to log a large amount of data into the Engine log
files.

• The size of the files should not be much bigger than the current default.
Appending information to large files is an expensive operation. It is
recommended to increase the amount of logs.

• When having more than one cluster node in the same machine, all the cluster
nodes will log information into the same log file with the possibility of losing
sensitive information due to concurrency by 2 Java processes storing
information in the same Oracle BPM Engine log file. To ensure proper state of
the information stored in the BPM Engine log files, it is recommended that
the cluster nodes are started with the following system property (-
Dfuego.server.log.file) that indicates each individual Java process running
the Oracle BPM Engine uses its own log file: -
Dfuego.server.log.file=/OracleBPM103/engine/logs/ClusterNode1.log
• In this case, Cluster Node #1, can specify a file with name “ClusterNode1.log”.
Additional Cluster Nodes running on the same machine could store the
Oracle BPM Engine log files in the same directory but with a different name
(i.e.: /OracleBPM103/engine/logs/ClusterNode2.log, etc).

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

Additionally, it would not be uncommon specially in development type

environments where software reusability maybe needed, it is possible to use the
following property: fuego.server.<engine_id>.log.file. In this case, we assume
there is more than one BPM Engine deployed on the same WLS instance
(possibly 2 Engine EARs on the same cluster for example). In this case, we would
like each engine to log in its own file. In this case, we would need to use this
property and specify in it the engine id (where it reads <engine_id>.

When the Engine needs to pick up the log file to dump information, it will use
the following order for resolving the log file location:

1. Will check if the system property fuego.server.<engine_id>.log.file is

available in the JVM that is executing the BPM Engine. If this property is
specified, it will use the file location assigned to this property.

2. Will check if the system property “fuego.server.log.file” is available in the

JVM that is executing the BPM Engine. If this property is specified, it will
use the file location assigned to this property.

3. If none of the previous system properties were specified, it will use the
file location specified in the Logging Tab of the Engine preferences
accessible from the Process Administrator.

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

Notifications and Updates

Introduction

When the OBPM Engine is up and running, it is very likely that it will need to
be notified when certain management operations take place such as project
deployment, project un-deployment or project deprecation. For this, a client
application running these management operations (like the Process Administrator -
but it could also be Ant Tasks) need to notify the OBPM Engine so that it can update
the information without needing to restart or recycle the OBPM Engine application
or the container where it runs.

JMX Configuration

When configuring an OBPM Engine in the Process Administrator that will be

deployed to WebLogic or WebSphere, it is important to properly configure the JMX
Engine Management settings. This can be done on the “Application Server” tab of the
OBPM Engine properties in the “JMX Engine Management Configuration” section.

The most import piece is the host and port as well as the needed Principal
and Credentials.

The host name is the host where one of container nodes (cluster node if you
are using a cluster configuration – managed server node in WLS and cluster node in
WAS) where the OBPM Engine EAR is deployed and up & running. It is important to
also specify the port where this container is running. This pair will uniquely identify
an OBPM Engine up and running. The principal and credential parameters are used
in case security is enabled.
It is important to note that we should not specify the WLS Admin Server or
WebSphere Admin Node. The host:port pair needs to be pointing to a node or
cluster node where the OBPM Engine is running. If for some reason, the node or
cluster node specified is not running, then the notification will be lost and it will be
necessary to restart the OBPM Engine to pick up the latest changes.

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

OBPM Engine Metadata Distributed Cache

Introduction

When the Oracle BPM Engine runs on a cluster configuration, it internally uses the
Apache Tribes framework to maintain and control cluster node behavior. Among the
things that this framework controls is the notion of a “Master Node”. This master
node will have certain specific administrative tasks that ONLY ONE node in the
cluster can execute. For example, which node in the cluster should manage project
undeployments and when a project is deployed, which is the node that initializes the
Global Automatic ToDo Items. These are just some of the engine behavior that are
managed and controlled through this framework. In previous releases of ALBPM,
JGroups was used.

Where is Apache Tribes configured in OBPM?

The Apache Tribes configuration is specified in the “Cluster” Tab in the Process
Administrator. In this Tab, the default settings are:

UDP(mcast_addr=224.8.8.8;mcast_port=45566)

This is the basic information used to allow the communication of OBPM Engine
cluster nodes through UDP. In addition, Apache Tribes uses TCP settings. As the
Ports used by the UDP and TCP protocols may generate some collision with existing
ones in the environment in which the OBPM Engine cluster nodes are deployed, it is
possible to change these settings by explicitly indicating the required ports. The
following would be the recommended entry settings when specific TCP values other
than the defaults are required. These are some possibilities:

UDP(mcast_addr=224.8.8.8;mcast_port=45566):TCP(ip_port=:5000)

In this case, it is assumed that the TCP/IP port to be used by Tribes is 5000 in all the
machines or hosts where there is a cluster node hosting the Oracle BPM Engine.

UDP(mcast_addr=224.8.8.8;mcast_port=45566):TCP(ip_port=host1:5000)
In this case, it is assumed that the TCP/IP port to be used by Tribes on host1 is 5000.
For all other cluster node machines hosting the Oracle BPM Engine, it will assume
the default port (4000)

UDP(mcast_addr=224.8.8.8;mcast_port=45566):TCP(ip_port=host1:5000;
ip_port=host2:6000)

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

In this case, it is assumed that the TCP/IP port to be used by Tribes on host1 is 5000
and on host2 it is 6000. For all other cluster node machines hosting the Oracle BPM
Engine, it will assume the default port (4000)

Additionally and specially on Solaris and HP-UX, it may be necessary to include the
usage of the “use_nio” setting. This value is needed for OBPM 10.3.0 and 10.3.1. For
OBPM 10.3.2, a newer version of the Apache Tribes library is being used and it is no
longer necessary to specify this parameter explicitly. To enable this setting, the
entry would look something like this:

UDP(mcast_addr=224.8.8.8;mcast_port=45566):TCP(use_nio=false)

Optionally, you can specify the timeout for cluster nodes to connect with each other.
The setting would be as follows:

UDP(mcast_addr=224.8.8.8;mcast_port=45566):TCP(timeout=3000)

Have in mind that the value assigned to the timeout parameter is specified in
milliseconds. The default value is 3000.

Bear in mind that if multiple of these parameters need to be enabled at the same
time, “;” is the separator.

Important Note on Apache Tribes port selection

For the TCP/IP port specified on the Engine setup, this will be the starting port
number. If for some reason, the initial port is taken, Apache Tribes will raise a
warning message and will move on to try with the following integer number until it
finds one is free. If after the first 100 ports, it was not able to get a free one, it will
report a failure and it will make the engine fail to start.

IMPORTANT Best Practices

If the physical machine has more than one network adapter, the hostname should be
the IP address rather than the name of the machine itself.

If setting multiple environments in the same network (for example Testing,

Development, QA and Production), it is mandatory that we explicitly specify a
different multicast port. The important thing is that no two environments have the
same value for the mcast_addr and mcast_port parameters.

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

PAPI-WS Tips
Introduction

This section will highlight some tips and considerations when using PAPI-WS
as a mechanism for client applications to talk to business processes deployed on an
OBPM Engine.

Session Pooling

The new PAPI-WS introduced in ALBPM 6.0 and also existing in OBPM 10gR3
moved from a stateful implementation to a stateless one. This feature also
introduced the notion of a dynamic and on demand session pool implementation
managed by the OBPM PAPI-WS Web Application.

The session pool can be tuned and configured by updating system properties
in the papiws.properties file. Some of these properties will have a direct impact on
how the PAPI Instance Cache is managed on the PAPI-WS Web Application context.
These aspects will be reviewed and highlighted on each properties description.

The following is the list of properties, their description and tips

recommendations.

Property Name Description Tips

fuego.papiws.pool.max Max number of cache Ideally, this parameter should
size entries for the pool. Each be set to the maximum
one of these cache entries number of different user ids
is associated in a one to that may attempt to connect
one relationship with the PAPI-WS. If you are
userid provided in the implementing a stress test
PAPI-WS requests. where you use the same user
id for multiple client threads,
then this property can be set
to 1.
fuego.papiws.pool.max Maximum number of This parameter is important
entrysize concurrent open sessions when you are creating
for a given userid. multiple concurrent sessions
with the same userid (typical
case in stress test situations).
For each one of the entries
specified in the
“fuego.papiws.pool.maxsize”

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

parameter, it is possible to
have a list of sessions
associated to this participant
id. This would ideally need to
be set to the max amount of
client concurrent threads
doing activity with the PAPI-
WS application.
fuego.papiws.pool.min Min number of sessions This parameter controls that
nodisposable per entry that are kept at least there will be one
alive after timeout. session one active all the time.
# May be useful to Not only this is advisable from
improve performance if a time to create a new session
there are many but also from a direct
concurrent users relationship with the lifecycle
connected with the same of the PAPI Instance Cache. If
participant. this value is changed from its
default 0 to 1, then once a
session has been created and
established, then the PAPI
cache will be filled for the
accessed processes. As there
will be at least one session
opened, it means that the
count of sessions will never
reach 0 and the PAPI Instance
cache will never be recycled
or cleaned which could save
significant time when
performing queries through
PAPI-WS.
fuego.papiws.pool.time Session timeout in This is the default time before
out minutes. idle sessions in the
# Related to the length of “fuego.papiws.pool.maxentrys
the interval between calls ize” are recycled and closed.
from the same user.
fuego.papiws.pool.wait Indicates whether session Generally speaking, this
acquisition should be parameter should be kept in
blocking or throw an false as it may block a client
exception when the cache connection in trying to
limit has been reached. perform an operation for a
long time. It is recommended
that the retry/timeout
mechanism is used.
fuego.papiws.pool.max Number of retries to This is the amount of retries

Oracle Corporation – Proprietary and Confidential

 Oracle BPM 10gR3 Clustering Configuration Tips

retries acquire a session from when it is not possible to

the pool. Only valid if acquire a session. This value
fuego.papiws.pool.wait is should not be too large as
set to false. large time may induce the
client to believe it is working
while still trying to acquire a
connection. Sometimes is
better to notify the client
sooner that there is lot of
contingency on the PAPI-WS
Web Application.
fuego.papiws.pool.retr Interval in milliseconds In the event that all the
ytimeout between retries. Only sessions (specified through
valid if fuego.
fuego.papiws.pool.wait is papiws.pool.maxentrysize)
set to false. are taken, this parameter
specifies what is the wait time
before attempting a acquire a
session again. This value
should not been set to a short
couple of seconds.

Oracle Corporation – Proprietary and Confidential