IBM Business Process Manager testing methodology,

Part 2: Environments and detailed testing

Jing Jing Wei (weijingj@cn.ibm.com)
Advisory Software Engineer
IBM

10 December 2014

Allen Chan (avchan@ca.ibm.com)

STSM, IBM Blueworks Live and IBM BPM
IBM
Jerome Boyer (boyerje@us.ibm.com)
Senior Technical Staff Member
IBM
Hong Yan Wang (whongyan@cn.ibm.com)
Staff Software Engineer
IBM
Peng Guo (guopeng@cn.ibm.com)
Staff Software Engineer, IBM BPM SVT
IBM
Hong Li Liu (lhongli@cn.ibm.com)
Software Engineer, IBM BPM SVT
IBM
This tutorial is part 2 of IBM Business Process Manager Testing Methodology series. It
describes details of various testing that should be included in IBM Business Process Manager
(BPM) project tests and specific information for environments.
View more content in this series

Introduction
This tutorial is the second part in the series IBM Business Process Manager testing methodology.
In Part 1 of this series, we discussed the general guidelines for IBM Business Process Manager
Copyright IBM Corporation 2014
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Trademarks
Page 1 of 14

developerWorks

ibm.com/developerWorks/

(BPM) project testing. This tutorial introduces detailed testing methods and the corresponding IBM
BPM environments to use for various testing.

Environments
Try the Workflow service
Create long-running, stateful workflows that orchestrate tasks and services with synchronous
or asynchronous event-driven interactions with the Workflow service from Bluemix. Try it for
free!

IBM BPM is a comprehensive business process management platform, providing full visibility and
insight to managing business processes. It provides tooling and a runtime environment for process
design, execution, monitoring and optimization, along with basic system integration support.
The components of IBM BPM provide a unified repository; tools for authors, administrators, and
users; and a runtime platform. You can configure IBM BPM to support various levels of complexity
and involvement with business process management. Figure 1 illustrates a typical IBM Business
Process Manager configuration.

Figure 1. A typical IBM BPM configuration

In Figure 1, there are four IBM BPM environments: Process Center (Development), Staging,
Test, and Production. As a good practice, set up different environments for different users and
for different uses. Of course, there might be variants. For example, you might not have a "Test"
environment due to resource limitations, but you can use the "Staging" environment for functional
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 2 of 14

ibm.com/developerWorks/

developerWorks

testing, nonfunctional testing, and regression testing. Or, you might set up more than one Test
environment, because you have different IBM BPM projects running in parallel.

Table 1. IBM BPM environments

ID

IBM BPM Environments

Expected Users

Expected Usage

Design time:
Development environment on a
Process Center

Developers

Central repository and unit testing

Run time:
Test environment on a Process
Server

Quality assurance team testers

Integration testing

Run time:
Staging environment on a Process
Server

User acceptance testers and

performance testers

User acceptance testing,

nonfunctional testing, and
regression testing

Run time:
Production environment on a
Process Server

Business users and other end

users

The runtime environment for all

end users

Detailed testing methods

IBM BPM project testing methods include many testing methods similar to other software testing.
However, IBM BPM has its own specific testing methods. Table 2 shows detailed testing methods
for IBM BPM projects.

Table 2. IBM BPM project testing methods

IBM BPM project testing methods

Detailed testing methods

Functional testing

Unit tests, integration tests, user acceptance tests, instance migration

tests, globalization tests, mobile and browser tests

Nonfunctional testing

Performance tests, high availability tests, disaster recovery tests,

stability tests, stress and endurance tests, security tests

Regression testing

Detailed testing methods depend on the fixes applied. Some of above

functional and nonfunctional testing methods are applicable.

Unit tests
In software, a unit is defined as the smallest building block of code that other code can be built
upon. In IBM BPM projects, the unit is the developer's test of individual components of a specific
process application prior to creating snapshots. For example, a unit could be one user interface, or
one generic system service.
Consider the following guidelines when performing unit testing in an IBM BPM project:
Developers perform unit tests in the IBM Process Designer, which is connected to the
Process Center environment. Developers can inspect the results with the debugger in the
following two ways:
The Process Inspector in Process Designer.
A playback in Process Designer.
Each unit should have a unit test (or a test harness), and each unit test should contain the
following paths:
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 3 of 14

developerWorks

ibm.com/developerWorks/

the good
the bad
the ungly
For the user interface (UI) test, it is sufficient to create a harness so you can test the UI
independent from other units.
For the generic system services test, developers can create automated tests that ensure the
expected results.
If possible, create mock services to test external endpoints.

Integration tests
Typically, after unit testing and playbacks, you perform integration tests. Integration tests focus not
just on individual units of work but also on the entire process, including systems that are external
to the IBM BPM project.
Consider the following guidelines when performing integration tests in an IBM BPM project:
Test the entire process application snapshot pathways using agreed-upon scenarios and data
sets (good, bad and ugly) to ensure all components of a process application snapshot.
Perform integration tests on the test environment on Process Server. Some functions, such
as scheduled undercover agents, can run only on Process Servers. Also, some process
application management and behaviors are different on Process Center and Process Server.
For example, the default snapshot of a process application on Process Center is the tip by
default, and on Process Server, you must set a specific snapshot to be the default. Therefore,
don't use Process Center to do the integration test. Make sure to choose a Process Server
environment.
Many processes include long-running process pathways and timers. For these types of
processes, special considerations must be made: use custom dates with variables for values
in timer settings to facilitate timer-escalated testing. Using custom dates allows the timers to
be adjusted on a sliding scale during testing so that a process that would normally take 80
days to test can be completed in 8 days. Divide all time values by 10 in integration tests.
Remember to include integration tests for the Process Portal and the end-user experience.
Decide whether you want integration tests to be completed manually or with an automated
tool.
For manual continuous integration testing, perform visual testing of user interfaces to ensure
that they meet requirements.
Consider at least the following three data sets as unit tests for integration testing:
Expected outcome of a specific process scenario, also known as the happy path.
Expected outcome of exception paths, also known as the unhappy paths.
Expected outcome for system failure cases, also known as the ugly paths.

User acceptance tests

User acceptance testing is the last phase of the software testing process. During user acceptance
tests, actual software users test the software to make sure it can handle required tasks in realworld scenarios, according to specifications.
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 4 of 14

ibm.com/developerWorks/

developerWorks

Consider the following guidelines when performing user acceptance tests in an IBM BPM project:
Perform user acceptance tests on the staging environment on Process Server. Functional
user acceptance testing occurs after a snapshot of a specific process application has been
deployed to a staging environment on Process Server.
Make sure that physical users test entire process application snapshot pathways using
agreed-upon scenarios and data sets from test scripts. Ensure that the following specific
business requirements have been met:
Subject matter experts (SMEs) have the accountability for the specific subject's quality.
Risk mitigation requirements have been met.
Replace physical users with an automated test tool if needed.
Learn to prioritize defects correctly.
Ask the project sponsor to review defect priorities for valuation.

Instance migration tests

When installing snapshots on a process server, consider how to handle the business process
definition instances that are running on the server from previous snapshots. You can migrate those
running instances to the new snapshot and consider how to handle orphan tokens, or you can
leave those running instances in old snapshot.
Consider the following guidelines when performing instance migration tests in an IBM BPM project:
Perform instance migration tests in the test and staging environments on Process Server.
Get a list of application changes from your application development tests.
In the test and staging environments on Process Server, make sure there is a good
representation of process instances in different states. Then you can perform an instance
migration and test for successful migrated instances.
For orphan tokens, verify that you have a valid token policy file according to business
requirements.
Make sure that you have a regression test plan for the application to be migrated.
If you choose to leave the instances in old snapshot, also make sure there is a good
representation of process instances in different states. Then you can ensure that all the
instances can be successfully completed in the old snapshot after new one is deployed and
made the default.

Performance tests
Performance tests determine how a system acts in terms of responsiveness and stability under
a particular workload. Performance tests also can investigate, measure, validate or verify other
quality attributes of the system, such as scalability, reliability and resource usage.
Usually before performance tests, you should tune your system to meet your performance
objective. Consider the following guidelines when tuning your IBM BPM system:
Apply a system-wide approach to performance tuning. Tuning encompasses all of the
following elements of the deployment topology:
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 5 of 14

developerWorks

ibm.com/developerWorks/

Physical hardware topology choices.

Operating system parameters.
The IBM BPM server, the IBM WebSphere Application Server, the database, and the
messaging engine settings.
Identify potential performance bottlenecks and tuning changes to mitigate. Be on the
lookout for early signs of a performance problem, for example, integrations with other
systems, scaling the Lightweight Directory Access Protocol (LDAP) repository, and Database
Management System capacity.
Identify issues with prerequisite dependencies and changes to mitigate, for example, network
compression to combat network latency.
Prioritize the process server network traffic, ensuring that appropriate rules are defined for
routers, load balancers, and firewalls.
Get the administrators for the database, the operating system, and IBM BPM on the same
page during load and performance testing. Work with the application designers to understand
the resource needs.

Before you start performance testing, consider the following lessons learned by our team:

Know your metrics.

Set goals.
Get current.
Give yourself plenty of time.
Simulate real life scenarios as best as possible.
You can age your database to simulate long-running environments.
Consider that performance may degrade as a database ages.
Track your results.

Consider the following systemic approach when performing performance testing in your IBM BPM
project:
1. Establish clear performance objectives that are representative of business requirements, for
example:
Refreshing the task list in Process Portal should take < 3 seconds using IE 9 with 100
milliseconds of network latency between the Process Portal and Process Server.
2. Establish a baseline after installation for both product level and process application
deployment, for example:
Refreshing the task list in Process Portal should take 10 seconds using IE 9 with 100
milliseconds of network latency between the Process Portal and Process Server.
3. Pick a reasonable set of initial parameter settings.
4. Tune the environment according to good practices.
5. Monitor running systems on an ongoing basis to guide future tuning.
6. For each new process application deployment, complete the previous steps 1 to 5.
7. For each new snapshot, complete the previous steps 4 and 5.
8. Isolate rogue applications by completing the previous steps 4 and 5.
In addition, consider the following guidelines for performance testing:
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 6 of 14

ibm.com/developerWorks/

developerWorks

Complete a performance test on the staging environment on Process Server that is a

near clone of the production environment, including using simulated production data and
production security configuration.
Ensure that the environment is warmed up before you obtain measurements.
Design workloads that are reliable, that producing repeatable results, and that are easy to run.
Understand that running performance tests manually is not practical, and automated testing
tools are necessary.
Remember the following tips for performance testing loads:
Use realistic think times between requests.
Maintain a consistent flow of work into the system. For example, don't place 20K input
requests on a Java Message Service queue and then start. Implement flow control to
place the input requests.
Maintain steady states in the databases: a consistent number of active and completed
tasks and instances.
After running the performance test, clean up the environment, especially queues and the
databases.
Check for errors, exceptions, and time out messages in logs and reports.

Stress tests
Stress test are normally used to understand the upper limits of capacity in a system. This kind
of test determines the system's robustness in terms of extreme load and helps application
administrators determine if the system will perform sufficiently if the current load goes well above
the expected maximum.
Consider the following guidelines when performing stress tests in an IBM BPM project:
Perform stress tests in the staging environment on Process Server.
Include the following measures in an IBM BPM project stress test:
Concurrent users
Instance loads
Task loads
Stress tests aim to find the capacity of the IBM BPM system, based on current hardware
and software configuration. With the test results, application administrators and system
administrators can determine whether to apply a better hardware or software configuration,
according to future needs.
Any above-peak testing can be done to help determine future needs for unexpected high-load
conditions. Look for bottlenecks and any of the following resources that become constrained:
the operating system, the database, the Java Virtual Machine, or the product.

Endurance tests
Endurance tests determine if the system can sustain the continuous expected load. During
endurance tests, memory utilization is monitored to detect potential leaks. Also important, but often
overlooked, is performance degradation, for example: ensuring that the throughput or response
times are as good as or better after a long period of sustained activity than at the beginning of the
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 7 of 14

developerWorks

ibm.com/developerWorks/

test. Endurance testing essentially involves applying a significant load to a system for an extended,
significant period of time. The goal is to discover how the system behaves under sustained use.
Consider the following guidelines when performing endurance tests in an IBM BPM project:
Perform endurance tests in the staging environment on Process Server.
For endurance and data-build-up testing, test how your system performs after a long time
frame of usage.
Change the following factors about the test to scale for an accelerated schedule so that the
test can model months of usage without running for months:
Think about how many processes and tasks will be created or closed in a week. Then
calculate how many that would be for a month and year. Then perform the same actions
in a consistent but shortened amount of time.
To avoid load-related failures, ensure that the items are completed in a similar manner
and that you do not exceed the peak load tests.
Look for database performance degradation due to data build-up, any space issues, and any
Java Virtual Machine memory problems.
Look at the file system and database growth to determine any needed maintenance tasks or
tuning to consider.
Look for growth in baseline memory usage that might lead to problems.
Endurance testing can help to identify areas that need to be considered for regular
maintenance.
Endurance testing can identify some application issues that will only occur after long time
frames.
Tracking the affect of a month, 2 months and 3 months of usage might help project for the
effect for longer time frames.
Check the impact of a new application on the environment by adding new applications and
system changes, which affects the overall picture.

High availability (HA) tests

High availability is a characteristic of a system. Availability is usually expressed as a percentage of
uptime in a given year.
Consider the following guidelines when performing high availability tests in an IBM BPM project:
Perform high availability tests in the staging environment on Process Server.
Consider the following 4 types of test cases:
HA failover testing, which simulates a server failure while executing a stress test.
Controlled shutdown of a server (for example, using the administrative console).
Immediate shutdown of cluster members (for example, terminating the related system
process).
Uncontrolled shutdown of a complete system hosting (for example, power lost).
Consider the following categories of test cases:
White-box test cases, which pause the system at a sensitive spot and initiate a failure
event.
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 8 of 14

ibm.com/developerWorks/

developerWorks

Black-box test cases, which initiate a failure event while executing a stress test.
Verify the following information:
Ensure that automatic recovery from HA events completes within established recovery
time objectives.
Ensure that transaction integrity is preserved.
Understand that IBM BPM customers typically focus their HA testing efforts on the following
types of tests:
Basic configuration testing.
Operational procedures required for fail-over.
Limited black-box testing.

Disaster recovery tests

Disaster recovery consists of policies and procedures that describe how to recover or continue
the technology infrastructure that is critical to an organization after a natural or human-induced
disaster.
Consider the following guidelines when performing disaster recovery tests in an IBM BPM project:
Perform disaster recovery tests in the staging environment on the Process Server.
Consider the following types of test cases:
Disaster recovery configuration testing, which simulates a site fail-over against an idle
system.
Disaster recovery function testing, which continues processing an instance after site failover.
Measurement of disaster recovery objectives, which determines how long it actually
takes to execute the steps. Practice and scripting help speed up this test.
Verify the following information:
Test whether the disaster recovery environment can successfully connect to the replica
and take over from the primary site.
Test whether the disaster recovery environment can continue processing any in-flight
work.
Test whether the disaster recovery operational procedures are expressed clearly, well
scripted, and function properly.
Confirm whether the disaster recovery team can execute recovery procedures within the
established recovery time objectives.

Security tests
Security tests should be performed according to your organization's general security principles.
Consider the following guidelines when performing security tests in an IBM BPM project:
Understand the differences in security setup in the different IBM BPM environments (test,
staging, and production).
Ensure that the security configuration on the staging environment is identical to or similar with
the production environment.
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 9 of 14

developerWorks

ibm.com/developerWorks/

Ensure that security testing involves all parts of the infrastructure, including the system
security, IBM WebSphere Application Server security, and IBM BPM security.
Test for application-specific security requirements, for example that only certain users are
allowed to modify certain tasks due to task assignments.
Perform functional and nonfunctional tests after all the base security configuration has been
done, like the following examples:
Disable unsecure HTTP access.
Configure third-party authentication to secure the IBM BPM infrastructure.
Establish a small set of functional tests to run again after security configuration is modified,
especially for regular changes like the following changes:
A password change for administrative users.
A password change for database users.

Maintenance and regression tests

Regression testing is a type of software testing that seeks to uncover new software bugs, or
regressions, in existing functional and nonfunctional areas of a system after changes (such
as enhancements, patches or configuration changes) have been made to them. The intent of
regression testing is to ensure that changes have not introduced new faults.
Consider the following guidelines when performing regression testing on an IBM BPM project:
Perform maintenance and regression tests on all the four environments: the development
environment on Process Center, the test environment on Process Server, the staging
environment on Process Server, and the production environment on Process Server.
Understand the list of fixes before applying a fix pack or an interim fix.
Back up the system (profile, database) before applying fix packs and interim fixes.
Have a set of regression test cases ready for your applications and infrastructure so that you
can test the system after applying the fixes.

Test for changes in deployment environments

As previously described, you must do different types of testing in different environments.
When you deploy the application in deployment environments, pay attention to the following
issues:
Understand and document the differences in your deployment environments.
Note that different environments might have different configuration for security and different
peripheral systems.
Development environments typically have relaxed security.
Staging environments do not fully simulate production infrastructure.
Different environments have different LDAP group filters and scale.
Prepare to typically find problems in moves from test environments to staging and production
environments, like the following examples:
Testers forget to modify the role binding.
IBM Business Process Manager testing methodology, Part 2:
Environments and detailed testing

Page 10 of 14

ibm.com/developerWorks/

developerWorks

Testers forget to modify the WSDL URL and endpoint for a web service.
Testers forget to modify the schedule interval.
Record the detailed steps of changes when you move from test environments to staging and
production environments.
Have a set of regression test cases to verify that suitable changes are applied to the IBM
BPM application after it is deployed to different environments.

Conclusion
This tutorial, Part 2 of IBM Business Process Manager testing methodology series, described IBM
BPM environments and detailed testing methods.
Read the final Part 3 of the series to learn good practices to make IBM BPM applications testable
and advice about automation and testing tools.

Acknowledgements
The authors would like to thank Dawn Akuhanna, David Booz, Todd R Deen, Matthew S
Luczkowiak, Pascal Wagner, Dan Eykholt, Weiming Gu, Chris Richardson and Ekkehard Voesch
for their reviews and contributions to this tutorial.

IBM Business Process Manager testing methodology, Part 2:

Environments and detailed testing

Page 11 of 14

developerWorks

ibm.com/developerWorks/

Resources
IBM Business Process Manager documentation on IBM Knowledge Center
developerWorks Business process management zone
WebSphere Application Server V7 advanced security hardening, Part 1: Overview and
approach to security hardening
WebSphere Application Server V7 advanced security hardening, Part 2: Advanced security
considerations
IBM Fix Central
Technote about collecting troubleshooting data for IBM Business Process Manager products
Technote about the new IBM Support Assistant data collector in IBM BPM V8.5
IBM Support Portal for IBM Business Process Manager Advanced
Send data to IBM Services when no PMR is open: use the Enhanced Customer Data
Repository (ECuRep)
IBM Software Support Handbook
IBM Business Process Manager documentation on IBM Knowledge Center
Application Integration Middleware Support Blog in the developerWorks community
IBM Business Process Manager good practices wiki in the developerWorks community

IBM Business Process Manager testing methodology, Part 2:

Environments and detailed testing

Page 12 of 14

ibm.com/developerWorks/

developerWorks

About the authors

Jing Jing Wei
Jing Jing Wei leads the IBM BPM end-to-end SVT team, working on customer
scenario in-house testing. She acts as an IBM BPM SVT technical leader in the
IBM BPM lifecycle, security, integration and web services areas. Previously, she
worked with the IBM BPM and Monitor SWAT team, supporting customers and
business partners across the world with IBM BPM architect reviews, IBM BPM project
development, and problem determination.

Allen Chan
Allen Chan is a Senior Technical Staff Member of IBM Smarter Process, and is
the currently the Lead Architect for IBM Blueworks Live and IBM BPM installation,
Configuration and Migration. Previously, he was the IBM BPM SWAT Technical
Lead where he led a team of experts to help ensure customer success in key IBM
BPM deployment and rollout scenarios. He is passionate about ensuring customers'
successful use of IBM products.

Jerome Boyer
Jerome Boyer is an IBM expert on Enterprise Business Rule Management Systems
in IBM BPM, SOA and Complex Event Processing deployments. As a Senior
Technical Staff Member, Jerome is the lead IBM ODM and IBM BPM solution architect
in IBM Software Services for WebSphere (ISSW). Jerome is the author of "Agile
Business Rule Development", published by Springer in 2011.

Hong Yan Wang

Hong Yan Wang is a system verification tester for IBM Business Process Manager at
the IBM China Software Development Lab. She has more than 10 years experience
in software development and testing. She mainly focuses on end-to-end testing and
HADR testing.

IBM Business Process Manager testing methodology, Part 2:

Environments and detailed testing

Page 13 of 14

developerWorks

ibm.com/developerWorks/

Peng Guo
Peng Guo worked in WebSphere Adapters team for 3 years, and then moved to the
IBM BPM SVT team, focusing on lifecycle testing. He was responsible for automation
implementation of the SVT scenario for several releases. He is currently working on
IBM BPM end-to-end testing and customer scenario in-house testing.

Hong Li Liu
Hong Li Liu works in the IBM BPM QA & Automation team in IBM WebSphere. In
her current role as a software engineer, she tests the IBM Business Process Manager
product. She writes automation code to make her testing more efficient, and she has
solid Java, shell, and testing experience.
Copyright IBM Corporation 2014
(www.ibm.com/legal/copytrade.shtml)
Trademarks
(www.ibm.com/developerworks/ibm/trademarks/)

IBM Business Process Manager testing methodology, Part 2:

Environments and detailed testing

Page 14 of 14