Vous êtes sur la page 1sur 9

2014 40th Euromicro Conference on Software Engineering and Advanced Applications

Towards Automated Process and Workflow Management: A Feasibility Study on

Tool-Supported and Automated Engineering Process Modeling Approaches

Dietmar Winkler Michaela Schnbauer Stefan Biffl

Vienna University of Technology, Institute of Software Technology & CDL-Flex*
Favoritenstrasse 9-11/188, A-1040 Vienna, Austria

Abstract [Context] Engineering workflows and processes typi- neering or manufacturing system development where sev-
cally include a wide range of organization specific best-practices eral disciplines have to collaborate, process management is a
required for successful project execution. In contrast to business key challenge to support successful engineering projects
administration processes, engineering processes are highly specific [11][23]. Various stakeholders, coming from different discip-
and can include various heterogeneous engineering tools to be lines, use different and highly specific tools and data models
integrated in an overall process and project environment. that are the foundation for collaboration and data exchange.
[Challenge] High complexity and volatility of engineering However, observations in industrial settings, e.g., at a large-
processes require specific solutions to define, model, implement, scale hydro power plant systems development and integra-
and evaluate individual engineering process steps. In this paper we
tion organization, highlighted additional challenges to make
focus on the first step of process management, i.e., the process
definition phase. Based on the well-established Business Process
processes work; engineering processes must overcome limi-
Management Notation (BPMN) various tools have been developed tations caused by various specific tools that are loosely con-
that aim at supporting process modeling and automation. The key nected and data models that are incompatible to each other
question is to what extent different tools support process automa- [13]. However, engineering processes and workflows have to
tion in a defined context, i.e., in heterogeneous engineering envi- support this heterogeneity of tools and data models in engi-
ronments. [Method] Following a common tool evaluation ap- neering projects.
proach, we developed an evaluation framework and conducted a
feasibility study to investigate process modeling and automation
capability of tools based on defined requirements and scenarios.
[Results] The developed tool evaluation process was found useful
for analyzing candidate process modeling tools. In addition, results
of the evaluation process highlighted a set of specific tools that can Figure 1: Sequential Engineering Process in Automation Systems
support modeling and process automation. [Conclusions] The Engineering Projects according to [23].
evaluation framework can enable benchmarking for process mod-
eling tools and support (a) process engineers in selecting most Figure 1 presents a sequential engineering process illu-
valuable tools and (b) tool vendors to better understand complex strated by the project flow that should manage various
process needs with respect to improving and/or extending current stakeholders and different tools and data models within a
solutions. heterogeneous engineering environment [23]. A strict separa-
tion of process steps might be useful for project management
Keywords software management, process configuration, and control, e.g., by following a plan-driven approach [3].
process automation, workflow modeling, tool evaluation, However, in industry practice individual engineering discip-
feasibility study, BPMN lines have to work concurrently because of time, budget, and
project constraints. Thus, there is strong need to synchronize
I. INTRODUCTION individual (and heterogeneous) data models to enable a
Business processes and workflows represent the back- sound foundation for technical development. As engineers
bone of successful organizations. A variety of tools exist for work in parallel and distributed, changes in engineering
correct implementation of those processes and workflows. In artifacts, e.g., in electrical, mechanical, and software plan,
ISO standards [10] several processes support organizations are success-critical issues for project managers and engineers
in their business, e.g., management processes, business [23], which have to be supported by an appropriate process
processes with focus on customer product development, approach.
customer relationship management, supply chain manage- The automation service bus (ASB) aims at bridging the
ment, and supporting processes such as quality management gap between various disciplines by providing a middleware
[10]. However, in software and systems development organi- platform that is capable of integrating heterogeneous tools
zations engineering processes are success-critical processes and data models in a flexible way to support engineering
within the organization [5] for product life-cycle support [9]. projects in complex and distributed environments [1]. To
Systems engineering processes, especially in automation address change management needs for engineers and project
systems engineering (ASE) e.g., hydro power plant engi- managers, there is the need to implement process rules and
CDL-Flex: Christian Doppler Laboratory for Software Engineering guidelines within the communication platform. Initially, the
Integration for Flexible Automation Systems. http://cdl.ifs.tuwien.ac.at. ASB approach used hard-coded process rules, derived from

978-1-4799-5795-8/14 $31.00 2014 IEEE 102

DOI 10.1109/SEAA.2014.60
discussions with industry partners, to enable collaboration els, e.g., Activiti [18]. The process engine is able to detect
and data exchange between different tools and data models. events and to process them according to guidelines and rules
However, different organizations include different re- defined in business process models. For example, if an initial
quirements in terms of applied tools and data models, and event in a process models occurs, the process engine starts a
involved stakeholders. Thus, there is a strong need to support new process instance and executes this process based on its
flexible process management to address (a) system integra- model [22]. Figure 2 presents an example of a model of
tion needs (e.g., enabling an efficient and effective collabora- business process management systems architecture.
tion), (b) individual requirements of engineers (e.g., using
well-known and established tools and data models even in a
heterogeneous and distributed environment), and (c) project
managers (e.g., enabling efficient and effective project moni-
toring and control in such ASE environments).
In this paper we present the concept of an automation
supported process and workflow approach that enables mod-
eling, implementation and evaluation of engineering
workflows and processes in the ASE domain (see Figure 3).
Further we focus on the first step of the workflow modeling
approach, i.e., how to derive process requirements, how to
model these processes and how to prepare processes for Figure 2: Model of a Business Process Management
implementation, verification, and validation. Because the System Architecture [22].
Business Process Modeling Notation (BPMN) [18][20] has
In Automation Systems Engineering (ASE), process
been identified as a proven and well-established approach for
models include additional challenges, especially if heteroge-
process and workflow modeling, the goal of the first step in
neous tools and data models and a variety of stakeholders are
this research direction focuses on a systematic evaluation of
involved, e.g., mechanical, electrical, and software engineers
tools that are (a) capable of modeling engineering
as well as project and quality managers that are responsible
workflows, (b) support a technical implementation of these
for the overall engineering project. Thus, we see process
workflows, and (c) provide mechanisms for simulation. To
management including tool support in heterogeneous engi-
support this goal, we derived three main research questions:
neering environments a crucial challenge in ASE projects.
RQ1: What are the requirements for tools to automate
Figure 3 presents a basic workflow management approach
workflow definition steps in ASE projects? RQ2: How can
with the ASB including four main parts: (1) process and
BPMN tools be evaluated effectively and efficiently? RQ3:
workflow definition and modeling, e.g., based on customer
To which extent do selected tools support the definition of
input; (2) process implementation that should be supported
executable processes and workflows? To identify the most
by the process and workflow engine; (3) process evaluation
promising tools that support workflow definition and auto-
with focus on verification based on captured process events
maton we developed an evaluation framework and conducted
and data (3a) and process validation (3b).
a tool feasibility study for selected candidate BPMN tools.
The remainder of this paper is structured as follows: Sec-
tion 2 presents related work on business process modeling,
BPM tool support and introduces to the tool evaluation
process. Section 3 presents the research issues. We present
the tool evaluation process approach in Section 4 and the
results in Section 5. Section 6 summarizes validity consid-
Tech. Interop.
erations. Finally, Section 7 concludes and identifies future
work with focus on the next step towards automation sup-
ported process and workflow modeling.
This section summarizes related work on business
process modeling in the context of automation ASE projects
(Section II.A), BPMN and tool support (Section II.B), and Figure 3: Process and Workflow Management Cycle
tool evaluation processes (Section II.C). in ASE projects based on ASB [21].
A. Business Process Modeling in the ASE context In this paper we focus on the investigation of tools that
Modeling tools have to support specific modeling lan- support process modeling (see Figure 3, step 1) as founda-
guages that can be used for creating graphical representations tion for automation supported transition, verification and
of business processes according to a defined standard, e.g., validation of process entities, relationships, and actions to the
using BPMN [20]. Tools that allow the execution of business process engine. However, it remains unclear which tools
processes include a so-called process engine (or workflow support basic requirements of ASE projects to which extent.
engine) that works with the graphical business process mod-

B. BPMN and Tool Support quired to focus on specific requirements in the ASE context.
A key challenge of process engineers is to provide a Thus, we adapt the basic process approach, proposed by
process and workflow description that (a) is easy to under- Poston et al. [17], to meet individual criteria in context of
stand for customers and engineers and (b) include the capa- our feasibility study.
bility to transfer process designs to a running process or
workflow engine. The Unified Modeling Language (UML) is
a well-established approach for modeling the static structure Based on the need for tool-supported definition processes
and the dynamic behavior of a system [1]. For instance, and workflows in BPMN, aligned with the process automa-
activity diagrams aim at describing processes and workflows tion framework, we derived three main research issues:
systematically. However, UML primary focuses on software
engineers rather than on business people. In addition there is RQ1. What are the requirements for tools to automate
no specification for mapping UML Activity Diagrams to a workflow definition steps in ASE projects? Based on a series
business process execution language [14]. of workshops with industry partners, developers, and expe-
BPMN is a well-established approach for business riences collected during process development for ASE
process modeling [20] representing business processes sys- projects in the hydro power plant domain, we elicited and
tematically in the form of business process diagrams. The analyzed the most important requirements as input for tool
graphical representation of BPMN enables stakeholders, who evaluation.
might not be familiar with formal process modeling ap-
proaches, to design business processes as well as engineering RQ2. How can BPMN tools be evaluated effectively and
processes more effective and efficient [8][15]. Because of efficiently? Based on Poston et al. [17], we developed a
the formal definition of BPMN it aims at bridging the gap context-specific evaluation framework that focuses a syste-
between process definition and implementation. matic analysis of candidate BPMN tools in the context of the
However, tools are needed to support process engineers ASE project based on the ASB platform.
in systematically designing and modeling processes and
workflows. In literature, several tool studies have been re- RQ3. To which extent do the identified tools support the
ported. For instance, Ramakrishnan1 provides results of an definition of executable processes and workflows? We ap-
automation-supported tool study which is based on vendor plied the framework to selected and most promising BPMN
information and defined selection criteria. It remains open modeling tools, i.e., tools that aim at supporting process and
how the tools perform in a practical environment. Pawar workflow modeling requirements (see Section V.C) towards
investigated several BPMN modeling tools with respect to business process automation, and analyzed the tools accord-
testability and interoperability. However, it remains open ing to the evaluation framework.
how BPMN results can be used for automation-supported
workflow implementation [16]. Nevertheless, existing tools IV. TOOL EVALUATION PROCESS
studies can be used as a starting point for evaluating selected This section summarizes the adapted tool evaluation
tools in context of ASE projects in the ASB context. process based on the basic process proposed by Poston el al.
[17] and applied by Winkler et al. [24] (see Figure 5). Adap-
C. Tool Evaluation Process tations of the initial tool evaluation process include (a) de-
A key question is how to evaluate tools in an effective velopment of scenarios relevant for the ASE domain
and efficient way. Poston et al. proposed a tool evaluation (process step 4a), (b) development of the evaluation frame-
process [17] which has been applied for evaluating software work (process step 4b), and (c) evaluation of the selected
testing tools. Figure 4 illustrates four steps of the tool evalua- candidate tools (step 4c). Note that 4c is similar to the origi-
tion process: (1) Identification of requirements needed by the nal process step 4 proposed by Poston et al. [17]. We believe
user with focus on expected tool features; (2) Definition and that these additional steps (4a and 4b) can help to (a) better
prioritization of selection criteria to focus on the most prom- evaluate tools in a given domain or context and (b) provide a
ising tool candidates; (3) Search for candidate tools to iden- systematic and repeatable approach towards benchmarking
tify available solutions for evaluation, and (4) Tool evalua- of tools.
tion that focuses on the execution of the evaluation process
according to identified selection and evaluation criteria.

Figure 4: Tool evaluation process based on Poston et al. [17]. Figure 5: Adapted Tool Evaluation Process.
This general approach can represent the foundation for Figure 5 presents the adapted tool evaluation process
evaluating tools in different areas, e.g., evaluating BPMN based on [17] and [24], including six steps:
tools in ASE contexts. However some adaptations are re-
Step 1: Identification of Requirements. Based on existing
Tool study reported by Ramakrishnan available at
work (see Section II.C) and a sequence of workshops ex-
cannot-ignore/; accessed in March 2014. ecuted by a team of process experts and ASB developers, we

derived a set of requirements as input for BPMN tool evalua- calculated per requirement and requirement category. The
tion. Note that we followed the EasyWinWin process ap- results are normalized by dividing them by the maximum
proach [4] for the workshops. In addition we categorized total score and can be interpreted as percentage values that
collected requirements and prioritized them according to indicate how well each tool scored. Calculated and norma-
individual benefits in context of process and workflow mod- lized total scores can be compared and aggregated to achieve
eling, simulation, and the capability for automation- a comprehensible ranking of the evaluated tools. See Section
supported implementation, verification, and validation. Prior- V for the evaluation results of the feasibility study.
ities follow the estimated importance according to process
design needs and include four levels: (a) Critical require-

Tool C
ments (weight 10) that are mandatory for every tool; (b)


Important requirements (weight 7 = high); (c) less important
requirements (weight 3.5 = medium); and (d) nice-to-have
features (weight 1 = low) that are considered as optional.

Step 2: Define and prioritize selection criteria. To effi-

ciently focus on the most relevant tools to be investigated,
we defined a set of success-critical criteria for every re- Figure 6: Tool evaluation Framework based on [24].
quirements category. Note that these criteria are based on Step 4c. Tool evaluation. By applying test scenarios, e.g.,
identified and prioritized requirements, i.e., critical require- the change management process approach, and the evalua-
ments (weight 10). See tion framework, we evaluated and analyzed every selected
for details in the context of our feasibility study. candidate tool. Note that the scenarios (from user perspec-
tive) are typically based on user interaction with the tool.
Step 3: Identification of available tools. Based on list of
current BPMN tool vendors provided by the OMG Group2, V. BPMN TOOL EVALUATION RESULTS
sources provided in Section II.B and a quick web search we
We applied the evaluation process by conducting an ini-
derived a set of 76 candidate tools that are promising candi-
tial tool study (i.e., feasibility study) for selected BPMN
dates for evaluation. See Schnbauer et al. [19] for a com-
tools with focus on the capability of tools in the ASE and
plete list of candidate tools. However, evaluating 76 tools
ASB context. The main goal of this study was to (a) elicit
takes a considerable effort. Thus, we applied the selection
requirements related to BPMN modeling, simulation, and the
criteria, defined in Step 2 of the adapted evaluation process
capability towards automation-supported workflow imple-
to focus on the most promising candidate tools.
mentation, verification and validation, (b) develop an evalua-
tion framework as initial benchmarking approach for users
Step 4a. Scenario development. According to our pre-
(e.g., to support tool selection) and tool vendors (e.g., to
vious work in the area of tool evaluation [24] common and
improve and evolve existing tools), and (c) the identification
typical scenarios derived from individual application do-
of most promising candidate tools for further usage in the
mains help analyzing candidate tools effectively and effi-
business process modeling life-cycle in the ASB context.
ciently [24]. Scenarios include typical activities from user
perspective to better identify benefits and weaknesses of the A. Tool Study and Arrangements
tool under investigation. In context of this work, one key Basically the feasibility study followed the defined
process derived from our industry partner has been selected process steps, described in Section IV. However, different
for evaluation, i.e., a change management workflow. Note groups of stakeholders are involved in the individual steps of
that scenario brainstorming can be supported by established the study, e.g., in the requirements elicitation and prioritiza-
methods and tools, e.g., EasyWinWin [4]. tion step a set of workshops with different stakeholders, i.e.,
customers, developers, and quality and process experts, has
Step 4b. Evaluation framework definition. The evaluation been executed; tool evaluation framework design and tool
framework typically consists of a table of categorized re- evaluation have been done by one master student supported
quirements and corresponding priorities in context of the by a supervisor and reviewed by two process experts from
scenario and/or application domain, a set of individual tools industry and research partners.
to be evaluated, assessment results, and the overall score. Requirements elicitation and prioritization. During
Figure 6 illustrates an example for an evaluation matrix. workshops with industry partners a set of process require-
Individual assessment results (per requirement) have been ments, mainly with focus on the process workflow, were
derived by expert estimation regarding the degree of fulfill- defined. We applied the EasyWinWin process approach [4]
ment, i.e., 0 if the tool does not support the requirement and to elicit, classify, and rate individual requirements. These
1 if the tool fully supports the requirement. However, de- workshops have been moderated by a quality manager and
pending on the requirement different values in the range of process expert, i.e., two of the authors. Result was a basic set
0% to 100% might apply. Based on weighted requirement of requirements (from customer perspective) and the evalua-
(i.e., assessment results multiplied by priority) a Score (S) is tion scenario to be implemented. The second part of the
requirements elicitation step includes the identification of
Object Management Group (OMG): http://www.bpmn.org

additional requirements and process constraints from devel- cited 5 critical requirements for pre-selection. One of the
opment perspective. Note that one goal is to support process most important criteria was the availability of tools for test-
modeling, simulation, and automation-supported implemen- ing purposes. Thus, we had to focus on available open source
tation of the workflow (see Figure 3). The second series of tools and on commercial tools, where test versions were
workshops involved 3 researcher and 5 developers to find available for evaluation. Table 1 summarizes the number of
additional requirements for process implementation as input requirements per category and Table 2 presents selected
for evaluating candidate tools. We used results of the indus- individual requirements that are used for pre-selecting re-
try partner workshops as input for the workshops. lated tools. However, we are aware that alternative tools
Tool evaluation design and evaluation execution. Based might fulfill defined requirements on a high level; neverthe-
on the workshop results, the evaluation framework has been less, they have been excluded in this study because of avail-
defined and the tool study was executed by a master student, ability reasons. Note that we did not include BPMS Activiti
who has experience in process modeling, supported by a Support in the list of critical requirements because there are
supervisor. The results of the study were reviewed by two alternative tools usable for workflow implementation pur-
process experts from industry and research partners. poses. However, BPMS Activiti has been initially selected
by our developers as candidate tool for the next step of the
B. Requirements and Tool Properties (TP) process automation approach and, thus, was included in the
Expected tool requirements and properties were collected evaluation section. Nevertheless, Activiti does not support
in two series of workshops involving different groups of simulation and was, therefore, excluded from evaluation.
stakeholders, (a) from engineering process perspective and
Table 2: Critical Tool Selection Criteria (weight 10).
(b) from development perspective. Overall, 39 requirements
Requirement Category Individual Critical Requirement
in 6 requirements categories were identified. Table 1 sum-
General Requirements Availability for Testing Purpose
marizes the identified requirements categories and the share BPMN 2.0 Support Support of the BPMN 2.0 standard
of requirements. The basic requirements classes focus on Export Functionality and Inte- Export model in xml that can enable
general requirements (10%) (e.g., availability or platform roperability interoperability with other tools.
support), tool usability (28%), and the support of BPMN 2.0 Usability Tool installation performance
(3%)). Because our main goals is to investigate the applica- Simulation Support Offering UI for specifying simulation
bility of candidate tools with respect to data exchange and parameters and executing a simulation
the capability for automation supported implementation, we We applied selected critical requirements on all of the 76
introduced the requirements categories Export Functionality tools based on individual information provided by the tool
and Interoperability (23%) and BPMS Activiti support, a vendors. Missing information or information contrary to the
promising tool for automating processes (5%). BPMS refers defined critical selection criteria lead to an exclusion of the
to a Business Process Modeling System or Suite. Finally, tool from further evaluation. After this step in the selection
modeled processes have to be verified and validated; simula- process of candidate tools a test version of each selected tool
tion plays an important role in this context. Thus, we intro- was downloaded and installed. The final result of the tool
duced a requirement category for this purposes, i.e., Process selection process included eight tools that were considered
Simulation Capability (31%). See Schnbauer et al. for a for further evaluation (see Table 3; in alphabetical order).
detailed list of all requirements and categories [19].
Table 3: Selected Tools for Evaluation.
Table 1: Requirements & Requirement Category Overview. Tool Tool
Requirements No of Critical 1. Abacus3 5. iGrafx Process4
Requirement Category No % Requirements 2. AccuProcess Modeler5 6. inubit BPM Suite6
General Requirements 4 10% 1 3. Agilian7 7. Logizian8
BPMN 2.0 Support 1 3% 1
4. Bonita BPM Suite9 8. Signavio Process Editor10
Export Functionality and Interoperability 9 23% 1
Usability 11 28% 1
BPMS Activiti Support 2 5% 0 D. Scenario Brainstorming
Process Simulation Capability 12 31% 1 Scenarios, i.e., typical use cases in a defined application
Total 39 100% 5 (13%) domain or discipline, can be used for an effective and effi-
cient evaluation of tools in a given context. Based on our
Collected requirements include different prioritities ac- experiences with scenarios in tool evaluations [24], scenarios
cording to the application area and user needs. As introduced fit also well to the purposes of evaluating BPMN tools and
in Step 1 of Section IV we applied four priority levels (i.e., their applications. Tool requirements and application scena-
critical, important, less important, and nice-to-have features) rios were elicited during a series of workshops with our
and used critical requirements for a pre-selection of candi-
date tools. Out of 39 requirements, 5 were identified as crit- 3
ical (13%). See Table 1 and [19] for details. Abacus: http://www.avolution.com.au/
iGrafx Process: http://www.igrafx.com
C. Identification and Selection of Candidate Tools AccuProcess Modeler: http://www.accuprocess.com
inubit BPM Suite: http://www.bosch-si.com
As described in Step 3 of Section IV we used several 7
Agilian: http://www.visual-paradigm.com/product/ag/
sources to collect individual tools and came up with a total Logizian: http://www.visual-paradigm.com/product/lz/
number of 76 candidate tools. In the previous step we eli- Bonita BPM Suite: http://www.bonitasoft.com/
Signavio Process Editor: http://www.signavio.com

industry partner, a large-scale systems integrator in the hydro son). Finally, human interaction tasks enable the decision
power plant domain (see Figure 1 for the basic engineering whether to accept or to reject the change (see subtask Hu-
process, implemented at our industry partner). man Interaction). These steps are executed sequentially for
However, in practice, engineers coming from different all available signals. The workflow concludes with a sum-
disciplines, work concurrently using a variety of different mary report of the synchronization process. See Schnbauer
tools and data models that need to be synchronized frequent- et al. [19] for a detailed description of the change manage-
ly to enable a sound technical foundation for system devel- ment process and related subtasks.
opment and a project that is less risky. This synchronization Based on this engineering workflow we focus on a set of
process requires a defined workflow that is able to handle three basic scenarios for evaluation related to the identified
changes in an efficient and effective way. See [23] for a requirements and categories:
detailed description and a pilot application of an initial ver- Definition of the workflow/process, i.e., the change
sion of the change management process, derived from our management process derived from our industry partner.
industry partner. After a pilot application we refined the Import and Export that focus on interoperability and
change management process (see Figure 7 for a process data exchange between similar tools, e.g., electrical, me-
overview). The synchronization process starts with the avail- chanical, and software planning tools.
ability of a new version, represented by a simple CSV table. Simulation Test Scenario that can enable verification
Query EDB refers to a central data base where the current and validation of the process, including the definition of
versions of the overall project are located, i.e., new docu- a simulation sequence of the workflow and an observa-
ment versions have to be compared to that baseline by the tion of the simulated behavior.
system. Depending on the comparison results, individual
actions have to be executed (see subtask Signal Compari-

Figure 7: Basic Use Case: Change Management Process Overview in BPMN 2.0 according to [19] and [23].

addresses enterprise architects, Logizian is designed for

E. Results of the Tool Evaluation
business analysts and therefore specialized on modeling,
The main goal of the tool evaluation was to identify the analyzing and improving business processes. Hence, they
strengths and weaknesses of selected tools with respect to also differ in some of their functionalities. Note that these
defined requirements and scenarios relevant for defining and functional differences are not in the scope of the tool study.
designing engineering and business processes and The main weakness of all evaluated tools can be found in
workflows. Based on the evaluation framework (Section IV) the field of interoperability. Only one half of the selected
every selected tool was evaluated according to requirements tools offer acceptable support for interaction with other tools.
and requirements categories. The rating schema follows a More than fifty percent of the tools offer over-average sup-
level of fulfillment on a scale von 0 to 100% or 0/1, depend- port for the actual BPMN 2.0 standard. The other tools like
ing on the scale to be applied. See Schnbauer et al. [19] for for example Bonita BPM Suite or AccuProcess Modeler
a detailed description of the different rating schemes, how support at least most of the important BPMN 2.0 elements.
they were applied to each requirement, and the complete set According the aspect of process validation by simulation
of evaluation results. more than two thirds of all selected tools could score with
Figure 8 presents the summarized results of selected re- quite comprehensive simulation features. The tool that re-
quirements based on the identified categories. Based on the ceived the lowest score in this requirement category was
results of the tool evaluation Logizian turned out to be the Abacus. It only offered simulation-similar features like cost
tool with the best overall support for the defined require- and performances analysis instead of the expected interface
ments. It offers comprehensive simulation support, several for specifying simulation parameters. Figure 9 summarizes
import and export functionalities, and a quite good usability. the aggregated results of all requirements categories and the
According to its design and functionalities Logizian is quite average score of all tools under investigation. The order of
similar to Agilian, another tool provided by Visual Para- the tools according to the overall results represent the rank-
digm. However, the user group is different: While Agilian ing (i.e., from left to right in Figure 9).

Figure 8: Evaluation of Selected Requirements of Related Categories.

Figure 9: Summarized Results based on Requirements Categories.

sults, i.e., requirements, priorities, and results, have been

VI. LIMITATIONS AND VALIDITY CONSIDERATIONS reviewed by experts from industry and research partners.
Every study has to deal with limitations and threats to va- Study participants. The requirements were elicited in
lidity. We identified a set of threats and addressed them: two series of workshops with industry partners and de-
Requirements and Scenarios have been developed in a velopers. The evaluation framework has been developed
set of workshops involving a set of different stakehold- by a master student supported by a supervisor. To over-
ers, i.e., industry people, quality and process experts, come a possible bias two process experts from industry
and developers. Thus, we believe that we covered a and research partners reviewed the evaluation frame-
wide range of stakeholders that addresses critical issues work and the results. This setting might not be repre-
properly based on a real world setting. sentative for an industrial setting. Nevertheless, this will
Tool evaluation process. The evaluation is based on a not be a strong limitation to evaluate the tool features
well-defined and established tool evaluation process in- with respect to BPMN requirements as the scenarios
troduced by Poston et al. [17] and already applied by have been derived from an industrial environment.
Winkler et al. [24] in a different application domain. BPMN Limitations. We selected the BPMN notation as a
Thus, we see the evaluation process as appropriate. well-established and proven approach for process and
Tool selection. In context of our initial feasibility study workflow modeling [18][20], being aware of the limita-
the selection of tools is based on the availability of exist- tions regarding notation, semantics, and process interac-
ing solutions, i.e., as open source tools and test versions tion of concurrent execution [6]. More recent approach-
of commercial products. We applied selection criteria to es, e.g., Subject-oriented Business Process Management
focus on the most promising tools that are available for (S-BPM) [7] or Adaptive Case Management (ACM)
evaluation. Thus, the list of evaluated tools is a snapshot [12] are promising candidates that aims at overcoming
in the given context and cannot be considered as com- BPMN limitations.
plete. Extending the list of evaluated tools will remain
for future work.
Data collection. Some estimated evaluation results, e.g., Engineering and development processes are success-
usability concerns, can be seen as subjective. However, critical issues in software and systems engineering processes.
we referred to best practices to enable an objective as- However, additional challenges arise in a heterogeneous
sessment in academic environment. In addition, the re- environment where stakeholders coming from different dis-

ciplines have to collaborate and exchange data. In addition hinder effective data exchange. See Section V.E and [19] for
different workflows and processes are applicable that have to details on the evaluation process and the results.
be managed in an effective and efficient way to make
projects successful. In order to support flexible engineering Table 4: List of evaluated tools ranked by their calculated
and normalized total scores.
processes within the process management life cycle,
Rank Tool Score
processes have to be designed, implemented, verified, and 1 Logizian 89 %
validated according to current needs. Figure 3 presented a 2 Agilian 87 %
process and workflow management life-cycle in ASE 3 Signavio Process Editor 80 %
projects that can be supported by the ASB approach [2]. 4 Bonita BPM Suite 78 %
However, a first step towards automation supported 5 iGrafx Process 76 %
6 inubit BPM Suite 70 %
process and workflow management is the definition of 7 AccuProcess Modeler 62 %
processes and workflows capable for automation-supported 8 Abacus 37 %
implementation within a workflow environment. In a first
step of tool support for process and workflow management, Table 4 summarizes the final score of the tool evaluation
the main research question focused on the systematic inves- study of the 8 selected tools. Logizian received the highest
tigation of tools for process and workflow management that score, followed by Agilian, and the Signavio Process Editor.
(a) enable the effective and efficient definition of processes Following these results, Logizian seems to be the most
workflows, (b) provide mechanisms for import and export promising tool to support effective and efficient process
functionality to support tool interoperability, and (c) enable definition, date exchange and interoperability and simulation
the simulation of processes and workflows for verification based on the results of the tool study. During the evaluation
and validation purposes. we observed strengths and weaknesses related to all tools
RQ1. What are the requirements to support automation- which offer space for improvement and extensions.
supported workflow definition steps in ASE projects? In a set
of workshops and discussions with industry partners, process Future Work will include two main research directions, (a)
and quality management experts we identified an overall towards extending and improving the evaluation framework
number of 39 requirements in 6 requirements categories that including a larger set of tools that will be considered in the
are used for evaluating selected candidate tools (see Table 1) evaluation study; and (b) the next step of the process man-
and a set of 5 success-critical requirements for setting up the agement framework, i.e., the automation-supported imple-
tool evaluation. The workshop format (based on the Easy- mentation of process modeling outputs in the ASB environ-
WinWin approach [4]) was found helpful in addressing and ment (see Figure 3).
eliciting important requirement from different perspectives, Extension and improvement of the evaluation process.
i.e., from industry partner and developer perspective. This research was based on preliminary studies (see Section
RQ2. How can BPMN tools be evaluated effectively and II.C) and an additional quick search on existing tools. How-
efficiently? Based on the work by Poston et al. [17] and ever, a formal research protocol and a systematic mapping
according to [24] we developed and extended a tool evalua- study could extend the currently available tool sets and lead
tion process based on a defined set of scenarios. The evalua- to more comprehensive results. In a next step we will also
tion framework and the scenarios, derived from real world include commercial tools to enable a more complete view on
settings, represent a systematic and adjustable approach to BPMN tools. Additional scenarios and a refinement of the
focus on most relevant aspects from different perspectives. evaluation process will also lead to more comprehensive
According to the evaluation framework, the systematic tool results towards a benchmarking approach of BPMN model-
study enabled an efficient, effective, and traceable evaluation ing tools. We believe that this initial feasibility study can
of selected BPMN modeling tools. Scenarios helped to focus support tool vendors to improve and extend existing tools
on most relevant industry partner needs leading to traceable and users to select appropriate tools in their individual con-
evaluation results. Alternative contexts or needs can lead to texts.
additional scenarios which can be addressed within the Process and Workflow Management Cycle. Finally, and
framework. Further identifying critical selection criteria the most important future work aspect will focus on the au-
helped to reduce evaluation effort as less promising tools can tomation-supported process and workflow management life-
be excluded early. cycle, i.e., the implementation, verification and validation of
RQ3. To which extent do the identified tools support the designed processes and workflows. Results of this research
definition of executable processes and workflows? The third direction will support flexible process and workflow man-
research question focused on the results of the tool evalua- agement, can enable automation-supported implementation
tion. The results of the BPMN tool study showed that there in the ASB context and provide mechanism to automatically
are several tools available that offer support for the BPMN verify and validate designed and implemented workflows.
2.0 standard as well as other required features for supporting
interoperability and simulation. However, the results also ACKNOWLEDGMENT
showed needs for improvement: there are only few tools that This work was supported by the Christian Doppler For-
are fully compatible to each other (i.e., limitation of interope- schungsgesellschaft, the Federal Ministry of Economy, Fam-
rability). Furthermore, it seems that there is no uniform BPM ily and Youth, and the National Foundation for Research,
format that can be interpreted by all tools which can also Technology and Development - Austria.

REFERENCES on Semantic Integration of Engineering Knowledge, In: Proc. of
ETFA, Toulouse, France, 2011.
[1] Ambler S.: Elements of UML 2.0 Style, Cambridge Univ Press,
2005. [14] Peixoto D.C.C., Batista V.A., Atayde A.P., Borges E.P., Resende
R.F., Pdua C.I.P.S.: A Comparison of BPMN and UML 2.0
[2] Biffl S., Schatten A., Zoitl A.: Integration of Heterogeneous Activity Diagrams, In: Proc. of VII Simpsio Brasileiro de
Engineering Environments for the Automation Systems Lifecycle, Qualidade de Software, Vol. 56, 2008.
In: Proc. of Industrial Informatics (Indin), 2009.
[15] OMG: Business Process Model and Notation (BPMN), Version
[3] Biffl S., Mordinyi R., Raidl G., Steininger H., Winkler D.: An SME 2.0, Object Management Group, 2011.
Transition from Plan-Driven to Hybrid Project Management with
Agile Software Development Methods; In: Proc. of EuroSPI, [16] Pawar S.N.: BPMN Tools A Comparative Analysis to Improve
Industrial Experience Paper, Luxemburg, 2014 (accepted). Interoperability, MA-Thesis, Purdue Univ., USA. 2011.
[4] Boehm B., Grnbacher P., and Briggs R.: Easy-WinWin: A [17] Poston R.M., Sexton MP.:. Evaluating and selecting testing tools,
Groupware-Supported Methodology for Requirements Negotiation, In: IEEE Software, 9(3), pp. 33-42, 1992.
In: Proc. of ICSE, 2001. [18] Rademakers T.: Activiti in Action: Executable Business Processes in
[5] Bondi A.B.: Foundations of Software and Systems Performance BPMN 2.0, Manning, 2012.
Engineering: Process, Performance Modeling, Requirements, Testing, [19] Schnbauer M., Winkler D.: A Feasibility Study on Tool-Supported
Scalability, and Practice, Addison Westley, 2014. and Automated Business Process Modeling Approaches, Technical
[6] Brger E.: Approaches to modeling business processes: a critical Report, QSE-CDL 14-02, TU Vienna, March 2014. Available at:
Analysis of BPMN, workflow patterns and YAWL, In: Journal of http://qse.ifs.tuwien.ac.at/publication/IFS-CDL-14-02.pdf.
Software and Systems Modelling, 11(3), pp.305-318, 2012. [20] Shapiro R., White S.A., Bock C., Palmer N., zur Muehlen M.,
[7] Fleischmann A., Schmidt W., Stary C., Obermeier S., Brger E.: Brambilla M., Gagne D.: BPMN 2.0 Handbook Second Edition:
Subject-Oriented Business Process Management, Springer, 2012. Methods, Concepts, Case Studies and Standards in Business Process
Modeling Notation, Future Strategies Inc., 2011.
[8] Geambau C.V.: BPMN vs. UML Activity Diagram for Business
Process Modelling, In: Accounting and Management Information [21] Sunindyo W.D., Moser T., Winkler D.: Process Model Validation
Systems, 11(4), pp.637-651. Bucharest Univ. of Economic Studies, for Heterogeneous Engineering Environments, In: Proc. of the
Romania, 2012. Software Quality Days, Practical Track, Vienna, Austria, 2012
[9] Grieves M.: Product Lifecycle Management Driving the next [22] Weske M.: Business Process Management: Concepts, Languages,
generation of lean Thinking, McGraw-Hill Professional, 2005. Architectures, Springer, 2007.
[10] ISO 9001:2008: Quality Management Systems -- Requirements, ISO [23] Winkler D., Moser T., Mordinyi R., Sunindyo W.D., Biffl S.:
Standard, 2008. "Engineering Object Change Management Process Observation in
Distributed Automation Systems Projects", Proc. of EuroSPI,
[11] Kossiakoff A., Sweet W.N., Seymour S., Biemer S.M.: Systems Roskilde, Denmark, 2011.
Engineering Principles and Practices, John Wiley & Sons, 2011.
[24] Winkler D., Biffl S., Kaltenbach A.: Evaluating Tools that Support
[12] Kurz M., Lederer M.: Subject-Oriented Adaptive Case Pair Programming in a Distributed Engineering Environment, In:
Management, In: Proc. S-BPM One, pp.123-132, Germany, 2014. Proc. of EASE, Keele, Great Britain, 2010.
[13] Moser T., Mordinyi R., Winkler D., Melik-Merkumians M., Biffl S.:
Efficient Automation Systems Engineering Process Support Based