Académique Documents
Professionnel Documents
Culture Documents
As digitalization becomes an increasing task for industrial In contrast, in the peer-to-peer-model, several services
production the requirements for developing new production communicate in a network equally with each other. A service
systems are increasing, too. Referring to the reference can send and receive data at the same time. The usually
architecture model for Industry 4.0, flexible communication asynchronous exchange of data leads to advantages regarding
and a distributed system architecture are prominent features of decoupling of communication partners and parallelization.
future production systems [1, 2]. All instances participating in The peer-to-peer model can be realized both as a central
the value creation must be linked to distribute relevant (broker) and as a decentral (brokerless) structure [6]. In the
information as fast as possible, causing service-oriented broker-oriented communication model, each component
architecture (SOA) to gain in importance. SOA is defined as a establishes a single connection to the broker, over which all
loose coupling of independently managed information other components can be addressed (1-to-n). Thus,
systems. Here, a service provides an atomic functionality, authentication and authorization take place only once with the
which can be executed via interface in a computational connection to the broker. Subsequent connections to additional
network [3, 4]. Realizing such an architecture results in a components are handled by the broker and therefore, no
many-to-many communication network (n-to-m). As shown in existing connection is changed. Due to the broker the target of
Fig. 1, there exist two typical kinds of models to communicate out-going or the source of incoming data does not need to be
between network participants, namely the client/server model known. When sending data from one service to several other
ones, the data are sent to the broker only once and then
Server Service 1 distributed from there to the receivers, like a broadcast.
EP EP EP To implement SOA with loose coupling, the advantages of
Service 2 Service 4
EP EP EP EP the peer-to-peer model, especially the independence of source
and target, outweigh the client/server model. For realizing the
Server/Client Server/Client
Broker peer-to-peer model, a message-oriented middleware (MOM)
EP EP EP EP (optional)
is used. A middleware is defined as a layer between software
applications and system resources, which abstracts
EP EP
EP EP EP EP
Service 3 Service 5 functionality of, for example, interfaces and operating system
Client Client Client EP to afford interoperability. Furthermore, a middleware
Service 6 implements and encapsulates additional functionality by
Key: EP Endpoint providing a reusable interface for the user. A MOM in
particular creates an interface for sending messages between
Figure 1. Communication models; Left: client/server; Right: peer-to-peer
* This research and development project is funded by the German Federal P. Sommer, F. Schellroth and M. Fischer are with the Institute for Control
Ministry of Education and Research (BMBF) within the Framework Concept Engineering of Machine Tools and Manufacturing Units, University of
”Innovations for Tomorrow’s Production, Services, and Work” (fund number Stuttgart, 70174 Stuttgart, Germany (phone: +49 711 685-82505; e-mail:
02K16C010) and managed by the Project Management Agency Projektträger Philipp.Sommer@isw.uni-stuttgart.de).
Karlsruhe (PTKA). The authors are responsible for the contents of this J. Schlechtendahl is with Bosch Rexroth AG, 97816 Lohr am Main,
publication. Germany.
1218
(SASL) framework is used, which defines different results in extra effort and must be weighed against the
authentication methods. After authentication, authorization technical advantages. Since the technical features are similar,
enables the possibility to grant rights to specific resources for the results of Table I and the unique features are considered in
specific clients, mainly by Access Control List (ACL) [18, 19]. a short outline for each MOM in the following sections.
1219
Kafka saves each message on the local hard drive by using a A. Quantitative benchmark setup
simple storage concept. Each partition is stored in multiple log The benchmark is executed by two PCs directly linked in
files with fixed sizes. Every time a producer publishes a new a 1Gb/s local network, see Fig. 3. The design of a test platform
value the broker appends the new log entry to a file. The values with producer and consumer on the same computer simplifies
can be stored forever in the broker, independently of the the task of measuring the sending and receiving time, as the
delivery to the consumer. Each consumer has its own reading local system clock can be used as reference. To ensure
offset. The offset enables clients to read all historical data comparability, all MOM are programmed in Java. All sending
stored in the broker [22]. Therefore, Kafka can be used as a and receiving timestamps are set in the benchmark program
database. Kafka also meets the requirements for secure and thus are independent of the MOM. All implementations
messaging and is widely used and supported. As the broker are as close to default settings as possible.
implementation by the Apache Software Foundation seems to
be the only one available, this implementation was used for the The first characteristic value to be measured is the
quantitative benchmark. maximum sustainable throughput (MST) as described by [9].
Therefor a set of payloads and sending rates is determined
C. MQTT from characteristics of industrial applications. The required
payload lp depends on the number and size of transmitted
MQTT is an open standard managed by the non-profit variables and is defined as:
consortium OASIS and was originally designed to enable a
stable communication for low power devices [23]. It also • 4 byte – minimal sensor with a single float value
utilizes the Pub/Sub pattern with a central broker. The MQTT • 64 byte – basic system with multiple values
standard is relying on TCP and describes the exchange of
MQTT control packets between client and broker. Also, the • 512 byte – complex system with larger
standard describes operational behavior and security of MQTT aggregation of values and meta information
sessions. Furthermore, there is a requirement for ordered • 4096 byte – bigger data set packaged in an
sending of MQTT packets in broker and client [24]. There are information model
implementations available focusing on scalable systems as
shown in [25]. Besides the MQTT on top of TCP there is the The sending rate, at which these payloads are transmitted,
MQTT for Sensor Networks running on a serial link. depends on the cycle, respectively on the update time, of the
Therefore, a gateway is necessary to route the messages to the system. Derived from typical industrial cycle times, the target
TCP stack [26]. For benchmarking MQTT, the open source sending rates Rt = {0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 5, 10, 15, 20,
broker implementation Eclipse Mosquitto was used. 25, 30, 35} in kilo messages per second (kmps) have been
selected.
D. ZeroMQ
ZeroMQ is a distributed socket system and does not use B. Quantitative benchmark results
any broker. All unprocessed messages are stored and managed Regarding MST, all payload sizes are producing
in generalized sockets above the transport layer. It is possible qualitatively similar graphs for one MOM. Therefore, Fig. 4
to use different transport protocols for communication like shows only one graph per MOM. The AMQP benchmark
TCP or Unix Domain Sockets (IPC) [27].The stack is very vividly demonstrates the behavior for the MST as described by
small, runs on embedded devices and supports multiple [9]. At an approximate rate of 𝑅𝑡 = 2000 mps the average
communication patterns. To establish a middleware consumer-rate Ravg is saturating and the transmission queue is
communication, all participants have to know each other or filling up. This thesis can be proved by Fig. 5, showing the
there has to be a central managing service, a broker, which can latency Δ𝜏 for a target rate beyond the saturation point. The
also be distributed [16]. Therefore, ZeroMQ is a messaging latency is strictly monotonically increasing. Thus, the
framework and not a fully MOM. This also means that many transmission queue is constantly filling up with messages,
interactions and features like a topic mechanism must be which are taking more time to traverse. The Kafka graph
implemented by the user. For the communication between shows a similar behavior and will therefore not be discussed
sockets the peer-to-peer ZeroMQ Message Transport Protocol further. The benchmark of MQTT reveals another bottleneck
(ZMTP) is used. To establish a connection, the ZMTP of MOM. Packaging the payload and sending a message for a
performs a handshake. Furthermore, ZMTP supports single producer consumes more time than unpacking the
encryption by SALS or Kerberos [28]. If the payload’s size message for a single consumer. For this scenario the definition
exceeds the maximum frame size, a message can be split into
multiple frames [29]. In the benchmark, the ZeroMQ
implementation JeroMQ was used.
1220
proven by identifying the mean maximum consumer rate of
̅ C,max = 11.98kmps with 4096 byte payload and by
R
calculating the data rate as:
̅ C,max * 2 * (lP +lO ) = 822 Mbit/s1
C=R ()
Table II shows the mean MST for all benchmarks.
ZeroMQ is left blank as it never reached computational
saturation in all tests, as described previously. The second
characteristic value to be measured is the latency, defined as
the time from finished sending to finished consuming of a
message. The target rate for the latency measurement is set as
Rt RMST / 2 to prevent a queuing delay. Table III shows the
results for all latency benchmarks. Lastly, the third
characteristic value to be measured is the overhead for sending
a message, defined as the frame size excluding the payload.
Figure 5. Latency for AMQP beyond the saturation point Table IV shows the resulting overheads for all MOM.
of MST is extended by an estimated producer saturation point Concluding from the previously mentioned tables,
to ensure comparability between all MOM. Finally, the ZeroMQ is by far the MOM with the highest MST and lowest
ZeroMQ graph reveals a third bottleneck. As ZeroMQ is latency, caused by the brokerless design. Furthermore, the
brokerless and messages are sent virtually immediate, neither highest MST over all broker-based MOM is reached by Kafka,
producer nor consumer are reaching their saturation point, but
the network card reaches its maximum of 1Gbit/s. This can be
(consumer rate)
(producer rate)
Figure 4. Real sending and receiving rates over target rate for all MOM
1
Flow control, collision handling and calculation intervals between
frames are preventing a data rate of 1Gb/s. The overhead 𝑙𝑂 will be defined
in the following paragraphs.
1221
pattern, AMQP or ZeroMQ (with some extra work) have to be
TABLE II. MEAN MST FOR ALL PAYLOADS AND MOM IN MPS used. AMQP has by far the most built-in functionality and the
Payload 4 byte 64 byte 512 byte 4096 byte biggest flexibility and can therefore be considered a very
MOM balanced MOM with neither outstanding performance nor big
AMQP 1983 1913 1840 1550 downsides. If performance is the main objective, ZeroMQ is
Kafka 10512 11445 12734 9153 by far the best choice due to its brokerless design and basic
MQTT 1056 1074 1056 827 functionality. Regarding message throughput, Kafka is also
ZeroMQ - - - - very powerful. Also, Kafka has low latency with small and
medium sized messages, but not with big messages. In terms
TABLE III. MEAN LATENCIES FOR ALL PAYLOADS AND MOM IN MS of latency MQTT is even a bit better with small massages than
Payload 4 byte 64 byte 512 byte 4096 byte Kafka but does not reach a high throughput due to limited
MOM publisher performance. AMQP shows balanced performance
AMQP 0.867 0.972 0.911 1.120 for both throughput and latency regardless of the message size
Kafka 0.533 0.491 0.441 1.238 and can therefore be considered a generalist. Fig. 6 gives an
MQTT 0.408 0.336 0.398 0.486 overview over the performance of the benchmarked MOM and
ZeroMQ 0.134 0.160 0.168 0.178
ranks them against each other taking into account the different
criteria.
TABLE IV. OVERHEAD OF A SINGLE MESSAGE FOR ALL MOM
MOM Overhead
To decide which MOM should be used for a specific
AMQP 105 byte industrial application the template in Table V gives some
Kafka 192 byte guidance. Here it is exemplary filled for the use case of Motion
MQTT 64 byte Control. As a first step the system architect has to decide which
ZeroMQ 52 byte* application criteria have to be applied to the application. He
*Messages with 4096 bytes payload are split in multiple frames.
For this scenario, no overhead is calculated
also has to rank these criteria between 0 and 10 (not part of the
table). Then, the architect has to decide if some criteria are
but the lowest latency is reached by MQTT. For higher more important than others and assign a criteria importance
payloads AMQP tends to be a good compromise between factor (CIF) between 0 (not important) and 3 (very important).
MST and latency. Lastly, ZeroMQ has the smallest overhead Afterwards, the architect can calculate the absolute difference
of all MOM needing only 52 bytes for framing a message between his application and the according MOM (see Fig. 6)
payload. Regarding the MOM having a broker, MQTT has the for every criterion multiplied by the importance factor. The
smallest overhead. All tested MOM are suitable for industrial sum of all differences per MOM can be compared resulting in
purposes regarding the previously described tests, but there the most fitting MOM with the lowest sum.
are clear favorites for each use case. For selecting the optimal
MOM there is a need of priorising each criterium (see chapter
V). For the example of Motion Control the latency is VI. CONCLUSION AND FUTURE WORK
considered as highest priorised criterium and therefore In this paper the performance of four well established
ZeroMQ or MQTT should be used. MOM was examined in terms of qualitative and quantitative
features, enabling a decision for a MOM in a specific use case.
V. SELECTING A MOM FOR AN INDUSTRIAL PRODUCTION For the qualitative features, a comprehensive table of features
SYSTEM has been presented. This table contains both objective and
subjective features like the usable communication patterns or
Selecting a suitable MOM for a new industrial application the ease of implementation. Additionally to the comparison
is not an easy and unambiguous task. Based on the results of table, a list of unique features for each MOM has been
the qualitative and quantitative comparison, this section identified. In a comprehensive benchmark the maximum
explains when to use which of the presented MOM. As each sustainable throughput, the latency and the overhead of these
of them has different advantages and disadvantages, the MOM has been measured. In the MST benchmark, all MOM
decision depends on the requirements of the use case. except MQTT have achieved a throughput of more than 1000
A first aspect to consider is the necessary communication mps for all tested payload sizes, which is relevant for industrial
pattern. MQTT, AMQP and Kafka are all based on a broker, AMQP Kafka MQTT ZeroMQ
whereas ZeroMQ is a brokerless MOM. Furthermore, Throughput
ZeroMQ has to be taken as a messaging framework rather than 10
as a full MOM. Any functionality beyond fundamental Development 8
6 Latency
transmission of messages has to be implemented by the effort
application developer. Therefore, for an out-of-the-box and 4
2
easy implementation, ZeroMQ is the most unsuitable choice.
0
Much better in terms of ease of implementation are Kafka, Bandwidth/
MQTT and AMQP, in that order, from more complex to easy. Standardization
overhead
A unique approach having durable saved message streams and
the ability to jump back and forth is taken by Kafka. With all
other MOM, data storage has to be solved separately. Bidirectional
Flexibility
communication
Regarding messaging types, neither Kafka nor MQTT support
the request/reply pattern. Consequently, if there is need for this
Figure 6. Performance of MOM
1222
TABLE V. SELECTION TEMPLATE MOM [7] L. Qilin and Z. Mintian, “The State of the Art in Middleware,” in
2010 International Forum on Information Technology and
Application Criteria Difference between MOM and Applications, Kunming, China, 2010, pp. 83–85.
criteria Importance Application (See Fig. 6) * CIF [8] S. P. Ahuja and N. Mupparaju, “Performance Evaluation and
Factor (CIF) AMQP Kafka MQTT ZeroMQ Comparison of Distributed Messaging Using Message Oriented
Throughput 1 4 8 2 10 Middleware,” CIS, vol. 7, no. 4, p. 9, 2014.
Latency 3 4 6 7 10 [9] P. Tran, P. Greenfield, and I. Gorton, “Behavior and performance of
Bandwidth / 3 4 2 7 8 message-oriented middleware systems,” in Proceedings 22nd
overhead International Conference on Distributed Computing Systems
Flexibility 1 9 4 5 7 Workshops, Vienna, Austria, 2002, pp. 645–650.
Bidirectional 1 10 2 2 7 [10] P. Dobbelaere and K. S. Esmaili, “Kafka versus RabbitMQ,” 2017.
communication [Online] Available: http://arxiv.org/pdf/1709.00333v1. Accessed on:
Standardization 0 10 0 10 0 May 30 2018.
Development 1 8 5 5 2 [11] J. Alquiza, Load testing Apache Kafka on AWS. [Online] Available:
effort https://grey-boundary.io/load-testing-apache-kafka-on-aws/. Accessed
Result - 55 43 56 80 on: Jan. 04 2018.
[12] A. Dworak, P. Charrue, F. Ehm, W. Sliwinski, and M. Sobczak,
“Middleware Trends And Market Leaders 2011,” Conf. Proc., vol.
applications, as most control cycle times are between 1 and 10 C111010, no. CERN-ATS-2011-196, FRBHMULT05. 4 p,
ms. Moreover, it is notable that all MOM are varying greatly http://cds.cern.ch/record/1391410, 2011.
in all quantitative features, but the three characteristic values [13] J. Kreps, Benchmarking Apache Kafka: 2 million writes per second.
[Online] Available:
are not necessarily correlated, clearly recognizable for Kafka, https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-
with a low latency but high overhead and MST. 2-million-writes-second-three-cheap-machines. Accessed on: Feb. 12
2018.
Resulting from the qualitative and quantitative features, a [14] J. Kuch, RabbitMQ Hits One Million Messages Per Second on
guideline to select a MOM for industrial use cases has been Google Compute Engine. [Online] Available:
developed. Therefore, the four described MOM are ranked by https://content.pivotal.io/blog/rabbitmq-hits-one-million-messages-
application criteria. By using the given guideline and template, per-second-on-google-compute-engine. Accessed on: Jan. 19 2018.
new use cases can be categorized and then be linked to the [15] S. MacMullen, RabbitMQ Performance Measurements, part 2.
[Online] Available:
matching MOM. http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-
The benchmarked MOM cover only the transport layer in measurements-part-2/.
[16] zeroMQ, Broker vs. Brokerless. [Online] Available:
the OSI model. The layers above are not discussed in this http://zeromq.org/whitepapers:brokerless. Accessed on: Jan. 05 2018.
paper. The discussion of standardization of interfaces to enable [17] N. Narkhede, Exactly-once Semantics are Possible: Here’s How
data exchange among arbitrary participants as revealed in [1] Kafka Does it. [Online] Available:
is not finished yet. VDI recommends the use of IEC 62541, https://www.confluent.io/blog/exactly-once-semantics-are-possible-
also known as OPC UA, which is more than a transport heres-how-apache-kafka-does-it/. Accessed on: Jan. 31 2018.
[18] I. Juma, Apache Kafka Security 101. [Online] Available:
protocol. It also covers the layer of data representation. With https://www.confluent.io/blog/apache-kafka-security-authorization-
its growing numbers of companion specifications of authentication-encryption/. Accessed on: Jan. 31 2018.
information models, OPC UA can be used for data [19] P. Hintjens, Using ZeroMQ Security (part 1). [Online] Available:
representation. But OPC UA is based on the client/server http://hintjens.com/blog:48. Accessed on: Feb. 01 2018.
pattern and its information model is not transport-independent. [20] “AMQP Advanced Message Queuing Protocol: Protocol
As discussed in the first chapter the importance of the peer-to- Specification Version 0.9.1,” Nov. 2008.
[21] Z. Wang et al., “Kafka and Its Using in High-throughput and Reliable
peer model in environments of loosely coupled participants is Message Distribution,” in The Eighth International Conference on
undeniable. The OPC UA standard is extended to support Intelligent Networks and Intelligent Systems: ICINIS 2015 : 1-3
AMQP (broker-based) and UDP broadcasts (brokerless) to November 2015, Tianjin, China : proceedings, Tianjin, China, 2015,
deliver data in a peer-to-peer model. The independence of pp. 117–120.
transport protocol and representation is not completely [22] J. Kreps, N. Narkhede, J. Rao, and others, “Kafka: A distributed
messaging system for log processing,” in Proceedings of the NetDB,
fulfilled [30]. Therefore, the representation of data 2011, pp. 1–7.
independently from the transport layer needs further [23] M. Houimli, L. Kahloul, and S. Benaoun, “Formal specification,
investigation. verification and evaluation of the MQTT protocol in the Internet of
Things,” in 2017 International Conference on Mathematics and
Information Technology (ICMIT), Adrar, Algeria, 2017, pp. 214–221.
REFERENCES [24] OASIS, “MQTT Version 3.1.1: OASIS Standard,” Organization for
[1] Bundesministerium für Wirtschaft und Energie, Ed., Netz- the Advancement of Structured Information Standards, Oct. 2014.
kommunikation für die Industrie 4.0: Diskussionspapier. Berlin, 2016. [25] P. Jutadhamakorn et al., “A scalable and low-cost MQTT broker
[2] VDI/VDE-Gesellschaft, Ed., Referenzarchitekturmodell Industrie 4.0 clustering system,” in 2nd International Conference 2017, pp. 1–5.
(RAMI4.0), 2015. [26] A. Stanford-Clark and H. L. Truong, “MQTT For Sensor Networks:
[3] S. Shekhar, H. Xiong, and X. Zhou, Encyclopedia of GIS. Cham: Protocol Specification Version 1.2,” IBM Corporation OASIS
Springer International Publishing, 2017. Standard 1.2, Nov. 2013.
[4] S. Mankovski, “Service Oriented Architecture,” in Springer [27] ZeroMQ Feature List. [Online] Available:
reference, Encyclopedia of Database Systems, L. LIU and M. T. http://zeromq.org/docs:features. Accessed on: Jan. 05 2018.
Özsu, Eds., New York: Springer, 2009, pp. 2633–2636. [28] ZeroMQ Message Transport Protocol. [Online] Available:
[5] A. Schill and T. Springer, Verteilte Systeme: Grundlagen und Basis- http://zmtp.org/page:read-the-docs.
technologien, 2nd ed. Berlin, Heidelberg: Springer Vieweg, 2012. [29] P. Hintjens, ØMQ - The Guide. [Online] Available:
[6] E. Curry, “Message-Oriented Middleware,” in Middleware for http://zguide.zeromq.org/page:all. Accessed on: Jan. 19 2018.
Communications, Q. H. Mahmoud, Ed., Chichester, England: John [30] OPC Unified Architecture Part 14: PubSub, RC 1.04.24, 2017.
Wiley & Sons, 2004, pp. 1–28.
1223