Académique Documents
Professionnel Documents
Culture Documents
com
Complex Event Processing Made Simple Using Esper
Introduction to Esper
Esper is an open-source CEP engine written entirely in Java and fully embeddable into any Java process - custom, JEE, ESB, BPM, etc. It recently has reached version 2.0, is backed by EsperTech under a professional open source / dual license business model, and triggers a growing interest in the Java and .Net community as well with NEsper, its full .Net/C# implementation. Both can be downloaded from esper.codehaus.org with complete examples and documentation. The Esper engine works somewhat like a database turned upside-down. Instead of storing the data and running queries against stored data, Esper enables applications to store queries (or statements) and run the data through them. Response from the Esper engine is thus real-time when conditions occur that match queries. The execution model is thus continuous rather than only when a query is submitted. One of the most attractive features in Esper is that is easy to formulate continuous queries: Esper provides an SQL-like language extended to handle event streams and patterns, an Event Processing Language (EPL). For developers more inclined to program against an API, a continuous query object model API is also available. The other key features are listed below and a few of them will be further described through a concrete example in this article: Continuous queries, filtering and computations (min, max, avg, stddev, vwap, ...) Continuous joins "Followed by" logic and detection of missing events Time and length based sliding window, output flow control
Continuous joins of streams and historical data stored in relational databases, with local caching Event representation as Java object, .Net object, Map or XML Continuous listener processing model and pull based iterator processing model
Assume that we have a service agreement with the payment processing endpoint: it must acknowledge such requests within 15 minutes of receiving a request. The provider answers back by providing a status indicating whether the payment was found in its own system. The acknowledgement looks like:
public class PayInqAck { private String customerId; private String paymentId; private long timestamp; private boolean isFound; }
The 'win:time' syntax in the above query declares a time window of 10 minutes. The query reports immediately when the condition is met. Knowing that a threshold has been reached as soon as it happens can thus increase operational efficiency and provides real- time business activity monitoring on such a non-trivial scenario. The use of a CEP engine like Esper represents a fair amount savings compared to trying to develop by hand such detection logic. It further provides great flexibility to change it and enrich it as business imperative evolves.
The sample code above specifies a package name; thereby our queries don't need to use the fullyqualified Java class name. We could also have declared Map or XML based events - without ever requiring the EPL query to be changed. Continuous query statements can then be created and registered at runtime:
EPStatement statement = service.createEQL( "select customerId, count(*) as cnt " +
"from PayInqAck(isFound = false).win:time(10 minutes) " + "group by customer " + "having count(*) >= 3";
A sample observer that only prints query results as they happen is shown below. Esper invokes the observer and passes strongly-typed statement results when conditions match:
public class MyStmtObserver { public void update(String customerId, int cnt) { System.out.println("customer: " + customerId + " ,count: " + cnt); } }
As an alternative, we can subscribe one or more generic listener components to a statement and use an API to access output events.
public class MyStmtListener implements UpdateListener { public void update(EventBean[] result, EventBean[] prior) { System.out.println("customer: " + result[0].get("customerId") + " ,count: " + result[0].get("cnt") ); } }
Code to receive actual events from the outside will very likely be integrated into a service endpoint in a MOM, an ESB, a WebService facade etc. where transport and protocol issues are dealt with. The code to send those events into the event processing engine is then very simple:
PayInqRequest event = new PayInqRequest(...); //could have read it from network etc epService.getEPRuntime().sendEvent(event);
Events can be domain representations of relevant steps in a business process or problem space, and remain agnostic to the technology used to accomplish integration - be it format, transport or protocol which ensures existing systems tailored for such tasks can fully be reused.
Resiliency and out of heap overflow EsperTech also provides a resiliency and high availability extension - EsperHA - under a commercial license - which ensures internal computed intermediate results survive a crash and further on that the stream window processing capabilities are not bound by memory limits.
Summary
This article has introduced Complex Event Processing using Esper, a technology for real-time continuous data analysis. It solved two use cases out of a BAM scenario in an extensible and decoupled fashion, where Esper significantly reduced a sizable development effort. Last, it presented some of the more advanced use cases and features.
Biographies
Thomas Bernhardt is founder and project lead of the Esper open source project and founder/CEO of EsperTech. Thomas is regularly architecting event-driven software systems at major financial institutions. He has more than 20 years of experience in software. Alexandre Vasseur co-leads the Esper open source ESP/CEP project and works and evangelizes on Event-Driven Architectures and Java middleware. He is is a member of Eclipse and CodeHaus communities, and also co-founded the AspectWerkz AOP framework prior its merger with AspectJ, now being leveraged in Spring 2. All Rights Reserved,Copyright 2000 - 2011, TechTarget | Read our Privacy Statement