Marco on CEP

No. I'm not dead. We're still in business. Working hard on various projects so I have very little motivation to write anything here right now. I'll be back when my blog writing motivation returns…

Still closed for summer, but I could not resist pointing out this presentation:

 

http://www.slideshare.net/opher.etzion/aaai-2011-event-processing-tutorial

 

A good summary of current CEP systems and what challenges they face.

Well, at least for summer…

Back in August (maybe September) again… A good Swedish tradition is to have a looong break in the summer. (A hint for you in the US, work for a Swedish company like Volvo and you'll might have five weeks of summer vacation…)

I did some googling to see what's new from the StreamInsight group at Microsoft. There seems to be some development but I did not find any revolutions there. Not in any way implying that there's something wrong with StreamInsight. I like the concept and they have a really cool product there. It's also nice to see that Microsoft is part of the CEP community. But I sort of expected more news as it was a long time since my last check. Maybe I missed something? Anyone from StreamInsight team that have the inside?

What was interesting is that I found a picture describing three kinds of CEP. I have been saying for long now that not all CEP is created equal.

The StreamInsight view:

1. CEP for lightweight processing and filtering at the edge of your network, close to your sensors and other event source.
2. CEP for aggregation and correlation of in-flight events closer to home
3. And at the center of your solution, you do more processing intensive CEP for complex analytics possibly including access to historical reference data.

That's one good way of describing three different types of CEP. Another perspective that one could add to that is that there's two main processing types at each of these three steps: Driving algorithms with event data or looking for interesting patterns in the event stream.

 

 

 

 

It might not come as a surprise that the definition of an event is central to any type of event processing. Call it Complex Event Processing (CEP) , Event Stream Processing (ESP) or just Event Processing, but the concepts are the same.

In some products, like Coral8 and StreamBase, they actually started with the notion of data and after a while re-named the tuples to events. So the notion of an event seems to be a bit flexible. Different vendors have different ideas on what an event is. Our ruleCore started from the other end, starting with an strict 'everything is event approach' with a more strict definition of an event and using events as the native interface for everything. Other products seems to end up somewhre in between. I don't think either of there approaches are wrong. They just reflect two different types of event processing.

In order to better understand all these different event models it's great to have some possible features that a conceptual event might have.

The event model used by an event processing system defines to great extent the feel of the system. Small details in the event model have a huge impact on the expressive power and capabilities of an event processor. Just compare StreamBase with ruleCore and you'll see two type of products even though both are marketed as CEP, and they are both CEP!

An event model consists normally of:

- Event semantics
- Event syntax

 

Let's break up the text with a spider:

 

In addition to headers and other meta-data an event normally contains some payload. It's the data which tells us more about the event. If the event type tells us what happened, the payload tells the details about the event that occurred.

This data can be in different forms. Key/value pairs, structured or unstructured. Popular event encodings such as XML enforces some restrictions on the payload. Using a binary event encoding opens up for all kinds of creative content in the event body; Pictures, video or other types of rich content can easily be added into the body of a binary event. Although many architects and designers favor keeping their events small.

 

So, what are the fundamental base properties of an event? That is, what kind of properties can we expect to find in most events?

We need to consider both the syntax and semantics of the properties. Syntax is actually the easy part. Semantics can be a bit harder to define.

We have some prime candidates for event core properties

• Event Id
• Event Type
• Detection timestamp
• Entity
• Probability
• Location
• Occurrence time stamp
• Event class or category
• Caused by.

 

Event Example

Lets look at some practical examples. Here's an event encoded in XML (actually in the ruleCore markup language, but the concept is the same independent of the exact format. Feel free to add others in the comment field):

<ProximityAlert id="bd79d85a-12c1-4a1f-9047-3dc34fcab2cc" detection_time="2011-06-06 08:29:22">
  <Header>

     <CausedBy>39493849384938984930</CausedBy>
  </Header>
  <Body>
       <EntityId type="semitrailer">AB-123</EntityId>

       <Location>-23.3495 34.4955</Location>
  </Body>
</ProximityAlert>

 

I’m using XML here as an example, but it could as well be something else… In this example we have an event of type ProximityAlert, it was detected just a couple of minutes ago and its globally unique id is bd79d85a-12c1-4a1f-9047-3dc34fcab2cc.


As we can see I envision the event to contains two containers, one if the header and the other is the body.

The header contains information about the event itself, for example which events it depends on, security related information, information about the event sending system and other meta data type of information.

The body is the actual payload of the event.

At a lunch discussion we talked about things that every architect and developer should now. Not knowing these will yield the "wtf? Where have you been the last ten years"-response

Just some examples that we talked about

http - foundation of the Internet. You should know pretty well how it works.

Java or C# - If you need to code, you really should be able to do it using one of these two.

SQL - For storing data there’s no excuse not to know SQL.

It’s pretty aparent that anything related to CEP has a long way before it finds its way onto that list.

If CEP is ever on the must-know list, in what form do you think we will talk about required CEP skills?

 

 

 

 

As you might know we (ruleCore that is) have a cloud only (except for the OEM Kit) solution for CEP. This means that I try to keep an eye on other non-CEP related developments and trends in the Cloud space.

A trend that I think I have spotted, is that there seems to be a number of cloud services with a special super cloud status. Amazon Elastic Compute Cloud EC2, Google App Engine and Microsoft Azure have a very special position in the market. Third parties are starting to adopt their software to these super clouds because it gives them credibility.

My idea of a super cloud is thus not purely based on technical merits. I’m sure rackspace and other local cloud providers have services that are as good (or better) as these super clouds.

But what is interesting is that software vendors which produce new cloud services seems to have discovered marketing benefits by running their solutions in a super cloud.

 I think trust is one of the most defining characteristics of any cloud service. If you don’t trust that the cloud works, you really don’t care about its technical capabilities. So any successfull cloud provider will mainly focus on trust and secondly on technology. With a good enough technical solution it all comes down to trust.

That’s why I think that you will see names in the cloud arena that you really did not expect to provide this kind of services, building on a strong brand instead. Maybe many of the telcos or other well know companies with a brand that says "you can trust us, we will be here forever".

 

 

Do CEP if your events come in a steady stream

Do CEP if temporal aspects of the event stream is important

Do CEP if you need to drive computation by live data

Do CEP if your data changes rapidly

Do CEP if you find yourself polling your database at high rates.

 [add more]

 

Don’t do CEP if performance is your only concern.

Don’t do CEP if you…

 

OK so I thought I had more time to write,  let’s call this a draft and I’ll return to it later…

 

Today I think I’ll read more about Scada and sensor networks.

Those two might be the next big thing for CEP…

I’m trying to understand what we need to do in ruleCore in order to support these. My first initial thought is that we are halfways there are we support a special kind of sensor already – The GPS

But there must be more features that are specific to Scada and sensor networks…. Stay tuned.? Any recomended links?

 

Would it not be nice if you could buy a stream of events?

http://www.play-project.eu/index.php/solutions/event-marketplace.html

What if  you had easy access to various event streams? 

I really hope that this project manages to produce something interesting in this area. What might be problematic is that this kind of functionality is not a technical problem. It’s mainly about business. Obviously you need the infrastructure to support this kind of funktionality, but I’m pretty sure the technology is here already.

But what is a greater challenge is to create an eco system where service provider can find profitable business in providing event streams. Without that, no event market…

 

 

As a follow up on my last post I have been thinking about what a complex action should do.

If the first part of the processing is the detection of something interesting, then the second one ought to be a decision about what to do as a result of the interesting thing….

In ruleCore CEP Server, the concept is similar in many other CEP tools, the main part of the processing is spent in trying to find intrresting patterns of events. We call them situations. Actually a situation is a combination of many patterns.

A situation normally represents something of interest to the line of business. Like a tanker just entering a processing plant or a freezer door open for more than five minutes.

If you are using ruleCore or any other type of CEP software to detect these interesting situations I’m pretty sure that you intent to do something as soon as the situation is detected.

So the logical step would be to add complex decision making that is triggered by the action part of the rule.

With decision making capability in place you would be able to automate not only the detection of interesting situations but also the reaction to them…

I think something like that would be a nice addition to ruleCore…

Lately I have been thinking that a nice addition to our ruleCore CEP Server would be to add complex actions.

Today the actions are very simple. They just gather the context at the point when a rule triggered. This is done using XSLT that can execute against the complete SIM (our tree like structure with all the rule instances and other meta data) and gather information about the rule that just triggered.

The action takes this info and generates a new event which is the output of ruleCore or used as input for rules at higher level.

My idea is to add a complex action execution stage which uses this event as input and trigger complex behavior

For example we could feed a rule engine (Drools for example) with the event and let all kind of interesting rule based actions happen as a result of a ruleCore rule.

With this concept ruleCore would be detecting when something interesting happens and the second action execution stage would execute all kind of things that need to happen as a response to the complex event that was detected.

It sounds like a good idea, what do you think?

 

 

First, Opher have an interesting post here.

The Cloud and CEP are two technologies that seems to have difficulties to catch on. Both are sold as the solution to practically everything. But still, business are slow to adapt them. Why so? Well, both are actually bigger changes that you might think. Both require you to think differently and that’s something most people don’t want to do.

But these two technologies are going to change how we view computing.

Perhaps not in the short term. But in 10 years from now I think we are going to have a completely new software stack or even better, we don’t think that much about software stacks anymore.

Why these two? Well, the cloud requires CEP and CEP requires cloud based computing to give you that ROI that the MBAs care of.

Just try to make the current technology to live in the clould. The impedance missmatch is massive. That’s why clould computing have a hard time to catch on. Basically everything need to be rewritten.

In the manifesto Opher talks about there a challenge, how to build a fabric for CEP. A combination of sofware based on CEP principles living in the cloud will take us there!

 

 

 

 

One thing that most software developers are really ‘good’ at is to prefer building most fun/interesting/looks nice on cv type of software themselves. Preferably using their choice of religion. Most commonly C# och Java these days.

In the world of CEP that manifest itself in two ways.

First, if possible. Most companies prefer to build CEP functionality themselves. Mostly because it’s the type of functionality that is great fun to build. They also build it using their regular software stack. So that’s two things that are not a good idea. So a Really Bad Idea.

It’s not that CEP is a bad idea or that any of the CEP products are bad out there. It’s just that CEP actually requires people to change their way of thinking. And we know how easy that is.

My guess is that in the early days of databases people did the same thing. So it’s just that we are very early in the process of changing how people think of data processing. We need to add another dimenstion. Now we have stored data and we need to add live data to it. That will take some time as it requires a slightly different mindset.

 

 

I have continued my googling session to see what’s up in the CEP world.

What strikes me is that CEP is not needed for most problems.

In the early years of CEP (that is, not that many years ago) one vendor boldly claimed that CEP would replace all databases and the era of stored data was to end very soon. That did not happen. And it will not happen in the future either.

Today many vendors, ‘experts’ and various projects have examples of what kind of problems their amazing CEP software is a good solution for.

Most of them are totally wrong, or at least not exactly right. Most of the problems can be solved with your traditional software stack containing app servers, datbases, messaging middleware and other types of more common software.

That’s why many CEP tools just don’t seem to catch on. They try to solve a problem which have a simpler solution.

There are offcourse areas where CEP is needed. Problems that don’t have a good solution using the traditional stack. Trading is one of them and there’s others. But CEP is a niche technology as it solves a very specific problem. It’s not a generic tool as some would like you to believe.

Said that. CEP is very  important and absolutely critical to have in your stack when you are facing the type of problems CEP is good at.  Most CEP products are just amazing when paired with the correct problem. Try to solve CEP problems with your traditional stack and you will spend 10 times the money on your implementation.

Why I bring up this is that many will be dissapointed on CEP technology if they think that CEP the solution for all kind of problems. It’s 10 times cheaper to use traditional software instead of CEP it you don’t need CEP capabilities…

Nothing new or magic here – Use the right tool for your problem..

 

 

 

 

At last it’s here:

http://drops.dagstuhl.de/portals/index.php?semnr=10201

 

The output from Seminar that a group of people held last year. We had a lot of both formal and more importantly informal evening discussions that created the foundation for this manifesto.

 

 

Interesting read from a CEP perspective:

http://www.efinancialnews.com/story/2011-03-03/rise-of-the-picosecond

 

From a technical viewpoint this is probably a really intereting problem to solve. From another viewpoint, this is another nail in the coffin… (is that correct english?)

If everything goes according to plan there will be lots and lots of new sources for events, starting with Saab and their Android based car system called IQon. So if  (when) this goes into production there will be lots of moving event sources out there. I would not be surprised if other car companies follows with systems on the same theme. Perhaps not open and Android based, but probably something that will generate lots of data streams.

With access to hundreds of sensors in the car that your own Android app can read from and send to some central location I envision that vechile tracking will go mainstream. Today it takes some work to install a gps tracker into vehicles and integrate these into the CAN bus in the car. But if all that becomes standard in most cars, we have amazing opportunities to build cool software.

 

 For a while I have been thinking about the possibility to integrate two things into our ruleCore CEP Server.

First, a high performance pre-processing stage and secondly the capability to make (take?) decisions.

As a first stage, before the events hit ruleCore, it would be rather nice to have some filtering, routing and preprocessing. The focus would be to do simple operations but at a high speed. For example:

• Enrich events
• Route events
• Partition events based on attribute values
• Filter out events which are not needed by the rules downstream
• Aggregation

That’s what was called ESP in the early days before everything went CEP. That would be simple, event by event, processing. Instead of building this I thought we could use Esper or some other library for stream processing. I think Esper could be a good first stage if we could integrate it with our rules.

Another addition that I have been thinking of is some kind of support for decisions. Perhaps in the form of production rules, the forward chaining type. Maybe Drools? Could it be an interesting option here? 

By integrating Drools into the various stages of processing I think we could do really interesting things. Like better actions and perhaps more ‘intelligent’ entities. For example it might be possible to automatically deduce that a container is loaded on a train if both the train and the container is moving along the same route.

So, what do you think? Could a combination of ruleCore+Esper+Drools be a winner?

 

 

 

Now I have done a fair amount of googling and looking through my CEP bookmarks. I’m happy to see that All Is Good in most part of the CEP Universe*. What I’m missing is new cool startups, new ideas, fresh concepts and most of all I’m missing a good fundamental theory on CEP.

I would expect to see a new wave of CEP in all areas. But I really can’t find any.

I would expect new cool startups to come out with refined CEP ideas and better implementations. But no, there’s the same old vendors doing the same old things. Well, more or less. I don’t want to be unfair to vendors or open source projects;  there’s lots of activity in most places. But mostly variations of what’s been out there for several years now.

What I also miss is a good solid theory on CEP. I would love to see somthing like the relational theory for databases.  A simple and elegant theory that can be implemented in a number of ways.

Also, looking at the current open source projects all I can see is variations of the commercial offerings. It’s not that they are bad in any way. There are a number of really great open source CEP projects. But many of them feels just like a gratis version of the commercial products with nothing exciting added, except that they are cheap.

I think it is obvious that the current state of concepts, tools and languages are not the end of the road. This is in my world the first attempt at CEP. I think we need at least four or five additional iterations to arrive at something really, really good. At least I have much more ideas than cash for new CEP related stuff that we could build that are way better than anything out there. I can’t be the only one with good ideas on TNG CEP.

 

* I’m reading a book about M-Theory and it seems that it’s actually a Multiverse we live in. Very strange indeed.

 

 

Last night I tried to do a quick check on so many CEP sites as I could find. I started with my own vendors list followed by my bookmark list.

I got the feeling that some of the vendors and their projects are doing just fine while others have very little activity. One vendor seems not to have updated their site for a couple of years now but others have lots of activity.

There seems to be very little activity in some open source projects too, while others are producing updates on regular basis. It might not be that fun to run an open source project when nobody contributes. The ones that are most succesfull are what I call formally open source, but with nobody else than the company behind it contributing makes me wonder if they are open source purely for marketing reasons. It’s understandable as many customers are looking for a tick in the gratis box. If the license is free, then there seems to be a huge budget to spend hours in configuring and coding. But who said the world is rational.

What strikes me is that the offers in the CEP are from many vendors are still very technical. Yes, there are exceptions but then it goes all the way to the other end with the vendor barely talking about CEP but instead concentrating on a specific market. Which seems like a good idea. StreamBase for instance feel more like a supplier to Wall St. than a CEP tech company. Which is probably a sane option for the smaller CEP companies. I can’t see how a smaller company could charge for this kind of infrastructure anymore. Gigants like IBM, Tibco and Progress might be able to sell infrastricture software but the rest would need to give it away or provide a vertical packaing.

But still I think most players in the CEP world are doing pretty OK. Most offerings are pretty solid and most projects and vendors have found their own area within the bigger CEP playing field.

What I’m waiting is for the next wave of CEP. It will have it’s roots in todays solutions but radically different in many ways. Just like the jump from all kind of concepts for storing data to the relational model and then SQL.

On an unrelated news I spent some time reading about M-Theory. I see to possibilities here. Either the world’s most smartest scientists have been eating mushrooms or the world is a very strange (but cool) place to live in…

 

 

 

I have been away for a while from the blog and the CEP community. I thought I’d do a quick status update to see what is happening in The World of CEP.

Starting at StreamBase I see that they are more or less transformed into a capital markets software provider, which seems like a good thing. During my absence there have been a lot of negative press about automated trading, even in the mainstream press here in Sweden. Almost everyone is calling for regulations. Well, apart from those doing hft or supplying cool software like StreamBase.

A short googling did not reveal the fate of Aleri/Coral8, but I suppose it might be that not much happened during the few months that I have been doing other stuff (Working for the worlds best car company).

Both TIBCO and IBM seems to have their Business[ ]*Events products. Tibco had something called tibbs which in true US marketing style is presented as the best thing since sliced bread. But it seems to be using event processing concepts heavily. Which might not be a suprise considering whos running company. But the information about Tibco BusinessEvents seems to date back to 2008 so I wonder what ever happened to that product.

At the ruleCore HQ there’s activity in the track&trace field and many other vendors seems to continue to work with their products and CEP inspired functionality is finding its way into many products.

I’ll continue my status update later on to see what’s going on with CEP.

Any insights, rumours or fancy press relases? Just send them my way…

 

 

 

It’s been a while since I updated my blog. I just wanted to say that I’m still here, the company is alive and ruleCore CEP Server is crunching along. It has just been a lot of work at one of our car companies around here so I have not had any time to attend to my blog. But soon I hope I can write something of interest and not this kind of meta stuff…

 

Introduction

The Rulecore CEP Language – Reakt! – When you need to react immediately. Reakt is a language for complex event processing, or CEP for short. It is designed to provide a powerful and productive tool for developers, allowing them to quickly create solutions for detecting critical business situations in real-time. Reakt is not a traditional programming language. Reakt is a business situation description language. By using a declarative approach it allows the developer to concentrate on what kind of situations to detect and not how The design goal of Reakt is to provide a way for organisations to react immediately to critical situations. It achieves this goal by finding related events from streams of inbound events. What events a critical situation consists of, and how the events are related, are defined by the user using Reakt. Reakt allows the user to specify conditions and temporal relationships which must hold in order for the events to be considered a noteworthy situation. When the situation is detected by Reakt, an user defined real-time report is created which provides actionable information about the situation. Allowing the enterprise to react immediately. This short white paper gives an informal description of the language and explains the different language elements using examples instead of formal definitions.

Overview

Rulecore Reakt is a small domain specific complex event processing language designed for detecting situations by observing streams of events. An event is a description of something that happens. For example ‘engine started’, ‘order created’, ‘invoice sent’ or other types of events which are relevant to your business. The events originate normally from your internal business systems, from your suppliers or other partners. You can also get events from public data sources like Facebook, Twitter or events extracted from news feeds or weather and traffic data feeds. A situation is a formal description of a combination, a pattern, of events as they happen over time along with optional restrictions among the events. A simple example of situations that you might want to detect:

"A vehicle has crossed River Thames in London more than 10 times during the last three hours"

or

"ParcelDelivery event came before Order Event unless delivery is of type priority one"

Main Reakt language features:

• Continuous Evaluation – Designed to operate on live streams of events as they happen.
• Declarative – Rules and other items are specified using a high-level declarative approach.
• Modular – Rules are defined using modular and re-usable building blocks.
• Temporal – Handling of time is an integral part of Reakt. Timing restrictions and other temporal conditions can be stated using a number of declarations.
• Location Aware – Reakt includes built in support to understand location information and other geographical objects such as zones.
• Implementation Independent – The elements of Reakt are defined using standard XML and are implementation independent. Different implementations can be created in order to provide specialized execution environments.
• Automatic Life Cycle Management – Rule instances are created and deleted automatically in response to events. Rule instances are analyzed and deleted if their situation is determined to be undetectable.
• Dynamic Context Management – Each rule instance maintains its own virtual view into the streams of inbound events. The view provides a window into the incoming event streams in order to create a dynamic context for rule evaluation.
• Separation of Concerns – Reakt uses reusable blocks for;
  • Selection of events from the inbound stream
  • Rule life cycle management
  • Situation definition
  • Rule trigger reporting

The design goals of Reakt is to provide a language which allows the user to describe complex situations consisting of multiple events with non trivial temporal and geospatial relationships. The language is declarative. Lacking any procedural elements provides a base for formal verification and automated optimization and analysis of the language. Machine Friendly. A design assumption made early in the language design was that it is likely that computer systems will use Reakt to create and modify rules more frequently than humans. The rate at which new rules need to be created and old ones modified is assumed to be high. Thus the language was designed to be easily understandable by machines and at the same time being reasonably human readable for system developers. Friendly XML. The current version of Reakt uses XML for its syntax. XML was chosen to take advantage of existing knowledge among developers, known standards and existing tools for generating and parsing. It would also be possible to create alternative syntaxes for Reakt. For example to create a more compact human friendly representation for business users. EDA Building block. The Reakt language was designed to support execution environments which fits into an architectural style called Event Driven Architecture (EDA). In order be a natural building block in an EDA, the ruleCore Reakt language and its supporting execution environment, the ruleCore CEP Server, was designed to be completely event driven. This purely event driven approach allows all systems in a distributed messaging architecture to integrate with ruleCore using simple events as their only communication mechanism.

The System Information Model

The System Information Model, or SIM for short, is both the name of the information model used by ruleCore CEP Server and the name of the run-time data structure modeled after the familiar DOM model, commonly used to represent information originating from XML. The SIM contains all information about different Reakt items such as rules, situations and other types of run-time information. The SIM is part of the ruleCore CEP Server implementation and not the language definition. But as all the language elements are executed against the SIM, it is described here briefly. Different implementations can implement varying information models. The appearance of the SIM can also vary  in ruleCore CEP Server depending on which dynamically loadable modules are present in the system. The SIM is used for various purposes, for example:

• Listing available definitions – All definitions of items such as rules, views, zones, entities, situations and actions can be found in the SIM. Definitions are created using the systems events, for example AddRuleDef, AddSituationDef and AddActionDef.
• Listing user defined events – Before an event can be sent into ruleCore CEP Server, it must be defined using the AddEventDef system event. The SIM contains all user defined events.
• List of rule instances – When a rule instance is created, it’s state and internal data structures can be monitored by querying the SIM. The ruleCore CEP Server provides a query mechanism using XPath. But other implementations might query the SIM using other methods.
• Monitoring situation progress – As a situation develops, its progress can be monitored by querying the SIM.
• Examine view contents – Each rule instance is executed in an unique context of events called the event stream view. The view contains a window into the inbound stream of events and contains only semantically related events.
• Referencing items – When an item such as a rule needs to reference other items it is done using references through the SIM.
• Component Specific Data – Each plug-in component that extends the engine creates an entry in the SIM and may add any component specific information.

The SIM consists internally of dynamic data structures which are updated as needed when the internal state of the ruleCore CEP Server changes in response to inbound events. These data structures have an external representation which is the user’s view of the SIM. The external representation of the SIM can be treated as a DOM document which can be queried using XPath and transformed using XSLT. XPath queries can be executed against the SIM using the SysInfoQuery system event. If is also useful to note that the ruleCore CEP Server uses events for all tasks. Special pre-defined system events are used to create rules and other items in ruleCore CEP Server in addition to monitor the run-time state of the rule evaluation.

Structure of a Rule

The main language element in Reakt is the rule. Reakt uses a class of rules called reactive rules to provide its situation detection capabilities. Reactive rules should not to be confused with production rules or other types of rules which must be explicitly evaluated. A rule in Reakt is never explicitly evaluated, its evaluation is automatically managed by the execution environment in response to inbound events. A rule consists of a number of elements, the main ones being:

• Initialization directives - Specifies which events can lead to the creation of a rule instance
• Termination directives – Specifies what causes a rule instance to be deleted.
• The rule level – Specifies on what level in the rule hierarchy instances should be created.
• Trigger Directives – Controls how and when to report Rule triggering.
• The View – A reference to a view, providing execution context for the rule.
• The Situation – A reference to a situation to track and detect situations.
• The Action – A reference to an action, which will generate an event when
  • the situation is detected, or
  • if the situation is determined to be undetectable.

General Rule Attributes

Each rule has a name and a attribute which can be used limit the number of rule instances that can be created of a particular rule. A common usage of the limit is to create a singleton rule, of which only one single instance can exist. Rules at every level can publish their trigger events externally so that they are visible outside of the ruleCore CEP Server. All rule declarations includes a level specification. Every rule instance is created on a specific level in a hierarchy of rules. Level zero is the level on which inbound events arrive. Thus the lowest level for a rule is 1 (one). A rule will only see events from a level directly below it. For example, three levels of rules where the rule at level one sees the inbound events:

<Rule name="New Location" limit="1000" level="1"> … </Rule>

<Rule name="Zone Entry" limit="1000" level="2"> … </Rule> <Rule name="Zone Exit" limit="1000" level="2"> … </Rule>

<Rule name="Zone Visit" limit="1000" level="3"> … </Rule>

 

Rule Life Cycle Management

Rule instances are automatically created and deleted based on directives in the rule declaration section. Create. The <Initialize> declaration controls when a new rule instance can be created. All rule instances are created in response to events. The <Initialize> section lists those events which can lead to new rule instances being created. Create a rule instance when an event of type GPSPosition arrives. <Initialize> <Assert> <Event> <base:XPath>sim:EventDef[@eventType="GPSPosition"]</base:XPath> </Event> </Assert> </Initialize> The <Key> declaration is the unique key for each rule instance. The key consists of one or multiple properties of the init events. <Key> <Property name="Vehicle"/> <Property name="Zone"/> </Key> The key declaration allows for easy detection of situations per something. For example, the above key declaration creates rule instances which are per vehicle and zone. So there would be rule instances created for each unique combination of vehicle and zone. Terminate. The <Terminate> section determines when a rule instance is deleted. The <Terminate> section consists of a number of directives which allows for creation of flexible termination strategies. A rule could for example be terminated when it has triggered three times, when it’s view is empty, on a specific date and time or when a particular event arrives. <Terminate> <Triggered count="3"/>

<ViewEmpty name="ZoneEntry"/>

<Time>2010-01-31T00:00:00Z</Time> <Assert> <Event> <base:XPath>sim:EventDef[@eventType="Location"]</base:XPath> </Event> </Assert> </Terminate>

Rule Triggering

A rule may trigger when one of two things happen; the situation is detected or it is automatically determined that the situation is undetectable in the future. Depending on the terminate declarations a rule can fire one or multiple times.

<Trigger> <SituationDetected/> <UndetectableSituation/> </Trigger>

 

When a rule triggers, its action part is executed. The action part creates an event and thus the visible result when a rule triggers is a new event, created according to your specification.

Trigger Reporting

Related to rule triggering is the reporting of the rule triggering. The triggering of the rule and reporting that triggering is separated in two stages. Thus it is possible to postpone the report of a rule triggering. For example, it is possible to report rule triggering, or multiple triggerings, each minute or when the rule has triggered five times or whichever comes first. It is also possible to limit the trigger reports so that only a specified number of reports are created. The most common case is to create rules whose triggering is reported only once. <Report>

<Every trigCount="10" timeframe="00:10:00"> <Max count="234"/> <AtTermination/>

</Report>

References

The rule also contains a reference to the rule’s view, situation and action. All references in Reakt are specified using XPath and executed against the SIM. The view reference contains a reference to a view declaration. The view provides the context in which the rule is executed

<Views> <ViewRef name="ZoneEntry"> <base:XPath>sim:ViewDef[@name='ZoneEntry']</base:XPath> </ViewRef> </Views>

 

The situation reference contains a reference to the situation which is evaluated in the context of the view.

<Situations> <SituationRef name="ZoneEntry"> <base:XPath>sim:SituationDef[@name="ZoneEntry"]</base:XPath> </SituationRef> </Situations>

The action reference contains a reference to the action part which generates a new event when the rule triggers.

<Actions> <SituationDetected situationName="ZoneEntry"> <ActionRef name="ZoneEntry" eventVisibility="external"> <base:XPath>sim:ActionDef[@name="ZoneEntry"]</base:XPath> </ActionRef> </SituationDetected> </Actions>

Complete Rule Example

Here is a complete example of a rule.

<Rule name="ZoneEntry" limit="1000" level="1">

<!– Create rule instance per Vehicle and Zone –>

<Key> <Property name="Vehicle"/> <Property name="Zone"/> </Key>

<!– Create a rule instance then the Location event arrives,

if one does not exists per the key above –>

<Initialize> <Assert> <Event> <XPath>EventDef[@eventType="Location"]</XPath> </Event> </Assert> </Initialize>

<!– Delete rule instance when rule has triggered 300 times, the view is

empty of at the specified time or when the Reset event arrives –>

<Terminate> <Triggered count="300"/>

<ViewEmpty name="ZoneEntry"/>

<Time>2010-01-31T00:00:00Z</Time>

<Assert> <Event> <XPath>EventDef[@eventType="Reset"]</XPath> </Event> </Assert> </Terminate>

<!– Report rule triggerings every 10th triggering of every tenth minute

or at rule termination, report a maximum of 234 triggerings –>

<Report>

<Every trigCount="10" timeframe="00:10:00"> <Max count="234"/> <AtTermination/> </Report>

<!– Trigger rule when situation is detected –>

<Trigger> <SituationDetected/> </Trigger>

<Views> <ViewRef name="ZoneEntry"> <XPath>ViewDef[@name='ZoneEntry']</XPath> </ViewRef> </Views>

<Situations> <SituationRef name="ZoneEntry"> <XPath>SituationDef[@name="ZoneEntry"]</XPath> </SituationRef> </Situations>

<Actions> <SituationDetected situationName="ZoneEntry"> <ActionRef name="ZoneEntry" eventVisibility="external"> <XPath>ActionDef[@name="ZoneEntry"]</XPath> </ActionRef> </SituationDetected> </Actions>

</Rule>

The View

The rule declaration contains a reference to a view. The view provides the context in which the rule is evaluated. The view keeps related events together and selects only interesting events from the inbound stream of events. Each rule instance contains its own private view instance. The view contains a collection of events. The view is kept updated based the stream of inbound events. The view provides a dynamically changing execution environment for the rule as each rule instance is executed in the context of its view. The view consists of a number of declarations, each specifying a property of an event that must be true in order for the event to be present in the view. All these declarations are evaluated to determine which events are present in the view of each rule instance. Example of properties available:

• The type view property is used to include events into the view only if they are of the specified type.
• The maxage view property defines the maximum age for each event in the view. You could for example use the age property to let each rule instance see only events from the past 10 minutes.
• The maxcount property defines the maximum number of events in the view. You could for example use this property to limit the view to contain only the last 100 received events.
• The match property defines which event properties must match in order for them to be included in the same view. You could for example require that the property CustomerID matches on all events in the view, thereby creating a view which contains events about the same customer.
• The assert property defines an XPath expression that must evaluate to true for all instances of a specified event type. The assert property could for example be used to compare a certain element in the event body with a constant.

<ViewDef name="ZoneEntry"> <Properties> <!– All events in the view must be of types InsideZone or OutsideZone –> <Type> <Event> <XPath>EventDef[@eventType="InsideZone"]</XPath> </Event> <Event> <XPath>EventDef[@eventType="OutsideZone"]</XPath> </Event> </Type> <!– All events in the view must be from the same vehicle –> <Match name="vehicle"> <Value> <Event> <XPath>EventDef[@eventType="OutsideZone"]</XPath> </Event> <Property name="Vehicle"/> </Value> <Value> <Event> <XPath>EventDef[@eventType="InsideZone"]</XPath> </Event> <Property name="Vehicle"/> </Value> </Match> <!– All events in the view must be from the same zone –> <Match name="zone"> <Value> <Event> <XPath>EventDef[@eventType="InsideZone"]</XPath> </Event> <Property name="Zone"/> </Value> <Value> <Event> <XPath>EventDef[@eventType="OutsideZone"]</XPath> </Event> <Property name="Zone"/> </Value> </Match> <!– Keep only the last 100 events in the view –> <MaxCount>100</MaxCount> </Properties> </ViewDef>

Situation

A situation describes an interesting combination of multiple events as they occur over time. The situation detection capability of Reakt have the following features.

1. Declarative – Complex situations are described using a high level declarative approach, avoiding complex procedural solutions using traditional programming languages.
2. Temporal support – Support for describing temporal relationships between events. Timing restrictions such as deadlines for detection can easily be added.
3. Extensible – The situation detector is designed to be extended with new task specific detector nodes.
4. Monitoring – The detector is designed to be monitored and thus the progress of situation detection can easily be tracked.

As a situation describes an interesting combination of multiple events as they occur over time, it must start somewhere. A situation starts developing when a specific initiator event occurs. This is the first event in what could possibly be a fully developed situation consisting of multiple events. This also means that a situation has an exact point in time where is starts developing, namely the timestamp of the initiator event. The situation develops over time and can either be detected at some point in time or automatically determined to be undetectable. A situation can develop over long periods of time and consist of a large number of contributing events. The ruleCore CEP Server provides a persistent rule state allowing rule instances and their situations to survive restarts and unexpected crashes. The progress of a situation, as it develops, can be followed by submitting a query event to the ruleCore CEP Server about the state of a particular rule’s situation detector. The degree of completion of a situation can be monitored to provide an early warning about a situation which could possibly be detected soon. For example, detect when a vehicle has been outside a zone for at least five seconds: <SituationDef name="OutsideZone5s"> <Detector> <After timeframe="00:00:05"> <Not> <EventPickup> <XPath>Views/View[@default='true']/Events/Event[@eventType="InsideZone"]<XPath> </EventPickup> </Not> </After> </Detector> </SituationDef> The above situation example is detected when the innermost event pickup does not pick up any events from the view five seconds after the rule instance was created. If InsideZone events have arrived into the view during those five seconds the situation is not detected. A situation is defined using a tree with detector nodes. The <Detector> node is the root of the situation detector. Each node can be either true or false depending on its semantics. One of the central detector nodes is the <EventPickup> node which selects events from the view. It selects events by executing a XPath query against the view. If the query returns any events from the view, the <EventPickup> node becomes true. The detector is evaluated each time it is possible that changes in the view or temporal conditions in detector nodes can affect the state of the detector.

Action

Actions are executed as the last part of a rule’s evaluation in response to a detected situation, or optionally when it is determined that a partially detected situation can’t be detected at any point in the future. An action is used to create a new event. The new event is called the reaction event and is used to notify external systems about a detected situation. Different events can be created for a detected situation and for undetectable situations. The outbound event created by the action is created using a XSLT stylesheet. The stylesheet is executed in the context of the rule and can access all the events that contributed to the detection of the situation. The stylesheet can also access the events present in the view at the time the action is evaluated.

Reakt is a trademark of Rulecore KB of Sweden. Rulecore is a registered trademark of Rulecore KB of Sweden

Just some quick thoughts of what I think will emerge in CEP in the next couple of years:

• A small tailor made language. Maybe looking something like Erlang or F#? 
• A formally defined model specially developed for event processing, think of the relational model as comparison, which provides a solid foundation.
• Small light weight libraries which contains the essentials needed for event processing. Used as a base for complete stand alone server of to implement CEP functionality in various kínds of applications.
• Services for processing live data targetting various domains. Like finance (already happening?), logistics, sensor networks, SCADA and that kind of things.
• More specialised CEP tools which solve one particular problem in the CEP domain. For example event enrichment or aggregation. One size does not fit all, as the database community already knows.
• Better GUI concepts, the ones today are too low level.

 

 

 

So, where’s CEP heading? 

That’s a question that I’m asked a couple of times each week. Those that know a bit about CEP are a bit worried that the current solutions have not caught on. Sure, in some sectors they have. For example in the capital markets there are now a decent number or installations from several vendors. But the real mainstream adaptation is not here if you ask the outsiders. We that work with CEP would really love to hear that CEP is now mainstream, but it is not there yet.

At least CEP have left the early adopter stage, the startups are now acquired or living a post-startup life. That is, those that are still alive.

It seems that those that made it as ‘independent’ vendors, have gone after a solution approach and sell more of a solution to a specific domain instead of pure technology.

Many of the bigger vendors have now some kind of CEP capability to offer. Most of them bought one of the smaller vendors a while ago. Notably Microsoft seem to have created StreamInsight inhouse. But from what I hear, none of the big vendors are selling lots of CEP licenses.

So, there is some activity both at the smaller and bigger vendors, but the real rush seems to be a couple of years away.

What I think is required is a better understanding of what CEP is and from that we need to create better tools, using better concepts. The current concepts in CEP looks to me as a first wave of something. Nothing wrong with them. But I would be very surprised if this would be the optimal way to do CEP.

We need better concepts, perhaps an underlying theory and some kind of collecting agreement on what it The Right Way to do CEP. That’s things that will take some time. Perhaps not a valid comparison, but it tool a while before SQL was The Way To Go in the data storage world.

But I’m sure there are interesting concepts coming. The current ideas seems all be rooted in the SQL or rules concepts. But I would love to see a new freash approach to CEP that treats CEP as an own field. I someone would do that I’m sure more creative ideas would emerge than reusing old concepts from other fields.

My be would be to use a small tailor made language as a base and then build some clever abstractions using that… The language has to be small and complete in order to be a good candidate to build future CEP solutions on. Flexible enough to give some headroom to build different kinds of product, but on the same base.

A good idea? A bad one? Do you have other ideas for CEP 2.0?

 

 

So, back from the summer vacation. This year I had a really long and nice time off and tried to do as little as possible.

I tried to look through my google alerts on CEP and there seems not be much exciting news in the world of CEP. The best news is that Event Processing in Action (the book) is ready. I tried to read the PDF, but the acrobat on Linux was so slow that it took a minute or so to turn the page. The pdf format seems to be seriously broken considering other problems like painfully slow search, slower than text search used to be in 1987.

Apart from the book there seems to be very little new cool stuff. The existing projects seems to progress with a steady pace. In some cases there seems to have been very little action lately, but it seems hard for both the commercial vendors and the open source projects to find a way to the minds of the mainstream. Not sure what this is a sign off, have to think about that…

My guess is that the next 12 months will be critical for most CEP vendors and open source projects. I have the feel that there’s a new wave of solutions and tools coming up. It would be interesting to see what v2.0 of the event processing concepts are.

Now time for some CEP googling to get updated…

 

The blog is taking a short break while I’m relaxing away from office… See you all in the spring..  in a couple of weeks…

 

 

Show your latest work here!

 

   http://2010.ruleml.org/ruleml-2010-challenge.html

Short summary: buy the book!

I just started to read "Event Processing in Action", the final version. It’s a brand new book coming to your favorite store any day.

I really like this book. It’s so good. This is the kind of book on event processing that I have been waiting for. I think the authors, Opher Etzion and Peter Niblett, really gets event processing.

Hopefully this book will teach many professionals and students that there is something more to event processing than just real-time processing of data. The fact that we are processing events and not data should be obvious as one reads this book.

I wish I had more time to read. But today Sweden is closed. Nobody works. So don’t even think of calling anybody here today. We are celebrating Midsummer’s Eve. One of those really old traditions up here in the north. If you live in a place where it’s dark and cold most of the year you surely appreciate a good party around the time for summer solstice. If I were younger, I would have been drunk all day. Now, it’s more like spending time with my wonderful family and good friends and their kids.

 

 

 

At least a few high-frequency traders have learned to make a killing by detecting the more simplistic algo strategies deployed by basic pension funds and mutual funds, buying the next stock the funds plan to buy, and then selling it to them at a higher price. This may not be illegal, but it’s almost certainly unfair to the funds’ investors. “It is increasingly clear that there are quite a number of high-frequency bandits in the high- frequency-trading community who pump up volume statistics, front-run investor orders, increase transaction costs, and hurt real liquidity,” David Weild, an adviser at Grant Thornton and a former vice chairman of Nasdaq, told me.  – http://www.theatlantic.com/magazine/archive/2010/07/monsters-in-the-market/8122/

 

When this is what the public reads about CEP, then I think we might have a minor image problem.

 

Microsofts entry into CEP space starts to look good and here’s some links (courtesy of Badrish Chandramouli from the StreamInsight team, thanks!) to those interested in reading more.

 

StreamInsight webpage (where you can find some more recent information such as white papers, help documents, examples, trial download):

• http://www.microsoft.com/sqlserver/2008/en/us/r2-complex-event.aspx

StreamInsight blog and public forum:

• http://blogs.msdn.com/streaminsight/
• http://social.msdn.microsoft.com/Forums/en-US/streaminsight

  

Research papers: 

The original CIDR paper (some of the stuff in this paper is out-dated, for example, the presence of transaction-time in the logical tables):

• http://www.cidrdb.org/cidr2007/papers/cidr07p42.pdf

 

A technical report about some more detailed implementation aspects:

• http://research.microsoft.com/pubs/70517/tr-2007-158.pdf

 

Two more recent papers from the StremInsight group: 

Recursive query processing in a streaming system with temporal algebra:

• http://research.microsoft.com/pubs/101567/FFP-2009-camera.pdf

 

A description of StreamInsight and behavioral targeted advertising:

• http://research.microsoft.com/pubs/101568/MicrosoftCEPDemo.pdf

 

 

Many new developments in the CEP space seems to be searching for other types of languages than the popular streaming extension to SQL. 

ETALIS is one of them, it is an open source engine for event processing based on the Logic Programming paradigm.

ETALIS engine implements a corresponding “ETALIS Language for Events” that is a rule-based declarative language. ETALIS is a research prototype created by Darko Anicic at FZI Forschungszentrum Informatik an der Universität Karlsruhe.

The ETALIS project is founded on novel algorithms, and implementing event-driven backward chaining rules. The rules enable derivation of complex events in a data-driven fashion. Apart from event processing, ETALIS features strong inference capabilities as well as easy integration with databases and transactions.

Future plans include to develop some unique features of ETALIS that can be particularly efficient or easy implemented in a *logic-based* EP.

For instance, event-driven workflows that are dynamically modified while executing can be one area of research. Workflow modifications, being done on-the-fly, need also to be verified at run time (as changes can disable workflow executions or make it infinite though workflows featured the finite termination property). This is one example where a logic approach can help. Event-driven dynamic workflows are useful in many areas including edBPM, Cloud Computing etc.

Another use of ETALIS logic approach in EP is related to event revision and world of database transactions. ETALIS can handle transactions in a logic framework. When transactions are mixed with event processing, event revision become necessary. Transaction are executed in the all-or-nothing fashion. Now events that are triggered during executions of some transactions, that later turned out to be unsuccessful, need to be retracted (in the same vain as transactions are rolled back). These events can be already used in building more complex events. This implies that also these complex events should be retracted too. ETALIS is using the logic to find out the consequences of events retractions on other more complex events.

It’s nice to seen the addition of another declarative CEP rule languate apart from the Reakt language implemented by the ruleCore CEP Server. Even though both languages are rule based and declarative it shows that ETALIS comes from a more formal and theorethically correct research world whereas the Reakt language was designed by software developers without too much concerns of formal correctness.

I will follow ETALIS with great interest as I think it could evolve into a really powerful language.

 

 

 

After listening for two days on incrediblty smart people, here’s what I think (at least at the moment) will happen with event processing in, say, 5-10 years…

Elements from sensor fusion, machine learning and other areas will creep into event processing and will start to get more and more specialized event processing solutions.

Most interesting problems in event processing will require that you can do things like "find all event patterns that deviate from the norm" from the inbound streams. That sort off implies that you need to elements from other fields.

The good news is that this will be rather easy, just take existing stuff and adjust it to the event processing context. No need to do re-invent things here…

 

4th International Workshop on Event-Driven Business Process Management

collocated with BPM 2010

Hoboken, NJ, USA from September 13-16, 2010

http://www.bpm2010.org/conference-events/workshops/edbpm10/

Introduction
————————–

The recently coined term "Event-Driven Business Process Management" (EDBPM)
is nowadays an enhancement of BPM by new concepts of Service Oriented
Architecture (SOA), Event Driven Architecture (EDA), Software as a Service
(SaaS), Business Activity Monitoring (BAM) and Complex Event Processing
(CEP). In this context BPM means a software platform which provides
companies the ability to model, manage, and optimize these processes for
significant gain. As an independent system, CEP is a parallel running
platform that analyses and processes events. The BPM- and the CEP-platform
correspond via events which are produced by the BPM-workflow engine and by
the – if distributed – IT services which are associated with the business
process steps. Also events coming from different event sources in different
forms can trigger a business process or influence the execution of a process
or a service, which can result in another event. Even more, the correlation
of these events in a particular context can be treated as a complex,
business level event, relevant for the execution of other business processes
or services. A business process – arbitrarily fine or coarse grained – can
be seen as a service again and can be "choreographed" with other business
processes or services, even between different enterprises and organizations.

Loosely coupled event-driven architecture for BPM provides significant
benefits:

    * Responsiveness. Events can occur at any time from any source and
processes respond to them immediately, whenever they happen and wherever
they happen.
    * Agility. New processes can be modeled, implemented, deployed, and
optimized more quickly in response to changing business requirements.
    * Flexibility. Processes can span heterogeneous platforms and
programming languages. Participating applications can be upgraded or changed
without breaking the process model.

Workshop Themes
————————–

Authors are invited to submit novel contributions in the prior described
problem domain.

    * Event-driven BPM: Concepts
          o Role of event processing in BPM
          o Business Events: types and representation
          o Event stream processing in business processes
          o Data- and event-driven business processes
          o Evaluation/ROI of event-driven BPM
          o Event-driven SOA
          o EDA and BPM
          o Real/time awareness in BPM
          o Context in BPM
    * Design-time CEP and BPM
          o Modelling languages, notations and methods for event-driven BPM
          o Event Patterns: Definition / Creation / Representation /
Learning
          o BPMN and event processing
          o Modelling unknown/similar events in business processes
          o Modelling events in human-oriented tasks
          o Semantics/Ontologies for event-driven BPM
          o Publish/subscription mechanism and process modelling
    * Run-time CEP and BPM
          o Event pattern detection
          o BPEL and event processing
          o Reasoning about unknown/similar events
          o Distributed event processing
          o Dynamic workflows
          o Ad-hoc workflows
    * Applications/Use cases for event-driven BPM
          o Event-driven monitoring/BAM
          o Event-driven SLA monitoring
          o Domains: Logistics, Automotive, …
          o Event processing and Internet of Services

Workshop Format
————————–

The Workshop is planned as a full-day event, including a keynote, paper
presentations, lightning talks, demos, posters, and a moderated, open
discussion with the clear goal of agreeing upon a research roadmap for
event-driven Business Process Management research, by taking into account
new challenges, described earlier.
A possible agenda:

    * 9:00 – 9:30 Opening and Keynote
    * 9:30 – 13:00 Paper Presentations
    * 13:00 – 14:00 Lunch
    * 14:00 – 15:00 Lightning Talks
    * 15:30 – 17:00 Moderated Community Discussion: A Roadmap for
Event-driven Business Process Management Research
    * 18:00 – 21:00 Poster and Demo Presentations and Get-Together

For the keynote, we aim at a high-profile speaker, who will give a rather
visionary view on the role of Future Internet for BPM and vice versa. For
the moderated community discussion, we will have senior experts from our
Program Committee and Experts from an industrial background. A clear
objective of that discussion is to yield a first draft of a respective
research agenda.

Important Dates
————————–

Deadline paper submissions:  21 May 2010

Notification of acceptance: 30 June 2010

Camera-ready papers: 25 July 2010

Workshops: 13 September 2010

Submission
————————–

The following types of submission are solicited:

    * Long paper submissions, describing substantial contributions of novel
ongoing work. Long papers should be at most 12 pages long.
    * Short paper submissions, describing work in progress. These papers
should be at most 6 pages long.
    * Use case submission, describing results from an edBPM use case. These
papers should be at most 4 pages long.

Papers should be submitted in the new LNBIP format
(http://www.springer.com/computer/lncs?SGWID=0-164-7-487211-0). Papers have
to present original research contributions not concurrently submitted
elsewhere. The title page must contain a short abstract, a classification of
the topics covered, preferably using the list of topics above, and an
indication of the submission category (Long Paper/ Short Paper).

For submission, please visit
http://www.easychair.org/conferences/?conf=edbpm10.

Organizing Committee
————————–

Nenad Stojanovic
FZI – Research Center for Information Technologies at the University of
Karlsruhe, Germany.
Haid-und-Neu-Str. 10-14
D-76131 Karlsruhe, Germany
nstojano (at) fzi.de
URI: http://www.fzi.de/ipe/mitarbeiter.php?id=483

Rainer von Ammon
CITT Regensburg/Germany
Konrad-Adenauerallee 30
D-93051 Regensburg, Germany
rainer.ammon (at) citt-online.com

Opher Etzion
IBM Research Lab in Haifa
OPHER (at) il.ibm.com

Adrian Paschke
Corporate Semantic Web, Free University Berlin, Germany and RuleML Inc.,
Canada
AG-CSW (Corporate Semantic Web)
Institute for Computer Sciences
Free University Berlin
Königin-Luise-Str 24/26
14195 Berlin, Germany
paschke (at) inf-fu-berlin.de

Program Committee
————————–

(see Website)
 

SAP acquires Sybase

Given that big companies buy other big companies left and right, this might not even be big news even if it is a big deal. For us event processing aficionados this now means that SAP owns Coral8 and Aleri code base. I would be surprised if this is not the end of those products as standalone CEP products…

 

I have not looked at what StreamBase do for a while now. But as you might read in my previous blog entry I found some financial information about StreamBase and thought I’d take a quick look to see what they are up to. Or, what can you do with $48,000,000 USD?

A new idea they have is the component exchange. I like. This is something all infrastructure platforms need. A place where you can find plug-in components to your favorite tool. For example the Eclipse platform have thousands of plug-ins available on their marketplace.

So I really like the idea. But 18 components? That’s sadly not much of an exchange. It seems that there’s no third party components and StreamBase don’t seem to put much into their own exchange either. But still a good idea.

Another good place to check what’s going on is in the documentation section of the StreamBase site. They have good documentation and a quick check reveals that there are updates published with regular intervals. While browsing the "what’s new" section I smell Financial Markets all over the place. If I would be looking at this for the first time I would put this into the "tools for traders and other people on Wall St." category and not into the ‘generic CEP tool’

I did not bother to download, which I think you still can (a gold star to StreamBase for that) , as there were not enough interesting stuff to look at. Maybe I would be more excited if I were working in the capital markets.

 

It seems that StreamBase needs more money: www.sec.gov/Archives/edgar/data/1315537/000131553710000003/xslFormDX01/primary_doc.xml 

I understand that there’s 5.5M involved. Rest is sort of MBA talk, anyone care to translate this to us mere mortals?

Last year they got $6M if I remember correctly. Does this mean that StreamBase burns about 5-6M more than they earn per year?

 

Edit: 

Following the link posted  by Marc in the comments section I think (noticing that I don’t really understand what I’m looking at) StreamBase have got about $48M from investors since 05. Is that lots of money compared to what they have achieved?

But having raised $48M (daydreaming on how much we could do with that sum  ), do that mean that they are spending more on development than customers are willing to pay for the software? Or do software companies have other reasons to take money from investors?

At least some investors think that a CEP company is a good place to put a $48M in, that must be a good sign!

As you might have noticed, I have not had any inspiration to write in my blog lately. This is partly due to a high work load over here at ruleCore HQ. Lots of small things to fix before a new release and the GUI stuff just sucks up all my time.

Also, there seems not much interesting happening in the world of CEP. StreamBase have turned into a financial solutions provider just as I predicted. IBM, Tibco and Progress seems to work on their offerings but there’s not much excitement but the steady flow of gradual improvements to their products. Which is a good thing. But not much to write about.

One inspiring news is the the new Event Processing in Action book by Opher and Peter is more or less ready. I knew it was a good idea to suggest Opher as a author to Manning. I did some non-rocket science analysis and decided rather quickly that writing a book will steal way too much time from my other work. Reading what Opher writes, I was right… And these two guys are way better authors than me anyway. Maybe I’ll return with a more practical hands on companion books some day which can be read after you got the theory in their book. Or hopefully someone else beats me to it, saving me the trouble…

So if anyone have some more inspiring news about the state of the event processing world, please post them!! 

Another good news is that I’m going to Dagstuhl next month (anyone wanna share a cab from Turkismuhle?) , that ought to be great fun. The last time I did not have the guts to give a presentation. But this time I think I will do, at least a short one. With that kind of audience, there’s not much that I can say that they don’t already know. But I realized that there’s at least one topic that I’m better at than they are. [insert cliffhanger]. As Opher would say, more later (in Dagstuhl)…

 

 

 

Introduction

The Rulecore CEP Language – Reakt! – When you need to react immediately.

Reakt is a language for complex event processing, or CEP for short. It is designed to provide a powerful and productive tool for developers allowing them to quickly create solutions for detection oriented CEP scenarios.

Reakt is not a traditional programming language. Reakt is a business situation description language. By using a declarative approach it allows the developer to concentrate on what kind of situations to detect and not how

The design goal of Reakt is to provide a way for organisations to react immediately to critical situations. It achieves this goal by finding related events from streams of inbound events. What events a critical situation consists of, and how the events are related, are defined by the user using Reakt. Reakt allows the user to specify conditions and temporal relationships which must hold in order for the events to be considered a noteworthy situation. When the situation is detected by Reakt, an user defined real-time report is created which provides actionable information about the situation.

Allowing the enterprise to react immediately.

This short white paper gives an informal description of the language and explains the different language elements using examples instead of formal definitions.

Overview

Rulecore Reakt is a small domain specific complex event processing language designed for detecting situations by observing streams of events. A situation is a formal description of a combination, a pattern, of events as they happen over time along with optional restrictions among the events. A simple example of situations that you might want to detect could be

"A vehicle has visited zone A more than 10 times during the last hour"

or

"Event A comes before event B, unless event C has been seen"

Main Reakt language features:

 

• Continuous Evaluation – Designed to operate on live streams of events as they happen.
• Declarative – Rules and other items are specified using a high-level declarative approach.
• Modular – Rules are defined using modular and re-usable building blocks.
• Temporal – Handling of time is an integral part of Reakt. Timing restrictions and other temporal conditions can be stated using a number of declarations.
• Location Aware – Reakt includes built in support to understand location information and other geographical objects such as zones. 
• Implementation Independent – The elements of Reakt are defined using standard XML and are implementation independent. Different implementations can be created in order to provide specialized execution environments.
• Automatic Life Cycle Management – Rule instances are created and deleted automatically in response to events. Rule instances are analyzed and deleted if their situation is determined to be undetectable.
• Dynamic Context Management – Each rule instance maintains its own virtual view into the incoming event stream. The view provides a window into the incoming event stream in order to create a dynamic context for rule evaluation.
• Separation of Concerns – Reakt uses reusable blocks for;
  • Selection of events from the inbound stream
  • Rule life cycle management
  • Situation definition
  • Rule trigger reporting

 

The design goals of Reakt is to provide a language which allows the user to describe complex situations consisting of multiple events with non trivial temporal and geospatial relationships. The language is declarative. Lacking any procedural elements provides a base for formal verification and automated optimization and analysis of the language.

A design assumption made early in the language design was that is is likely that machines will use Reakt to create and modify rules more frequently than humans. The rate at which new rules need to be created and old ones modified is assumed to be high. Thus the language was designed to be easily understandable by machines and at the same time being reasonably human readable for system developers.

The current version of Reakt uses XML for its syntax. XML was chosen to take advantage as much as possible of existing knowledge among the developers, known standards and existing tools for generating and parsing. It would also be possible to create alternative syntaxes for Reakt. For example to create a more compact human friendly representation for business users.

The Reakt language was designed to support execution environments which fits into an architectural style called Event Driven Architecture (EDA). In order be a natural building block in an EDA, the ruleCore Reakt language and its supporting execution environment, the ruleCore CEP Server, was designed to be completely event driven. This purely event driven approach allows all systems in a distributed messaging architecture to integrate with ruleCore using simple events as their only communication mechanism.

 

The System Information Model

The System Information Model, or SIM for short, is both the name of the information model used by ruleCore CEP Server where all information about different items (rules, situations etc.) can be found and the name of the run-time data structure modelled after the familiar DOM model, commonly used to represent information originating from XML. The SIM is part of the ruleCore CEP Server implementation and not the language definition. But as all the language elements declared are executed against the SIM it is described here briefly. Different implementations can implement varying information models.

The SIM is used for various purposes, for example:

• Listing available definitions – All definitions of items such as rules, views, zones, entities, situations and actions can be found in the SIM. Definitions are created using the systems events, for example AddRuleDef, AddSituationDef and AddActionDef.
• Listing user defined events – Before an event can be sent into ruleCore CEP Server, it must be defined using the AddEventDef system event. The SIM contains all user defined events
• List of rule instances – When a rule instance is created, it’s state and internal data structures can be monitored by querying the SIM. The ruleCore CEP Server provides a query mechanism using XPath. But other implementations might query the SIM using other methods.
• Monitoring situation progress – As a situation develops, its progress can be monitored by querying the SIM.
• Examine view contents – Each rule instance is executed in an unique context of events called the event stream view. The view contains a window into the inbound stream of events and contains only semantically related events.
• Referencing items – When an item such as a rule needs to reference other items it is done using references through the SIM.
• Component Specific Data – Each plug-in component that extends the engine creates an entry in the SIM and may add any component specific information.

 

The SIM consists internally of dynamic data structures which are updated as needed when the internal state of the ruleCore CEP Server changes in response to inbound events. These data structures have an external representation which is the user’s view of the SIM.

The external representation of the SIM can be treated as a DOM document which can be queried using XPath and transformed using XSLT. XPath queries can be executed against the SIM using the SysInfoQuery system event.

If is also useful to note that the ruleCore CEP Server uses events for all tasks. Special pre-defined system events are used to create rules and other items in ruleCore CEP Server in addition to monitor the run-time state of the rule evaluation.

 

Structure of a Rule

The main language element in Reakt is the rule. Reakt uses a class of rules called reactive rules to provide its situation detection capabilities. Reactive rules should not to be confused with production rules or other types of rules which must be explicitly evaluated. A rule in Reakt is never explicitly evaluated, its evaluation must be automatically managed by the execution environment in response to inbound events.

A rule consists of a number of elements, the main ones being:

• Initialization directives - Specifies which events can lead to the creation of a rule instance
• Termination directives – Specifies what causes a rule instance to be deleted.
• The rule level – Specifies on what level in the rule hierarchy instances should be created.
• Trigger Directives – Controls how and when to report Rule triggering.
  
• The View – A reference to a view, providing execution context for the rule.
• The Situation – A reference to a situation to track and detect situations.
• The Action – A reference to an action, which will generate an event when
  • the situation is detected, or
  • if the situation is determined to be undetectable.

General Rule Attributes

Each rule has a name and a attribute which can be used limit the number of rule instances that can be created of a particular rule. A common usage of the limit is to create a singleton rule, of which only one single instance can exist.

Rules at every level can publish their trigger events externally so that they are visible outside of Reakt.

All rule declarations specifies a level. Every rule instance is created on a specific level in a hierarchy of rules. Level zero is the level on which inbound events arrive. Thus the lowest level for a rule is 1 (one). A rule will only see events from a level directly below it.

For example, three levels of rules where the rule at level one sees the inbound events:

<Rule name="New Location" limit="1000" level="1"> … </Rule>

<Rule name="Zone Entry" limit="1000" level="2"> … </Rule>
<Rule name="Zone Exit" limit="1000" level="2"> … </Rule>

<Rule name="Zone Visit" limit="1000" level="3"> … </Rule>

 

Rule Life Cycle Management

Rule instances are automatically created and deleted based on directives in the rule declaration section.

The <Initialize> declaration controls when a new rule instance can be created. All rule instances are created in response to events. The <Initialize> section lists those events which can lead to new rule instances being created.

Create a rule instance when an event of type GPSPosition arrives.


    <Initialize>
        <Assert>
            <Event>
                <base:XPath>sim:EventDef[@eventType="GPSPosition"]</base:XPath>
            </Event>
        </Assert>
    </Initialize>

The <Key> declaration is the unique key for each rule instance. The key consists of one or multiple properties of the init events.

    <Key>
        <Property name="Vehicle"/>
        <Property name="Zone"/>
    </Key>

The key declaration allows for easy detection of situations per something

 

For example, the above key declaration creates rule instances which are per vehicle and zone. So there would be rule instances created for each unique combination of vehicle and zone.

The <Terminate> section determines when a rule instance is deleted. The <Terminate> section consists of a number of directives which allows for creation of flexible termination strategies. A rule could for example be terminated when it has triggered three times, when it’s view is empty, on a specific date and time or when a particular event arrives.

 
    <Terminate>
        <Triggered count="3"/> 

        <ViewEmpty name="ZoneEntry"/>

        <Time>2010-01-31T00:00:00Z</Time>
        <Assert>
            <Event>
                <base:XPath>sim:EventDef[@eventType="Location"]</base:XPath>
            </Event>
        </Assert>
    </Terminate>

 

Rule Triggering

A rule may trigger when one of two things happen; the situation is detected or it is automatically determined that the situation is undetectable in the future. Depending on the terminate declarations a rule can fire one or multiple times.

   

    <Trigger>
        <SituationDetected/>
        <UndetectableSituation/>
    </Trigger>

 

When a rule triggers, its action part is executed. The action part creates an event and thus the visible result when a rule triggers is a new event, created according to your specification.

 

Trigger Reporting

Related to rule triggering is the reporting of the rule triggering. The triggering of the rule and reporting that triggering is separated in two stages. Thus it is possible to postpone the report of a rule triggering. For example, it is possible to report rule triggering, or multiple triggerings, each minute or when the rule has triggered five times or whichever comes first. It is also possible to limit the trigger reports so that only a specified number of reports are created. The most common case is to create rules whose triggering is reported only once.

   <Report>        

        <Every trigCount="10" timeframe="00:10:00">
        <Max count="234"/>
        <AtTermination/>
    </Report>

 

References

The rule also contains a reference to the rule’s view, situation and action. All references in Reakt are specified using XPath and executed against the SIM.

The view reference contains a reference to a view declaration. The view provides the context in which the rule is executed

 

    <Views>
        <ViewRef name="ZoneEntry">
            <base:XPath>sim:ViewDef[@name='ZoneEntry']</base:XPath>
        </ViewRef>
    </Views>

 

The situation reference contains a reference to the situation which is evaluated in the context of the view.

    <Situations>
        <SituationRef name="ZoneEntry">
            <base:XPath>sim:SituationDef[@name="ZoneEntry"]</base:XPath>
        </SituationRef>
    </Situations>

The action reference contains a reference to the action part which generates a new event when the rule triggers.

   

    <Actions>
        <SituationDetected situationName="ZoneEntry">
            <ActionRef name="ZoneEntry" eventVisibility="external">
                <base:XPath>sim:ActionDef[@name="ZoneEntry"]</base:XPath>
            </ActionRef>
        </SituationDetected>
    </Actions>

 

Complete Rule Example

Here is a complete example of a rule.

<Rule name="ZoneEntry" limit="1000" level="1">

    <!– Create rule instance per Vehicle and Zone –>

    <Key>
        <Property name="Vehicle"/>
        <Property name="Zone"/>
    </Key>

    <!– Create a rule instance then the Location event arrives,

         if one does not exists per the key above –>

    <Initialize>
        <Assert>
            <Event>
                <XPath>EventDef[@eventType="Location"]</XPath>
            </Event>
        </Assert>
    </Initialize>

    <!– Delete rule instance when rule has triggered 300 times, the view is

         empty of at the specified time or when the Reset event arrives –>

    <Terminate>
        <Triggered count="300"/> 

        <ViewEmpty name="ZoneEntry"/>

        <Time>2010-01-31T00:00:00Z</Time>

        <Assert>
            <Event>
                <XPath>EventDef[@eventType="Reset"]</XPath>
            </Event>
        </Assert>
    </Terminate>

    <!– Report rule triggerings every 10th triggering of every tenth minute

         or at rule termination, report a maximum of 234 triggerings –>

    <Report>        

        <Every trigCount="10" timeframe="00:10:00">
        <Max count="234"/>
        <AtTermination/>
    </Report>

    <!– Trigger rule when situation is detected –>

    <Trigger>
        <SituationDetected/>
    </Trigger> 

    <Views>
        <ViewRef name="ZoneEntry">
            <XPath>ViewDef[@name='ZoneEntry']</XPath>
        </ViewRef>
    </Views>

    <Situations>
        <SituationRef name="ZoneEntry">
            <XPath>SituationDef[@name="ZoneEntry"]</XPath>
        </SituationRef>
    </Situations>
    

    <Actions>
        <SituationDetected situationName="ZoneEntry">
            <ActionRef name="ZoneEntry" eventVisibility="external">
                <XPath>ActionDef[@name="ZoneEntry"]</XPath>
            </ActionRef>
        </SituationDetected>
    </Actions>

</Rule>

 

The View

The rule declaration contains a reference to a view. The view provides the context in which the rule is evaluated. The view keeps related events together and selects only interesting events from the inbound stream of events.

Each rule instance contains its own private view instance. The view contains a collection of events. The view is kept updated based the stream of inbound events. The view provides a dynamically changing execution environment for the rule as each rule instance is executed in the context of its view.

The view consists of a number of declarations, each specifying a property of an event that must be true in order for the event to be present in the view.

All these declarations are evaluated to determine which events are present in the view of each rule instance.

Example of properties available:

 

• The type view property is used to include events into the view only if they are of the specified type.
• The maxage view property defines the maximum age for each event in the view. You could for example use the age property to let each rule instance see only events from the past 10 minutes.
• The maxcount property defines the maximum number of events in the view. You could for example use this property to limit the view to contain only the last 100 received events.
• The match property defines which event properties must match in order for them to be included in the same view. You could for example require that the property CustomerID matches on all events in the view, thereby creating a view which contains events about the same customer.
• The assert property defines an XPath expression that must evaluate to true for all instances of a specified event type. The assert property could for example be used to compare a certain element in the event body with a constant.

<ViewDef name="ZoneEntry"> 

  <Properties>

 

    <!– All events in the view must be of types InsideZone or OutsideZone –>

    <Type>

      <Event>

        <XPath>EventDef[@eventType="InsideZone"]</XPath>

      </Event>

      <Event>

        <XPath>EventDef[@eventType="OutsideZone"]</XPath>

      </Event>

    </Type>

    <!– All events in the view must be from the same vehicle –>

    <Match name="vehicle">

      <Value>

        <Event>

          <XPath>EventDef[@eventType="OutsideZone"]</XPath>

        </Event>

        <Property name="Vehicle"/>

      </Value>

      <Value>

        <Event>

          <XPath>EventDef[@eventType="InsideZone"]</XPath>

        </Event>

        <Property name="Vehicle"/>

      </Value>

    </Match>

    <!– All events in the view must be from the same zone –>

    <Match name="zone">

      <Value>

        <Event>

          <XPath>EventDef[@eventType="InsideZone"]</XPath>

        </Event>

        <Property name="Zone"/>

      </Value>

      <Value>

        <Event>

          <XPath>EventDef[@eventType="OutsideZone"]</XPath>

        </Event>

        <Property name="Zone"/>

      </Value>

    </Match>

    <!– Keep only the last 100 events in the view –>

    <MaxCount>100</MaxCount>

  </Properties>

</ViewDef>

 

Situation

A situation describes an interesting combination of multiple events as they occur over time.

The situation detection capability of Reakt have the following features.

1. Declarative – Complex situations are described using a high level declarative approach, avoiding complex procedural solutions using traditional programming languages.
2. Temporal support – Support for describing temporal relationships between events. Timing restrictions such as deadlines for detection can easily be added.
3. Extensible – The situation detector is designed to be extended with new task specific detector nodes.
4. Monitoring – The detector is designed to be monitored and thus the progress of situation detection can easily be tracked.

 

As a situation describes an interesting combination of multiple events as they occur over time, it must start somewhere. A situation starts developing when a specific initiator event occurs. This is the first event in what could possibly be a fully developed situation consisting of multiple events. This also means that a situation has an exact point in time where is starts developing, namely the timestamp of the initiator event.

The situation develops over time and can either be detected at some point in time or automatically determined to be undetectable. A situation can develop over long periods of time and consist of a large number of contributing events. The ruleCore CEP Server provides a persistent rule state allowing rule instances and their situations to survive restarts and unexpected crashes.

The progress of a situation, as it develops, can be followed by submitting a query event to the ruleCore CEP Server about the state of a particular rule’s situation detector. The degree of completion of a situation can be monitored to provide an early warning about a situation which could possibly be detected soon.

For example, detect when a vehicle has been outside a zone for at least five seconds:

<SituationDef name="OutsideZone5s">

  <Detector>

    <After timeframe="00:00:05">

      <Not>

        <EventPickup>

          <XPath>Views/View[@default='true']/Events/Event[@eventType="InsideZone"]<XPath>

        </EventPickup>

      </Not>

    </After>

  </Detector>

</SituationDef>

The above situation example is detected when the innermost event pickup does not pick up any events from the view five seconds after the rule instance was created. If InsideZone events have arrived into the view during those five seconds the situation is not detected.

A situation is defined using a tree with detector nodes. The <Detector> node is the root of the situation detector. Each node can be either true or false depending on its semantics.

One of the central detector nodes is the <EventPickup> node which selects events from the view. It selects events by executing a XPath query against the view. If the query returns any events from the view, the <EventPickup> node becomes true.

The detector is evaluated each time it is possible that changes in the view or temporal conditions in detector nodes can affect the state of the detector.

 

Action

Actions are executed as the last part of a rule’s evaluation in response to a detected situation, or optionally when it is determined that a partially detected situation can’t be detected at any point in the future.

An action is used to create a new event. The new event is called the reaction event and is used to notify external systems about a detected situation. Different events can be created for a detected situation and for undetectable situations. 

The outbound event created by the action is created using a XSLT stylesheet. The stylesheet is executed in the context of the rule and can access all the events that contributed to the detection of the situation. The stylesheet can also access the events present in the view at the time the action is evaluated.

 

Reakt is a trademark of Rulecore KB of Sweden.

 

DataMentors is a company I was not aware of previously. Apparently they are branding some of their products as CEP, with some interesting claims:

GEMS technology is the first commercial event pattern language that includes an operator to express causality in the moment. This new real-time CEP technology enables companies to simultaneously examine historical, current, and predictive data.

Not sure what causality in the moment is. But I’m pretty sure that several other vendors would not agree them being first commercial to do anything with causality.

 

Furthermore:

GEMS ability to instantly compile data from thousands of disparate data-streams and conduct nesting pattern evaluations quickly and accurately is dependent on DataMentors’ software. DataMentors’ processing speeds are necessary to enable the GEMS technology to deliver insights and models instantly to end users, which result in higher levels of accurately predicting future events.

which a description one would expect from a company doing CEP.  Once again interesting to see that another company chooses to position their products as CEP-something. It’s probably not done on random, so they must think that there is something to gain from calling themselves CEP and not something else. Which I like, as it shows that CEP is, slowly, gaining some traction.

See rest here

 

 

I have tried to google around for something about the Sybase/Aleri deal. There is lots of the normal "information" which is corporate blahblabla without any substance. But I found one piece of info that actually have something interesting in it about what Sybase might have in mind for Aleri:

"In addition to slashing latency, CEP can also help to reduce warehouse storage requirements and help tame problems that were heretofore too costly to solve. By conducting analysis of a stream as it arrives, much of needed analysis and aggregation can be done without persisting details to the warehouse at all. If your data has no retention requirements, other than your analysis needs, you may be able to use CEP to greatly reduce your warehouse costs."  – http://www.beyeblogs.com/sybaseiq/archive/2010/02/inastreamofgood.php

Analytic BI + nifty real-time front-end for the datawarehousing guys would seem like a pretty good use for the Coral8 engine, I just assume that they will use the Coral8 engine and throw away the rest.

Apart from that, I don’t think this deal will turn into anything significant for the CEP community. Sybase will add some real-time features and give some nice shiny toys for their capital market customers that can be used to game the markets even further. But I don’t think they will have a generic CEP engine to offer in the future.

 

"SenSage 4 includes a highly scalable real-time correlation engine, the Scalable Alert Server (SAS). Correlation is based on the application of threshold and scenario-based rules against multi-source, real-time event streams" – http://www.sensage.com/products/sensage-40.php

I wonder how long it takes before this product carries the CEP label…