Marco on CEP

I’ll be away for the keyboard for some well earned rest during the summer. Meanwhile, here’s a summer CEP quiz:

When detecting nested sequences of events, how many different types of sequences do we need to completely be able to express all kinds of sequence semantics?

Nested sequences are sequences where the items of the sequence can consist of other sequences. Something like:

Sequence( Sequence(E1 or E2), Sequence(E2 or E3) )

is nested…

Answers can be supplied as comments to this post and I’ll give you the right answer in a number of weeks.

Complex Event Processing

In Event Processing (EP) and it’s sub-disciplines CEP and ESP there’s a very important concept called context. For some reason the IT world chooses as generic names as possible to all kinds of important concepts which would benefit from a more specific and inspiring name.

The context in is one of the defining features of EP. It’s something you need to understand and be friends with in order to really appreciate the power of EP.

Unfortunately it seems that there are some confusion about what a context is. Which I understand, being such a generic term and a concept used in various places outside CEP.

The event processing context is, in short, the set of events that you see when executing your CEP rules, procedural code or your streaming version of SQL.

The most trivial context is the event that you just received, for simplicity just assume that you receive one event at the time. The concept of "events that happen at the same time" is too confusing anyway.

A more complex context can be constructed by keeping any number of the events that you received. For example the last 10, or the events received during the last 10 minutes. Commonly this kind of context is referred to as a window in many products.

An even more advanced context can be constructed by applying different kinds of conditions to events in order to determine if they should be included in the context. For example, that they should be emitted by a car which is currently in Stockholm. You could also use various types of event relationships to determine if events should be included in the context.

The context can include events which you receive as time goes by, you select some of these to be included in your context. The context can also contain historical events which have occurred before you started your processing. As time goes by, events are inserted and removed from the context.

If you are in a hurry, read only this: an event window is not the same thing as a context. An event window is a type of context!

To decide what events should be included in the context, a large number of factors can be considered. Depending on your type of CEP system you might find context management which takes into account:

• Semantics
• Location
• Event causality
• Temporal relationships
• Historical events
• State
• Reference data

and so on…

A rich context could for example be, all events that:

1. Are notifications about new GPS locations of entities of type heavy vehicle, and
2. The events occurs during working hours, and
3. The vehicle is located in Construction Zone X. and
4. The cargo of the vehicle weights over 40 tons, and
5. It travels on a road classified as temporary, and
6. The driver is listening to Iron Maiden

Well, ok. I might made the last one up. But I hope you get the general idea; All of the above must be true at any point of time for the event to be included in the context.

Also note that in this case we would probably have one context per vehicle.

In the context you have a bunch of events which are all related somehow, possible in very complex ways. By using these events as an input to whatever processing your system is doing, your processing logic will be much simpler than if you were operating on the raw stream of events.

So, rich context management avoids processing the raw stream of events and gives you a easy way to select only those events in which you are interested.

Complex Event Processing

As a part of an interesting discussion in our plans to take over the world^H^H^H^H^H^H to understand what our clients think is difficult with CEP I ended up in doing a improvised presentation on what EDA and CEP is good at.

 I ended up with the following bullets which I made up along the way:

Detect when a complex state change happens.

Here I tried to do an analogy to a more traditional RDBMS system. How would you detect that a number of tables have been changed over time in a very specific way? I hoped that this would give a good mental model on what CEP is trying to achieve.

In database terms, it rather messy to write the SQL and/or procedural code to detect changes to multiple tables. All agreed that things get even more messy when there’s timing constraints involved. Suggestions ranged from triggers to flashback queries but even the hardcore DBAs and database coders agreed that they would rather avoid looking at these type of queries after a couple of years – "What the hell did I do here??"

Drive calculations

 Another good example of what can be done with CEP is to drive online calculations. Rapidly changing data fed to an algorithm which produces a constant stream of output values. I showed some example of Coral8, StreamBase to get an overall understanding of how these could be used in the capital markets. As no one worked with anything even remotely related to the world of finance we sortof had to invent use-cases.

Instead driving calculation of business metrics was agreed to be one of the major areas for this type of processing as these can be found in more or less all companies. There’s always a dashboard which needs to be as close to real-time as possible. For example "Tempo of production during the latest 13 minutes."

Event warehouse

A database full of events can be used to do some really interesting analytics. If we have long sequences of events which can be related to each other using different kinds of correlation rules we can find lots of interesting things. Here I tried to make people understand that there’s nothing magic about CEP. If you know how to do data mining you can apply the same techniques to events. Events are just a special type of data from this point of view. Just call the events for data structures of tuples (which actually some CEP vendors do) and you can pretend that you are processing normal data.

Simulation, what-if?

If everything is driven by events then we can save the stream of events a reply it. This is like being able to tweak the history and do it all over again to understand how we should have done in order to avoid certain situations. We can also replay history and try to build rules which detect new types of situations.

Control systems

 High performance processing of sensor data is something that has been done for ages. Google for SCADA and you’ll find tons of systems doing that. The ‘problem’ is that all these systems are custom built from ground up. There has been missing critical infrastructure software to build upon. When building a traditional dataprocessing system one normally starts with a database and perhaps an application server. These kind of system components have been missing for real-time processing of data. Within CEP, there’s now a category of CEP software which is suitable for this.

The response to this "CEP thing" was mixed. A couple of persons realized the value of CEP at once and a large portion remained very sceptical. Which partly validates my theory that CEP is not for everyone (no it will not replace databases even though some CEP vendors say (said) so). A good subset of business problems are suited for CEP and that’s enough to keep at last a couple of CEP vendors running.

Complex Event Processing