Marco on CEP

I was asked a time ago why CEP (Complex Event Processing) is so difficult?

Or rather, why is it so hard to build your own generic and functionally complete CEP server?

I think one of the answers is that you need to do two conflicting things:

1. Process events in memory only
2. Store events and state in persistent storage

 

You need to do 1. because of your need for speed.

You need to do 2. in order to build a reliable and predictable system.

 

The magic blend of these two are required for most CEP systems. It’s here where you need better than average architects and developers to produce anything usable.

 

Example – Why..

Just a simple example to understand why you need both. Say you are looking for a sequence of events:

     A,B,C,D,E

You expect this sequence to occur during a seven day period. When you see event A, you start looking for the rest of the events.

Lets say you saw events B and C after a couple of days. Then the server is re-booted suddenly, just quickly, but your system is killed unexpecedly.

When your event processor is brought back online again you need to remember that you have seen A, B and C and are looking for the next event in the sequence which is D.

This forces you to save state information in some kind of persistent storage (database). If you are looking for thousands and thousands of long event sequences you will quickly get really bad performance if you keep writing state information to the database constantly.

 

When You Don’t Need Persistence

There’s a special type of event processing where you don’t need persistent storage

A class of CEP solutions where the problems can best be described as data driven algorithms do rarely require persistence. Tools like Coral8/Aleri, Esper, StreamBase are commonly used to do this kind of processing. This type of processing, commonly done using some kind of extension to the SQL concept. Typical uses that I have seen have mostly simple queries which constantly produce output. If there are patterns in the queries they tend to be simple and short lived.

If you only need this kind of state-less processing then you can get really high performance instead of systems like ruleCore where the persistence is a fundamental concept and can’t really be switched off.

If you are shopping for a complete event processing platform you should at least evaluate everything with full persistence switched on. Provided that the CEP system provides real usable persistence, not all do…

 

 

Complex Event Processing Add new tag

So this is interesting. If I understand this correctly, Sybase now has the same engine as the merged Aleri+Coral8. Or at least one of the Aleri’s engines, I think they have two engines which they have plans to support.

It will be interesting how Aleri handles competetion from Sybase as they could now, at least theorethically, build the same kind of solutions.

So first Coral8 sells it’s source code to Sybase and then Aleri buys Coral8. Interesting times…

"Sybase is entering the Complex Event Processing business with the introduction of Sybase CEP, which is being offered – initially at least – as an option to its RAP – The Trading Edition capital markets analytics platform. The product results from the vendor’s licensing of the source code for Coral8’s CEP engine," – from here

 

 

Complex Event Processing

So it seems that there’s now one less company in the event processing space.

Aleri bought Coral8, in what is called by the vendors as a merge. One company less in CEP space is not necessarily a bad thing as the resulting Aleri is much larger. Which can be a good thing.

If someone had asked me last week which companies in the CEP space would merge – Aleri and Coral8 would been my last guess. My best guess would have been a dashboard focused BAM company acquiring an event processing company to get a good CEP processing engine.

This is a total surprise to me. I like both these companies and their approach to CEP. Both places have lots of nice people working for them too.

But I really don’t get this merger. Someone somewhere has a plan which they fail to communicate. I read the official merger announcements and read a lot of comments on blogs and on the web. But I still don’t get the point, and I would think I know more of these companies and their technology than the average Joe Programmer out there.

I’m pretty sure that there were a lot of intelligent people doing lots of thinking before they agreed on this one. So my guess is that there’s a clever plan for all this somewhere. I just don’t get it.

What I see is (was) two companies with similar approach to event processing and products which did more or less the same thing – if we look at the big picture. So why merge two companies which are so similar? Maybe there’s something in their marketing approaches and existing customer base that makes this a good idea?

From a technical point of view, I think the new merged Aleri will have tons of work in trying to figure out how to merge these two codebases. Something which is commonly incredibly difficult to do taken into account company politics, developer egos, and the fact that the companies are far away from each other. Something like this can take years to complete and consume insane amounts of resources. So, how are Aleri going to keep up the speed needed while doing this merge.

What code to keep, what to throw away? At the same time keeping those few customers happy and avoiding internal politics.

But hopefully Coral8 and Aleri can show us how a merger is done in an efficient way.

As I see it the real value of Coral8 is their CCL execution engine. Not the portal and other fancy graphical stuff added recently. Execution of CCL is what made Coral8 rock. So buying Coral8 and not using it would be total waste of efforts. To get a really good streaming data processing engine is more or less the only thing I see which would make this merger make sense. If, and now I’m speculating, the Coral8 engine was seen as superior to Aleri’s own one, I would make a great base to build on. The Coral8 engine is really good at processing data streams and probably the best data stream processing engine out there. Maybe complemented with a Splash module from Aleri?

Just some speculations, I’m excited to see what comes out of this.

 

 

Complex Event Processing

 

Coral8+Aleri

Complex Event Processing

A common task in many event processing systems is to detect patterns of events.

If combined, these patterns will eventually form a situation consisting of multiple patterns over time.

So basically a detected instance of a situation is a specific sequence of events.

For example:

Situation definition:

order(door_id, command), order(door_id, command), error(door_id, msg), or


order(door_id, command), error(door_id, msg), order(door_id, command)

A specific instance of this situation could be:

order(123, close)
order(123, open)
error(123, open_fail)

Telling us that the door closed but the open failed… So far, pretty simple concept…

But what about an environment in which we have thousands of doors?

Do we like our situation definition to apply to all of them?

Most likely no… Here’s where the context management comes in.

 

The idea is to detect the situation in a very specific and limited context. The context restricts which events are considered when evaluating the situation detection process.

A context could be defined in a number of ways:

• The most obvious restriction is that the events talk about the same door. Here its encoded in the first parameter 123.
• The time of the day could be a restricting factor. Consider only events between 7am and 5pm.
• Include only events in the context only if the events brings us information about doors which we have a service agreement on
• Include only events from doors which are closer than 3 miles from Gunnar (the guy who will fix any problems).

 

As you can see from above, the definitions of the situation and context are independent. This is important as it allows the situation to be detected in multiple contexts.

Also a rule, maybe a decision rule (not CEP style activation rule), could be used to determine which context should be used or you could simply pair up a situation with a context.

So, simply:

Situation – A sequence of Events.

Context – Limits which events are considered for the situation

 

 

 

Complex Event Processing