Why CEP is so difficult?
I was asked a time ago why CEP (Complex Event Processing) is so difficult?
Or rather, why is it so hard to build your own generic and functionally complete CEP server?
I think one of the answers is that you need to do two conflicting things:
- Process events in memory only
- Store events and state in persistent storage
You need to do 1. because of your need for speed.
You need to do 2. in order to build a reliable and predictable system.
The magic blend of these two are required for most CEP systems. It’s here where you need better than average architects and developers to produce anything usable.
Example – Why..
Just a simple example to understand why you need both. Say you are looking for a sequence of events:
A,B,C,D,E
You expect this sequence to occur during a seven day period. When you see event A, you start looking for the rest of the events.
Lets say you saw events B and C after a couple of days. Then the server is re-booted suddenly, just quickly, but your system is killed unexpecedly.
When your event processor is brought back online again you need to remember that you have seen A, B and C and are looking for the next event in the sequence which is D.
This forces you to save state information in some kind of persistent storage (database). If you are looking for thousands and thousands of long event sequences you will quickly get really bad performance if you keep writing state information to the database constantly.
When You Don’t Need Persistence
There’s a special type of event processing where you don’t need persistent storage
A class of CEP solutions where the problems can best be described as data driven algorithms do rarely require persistence. Tools like Coral8/Aleri, Esper, StreamBase are commonly used to do this kind of processing. This type of processing, commonly done using some kind of extension to the SQL concept. Typical uses that I have seen have mostly simple queries which constantly produce output. If there are patterns in the queries they tend to be simple and short lived.
If you only need this kind of state-less processing then you can get really high performance instead of systems like ruleCore where the persistence is a fundamental concept and can’t really be switched off.
If you are shopping for a complete event processing platform you should at least evaluate everything with full persistence switched on. Provided that the CEP system provides real usable persistence, not all do…
Hi Marco,
there’s an emerging design pattern to approach this problem: a data grid. Products like IBM’s ObjectGrid (=> http://www.ibm.com/developerworks/wikis/display/objectgrid/Getting+started ) or GigaSpaces ( => http://www.gigaspaces.com/ ) provide this capability. Basically this revolves around having distributed in-memory stores with sync or async replication/synchronization (over very very low latency network protocol, in the case of IBM Websphere MQ Low Latency Messaging => http://www-01.ibm.com/software/integration/wmq/llm/ ) as well as async persistance into a database. The distributed in-memory storage can be architected so that very high availability and high performance is achieved. I know that IBM has implemented this pattern in several customers with success.
Regards,
It’s nice to see that there starts to pop up infrastructure components to handle this very problem. The complexity of all these are a good illustration how difficult this problem is to solve.
Summary of Issues -> http://www.slideshare.net/opher.etzion/event-processing-the-next-generation-march-2009
Hi Marco. The issue of memory based vs. persistent processing is indeed one of the major implementation issues for event processing engines; IBM’s “WBE/XS” (Websphere Business Events Extreme Scale)
see: http://www-01.ibm.com/software/integration/wbexs/ is a grid-based implementation that addresses exactly this issue; as my presentation about the next generation of event processing that was cited above shows, we (in Haifa Research Lab) are working on some related optimization issues. I’ll blog about some of them in the future. Other vendors also employ grid platforms / in-memory databases for this purpose.
cheers,
Opher
After a quick read I got the impression that these data grids are based on memory replication and assumes that there are at least 2 servers. For added reliability it sounds that you need even more than that. My first thoughts is that this is great for building huge systems, but it scales down badly.
I’ll put those links into my reading list to see if I can evaluate them with my CEP architect hat on…
A while ago I did some work with MySQL Cluster edition and it takes a similar approach. Assume a large number of reliable servers and assume that they will not all go down at the same time. No power outages in the server room then. It will then allow itself to sync a commit in memory only to all nodes in the cluster Then some time later when it has time it will flush the data onto disk. To hardcore RDBMS guys this is a bit scary as a commit is suddenly not a commit, it’s a request to do a commit…
In a year so big and inexpensive SSD disks will start to help a bit, but persistent storage will always be slower than RAM.
Cool solutions to this would probably use clever algorithms designed specially for CEP and be usable with even with small server and small and slow disks.