When CEP Might be a Bad Idea
Recently I was in discussion with a group of developers. They had spend some days in trying to understand what’s behind the CEP TLA.
By some creative googling they had found all the vendors that I expected them to find when researching CEP: StreamBase, Aleri, ruleCore, Apama, Tibco BusinessEvents, IBM Business Events and Agent Logic.
This clever bunch of developers seemed to have a pretty good understanding on what CEP was. But something they really could not agree on what kind of problems CEP was unsuitable for.
As one said, "From the vendors web site I gather CEP can be used for just about anything apart from brewing coffee"
The rest of the discussion circled around the topic: When is CEP a bad idea?
To make a long dicsussion short, here’s (I might forgotten something though…) what we came up with…
CEP is not a Good Idea:
- When polling SQL will do the jobb
- If you could do a normal report and mail it to people instead. Requires fairly stable data and willing to use old booring (translation: productive and stable) technology
- When nobody cares about getting the results in real-time
- If your system is driven by user requests and there’s always a user waiting for an answer for each request.
- When 100 lines of code will solve the same problem.
- If you need an answer to a question at a specific point in time (this surprised me until I learned that the guy that came up with this was heavily into production rule systems)
- Anything that needs hard real-time. All agreed that CEP was more for business type of stuff than actually controlling plants and machines.
I think this was a good coffee break exercise as the biggest problem in any real systems that I have seen comes from using tools and products which are totally unsuitable for the task. So understanding when not to use a specific type of building block is critical for any architect and developer.
If by CEP, you mean one of the vendors mentioned above, then I disagree with two of these:
(1) If someone is waiting for a response – I’ve used a CEP engine to build a request-response system, works great.
(2) Answers to a question at a certain point in time. Actually, production rules are even worse at this than CEP engines. At least with CEP engines, you can build your own data structures and keep whatever data you need. I have used a CEP engine to answer such temporal queries, although not in a “generic” way that would handle all temporal queries, but rather to answer specific kinds of temporal queries.
In terms of controlling plants and machines, many financial use cases are not exactly real-time but do have low maximum latency requirements. This might indicate that CEP engines could be used for other fields with similar latency maximums.
Here is my advice: do not use any technology if you have to convince yourself that it is right for you. Most use cases do not require instantaneous responses. But developers that are used to working with systems that do require such things usually immediately understand where CEP engines should apply.
I think it’s a simple as 3 rules:
1. If the data you are analyzing is static, you don’t need CEP
2. If you don’t need immediate insight/response – if you plan to collect the data and analyze it later, you don’t need CEP
3. And if you can do it faster/easier with something else, don’t use CEP
(I’m sure someone will point out that CEP can be used to analyze historic event data for event patterns – but that’s not what the field is focused on or what people are enthused about)
Jeff, I think those three points are a good starting point for any CEP vendor, trying to solve problems already solved (by data warehousing tools, BI tools, reporting tools and so on) might not give you too many customers. But things that are hard to do with the current set of tools are something else…
Hans, when discussing we sort of assumed a completely event driven architecture. A architecture where we felt request-reply did not fit in too well.
I’m surprised polling a database came up. Most DBAs I know hate when apps poll…poll…polll..do something…poll…polll
I supposed the polling delay is key. Most situations I’ve been in have not waited very long between
checks – pseudo-realtime.
I think the key is “When nobody cares about getting the results in real-time”
If done carefully polling does not add too much stress to the database from my experience.
DBAs hate it, because it’s ugly and a no-no. But most polling applications can solve a large number of problem with a limited amount of queries per minute to the DB.
You have to be clever about what to poll. Executing all queries you know of blindly and checking results is generally a bad idea. But having special polling queries with matching a index can take you a long way.
But this solves only a fraction of all problems, if the load on the DB gets too high, consider a CEP solution instead. You might want to consider a CEP solution if your queries tend to be very complex and you have the gut feeling that SQL was never meant do do whatever you are trying to bend it to do.
@Dan
I think you found the key point there! Not even all CEP vendors understand this