Marco on CEP

I have now added ActiveInsight into the vendors list.

So now there’s five open source projects to choose from when you something really cheap* (= zero dollars in license fee). I have not yet looked into their product in detail, but their web page says all the right things, so it seems that they understand this CEP thing at least on presentation level. If someone have used their product, please share your expericeces. What would be most interesting to learn is how they compare to the other open source products (Pion, Esper, Drools Fusion and IEP) . 

It’s a good start for the next  year to see another open source product in the CEP space, I hope there will be a couple of more in 2010. At least I heard rumours about a major fork to Esper that is being developed with a rather large number of developers on the project.

2010 is soon upon us. Maybe it’s time to be a Gartner wannabe for a while and do a bunch of predictions on CEP… Maybe in next post, so stay tuned

 * If you still prefer to pay for your CEP software, there’s Progress Apama, Tibco Business Events, IBM BusinessEvents, Rulecore, Aleri, StreamBase to choose from.

Complex Event Processing

Soon it’s Christmas, at least in some parts of the world. The holiday that once had some religious meaning. Can’t even remember what it was all about, says a bit about my religion skills…

A couple of weeks ago I took a look at StreamBase, who are kind enough to provide free evaluation downloads, and they are certainly starting to get all bits and pieces in place. If you need to do real-time calculation on a data stream, this is probably one of the best ways to go about it. So stop all that custom development that you first thought of. Aleri/Coral8 is in the same camp so is might be a matter of taste which one to go for.

So if you have a continuous stream of data, perhaps some financial information that changes all the time like prices, then stop developing yourself and look at these tools instead.

Next I tried to see if I could find more about Apama, and Progress is going in the right direction with Apama. Not only that Mr. Apama is now CTO of Progress, but I like their approach with rules (working at ruleCore that would make sense, would it not? ;)  ) Expect lots of interesting developments from the Apama team next year. Maybe integration with other technologies like BPM engines?

A friend I know at IBM said that we could expect that it takes a couple of years to IBMify anything that they buy. The start is slow but when they get the speed up they can at best be unstoppable. So next year should be the year of AptSoft, er, IBM Business Events (with or without space? I can’t seems to never remember if it’s IBM or Tibco that uses the space there)

Speaking of Tibco, I think Tibco’s interest in selling CEP software is a good sign that there’s money to be made in the CEP field. Being the masters of building wealth by selling infrastructure software.

The presence of these big players in the CEP field is what actually makes "CEP Software" its own segment. Now we see that customers are actually asking for "solution based on CEP", not that our beloved customer always know that they are asking for. But at least they are asking for it. I would say that this formation of a CEP market is thanks to the collective marketing efforts of all these big vendors. We smaller vendors can’t create a new field within IT. But we can work in one, created by the big ones.

In addition to the bigger ones, there are a number of small vendors too. Providing parts of the solution which sometimes might be a better solution, for many reasons, than going with the bigger vendors. There’s of course our very own ruleCore, there’s the free and gratis open source Esper, we have EventZero, SQLStream, RTM,  Pion, WestGlobal, Truviso and other small vendors all building companies around CEP technology. All with their own twist to it.

What about Microsoft? Did I forget them? No, not at all. There’s StreamInsight. But it is way too early to say in what form it will be available in the future. But at least someone at Microsoft thinks that CEP is worth putting developer dollars into. Somebody did convince his manager to give him a budget to build this CEP thingy and actually got an approved budget to work with. Another very good sign. But I would expect that, following Microsoft tradition, that their offering will be mostly usable in a pure Microsoft world. Don’t expect something that fits smoothly into your brand new EDA/SOA master piece. I would guess that the same is true for Oracle’s CQL. CQL adds interesting capabilities into their RDBMS, which is good. Provided that you use it. For others, it might not be that interesting. I don’t think anyone buys a Oracle license just for CQL.

That sort of sums it up. If we can avoid abandoned open source projects, companies that run out of cash or products being dropped we can look forward for a very exciting year for CEP in 2010.

Complex Event Processing

With Christmas approaching rapidly I struggle to get a ton of small tasks done before it’s time to take a couple of days off and eat way too much food and candy. This year I actually hope for some snow. Here it starts to get dark at 3pm or so, with some snow it would not at least be completely dark. And of course everything looks prettier with some snow and the sounds of the city are much more softer with snow on the ground.

Lately I have been reading Event Processing : Designing IT Systems for Agile Companies by Mani Chandy and Roy Schulte. They present a good list of principles an event driven application should follow in order to actually implement EDA:

• Report current events. Things that happen right now. Not (old) historical data
• Pushes event - A producer creates and event and publishes it. Not the other way around. In EDA, a system never queries for event. It receives them.
• Process events immediately – Don’t store events and query them later, that’s a task better left for databases
• Know nothing of event destinations – Just fire away events and assume someone else will route them to proper destinations.
• Events are not commands – You don’t know anything about event destinations, per above bullet, so you can’t command them to do anything.

So if you look at a CEP system, which should naturally fit into an EDA. Then the CEP system should obviously follow the principles above. (advertisement: ruleCore CEP Server does, it was built to be a component in an EDA)

With an EDA and lots of available event processing systems that you can use as high level components, we have a very smart way to assemble enterprise wide systems. This is something SOA is not very well suited for. Everything is too coupled in SOA. But with a nicely designed EDA and a bunch of CEP, or other event processing agents, you are a step closer to actually creating an EA which your CEO will love.

So to my original question. What is the difference between data and events? Data is something that is your primitive entity that holds business information in a tightly coupled SOA or database centric architecture. Events are something carries around your business data in an EDA.

When moving from databases and SOA to EDA, you need to stop having a data centric view of the world and start thinking of events instead. Otherwise your EDA project is doomed.

If you look at the bullets above. You can learn a couple of things:

Report current events: If multiple systems receive the same report (event), they must somehow be able to know that it is the same report they are looking at. So events must have globally unique id.

Pushes events: If some other systems are to push an event to me, I need to know how the event is defined in order to make sense out of it. So everyone will need to share event meta data 

Process events immediately:  Later is too late. So I can’t use a query to process the event I receive as I need to do it now. Not query later.

Know nothing of event destinations: The implication of this is that I really don’t know who is listening and on what. So I need to publish events whenever I do something some other entity might be interested in. Now or sometime in the future. So basically you need to publish every potentially interesting event from every system.

Events are not commands: SOA style you know lots of things about other systems and know what you can ask them to do. In EDA, you just sprinkle around events and hope for something interesting to happen. This scales way better and does not cost a fortune to manage.  

So before you can ever dream of having a nice EDA in place with CEP components in place doing all kinds of interesting processing you need to start thinking about events and not data. Programmers are used from kindergarten to think in query/respond patterns so this is something that will take some time to get used to. So I still think EDA and CEP in the mainstream is at least a couple of years away.

Complex Event Processing

Note to self – look into these when time permits: http://www.activeinsight.net/

Complex Event Processing