Marco on CEP

It seems that Syndera – www.syndera.com – was completely bought off* by Tibco and is now shut down.  My earlier speculations about Syndera was right. So now we have one less CEP company alive.

Hopefully the rest of the gang is doing fine. My Google Alerts have recently brought me some good news on Aleri and Coral8 so they are hopefully doing fine.

 [Edit: Here's a nice summary of Syndera]

 

*)  What is amazing is that a company like Syndera, which actually has (had) an idea on how make money and had a pretty good story on the value they were providing is only worth $1M (That’s what Tibco bought them for). I just don’t get this.( I’m not supposed to either, so it’s ok.) Compared to Twitter which is valued at $250M. To me Twitter has more or less no apparent value and there seems to be no idea on how to make money. Still investors push money into it. Are the investors totally insane? They could as well burn their money on a nice bonfire. Hopefully there are some investors left which are not on acid. I have the same distinct feeling now as I hade just before the .com bubble burst. How long before this web 2.0 bubble bursts. There seems to be tons of companies like Twitter getting VC money currently. All these will die soon if you ask me.

 

 

Complex Event Processing

Soon (whatever that means to a software developer…) we are releasing a new feature set for ruleCore. 

Note – I said feature set, not a new version.

One of the things about delivering software as a service (SaaS) [or use your favorite term for this] is that we should not bother our users with things like versions and upgrades. Versions, patches, upgrades, hardware sizing and all that are a thing of the past. It’s sooo 2005.

If you are to deliver software as a service, sorry I mean as a Cloud (these things change names faster than I can type), then I think you should go all the way and do only services and nothing else. I’m starting to realize that Cloud software (if it is still called Cloud) must be designed differently from ordinary Box software. So it’s best to do either one and not mix them.

Anyway. To my point. In the new feature set we are introducing a new format for events. It’s something I’d like to get comments on.

Previously you did something like:

Event
  Header
    Fixed stuff
  Body
    Any valid XML


 

It’s the any valid XML part which is modified. So now you have:

Event
  Header
    Fixed stuff

  Body
     Property A: 'value 1'
     Property B: 'value 2'
     ...

 

 

The change might not look so big, but the implications of it are.

Instead of seeing the event body as a block of XML we now see it as a collection of named properties. The properties can be any valid XML but must be mapped to a property name.

The old approach is to write an XPath which gets us an anonymous value. This value is directly used in rule evaluation.

The new approach uses the properties in rule evaluation. The value of the property is found in the same old way with an XPath.The difference is that the XPath is defined in the event definition. So for each even definition we have a number of XPaths which are used to find the values of some named property.

Another interesting feature is that ruleCore knows how to derive a property from another one. So if you say something like "vehicle SameColor color", where vehicle and color are properties, ruleCore knows how to go from the vehicle property to the color property. Which means that you can apply different relations to vehicles and ruleCore knows how to look up the appropriate property. For example:

- Entity SameSize Entity
- Entity InFrontOf Entity
- Entity Inside Entity
 

Here the relation SameSize would need to find the Weight property of both entities and the InFrontOf entity would need to find the Location property. The Inside relation would need to find the location and bounding box of the entities in order to figure out if the left hand entity is inside the right one.

This makes things a little more complicated to set up, but much more flexible and powerful.

The point is that the properties are not just names. They have well defined semantics. Semantics that ruleCore knows a lot of, and can do a lot of interesting stuff with.

For example:

Let’s say that a property tells us that this is an event with a new location of a truck.

RuleCore would know the actual semantics of a real-world trucks and can based on that information assume a lot of things. For example we can make assumptions of maximum speed and we know that trucks can physically move faster than that speed.

So if we see a truck position in London at noon we know it’s an error to see it in Barcelona one hour later.

 

Weaving in semantics in this way opens up a lot of interesting ways to create rules. If we know that one event is from a tractor and the other one is from the trailer we can use the relation coupled to understand if the tractor and trailer is attached to each other. Or rather, should be attached to each other. If they are, the we can assume that they should for example travel in the same direction with just a couple of meters from each other. If they are not, someone just stole the trailer.

These properties can also be used to create a very rich context in which to evaluate the situation detection.

We could for example have an event view (that’s what we call the context) which contains only

  "location update events from heavy vehicles currently northbound on route E20 which have not stopped for the last hour"

As some other of the new features are that ruleCore knows how to read a map it actually knows which vehicles are on a certain road. Neat if you ask me.

So what I’m currently thinking of if this new functionality, and the extra work involved in defining rules, are worth the effort. What do you think? Does this make any sense to you?

Being from Scandinavia, I like simple and elegant things. This adds some complexity but might actually make things more elegant if we manage to get it right…

 

 

Complex Event Processing

There seems to be faith in the SQL based approach to stream processing. Here’s another vendor with a variation on the same concept: SQL stream. I wish them the best of luck!

Now we have a number of SQL based approaches. Quickly, can you tell the difference between Coral8, StreamBase, Aleri, Esper and Oracle’s tools (whatever brand name they have currently)? It seems that the language concept is more or less the same. I know, the vendors are unlikely to agree, citing features which are unique to them. That is certainly true for a specific problem for which one of these tools might be preferable over another.

But still, for the prospective customer, the difference is not in the languages as such. But in everything that surrounds them. The execution engines, tools, user interfaces, fancy dashboards. Not to forget soft factors as quality of support and perceived company stability.

It’s interesting to see that so many vendors go for the SQL based approach as there are many voices in the community that really don’t think SQL is a good idea for event processing. Some bloggers are a bit subtle about their opinions (company politics perhaps?) but others more independent are rather clear where they stand. For example, David Luckham, the father of the whole complex event processing field, writes in:

The features for defining event patterns have improved, although I do wish some of them would shovel SQL below decks and provide higher level event pattern definition languages!

I both agree and disagree.

First, the SQL based tools today are really good at what they are used for. And that is NOT event processing. It is marketed as such, but it’s actually DATA STREAM processing. Which is sounds similar, but is not event processing. For stream processing purposes, SQL rocks! For event processing, it can be make to do the job. But not in a natural way.

Personally I think that you need a better way to do EVENT processing than SQL. Using a query/poll based (SQL) concept and tweaking that into submission to handle event processing might not be the best option for processing in a event driven world.

I’ll repeat: Data stream processing is NOT event stream processing!

But as always, the users are always right. If they prefer SQL. Then SQL it is.  We’ll see when the market is mature enough to start doing real event processing instead of todays stream processing.

The weird thing is that in the early days, say 2004 or so, many of the vendors actually marketed their tools as data stream processing tools. Then somehow marketing took over and decided that they should market the same tools as event processing tools instead. So in many eyes they now have a lousy event processing tool instead of an incredible data stream processing tool. The products are more or less the same though. Only marketing changed as I can see it. Confusing.

Anyway, that’s in theory. Maybe the users really don’t care is the vendors call it CEP, the CEP label seems to be stuck to so many things so it starts to mean "any kind of data processing which is not like traditional database processing"

 

 

Complex Event Processing