Marco on CEP

Still Here

Marco — Sun, 01 Jan 2012 08:34:58 +0000

No. I'm not dead. We're still in business. Working hard on various projects so I have very little motivation to write anything here right now. I'll be back when my blog writing motivation returns…

CEP Tutorial

Marco — Mon, 08 Aug 2011 09:39:58 +0000

Still closed for summer, but I could not resist pointing out this presentation:

http://www.slideshare.net/opher.etzion/aaai-2011-event-processing-tutorial

A good summary of current CEP systems and what challenges they face.

The Blog is Closed

Marco — Thu, 14 Jul 2011 19:46:38 +0000

Well, at least for summer…

Back in August (maybe September) again… A good Swedish tradition is to have a looong break in the summer. (A hint for you in the US, work for a Swedish company like Volvo and you'll might have five weeks of summer vacation…)

Types of CEP – From StreamInsight

Marco — Tue, 21 Jun 2011 18:08:10 +0000

I did some googling to see what's new from the StreamInsight group at Microsoft. There seems to be some development but I did not find any revolutions there. Not in any way implying that there's something wrong with StreamInsight. I like the concept and they have a really cool product there. It's also nice to see that Microsoft is part of the CEP community. But I sort of expected more news as it was a long time since my last check. Maybe I missed something? Anyone from StreamInsight team that have the inside?

What was interesting is that I found a picture describing three kinds of CEP. I have been saying for long now that not all CEP is created equal.

The StreamInsight view:

CEP for lightweight processing and filtering at the edge of your network, close to your sensors and other event source.
CEP for aggregation and correlation of in-flight events closer to home
And at the center of your solution, you do more processing intensive CEP for complex analytics possibly including access to historical reference data.

That's one good way of describing three different types of CEP. Another perspective that one could add to that is that there's two main processing types at each of these three steps: Driving algorithms with event data or looking for interesting patterns in the event stream.

On Event Models

Marco — Mon, 06 Jun 2011 06:28:53 +0000

It might not come as a surprise that the definition of an event is central to any type of event processing. Call it Complex Event Processing (CEP) , Event Stream Processing (ESP) or just Event Processing, but the concepts are the same.

In some products, like Coral8 and StreamBase, they actually started with the notion of data and after a while re-named the tuples to events. So the notion of an event seems to be a bit flexible. Different vendors have different ideas on what an event is. Our ruleCore started from the other end, starting with an strict 'everything is event approach' with a more strict definition of an event and using events as the native interface for everything. Other products seems to end up somewhre in between. I don't think either of there approaches are wrong. They just reflect two different types of event processing.

In order to better understand all these different event models it's great to have some possible features that a conceptual event might have.

The event model used by an event processing system defines to great extent the feel of the system. Small details in the event model have a huge impact on the expressive power and capabilities of an event processor. Just compare StreamBase with ruleCore and you'll see two type of products even though both are marketed as CEP, and they are both CEP!

An event model consists normally of:

- Event semantics
- Event syntax

Let's break up the text with a spider:

In addition to headers and other meta-data an event normally contains some payload. It's the data which tells us more about the event. If the event type tells us what happened, the payload tells the details about the event that occurred.

This data can be in different forms. Key/value pairs, structured or unstructured. Popular event encodings such as XML enforces some restrictions on the payload. Using a binary event encoding opens up for all kinds of creative content in the event body; Pictures, video or other types of rich content can easily be added into the body of a binary event. Although many architects and designers favor keeping their events small.

So, what are the fundamental base properties of an event? That is, what kind of properties can we expect to find in most events?

We need to consider both the syntax and semantics of the properties. Syntax is actually the easy part. Semantics can be a bit harder to define.

We have some prime candidates for event core properties

Event Id
Event Type
Detection timestamp
Entity
Probability
Location
Occurrence time stamp
Event class or category
Caused by.

Event Example

Lets look at some practical examples. Here's an event encoded in XML (actually in the ruleCore markup language, but the concept is the same independent of the exact format. Feel free to add others in the comment field):

" detection_time="2011-06-06 08:29:22">


     39493849384938984930



	  

	       AB-123
       -23.3495 34.4955

	  

	
 
I’m using XML here as an example, but it could as well be something else… In this example we have an event of type ProximityAlert, it was detected just a couple of minutes ago and its globally unique id is bd79d85a-12c1-4a1f-9047-3dc34fcab2cc.

	
As we can see I envision the event to contains two containers, one if the header and the other is the body.
The header contains information about the event itself, for example which events it depends on, security related information, information about the event sending system and other meta data type of information.
The body is the actual payload of the event.



Things You Should Know
Marco — Mon, 16 May 2011 20:16:50 +0000
At a lunch discussion we talked about things that every architect and developer should now. Not knowing these will yield the "wtf? Where have you been the last ten years"-response
Just some examples that we talked about
http - foundation of the Internet. You should know pretty well how it works.
Java or C# - If you need to code, you really should be able to do it using one of these two.
SQL - For storing data there’s no excuse not to know SQL.
It’s pretty aparent that anything related to CEP has a long way before it finds its way onto that list. 
If CEP is ever on the must-know list, in what form do you think we will talk about required CEP skills?
 
 
 

    

	


Into The Cloud
Marco — Sun, 08 May 2011 06:28:46 +0000
 
As you might know we (ruleCore that is) have a cloud only (except for the OEM Kit) solution for CEP. This means that I try to keep an eye on other non-CEP related developments and trends in the Cloud space.
A trend that I think I have spotted, is that there seems to be a number of cloud services with a special super cloud status. Amazon Elastic Compute Cloud EC2, Google App Engine and Microsoft Azure have a very special position in the market. Third parties are starting to adopt their software to these super clouds because it gives them credibility.
My idea of a super cloud is thus not purely based on technical merits. I’m sure rackspace and other local cloud providers have services that are as good (or better) as these super clouds.
But what is interesting is that software vendors which produce new cloud services seems to have discovered marketing benefits by running their solutions in a super cloud.
 I think trust is one of the most defining characteristics of any cloud service. If you don’t trust that the cloud works, you really don’t care about its technical capabilities. So any successfull cloud provider will mainly focus on trust and secondly on technology. With a good enough technical solution it all comes down to trust.
That’s why I think that you will see names in the cloud arena that you really did not expect to provide this kind of services, building on a strong brand instead. Maybe many of the telcos or other well know companies with a brand that says "you can trust us, we will be here forever".
 

    

	


Event Processing – Dos and Don’ts
Marco — Fri, 15 Apr 2011 04:38:25 +0000
 
Do CEP if your events come in a steady stream
Do CEP if temporal aspects of the event stream is important
Do CEP if you need to drive computation by live data
Do CEP if your data changes rapidly
Do CEP if you find yourself polling your database at high rates.
 [add more]
 
Don’t do CEP if performance is your only concern.
Don’t do CEP if you…
 
OK so I thought I had more time to write,  let’s call this a draft and I’ll return to it later…

    

	


Scada & Sensors
Marco — Thu, 14 Apr 2011 04:28:15 +0000
 
Today I think I’ll read more about Scada and sensor networks.
Those two might be the next big thing for CEP… 
I’m trying to understand what we need to do in ruleCore in order to support these. My first initial thought is that we are halfways there are we support a special kind of sensor already – The GPS
But there must be more features that are specific to Scada and sensor networks…. Stay tuned.? Any recomended links?
 

    

	


Event Store
Marco — Fri, 08 Apr 2011 05:17:18 +0000
Would it not be nice if you could buy a stream of events?
http://www.play-project.eu/index.php/solutions/event-marketplace.html
What if  you had easy access to various event streams? 
I really hope that this project manages to produce something interesting in this area. What might be problematic is that this kind of functionality is not a technical problem. It’s mainly about business. Obviously you need the infrastructure to support this kind of funktionality, but I’m pretty sure the technology is here already. 
But what is a greater challenge is to create an eco system where service provider can find profitable business in providing event streams. Without that, no event market…