Marco on CEP

In real life it’s common to see how different messages get a varying meaning when presented in different contexts.

For example, a quote out of context can more or less be turned into whatever the messenger wants it to be. The same goes for pictures where the photographer can create totally different pictures of a scene by simple placing them into a different context, in photography this is done by carefully selecting your angle, background and crop.

That brings us to a very interesting concept in event processing. Namely context aware event processing, its one of the two major ingredients in event processing. The other is the art of detecting interesting combinations of events.

The context is apparent when you process static data. The data you query forms the context. Your SQL query is executed in the context of the data in the database. It’s obvious that the same query results in different sets of answers when executed against different data. In event processing, the context is a bit more complex concept.

Creating Context

In event processing we need to create a context for us. Normally an event processing system is operating on a stream of events. This have to be turned into a meaningful context. You could ofcourse process events one by one, but that tends to be so simple so you won’t actually need specialized CEP software for that.

For example

Let’s say you would like to be informed about trucks which stay still for more than one minute.

Depending on the context, this could mean different things:

• If detected on a large road with heavy traffic it’s something that need to be attended to immediately
• If detected inside a city during rush hour, it’s not too much to care about and might only lead to and informative message that the transport is late.

These two contexts give the same situation completely different meaning and it can also be used to ignore the situation if it happens in certain context like inside company premises.

Context Management

The context can viewed as your dynamically managed virtual view into the stream of inbound events. The context contains a sub set of events from the view which, at any point in time, are somehow relevant as a group to the situation you are detecting.

The events which make up your context are normally somehow related using a number of common properties. Some examples:

• Time – Manage the events in the context based on time constraints. For example only events from the last hour or those between nine and five.
• Location – Events which notify us about changes in entities located in certain geographical areas, or perhaps those which are close to each other.
• Content – Event content such as primary keys of different kinds can be used to keep only semantically related events available.
• Relation – Include only events which are related somehow.  Relations can be dynamic and change over time. Example relations are caused by, is sub class of, is super class of, belongs to same category and so on.
•  Entity – Include only events which are notifications about state change in the same business entity. For example the same person, truck, order, business process, customer or server

Your Evaluation Context

So what’s the point of all this context definition and management?

Situation detection (aka pattern detection) and other processing is performed in this context! 

This means that the same situation can be defined independently of the context.

Thus allowing same situation to be evaluated against different context in provide a modular and powerful processing model.

In order for this to work smoothly, the context must be treated as a first class citizen in the event processing model used. A context should be defined totally independently of any other processing which is performed using it.

When a situation is detected it is in turn used as a context to any action you like to perform as a result of the detection. This new context together with the original context in which the situation is detection provided a good base for creating meaningful result events.

The context is very important when reporting about a detected situation. In the truck stop example above, it would not be terribly valuable to know "A truck stopped". You need more context! At least the id of the truck and the location and perhaps road number where it stopped and the drivers name. Nice additions would also be how long it stood still and perhaps its movements during the last couple of minutes. All this information is picked up from the context while assembling the response.

All this context in the outbound (or whatever method your event processor uses) action event can be used to initiate a business process to handle the exceptional situation.

Example

 I think most event processing tools have some kind of context management available or at least functionality to create and manage your own context. I’ll use Coral8 and ruleCore here as example just because I know them best and don’t need to look up how they work. Maybe you can add more examples?

In Coral8 there’s a concept called windows. Normally each query contains a window which the query is executed against. There’s a extension of this in the later versions called named windows which are defined independently of the query. These named windows can be used to hold a context. A number of queries can write data into the and they can also be read by multiple queries. The named window can be maintained using CCL and thus there’s a very powerful and flexible way of deciding what events are present in the named view. For example (Copied from the Coral8 docs):

CREATE WINDOW WindowTradesMicrosoft
SCHEMA ‘WindowTradesMSFT.ccs’
KEEP 3 ROWS;

INSERT INTO WindowTradesMicrosoft
SELECT volume, price
FROM StockTrades
WHERE symbol = ‘MSFT’;

INSERT INTO AvgPriceMicrosoft
SELECT AVG(price)
FROM WindowTradesMicrosoft;

 

In ruleCore the context is called event stream views. A view is defined to include only events with certain properties. For example that they come from the same geographic zone or that they are all about the same entity. For example you could write something like:

 

<View name="Interesting Vehicles according to XYZ">

    <Type>
        <Event>Location</Event>
        <Event>X1</Event>
    </Type>

   <ZoneMatch/>

    <PropertyMatch property="Color"/>

    <MaxAge>00:10:00</MaxAge>

</View>

 

To create a view (context) which contains events of type Location and X1 from the same zone with the same color during the last ten minutes.

In ruleCore we use a declarative approach to context management. In the view definition you specify what properties the events must have in order for them to be present in the view.

Other event processing tools might also give the user the option to maintain the context using a procedural approach. You could basically have small piece of code (or other procedural logic) which updates the view continuously. Using some kind of pseudo code from a fictional event processing system this could look like:

if event.type ="Location" and someComplexFunction(event)   
     context.insert(event) 

if event.color = "Green" and databaseLookUp(event) 
    context.insert(event)

if event.color = "Red" and event.entity = "server1" 
    context.insertOrReplace(event)

 

 

Rich context management is one of the things I think sets event processing apart from traditional data processing and you should spend some time in getting it right when designing an event processing system or using one!

 

(When put into context, the boring picture of the Swedish "no parking sign" gets a bit more interesting with the two cars parked in front of it. Also, published in the context of a blog written in English it can be a bit amusing too. (Utfart is Swedish for Exit, nothing else…))

 

 

 

 

Complex Event Processing