Many event processing tools and engines contains some kind of processing element which takes events as input and produces events as output.
Currently* there seems to be to two concepts which are popular among the CEP tool vendors. There’s the SQL based approach, and the rule based approach. Both normally share the powerful mechanism of taking events as input and producing events as output.

The processing elements, SQL queries or rules, can as a result of this easily be organized in hierarchies. Organizing things in hierarchies is certainly nothing new. But I would like to show how powerful hierarchies are when applied within event processing.

I’m a bit surprised that there have not been more talk about hierarchical event processing in the CEP community. It’s a concept which is worth explaining in detail to CEP users.

Recently we have created a geofencing application using ruleCore, or rather the new geospatial ruleCore. It’s intended for fleet management, mobile asset management, telematics and all kind of location based solutions which need a quick alert capability to immediately get notification about unexpected vehicle behaviour.

We used rules organized in a hierarchy to implement all the geofence logic:

This hierarchy of rules allowed us rapidly to create the initial types of geofences found in the requirements list. It gives us a nice structure which can easily be extended when we add new types of geofences, like the off-route (Alert when a vehicle is not following its designated route) geofence we added recently.

It also allows us to add new rules in a modular fashion to monitor a specific property of a geofence. For example, there are rules which monitor some of the geofences and create alerts they are not triggered during a specified period. Great for arrival and departure alerts.

At the bottom level we feed the rule hierarchy with the GPS positions. The Level 1 rules trigger when the vehicle is within a zone. Level 2 in turn has rules which trigger when a vehicle is considered to entered a zone, which is not the same as being inside the zone.

In this way the events go from simple low level position update events to rather high level events at the upper level.

So for your next (or first) event processing project, spend some time in the design process and try to see if your rules are naturally hierarchical. If organized in this way you have a flexible and powerful abstraction which also tends to be easy to understand and explain to new crew members.

Most of the time the levels can also be give descriptive names as each level tends to handle rules at a certain abstraction level. For example, monitoring a web site and its infrastructure could use rules on four levels:

• Level 0 - Network
• Level 1 - Servers
• Level 3 - Services
• Level 4 - Tasks
• Level 5 - Business Processes

and so on.

Each level can emit alert events to the outside world in addition to the level above. This would allow for example the network manager to only listen to events from level 0 in the above example.

* The current set of vendors would obviously want you to believe that the current tools are mature and there are nothing new coming just around the corner, that is: buy now, don’t wait. But the event processing market is a very young one. The concepts are new. We are lightyear away for anything called mainstream and don’t even think of finding standards. In the years to come I’m sure there will be new and interesting concepts (which will not carry the legacy of SQL or rules) and implementations created for specially for event processing. The current tools are inspired by existing research and uses extensions to the SQL concept and the concept of rules. If we were to start over with a clean sheet I’m sure there would be more exiting tools than the current generation.

One of the difficult things when it comes to marketing CEP in general and specifically the ruleCore CEP Server is terminology.

The ruleCore CEP Server uses as its main processing concept something we call “rules” or “active rules”. The problem with that is that my rules are not your rules. The odds for misunderstanding me when I talk about rules are astronomical. Rules mean different things to everyone.

But we still call our the thing inside ruleCore for rules, because technically it is an active rule engine. People just hear the rule part and assume that we have a variation of the traditional rule systems based on RETE och similar algorithms. Nothing could be farther away from the truth. And we really can’t blame the customers either, it’s totally our fault as a vendor if customers don’t understand what we are selling. But we don’t have a better name for them so we are kinda stuck with ‘rules’

On the positive side, at least some people, understand what we are up to. There are basically two main kinds of rules in my world:

• Rules which make decisions for you, or at least tries to help you to make better decisions - not ruleCore
• Rules which alert you when it is time to make a decision.  - ruleCore

The first ones I would like to call (inspired by James Taylor) Decision Rules,

The second category is something I would like to call Activation Rules.

Both rules are important for a complete solution. The Activation Rule tells you when there is a decision to make and the Decision Rule makes or helps you to make the decision.

The two types of rules solves two different problems:

• Activation Rules - When should I start my decision process?
• Decision Rule - What decision should I make?

By using Activation Rules you can start your decision process much more quickly. A real-world example could be to alert an operational fleet manager that a truck is not following its designated route. Without an Activation Rule the fleet manger would manually have to check periodically if all trucks are on track. A Decision Rule would then be used by the fleet manager to decide how to manage the exception.

So my problem is currently how we should communicate this fact to potential buyers. Maybe we should just rename our rules to something else. Or perhaps create a higher level abstraction which we use when talking about the system. The rules could just be an implementation detail.

One particular idea for a marketing message would be to talk only about business event tracking. That’s basically what ruleCore is good at anyway. The rules are just how it is done. For example we could talk about tracking vehicles with GPS or tracking a zone for the presence or absense of a allowed number of containers (also with GPS). Combined with ERP data we can track trucks, business processes and other things as they move around. Might be a good idea, have to think more about this.

For a while now we have been busy adding support for tracking GPS devices to ruleCore. Basically we have added native processing of GPS locations according to WGS84 into ruleCore.

So now it is possible to send an event into ruleCore which not only carries information about an event, but also it’s location on earth. By adding geospatial support we now have a rather nice system to track mobile assets like vehicles, ships, GPS equipped mobile phones or basically anything that moves and for which you can determine its position.

Another addition is that is is now possible to describe zones to ruleCore using KML. This makes rules like:

• Trigger when any vehicle is in zone A
• Trigger when vehicle enters zone B
• Trigger when zone C contains more than 30 vehicles.
• Trigger when vechicle xyz is not on route 123.
• Trigger when vehicle xyz has not arrived to zone D between 3-6 am
• Trigger when a vechile been in zone E for more than 10 minutes
• Trigger when vechicles enter zones A, B or C more than 100 times an hour.

trivial to create.

In addition we have created a little GPS specific web app which we host and offer as a service. Just send the GPS location to us using a web service call and receive geofence violation alerts back using web service too. So adding rather advanced geofencing support to any telematics application is just two web service calls away.

The service also gives you a web interface which can be used to define a number of different types of geofences and gives a graphical editor to create zones and routes. So you don’t have to create the rules manually anymore, just fill in a form on a web page and the rules are create for you automatically.

The target audience for this new service would be any developer of fleet management systems, mobile asset tracking system or telematics solution providers.

Everything is of course delivered as SaaS! Pay as you go is the ruleCore way. For mobile asset tracking we have taken the business model a bit further and charge per vehicle/month.

A typical setup would look something like this:

Where the ‘Connector’ is a fancy word for our web service API. We expect someone else to gather the GPS position events for us, for example a fleet management or telematics system. The system then forwards the events into our service which we call ruleCore:Geofence. The end-users have before that created rules using a  simple web interface, the rules are then deployed as geospatial rules into the ruleCore service automatically.

So the point here would not be to offer a complete geospatial event processor for generic use. Those kinds of systems are better built onto of stuff from ESRI et. al.

But it allows adding advanced geofencing support in just an hour of coding time (The time it takes to write/generate code to call two web services). Building live geo tracking things like this with advanved GIS toolkits it a much more time consuming experience.

There’s lots of rather complex geosptial processing going on inside ruleCore now with the geo additions. I’ll write more about our geospatial event routing and indexing later on. Technically I think this is a very neat example of what kind of applications is possible to create with CEP. All our geofencing logic, for example, is completely implemented using a hierarchy of rules. Hierarchical event processing is really powerfull. Nothing is hardcoded so adding new types of geofences is very easy.

A key feature of any SaaS solution is to give the user the illusion of having her own dedicated server which runs the software. This illusion is required for any user to trust a SaaS solution, otherwise it is very hard to ask an user to send her sensitive data into the SaaS solution for processing.

Sadly, economics dictate that this illusion is just that - An illusion.

To create multiple virtual copies, in a cost efficient manner, of any application you need some help from you basic building blocks such as databases and application servers. These two components are commonly found in many software solutions today.

But most SQL databases are normally not very good at providing support for easy isolation of data between multiple users. One of the most important technical considerations of a SaaS solution is this isolation of the different users data. SaaS requires that your users trust you. This means that they trust that their data is never shared with other users in unexpected ways.

One extreme solution to this is to give each of your SaaS customers their own server.

This would pretty much guarantee that data is not seen by the wrong user. But this solution does not scale, it’s good for the customer, but bad for the SaaS solution provider. By using one dedicated server for each user, you really can’t offer your service for a reasonable fee.

So there need to be some kind of compromise. Compromises range from putting everyones data into the same tables and doing ’select from where userid=…’ to separate database instances of private schemas for the different users. Independent of the chosen solution you will probably realize that your database is not built for SaaS deployments…

Another problem is that most application servers are not terribly good at this either. The main way of providing database connections is to log in everyone into the database using one shared login and then provide ’security’ on the application server level. This opens up for lots of mistakes.

What we try to do is to assume that at some point in the life cycle of the ruleCore Service someone will make a mistake and introduce a bug which could lead data seen by the wrong user. So we try to design everything so that the consequence of such a bug will only halt the application and not show any data at all. Bugs should not lead to data being mixed up.
This is pretty much the hardest technical challenge of a SaaS solution. The elegant and best technical solution is not viable as it would cost too much to maintain and deploy.

So once again developers need to consider the economics of their solution when trying to create a clever technical architecture. This is something that is pretty new to most developers. Commonly the only financial decisions a developer had to make was to make sure that license cost of tools, database server and other components would not be too high. But not developers have to suddenly think about the operational costs of their software.

I’m pretty convinced that SaaS is the way to go. It will take some time as software basically have to be designed from scratch to fit nicely into the SaaS model. But easy and cheap access to software providing valuable functionality is just too good to not happen…

For example, to add ruleCore situation detection capability into any application, just call us with a web service/JMS and your have CEP capability in your application. I’m pretty sure we will see more CEP vendors with SaaS offerings soon…

Billing - This is something that most developers never have to worry about and not spend a second of valuable development time to implement code for. Normal software (when will marketing start to call this ‘legacy’ systems) is normally sold in a way where the vendor ships a more or less functional piece of software to the customer, followed by a invoice from the bean counters. No need to involve developers in the money gathering expedition. Normally the developers don’t even know how much their software is sold for.

So, now enter the wonderful world of Software as a Service. Compared to legacy (oops, sorry) software, it’s the developers which knows what to bill for. So for the first time the guys from the financials have to come down to the development floor and work together on how to charge for the software (as a service). A refreshing exercise.

For the first(?) time the developers are actually part of the strategic discussions on how to charge for the services of the company. In order to build a practical billing system for any SaaS solution you need to build in different kinds of monitoring points in your software from which you can gather data. Data which is used to create an invoice.

Most systems are just not built with billing in mind. So trying to use a legacy (sorry again) system for creating SaaS system could be a nightmare when it comes to billing. Who uses the service and how much?

Creative business models should be part of any good SaaS offering. If we are moving away from the install, test,deploy, upgrade cycle required for normal systems, we could as well do some clever thinking when it comes to our business model. This requires developers, they know what can be monitored and what metrics can be used to create the invoices, and the financial guys to figure out *together* how to use the available data in a good way. This is why I predict that most SaaS solutions will be less than optimal when it comes to innovative ways of charging for the service.

I’m thinking of introducing a non-technical model of charging for our ruleCore Service. It’s a bit tricky though. What costs us is basically servers, storage and bandwidth. The first idea would obviously to charge for some combination of these. Like number events, as that would give us bandwidth consumption and to some degree CPU usage. But from the customers (business) viewpoint this is as bad as Oracle charging per core. Old company, old business model.

So let’s try something new….

1. What about charging each time a rule triggers? Each time a rule triggers it means that there is a business situation detected and reported by ruleCore. How much is it worth to know when a critical business situation occurs? That’s a question I think might be easier to answer by a customer than to pay for some abstract technical feature of our solution.
2. Another option would be to charge for each rule which is defined, independently of the usage. A sort of flat rate option. Each rule represents a business situation of business entity which is tracked. So paying for a rule would give the customer the ability to track a certain class of situations.
3. This leads to a third billing option. Charge for each business entity which ruleCore tracks. Don’t care about the number of events, number of rules or rule triggering.  Just charge for each tracked business entity, like a business process. So if you would use ruleCore to track 100 business processes, 300 servers, 100 containers, 300 trucks - We would just charge a tracking fee for 800 business entities.

The third option would obviously require the software (ruleCore CEP Server in our case) to be able to understand the concept of business entities and remember how many of them it has seen in a month. (lucky us - ruleCore can already do this)
Another idea would be to charge a penny for tracking a order process and two pennies for tracking truck and maybe 10 pennies for tracking an aircraft. Small less important things should perhaps be inexpensive to track and larger ones might be ok to pay more for.

Now I got my blogging motivation back after a long summer break. Summer here in Sweden consists of doing anything possible to take advantage of the 2-3 months with good summer weather we get every year. So if you meet someone from Sweden, or any part of Scandinavia, and they seem a bit down - Try living in a fridge 9 months of the year. Where I live, add to that a constant wind. My home town Göteborg feels, or rather sounds, very eery when the wind is not blowing. Suddenly you hear everything that’s normally drowned in the sound of the wind. Anyho, enough bitching…

During the last six months or so since we created our hosted ruleCore offering, “The ruleCore Service”, I have been listening to all kinds of objections to use a hosted service. I actually thought that the idea of using hosted services would be more mature than it is.

Basically the most objections against anything (not only ruleCore) SaaS seems to be:

1. It’s not reliable.
2. It’s too slow.
3. I don’t trust you.

Reliability is somewhat dictated by the fact the we do everything over the Internet. On our end we have a super-hyper datacentre with all kinds of duplication, replication, redudancy, cooling and other things you might expect to find in a very expensive datacentre. So that part is under control. But what we simply can’t do anything about is the Internet in general and the client’s connection to the Internet. Not sure how we are going to solve this. We can offer dedicated lines into our datacentre in old school style, but these are pretty expensive.

What about the too slow part… Compared to what? I think our datacentre approach can scale better than what most of our clients would manage on their own. But still, if you don’t trust us to be able to do this it’s not any good. Maybe we need to set up a public demo system running on some really wicked hardware. Pretty expensive for us, but might be worthwhile…

So, 3) - I don’t trust you. This is a bit harder. Trust takes time to build. If we were IBM this would not be an issue.

The biggest challenge we have is that we are taking toys away from the IT-departments at our client’s sites. So basically for every SaaS customer we have to take a fight with the IT-department. The traditional way is to deploy a huge ruleCore installation and let the IT guys manage it. A typical installation might consist of a couple of ruleCore processing servers, a database server or two for redundant operation and a couple of Event I/O nodes. Enough gadgets to keep a couple of techies busy. With a ruleCore SaaS deployment the IT-department is barely needed. So guess what solution they prefer?

Apart from these initial problems I still think that a hosted solution is the right way to go. The next version of ruleCore might only be available as “ruleCore Online” or whatever we call it.  Especially in the new era of event driven architectures we need easily available building blocks like hosted event processors. Technically we are there, message based integration is the foundation for all this and it starts to get mainstream. But it seems that we still have issues how clients perceive a hosted service…

Event is one of those wonderfully overloaded terms we using in the world of computing. Combine event with another genereic term, namely processing, and you have a brand new world of new tools and vendors all trying to sell event processing solutions to you.

Basically this means that more or less anything can be called event processing. Some vendors have already begun this magic marketing trick and starts calling various random bits and pieces for event processing solutions and puts the CEP (Complex Event Processing) label to just about anything to see if sticks.

But Event Processing have a very specific meaning and is actually something new. Not 15 years old as some vendors would try to make you believe. As CEP moves up the hype cycle expect just about anything be redressed as CEP and resold as something brand new.

In order find the right tool for the right job it’s rather important to understand some basic characteristics of the field we call (Complex) Event Processing.

Events are not just a special type of data structure. If they were, you could just handle them as data using your traditional and more well known tools.
First of all, events happen. Data is created and modified.

Events are actually a notification about some detectable activity. The activity can be virtual or something that happens in the world around us.

- Example of activity -

Door opened - page printed - ping failed - process started - elevator stopped - train arrived

The fact that an event describe some activity which just happened is the main difference with plain old data. This actually makes event and data semantics very different. Events are much more complex and on a higher abstraction level.

Let’s see what follows from the simple fact that we are describing something that happens:

• Things happen at a particular point in time. Thus the event must have an exact timestamp. A data item is not related by itself to time in any way. If you insert data into a database, the insert itself has a timestamp attached to it, but the data does not.
• Events are also related to each other in time. Data is not. If looking at two values in a database you can not, by looking at the values alone, determine which one was created first. Events on the other hand have this property as they are created at particular times.
• Multiple events can’t happen at an exact point in time. If you think they do, your timer resolution is too low. In any practical setting you might need to deal with two events which seems to happen at the same point in time even if events actually can’t happen simultaneously.

From that follows a number of interesting properties. For example it is always possible to determine if an event have occurred before another one. Also the duration between two events is easily calculated.

The fact that events are temporally related and created at specific points are one of the single factors which makes event processing a bit tricky with traditional data processing tools.

Another interesting feature (see last post) of an event is that they have a globally unique id, often implemented using uuid.

The fact that an event describes something that happened a while ago also means that once an event is created it can’t be changed. Changing it would imply that the history changed. There is also no way to, at least conceptually, delete an event. For practical purposes you would of course allow removing old events. If an event happens, it happens. Nothing to do about it. A data item can on the other hand exist for a short period of time and then vanish. The action of deleting an data item on the other hand could lead to the generation of an event. That event can’t magically disappear as it describes an historical events, and as far as I know you can’t change history.

This is a few more characteristics of an event processing solution you could look for in order to understand it the system is actually processing events or normal data. If you really need event processing, select a true event processing solution and not a re-named data processing tool.

In summary:

• Anything that happens, happens.
• Anything that, in happening, causes something else to happen, causes something else to happen.
• Anything that, in happening, causes itself to happen again, happens again.
• It doesn’t necessarily do it in chronological order, though.

One of the features which are used to sell event processing systems are their ability to keep track of which event caused something to happen.

This might sound simple. But it’s actually something which is more or less impossible to do if you are not treating events as events. Events could just be data which is called events to make things sound more impressive. Data processing is old, but event processing is new and cool.

Events have an id. Data does not.

For example, If I write two SQL queries which both return the value 12, if I then store these both values, then I would later on have no way of knowing which value came from which query. Data is simply anonymous. Events on the other hand have distinct and unique identities. Two stored events can always be identified.

This is one of the fundamental differences between data processing and event processing. Continuous and real-time processing of data streams can be easily confused with event processing. Although a great number of problems require data-driven algorithms which are executed continuously in real-time - That’s not event processing.

This is one of the first and fundamental issue that one should sort out before starting to build event processing systems. Is it really event processing you need or is it data driven processing?

One of the best ways to try to understand your requirements is to ask yourself if your basic processing entity has an id or not. If it has an id, you might need and event processing solution. If not, go for data processing.

For example. A stream of price information for iron ore could contain values like 65.0, 73.5, 80.0, 70.2, 63.0 and so on. If you need to react immediately when the price goes below 44.5, then you would probably be doing data-driven processing. Not event-processing. Each price update is totally anonymous.

If on the other hand you are monitoring the iron ore market to see when three suppliers can offer iron ore for less than 55.0/whatever iron ore is sold as, then you might have an event stream with events like PriceOffer(55.0, SupplierX, TermZ, ValidUntil(2222-01-01)). This would be an unique offer with an id. This is event processing.

Why is this important? Currently the event processing market consists of tools which can do data processing (but sold as event processing tools), a mix of event processing and data processing and pure event processing solutions.  Pick the wrong one and you get the same type of problems as you would when you try to do hard core data analytics with an OLTP database. Simply the wrong tool in a shiny box.

This is also important for the event processing community. If customers try to use event processing for the wrong kind of problems they will just be disappointed and event processing can easily be regarded as nothing more than hype. There is great value in true event processing, the vendors know it. But we need to make sure that the customers get some real business value of our systems and not just another toy.

Apparently there are a increasing number of international companies which consider Sweden the cheapest place to develop software.

I had a interesting lunch conversation with a client and he told me that a number of companies are developing their software in Sweden in order to reduce cost. If you know a bit about the system over here this might come as a surprise. It is totally unintuitive when looking at the facts:
Total tax ratio as percentage of GDP is over 50%. Some highlights:

• VAT (sales tax) 25% on most items.
• 70% of the price of fuel is tax. Diesel is now almost at 15kr/litre ($9.4/gallon). With the nice twist that we pay sales tax in the price including tax. Tax on tax must be a Swedish innovation.
• about 33%-52% of your salary goes to income tax
• A company pays an additional tax of about 30% on a employees salary.
• A company pays normally an additional 5% of the salary into a retirement fund
• 28% of a company’s profits goes to tax.
• Six weeks of summer vacation is normal
• You get payed to stay home with sick children.
• You can stay at home, also payed time, with your children for over a year.

[A common bumber sticker: Born free, taxed to death]

So how could anyone think a country with this kind of structure to be the most cost effective place on the earth to develop hi-tech software?

In order to understand why Sweden might actually be the worlds best place to develop software you have to look beyond hard facts. It’s a result of the Swedish mentality and the way our system works.

First of all, senior software developers are very inexpensive here. The difference in pay between a newbie and someone with 10+ years of experience is rather small. A fact of our socialist history where all swedes were thought that everyone is equal and nobody should receive special treatment, including high salary. The tax on work is also progressive so you will keep less and less of your earnings if you increase your salary. So there is more or less not point in asking for a raise when 50% of that goes to tax. Highly skilled inexpensive developers are the foundation.

The second interesting observation is that the Swedish labor laws makes it more or less impossible to fire someone. This sounds like a bad thing. But consider how software is created. It is a creative process where everyone must be able to argue against a design proposal, even if it comes from your managers manager, and criticizes all bad decisions. When you can’t get fired, this is easy to do and comes naturally. So we have a way of working where everyone feels that they can say whatever they like and still have a job to go to the next day. This creates extremely efficient software development teams.

A consequence of that is that every software engineer, independent of their formal position, can normally make their own decisions in many matters without consulting a manager. The managers would not be able to make an informed decision anyway. Our style is more that we inform our managers what we are up to. This is also a good thing in a software project where every single staff member have to make dozens of decisions every day. Clearing them all with a manager just takes too long time. Also a very efficient way of working.

Also, the high taxes of Sweden forces us to be really good at things where you can create lots of value per hour. Meaning high-tech. Sweden has two car companies, two truck manufacturers. Produces the worlds deadliest sub-marine, have created third generation fighter jet, have a world leading telco, home of IKEA, ABBA, inventor of the tetra pack and so on… All done with a population where the work force consists of five million people. A country of the size of Bangalore and half of the size of New York City.

My own international experience supports all this. A common frustration when we deal with other countries is that everything is so slow and bureaucratic. Think of Sweden as a whole country where the modus operandi of most companies are the one of the innovative and fast moving start-up. Add to that the fact the almost everyone speaks good English.

So if you are a international company with a well defined project and you ask how much it costs to produce a piece of software to a fixed price. Then amazingly we are the world leaders in this. A given specification built to match given quality standards is inexpensive to build in Sweden IF the project is hi-tech and requires substantial amount of creative thinking.

What has this to do with CEP? Probably not a thing, just some personal reflections…

But when dealing with swedes, be aware that we still have some pre-Christian celebration going around here from time to time. So for example, don’t expect anyone to be accessible tomorrow as it’s a public holiday spent in celebrating Midsummer’s Eve. Valborg is another one. Apart from those we are pretty normal.

Not a revolutionary piece of news, but anyway:

Progress Software Corporation (NASDAQ: PRGS), a global supplier of application infrastructure software used to develop, deploy, integrate and manage business applications today announced a technology development collaboration in which the Progress Apama complex event processing (CEP) platform has been ported to the Sun x64 system, running the Solaris 10 Operating System. 

It seems to be CEP release season, a new one from both Progress and StreamBase

What is interesting that the three major points highlighted in the StreamBase release are all ease-of-use features. No mention of new event processing functionality in the release. And no stupid looking performance claims. One interesting point is:

“As the CEP market continues to grow, business analysts and developers demand more sophisticated tools.  StreamBase has always had a singular focus on ease-of-use for business analysts and IT developers to graphically build, deploy, and evolve applications.”

Event processing is difficult to understand so this is a great move for the users. It also seems that StreamBase thinks that business analysts, in addition to developers, are a typical user of CEP tools. Interesting. Great move StreamBase!
So now I’m just waiting for 6.0 of Coral8 and some activity from IBM to see what they have done with AptSoft… And why not something new from Syndera while we are at it… (We have already released ruleCore 9 preview, we just did not tell anybody outside our client base about it… )

I’m just waiting to see amazing GUI tools from all vendors this year. For the sake of the customers, I hope great GUI tools will be the next race. Instead of claims of performance, which Moore will solve for us anyway soon, so anyone trying to live purely by adding performance to the race will loose badly.

Aleri is preparing to release a new version of their streaming data processing platform. There has been lot of discussion in the CEP community if the SQL based approaches (Coral8, StreamBase etc) are suitable for event processing. One of the key features highlighted by Aleri is:

Enhanced scripting – Aleri’s procedural scripting language, SPLASH, which overcomes the limitations of SQL, has been enhanced with new data structures and greater control and is the industry’s only hybrid CEP platform that combines SQL with relational operations. Here

Apparently they feel that SQL is not enough so they will introduce a new scripting language to improve upon the SQL based approach. Sounds like a good idea to me, I hope I can get my hands on this new version and play around a bit with SPLASH (who came up with this name?). It will be interesting to see how this new language feels and what approach they are using.

Visit the Event Processing Technical Society - Here

How error handling is done is one of the most defining aspects of any development language/platform/system.

For example, one of the benefits C++ has over C is the structured (well, more structured than C) error handling with exceptions. Exception handling is a common error handling concept in many of the traditional programming languages. It fits nicely into the call/response cycle. Just call any number of functions/methods and exceptions are raised in case something goes wrong:

try:

function1(1,2,3)

function2(a,v,b)

function3("hello")

except FatalError:

print "Error"

This is an improvement for the old method where the programmer had to check for errors after each call to a function, otherwise the errors were not noticed. At last not directly.

Although clever, this style of error management don’t fit into the asynchronous event driven world of CEP where loosely coupled components acomplish a task by using the publish/subscribe way of indirect communication.

If you are used to error management in a traditional request/response world, be prepared for an unpleasant surprise. Detecting and handling errors are not that simple in the complex world of CEP. If done properly it’s even simpler. If you get your event architecture wrong - It’s a complete nightmare.

The basic processing model of an event driven architecture is:

1. receive event
2. process it
3. optionally, publish event

It’s that simple. Now, things might get FUBAR in completely new ways to a traditional distributed programmer used to web services, (xml)-rpc and other synchronous ways of achieving something:

Missing Events - When creating your processing logic you use more or less implicit assumptions that you will actually receive the particular type of events needed by your logic. What if that kind of event never shows up? Is it ok? Is it annoying? Is it completely wrong? Is it fatal for your continued processing?

Unexpected events - What if you receive an event you don’t expect? Just ignore it? Note it and then ignore it? Just throw it away? Events that you did not anticipate in your initial design might or might not be a problem.

Assumptions not shared by others - If you publish an event and expect that event to lead to some kind of processing and eventually resulting in an event you are waiting for, you might be in trouble. Other components might not at all care about the event you publish. There might not be a single node anywhere expecting the kind of events you publish (see above).  This kind of  (lack of) coordination requires carefully designed event driven systems in order to work properly.

Timing assumptions - Event if you manage to get everything logically connected in the right way, there is another dimension in event processing, and that is time. Do you assume that processing is done in reasonable time as part of your design? Bad idea when it comes to loosely couple components.

Loosing track of state -  If your processing is state dependent, say you like to drive your state forward for each event you receive, then there’s the possibility that you get out of sync with the flow of events. Say you like to detect a sequence of 10 events. When you have seen 8 of them someone kills your application for just a second. During that second you miss event number 9 and will this be waiting for event number 10 forever.

So new failure modes is just one of the things you have to re-learn when getting into event processing. But still, it’s worth it. Event processing is more or less the only way to build really scalable, huge (and I mean huge) system with insane amounts of complexity and processing power.What we see in CEP today is just the first step. If you were around in the early days of databases, think this as the pre-SQL era.

I have been watching the marketing efforts of a number of tech companies for the past years. What strikes me is that I commonly have problems in understanding what their products actually do. Most marketing material seem to be written on a really high level so it could be re-used basically for any tech company. The Dilbert mission statement generator is not too far off.

When there’s some substance, most marketing messages tries to convince me as a reader that their amazing software can do just about anything. Just plug it in and you start making more money. Not likely.
When the marketing is good, and actually manages to convey some usable information about the features and benefits of a product there’s always something missing.

What doesn’t the product do?

When designing software, you always have to draw the line somewhere.

This is probably the number one no-no of marketing. Never tell anyone about the limits of your solution. But I never attended marketing school, instead I studied systems programming. So…

Here’s what ruleCore don’t do, I might be the worlds worst marketing guy:

• No event pull - There’s no way ruleCore can go out and fetch an event. It just sits there and waits for events to arrive. It’s event driven. Also, there is no way to pull an event out of ruleCore. You just have to sit there and wait for a event to arrive from ruleCore.
• No event routing - It’s a black box accepting, processing and producing events. It has no knowledge  of where the events come from or who wants them. We just assume that ruleCore will not be used in a vacuum. Tools like message brokers, ESBs or other MOM infrastructure is much better at dispatching events so we’ll just let them do that instead of re-inventing parts of their functionality inside ruleCore.
• Not a programming language. - Using ruleCore should not be a programming exercise. Actually you can’t write anything that looks like procedural code. The concepts are on a much higher level. Avoiding programming also make things simpler, not as generic as a full blown turing complete programming environment. But more suited for the task for which it was designed for, namely looking for complex event patterns. The drawback is that it’s hard to do things with ruleCore for which it was not designed for. Like brewing coffee.
• Machines create rules. Rules are created by other machines. Albeit in response (sometimes) to human action. But the concept of detecting complex event patterns is just too complex to require end-user to understand it, they should have their mind on their business and not on technicalities of event processing. It’s better left specialists who create small machines which actually create the rules. Rule creation and management should also be very dynamic process. So there could possibly be thousands of new rules created, changed and deleted each day. A task not suitable for humans. So the rule management interfaces in ruleCore are created to be machine friendly, not for humans (programmers love them but they are not the average humans anyway). This means XML with XSD and allowing every possible rule management task to be done remotely using events.

I’m a strong believer that a software solution should stick to doing one thing really good and be designed to nicely fit into a larger picture with other tools which all solve their part of the whole problem. This allows customers to create powerful enterprise architectures by mixing different kind of components. We strive to make ruleCore one of those components focused on tracking and detecting complex event patterns.

This is the kind of job I spent hour daydreaming of when at the university:
“This position requires hands-on development expertise, developing innovative solutions for scalable, high performance distributed systems.” - found here

Now, 15 years later of server side coding I start to realize how hard this stuff is to build…

So if you like to work with CEP and like a challange, apply at StreamBase. Now. My guess is that this is one of the most interesting jobs a software developer could have.

“We are extremely pleased with Coral8 Engine’s agility and flexibility, as well as the responsiveness of Coral8’s engineering organization and technical support team,” said Dan Zulu, head of IT at Nedbank Capital. “During the evaluation period, we developed and deployed in just three months an initial set of market-making applications based on Coral8’s CEP platform. Our next step is migrating existing strategies and algorithms from our legacy arbitrage and basket trading applications to the new architecture. This daunting task would typically take five man-years to complete but can be done in approximately three months on Coral8’s platform. The new CEP infrastructure reinforces our customer-centric business model, allowing us to provide a superior customer experience by delivering more new products more often.”

Nedbank Capital evaluated many US- and UK-based CEP vendors to address a number of business challenges. For instance, having to custom code applications led to costly, long development cycles and slowed time-to-market of new services. Also, Nedbank Capital lacked a shared infrastructure for responding to evolving market dynamics and needed to address the stovepipe applications that were costly to maintain. Finally, limited availability of programmers skilled in both programming and industry-domain knowledge in South Africa made it difficult to staff projects.

Nedbank Capital clearly needed a new platform that would support the development of next-generation market-making applications and the bank’s expansion.

Coral8 Engine was selected after a rigorous evaluation because it meets Nedbank Capital’s requirements:

• Offers a flexible, dynamic server architecture with an agile infrastructure capable of responding immediately to changing business requirements
• Easily integrates with multiple data sources and different trading applications, in a matter of one to two days
• Is powered by familiar, SQL-based Continuous Computation Language (CCL) that reduces the learning curve and speeds development cycles for even the most sophisticated market-making strategies and algorithms
• The CCL syntax relieves the staffing pain by enabling the bank to hire developers with business domain expertise but who may have only high-level programming skills

Equally important to the bank was Coral8’s open evaluation and development business model, giving businesses access – even during the trial phase – to production-grade software and to the technical support team.

“We are delighted to be a strategic part of Nedbank Capital’s expansion and provide the CEP platform that is helping to transform the bank’s trading environment,” said Terry Cunningham, Coral8 CEO. “With Nedbank Capital, Coral8 now serves customers on every continent. Our global deployment gives us insight into a wide range of business and technology issues enterprises are facing worldwide. Ultimately, the knowledge we are building will benefit all our customers.”

Nedbank Group Limited is one of the four largest banking groups in South Africa offering a wide range of business, corporate and retail banking, property finance, investment banking, private banking, foreign exchange and securities trading services. Nedbank Capital comprises the group’s investment banking businesses that together manage the structuring, lending, underwriting and trading businesses.

About Coral8

Based in Mountain View, Calif., Coral8, Inc. is a leading provider of software for Complex Event Processing (CEP). Bringing together high-performance, innovative SQL-based programming language and enterprise-class features, Coral8 Engine is the fastest, lowest cost way to build and deploy powerful, sophisticated CEP applications. Coral8 is speeding the delivery of critical business information for customers worldwide, including Fortune 500 companies and global leaders within financial services, e-commerce, telecommunications, transportation, government and other rapidly growing vertical applications. For more information, visit www.coral8.com or call (650) 210-3810.

Coral8, Inc.
Anna Luo, 650-210-3819
Director of Marketing
anna@coral8.com
or
Schwartz Communications
Jill Reed or Katy Garlinghouse, 415-512-0770
coral8@schwartz-pr.com

In an article in Wall Street Technology, Iwy Schmerken writes about end user’s perception of CEP:

“One of the main myths is that the technology is mature. According to panelists, CEP technology is immature. Another myth is that it’s a sell-side only technology. A third myth is that it’s out-of-the box and easy to use.”

It seems that the end-users are not swallowing the marketing hype. Which is good. Realistic expectations on CEP technology is the best thing that can happen to the CEP vendors. Even if some marketing efforts wants the customer to think that CEP is the solution to basically all problems, except perhaps carbon dioxide emissions.

We have a slightly different marketing culture here in Europe. So marketing messages from US companies are also many times perceived to be full of, eh, a bit unrealistic… A hint to CEP vendors, marketing is a bit different over here so use a local firm to adjust your message. And no, EU is not an unified entity. All countries are different and need local representation. You need to support local languages to be really good over here. There are actually laws in some countries that local language must be used in official documents.
Back to CEP - CEP marketing of some of the leading vendors says that their technology is mature. Using corporate marketing math that might be true. But customers tend to favor the more traditional arithmetic methods. So when a company founded some five years* ago talks about mature technology most customers sees a problem with that message…

Also you might want to consider that the Internet generation is now in charge for many decisions. Easy access to evaluation software and fairly transparent companies gets high scores in evaluations. We are used to open source and total insight into everything around a product. Like future plans, bug databases and access to developers. Don’t miss this one. It’s more important than you think.

Enough marketing. Back to more interesting technical issues like how to scale ruleCore on a 100 server grid…
* I have a personal theory that it takes ten year to build a really good (market leading) piece of software.

By way of my trusted friend Google Alerts I just found this article:
http://www.securitiesindustry.com/news/22370-1.html:

“Sybase RAP Trading Edition streams real-time market data with submillisecond latency, while transaction information moves at speeds ranging from sub-seconds to seconds and analytical data on disks can take minutes to reach applications. Sybase expects the platform will be used for high-performance quantitative analysis, real-time trade analytics, intraday risk analysis and regulatory-driven analytics.”

Sounds like CEP to me. I see two options. Either they built something themselves or did something smarter like OEMed a CEP engine under the hood and just added some user visible parts. My best guess would be Coral8. That’s the only engine in the marketplace that I know of that would be a good solution to build something like this on.

Even if I’m totally into CEP technology and like CEP partially for the cool technology I like when the benefits of CEP is visible in new software offerings, but presented in a more customer friendly way. Customers tend to favor spending € on solutions to problems and not just another piece of cool technology.

Another totally CEP unrelated reflection is that this is available for Linux and Solaris. Where did Windows go? It seems not even be an option anymore for new software offerings. When I think of it, I can’t remember a single startup building their future on Windows.

If you think current CEP products are fast, think again:

“The PL10000 can handle up to 5 million subscribers and can track 48 million real-time data flows.” - here

ok, so the thing is built to inspect network packets. But that’s pretty close to event processing. Would it not be nice if this thing could be updated with new software for algo trading or some other high performance CEP task:

“And because the PL10000 doesn’t use custom ASICs to get the job done, protocol identifications can be updated in software just as fast as Procera can issue them”

Suddenly all the existing CEP products seems veeery veeery slow. When Procera realizes that they can slap a new sticker on this beast saying “Monster Event Processor” and write some new software they have a really cool CEP product.

http://www.proceranetworks.com/products/pl10000.html

Or maybe they already have half of that in place:

“DRDL™ (Datastream Recognition Definition Language) is at the core of what makes PacketLogic stand out from the competition. This state-of-the-art traffic identification and classification engine drives all PacketLogic platforms from the PL10000 to the PL5600.”

Maybe I should OEM this thing and turn it into a nice cool CEP product myself…

It seems that the ‘ol Mighty Gartner started to look at this side of the great pond too:

“Thanks to its fascinating BI real-time solution Gartner Inc. has just voted Austrian software company SENACTIVE most convincing and innovative solutions provider for business intelligence and performance management in their Cool Vendor Report 2008.” - Senactive Press release

It’s nice to see that European companies also get some attention even though the center of gravity still seems to be on the west coast of US.

Dennis Byron from ebizQ have also found us Europeans, he has written a small summary of event processing and open source and you can find his findings here. Annoyingly you have to register to read that article. Short summary would be:

“As with many open source activities, there is a strong European flavor with Paremus and Seewhy based in the UK, ruleCore based in Sweden, and Senactive in Austria. On the other hand, you cannot get more American-based than EsperTech of Wayne, NJ, USA.”

But it’s worth registering to read the complete article.

One positive trend that I have spotted when reading articles like these from analysts is that they are at last starting to get it right! Now the writers own conclusions are more or less consistent with reality and the way vendors, products and the CEP field is presented starts to sound really good with less misunderstandings about what we actually do. A great improvement from a couple of years back.

I just got an e-mail from Dr. Jürgen Krämer asking their company to be included in my listing of CEP vendors. Which I gladly do. It’s nice to see that the CEP market is growing especially here in Europe. Their product seems to be using the SQL approach, like Coral8, Esper, BEA, StreamBase and Aleri. I have not checked in detail how their offering looks like but it’s starts to be interesting to see that there are so many SQL based products now so that it would actually be a good idea to compare them all. Here’s an short description of their company - I wish them best of luck.

We are currently looking for a person to join us to take charge of our sales and marketing effort.

We have a really great product and powerful concept. Now we need great marketing and sales.

The right person would know a lot of the CEP world and would have lots of ideas on how to market and sell  something like ruleCore. We need someone who can work both on our strategic goals and is really good at closing deals. You probably know a bit about ruleCore already and have tons of ideas on how to turn this into a profitable company.
What we can offer is a partnership into a company well positioned to be part of what Gartner says will be
the Next Big Thing. We have a great product and there’s more exciting things developed as we speak.

We like to think in new creative ways - after all its 2008, and marketing and sales can be done in many creative ways these days. New ideas are very welcome, we appreciate new creative minds. For example your geographical location is not very critical and this might also be an opportunity for companies willing to take to role of our sales/marketing partner.

Interested? Contact me at marco@rulecore.com

As I wrote earlier I have been playing around with some ideas around the Semantic Web and how to turn these into Semantic CEP. I got a comment from Jack Rusher from Aleri who also seems to have some Sementic CEP ideas.

My first idea how to use this is to be able to categorize events dynamically as they happen. For instance assigning dynamic roles to entities from which the events are notifications from. This could allow us to create semantically related groups of events which are dynamically updated as new inbound events arrive. (in the ruleCore world time is events too so we don’t have to think about temporal aspects separately).

This is one of the simplest uses I have in mind for adding “Semantic CEP” into ruleCore. What I’m experimenting, still a napkin product so don’t hold your breath, with it to find a simple model where the user don’t have to care about the details of all the semantic technology beneath.

It will be interesting to see if Tibco, BEA, Apama or IBM will produce something in this area soon, which I predict they will. Or perhaps the smaller vendors - Aleri, Coral8, StreamBase, Syndera - will get there first as innovation is not what the big ones are good at. But there’s lot of thinking in the semantic web community so for this there would possible by less innovation to do and more engineering, which at the big companies are good at.

Was Apt-Soft, now  WebSphere Business Events
http://www-306.ibm.com/software/integration/wbe/

Another proof that, some some reason, innovation seems impossible to achieve in big companies. It’s easier to buy a small innovative company than to build something yourself, even if you have great research (IBM Labs Haifa) in the area…

Sometimes it’s nice to get verification from third parties that we are on track:
http://www.ebizq.net/blogs/bethgb/archives/2008/04/bi_meets_event.php
As you might have noticed our marketing message is centred around operational BI. We designed ruleCore just for this particular area* so it’s nice to see that others agree with us that operational BI is one interesting area for CEP.

Many of the offerings from different vendors are complete event processing solutions usable for almost any event processing task. We have on the contrary designed ruleCore to target operational BI developers.

The drawback is that ruleCore is not too good at, for example, algo trading, but the benefit is that we try to be really, really good at supporting operational situation tracking and instant detection of exceptional conditions.

* Actually it was called BAM - Business Activity Monitoring  - back then. But the concepts are pretty much the same. Perhaps with a shift from real time dashboards (who has the time to watch those anyway?) to automatic tracking and tracing of interesting things.

Company blogs tends to be rather boring. Nothing much than poorly camouflaged attempts at marketing. Lately I have been reading Aleri’s blog which actually contain a lot of interesting stuff.

For example, Jack Rusher writes about Semantic CEP, or whatever we should call it. I think this is the next cool thing from many CEP vendors. I’m pretty sure that there’s activity within this field at many of the CEP vendors as we speak.

It will be interesting to see who will get there first and announce support for true semantic processing. I have been having the same ideas on my drawing board for a some years now. I got a lot of ideas by taking part in the REWERSE, a “network of excellence” funded by the EU commission and Switzerland.

There’s lots of things happening in semantic technology in the scope of the web. I think we in the CEP community can use lots of this stuff in our world. Like OWL, RDF, SPARQL.

I think this is a interesting area and will give a lot of new interesting capabilities. For example it could use ontologies to process events related to certain areas, for example everything considered to be a threat to public health.

With Semantic CEP in place we could easily describe situations like “more than five events related to public health during the last hour”.

Semantic CEP will allow us to describe situations in a way that is more relevant to end-users.

I’ll continue looking into this field to see how we can put some of this stuff into ruleCore. The hard part is how to create usable abstractions which are not too hard to use without being an semantic technology expert. The terminology in this field can be a bit confusing and terrifying.

My first idea is to put some semantic technology into our event stream views, so one could maintain dynamic views into the inbound event stream based on semantic properties found in the event payload.

I just stumbled (well, Google Alerts helped out a bit) onto a job ad for a pretty cool job, just look at the spec:

1. Experience with Tibco BusinessEvents or other event process tool - 2+ years
2. Experience with JMS and Message Queues Architectures - 4+ years
3. Experience with Java and J2EE - 2+ years
4. Development experience in XML messaging, XLST, XSD and XPATH.
5. Experience of using and interpreting UML / Use-Case analysis and functional specifications to design solutions a plus - 2+ years
6. Experience in SOA framework architecture.

Two year CEP plus some additional pretty heavy stuff. This is for me typical senior consulting stuff. And what’s the pay? $50 (about €32)!!! I would expect closer to $200/h for this kind of position… Is the consulting business in the US this bad?

The job

But anyway, I’m glad to see that CEP experience found its way into wanted ads….

The last weeks have been very exiting with great response to our announcement of ruleCore CEP Server 2008. Hopefully all the work that’s been put into the product, which I personally think is really great, can solve some interesting problems at our clients.

It seems that many CEP vendors are pushing performance as their main selling point. The ruleCore as a Service or CEP as Saas as the marketing nightmare term is, tries to go the other way. Bring inexpensive and simple event processing to everyone. The only fear is that we will be seen as a simpler low cost alternative. Low cost is ok, but as for situation detection I actually think we are one of the leaders among the CEP vendors.
It will be very interesting to see how this is received by our customers. Just pay as you go and don’t care about servers, upgrades or anything. We host everything for them.

At 0.0001 euro/event I think it’s a pretty nice price too.

The most common question is about security and I think we have to create a ssh tunnel or something like that for those customers that likes to have some privacy. It’s been on the drawing board for some time so it might be the time to actually implement that feature now.

There’s more to come from us too…. We are working on a dot release due in just a month or so. A collection of the normal fixes and improvements that did not make it to the 8 release and a few minor (but usable) new features. It’s mainly a performance, stability and documentation improvement release as I see it.

On a unrelated note, SPRING is at last here in Sweden!!! Now it’s time to go out and enjoy the sun for a while