Summer Break
July 12th, 2010
The blog is taking a short break while I’m relaxing away from office… See you all in the spring.. in a couple of weeks…
Up for a Challenge?
July 8th, 2010
Event Processing in Action – The Book
June 25th, 2010
Short summary: buy the book!
I just started to read "Event Processing in Action", the final version. It’s a brand new book coming to your favorite store any day.
I really like this book. It’s so good. This is the kind of book on event processing that I have been waiting for. I think the authors, Opher Etzion and Peter Niblett, really gets event processing.
Hopefully this book will teach many professionals and students that there is something more to event processing than just real-time processing of data. The fact that we are processing events and not data should be obvious as one reads this book.
I wish I had more time to read. But today Sweden is closed. Nobody works. So don’t even think of calling anybody here today. We are celebrating Midsummer’s Eve. One of those really old traditions up here in the north. If you live in a place where it’s dark and cold most of the year you surely appreciate a good party around the time for summer solstice. If I were younger, I would have been drunk all day. Now, it’s more like spending time with my wonderful family and good friends and their kids.
Technology for the Benefit of Mankind (at least some selected men)
June 17th, 2010
At least a few high-frequency traders have learned to make a killing by detecting the more simplistic algo strategies deployed by basic pension funds and mutual funds, buying the next stock the funds plan to buy, and then selling it to them at a higher price. This may not be illegal, but it’s almost certainly unfair to the funds’ investors. “It is increasingly clear that there are quite a number of high-frequency bandits in the high- frequency-trading community who pump up volume statistics, front-run investor orders, increase transaction costs, and hurt real liquidity,” David Weild, an adviser at Grant Thornton and a former vice chairman of Nasdaq, told me. – http://www.theatlantic.com/magazine/archive/2010/07/monsters-in-the-market/8122/
When this is what the public reads about CEP, then I think we might have a minor image problem.
Wanna Read more about StreamInsight?
June 11th, 2010
Microsofts entry into CEP space starts to look good and here’s some links (courtesy of Badrish Chandramouli from the StreamInsight team, thanks!) to those interested in reading more.
StreamInsight webpage (where you can find some more recent information such as white papers, help documents, examples, trial download):
StreamInsight blog and public forum:
Research papers:
The original CIDR paper (some of the stuff in this paper is out-dated, for example, the presence of transaction-time in the logical tables):
A technical report about some more detailed implementation aspects:
Two more recent papers from the StremInsight group:
Recursive query processing in a streaming system with temporal algebra:
A description of StreamInsight and behavioral targeted advertising:
ETALIS – A new Declarative CEP Rule Language
June 2nd, 2010
Many new developments in the CEP space seems to be searching for other types of languages than the popular streaming extension to SQL.
ETALIS is one of them, it is an open source engine for event processing based on the Logic Programming paradigm.
ETALIS engine implements a corresponding “ETALIS Language for Events” that is a rule-based declarative language. ETALIS is a research prototype created by Darko Anicic at FZI Forschungszentrum Informatik an der Universität Karlsruhe.
The ETALIS project is founded on novel algorithms, and implementing event-driven backward chaining rules. The rules enable derivation of complex events in a data-driven fashion. Apart from event processing, ETALIS features strong inference capabilities as well as easy integration with databases and transactions.
Future plans include to develop some unique features of ETALIS that can be particularly efficient or easy implemented in a *logic-based* EP.
For instance, event-driven workflows that are dynamically modified while executing can be one area of research. Workflow modifications, being done on-the-fly, need also to be verified at run time (as changes can disable workflow executions or make it infinite though workflows featured the finite termination property). This is one example where a logic approach can help. Event-driven dynamic workflows are useful in many areas including edBPM, Cloud Computing etc.
Another use of ETALIS logic approach in EP is related to event revision and world of database transactions. ETALIS can handle transactions in a logic framework. When transactions are mixed with event processing, event revision become necessary. Transaction are executed in the all-or-nothing fashion. Now events that are triggered during executions of some transactions, that later turned out to be unsuccessful, need to be retracted (in the same vain as transactions are rolled back). These events can be already used in building more complex events. This implies that also these complex events should be retracted too. ETALIS is using the logic to find out the consequences of events retractions on other more complex events.
It’s nice to seen the addition of another declarative CEP rule languate apart from the Reakt language implemented by the ruleCore CEP Server. Even though both languages are rule based and declarative it shows that ETALIS comes from a more formal and theorethically correct research world whereas the Reakt language was designed by software developers without too much concerns of formal correctness.
I will follow ETALIS with great interest as I think it could evolve into a really powerful language.
RuleML-2010, Final Call for Papers
May 24th, 2010
Comments Off
Final Call for Papers RuleML-2010 4th International Web Rule Symposium: Research Based and Industry Focused October 21-23, 2010, Washington, DC, USA Co-located with the 13th Business Rules Forum http://2010.ruleml.org/ Overview and Aim ====================================================== The International Web Rule Symposium has evolved from an annual series of international workshops since 2002, international conferences in 2005 and 2006, and international symposia since 2007. This year, the 4th International Web Rule Symposium (RuleML-2010) will be held near Washington, DC, USA, co-located with the 13th Business Rules Forum, the world's largest Business Rules event. RuleML-2010 is devoted to practical distributed rule technologies and rule-based applications, which need language standards for rules (inter)operating in, e.g., the Semantic Web, Enterprise Systems, Intelligent Multi-Agent Systems, Event-Driven Architectures, and Service-Oriented Applications. RuleML-2010 is a research-based, industry-focused symposium: its main goal is to build a bridge between academia and industry in the field of rules and semantic technology, and so to stimulate the cooperation and interoperability between business and research, by bringing together rule system providers, participants in rule standardization efforts, open source communities, practitioners, and researchers. The concept of the symposium has also advanced continuously in the face of extremely rapid progress in practical rule and event processing technologies. As a result, RuleML-2010 will feature hands-on demonstrations and challenges alongside a wide range of thematic tracks. It will thus be an exciting venue to exchange new ideas and experiences on all issues related to the engineering, management, integration, interoperation, and interchange of rules in distributed enterprise intranets and open distributed environments. Conference Theme ====================================================== This year, we particularly welcome submissions that address applications of Web rule technologies for business and information systems. We invite you to share your ideas, results, and experiences: as an industry practitioner, rule system provider, technical expert, developer, rule user or researcher, exploring foundations, developing systems and applications, or using rule-based systems. We invite high-quality submissions related to (but not limited to) one or more of the following topics: Track Topics ------------------------------------------------------------------------------ - Rules, Semantic Technology, and Cross-Industry Standards Rules in current industry standards, including: * XBRL: Extensible Business Reporting Language * MISMO: Mortgage Industry Standards Maintenance Org * FIXatdl: FIX Algorithmic Trading Definition Language * FpML: Financial products Markup Language * HL7: Health Level 7 * Acord: Association for Cooperative Operations Research and Development (Insurance Industry) * Rules for Governance, Risk, and Compliance (GRC), e.g., rules for internal audit, SOX compliance, enterprise risk management (ERM), operational risk, etc * Rules and Corporate Actions - Rule Transformation and Extraction * Transformation and extraction with rule standards, such as SBVR, RIF and OCL * Extraction of rules from code * Transformation and extraction in the context of frameworks such as KDM (Knowledge Discovery meta-model) * Extraction of rules from natural language * Transformation or rules from one dialect into another - Rules and Uncertainty * Languages for the formalization of uncertainty rules * Probabilistic, fuzzy and other rule frameworks for reasoning with uncertain or incomplete information * Handling inconsistent or disparate rules using uncertainty * Uncertainty extensions of event processing rules, business rules, reactive rules, causal rules, derivation rules, association rules, or transformation rules - Rules and Norms * Methodologies for modeling regulations using both ontologies and rules * Defeasibility and norms: modeling rule exceptions and priority relations among rules * The relationship between rules and legal argumentation schemes * Rule language requirements for the "isomorphic" modeling of legislation * Rule based inference mechanism for legal reasoning * E-contracting and automated negotiations with rule-based declarative strategies - Rules and Inferencing * From rules to FOL to modal logics * Rule-based non-monotonic reasoning * Rule-based reasoning with modalities * Deontic rule-based reasoning * Temporal rule-based reasoning * Priorities handling in rule-based systems * Defeasible reasoning * Rule-based reasoning about context and its use in smart environments * Combination of rules and ontologies * Modularity - Rule-based Event Processing and Reaction Rules * Reaction rule languages and engines (production rules, ECA rules, logic event action formalisms, vocabularies/ontologies) * State management approaches and frameworks * Concurrency control and scalability * Event and action definition, detection, consumption, termination, lifecycle management * Dynamic rule-based workflows and intelligent event processing (rule-based CEP) * Non-functional requirements, use of annotations, metadata to capture those * Design time and execution time aspects of rule-based (Semantic) Business Processes Modeling and Management * Practical and business aspects of rule-based (Semantic) Business Process Management (business scenarios, case studies, use cases etc.) - Rule-Based Distributed/Multi-Agent Systems * rule-based specification and verification of distributed and multi-agent system * rule-based distributed reasoning and problem solving * rule-based agent architectures * rules and ontologies for semantic agents * rule-based interaction protocols for multi-agent systems * rules for service-oriented computing (discovery, composition, etc.) * rule-based cooperation, coordination and argumentation in multi-agent systems * rule-based e-contracting and negotiation strategies in multi-agent systems * rule interchange and reasoning interoperation in heterogeneous Distributed/Multi-Agent Systems We also welcome submissions on miscellaneous rule topics, such as - Rules and ontologies - Execution models, rule engines, and environments - Graphical processing, modeling and rendering of rules Case studies, experience reports, and industrial problem statements are particularly encouraged. RuleML-2010 Challenge ====================================================== The RuleML-2010 Challenge is one of the highlights at RuleML-2010 with prestigious prizes. Submissions of benchmarks/evaluations, demos, case studies / use cases, experience reports, best practice solutions (e.g. design patterns, reference architectures, models), rule-based implementations/ tools/ applications, demonstrations engineering methods, implementations of rule standards (e.g. RuleML, RIF, SBVR, PRR, rule-based Event Processing languages, BPMN+rules, BPEL+rules, ...), rules + industrial standards (e.g. XBRL, MISMO, Accord, ...), and industrial problem statements are particularly encouraged. This year, the RuleML-2010 Challenge will have a special focus theme: * Modelling Rules in the temporal and geospatial applications - temporal modelling and reasoning - geospatial modelling and reasoning - cross-linking between temporal and geospatial knowledge - visualization of rules with graphic models in order to support end-user interaction Other themes of the RuleML-2010 Challenge will include the following: * Demos related to the RuleML-2010 Track Topics<http://2010.ruleml.org/topics.html> * Extensions and implementations of W3C RIF * Editing environments and IDEs for Web rules * Benchmarks and comparison results for rule engines * Distributed rule bases and rule services * Reports on industrial experience about rule systems See, for more details (including submission guidelines) http://www.csw.inf.fu-berlin.de/ruleml2010/ruleml-2010-challenge.html Conference Language ====================================================== The official language of the conference will be English. Submission ====================================================== Authors are invited to submit original contributions of practical relevance and technical rigor in the field, experience reports and show/use case demonstrations of effective and practical rule-based technologies, or applications deployed in distributed environments. RuleML-2010 Submission Guidelines, Springer Proceedings, Best Paper Award ------------------------------------------------------------------------ Papers must be in English and may be submitted at http://www.easychair.org/conferences/?conf=ruleml2010 as: Full Papers (15 pages in the proceedings) Short Papers (8 pages in the proceedings) Please upload all submissions by also conforming to the following guidelines: * Papers must be uploaded as PDF files in LNCS format (http://www.springer.de/comp/lncs/authors.html); * Whenever possible, indicate as a first keyword the track to which the paper is submitted. To ensure high quality, submitted papers will be carefully peer-reviewed by 3 PC members based on originality, significance, technical soundness, and clarity of exposition. Authors are requested to upload the abstracts of their papers before May 25, 2010 and to upload their complete papers by June 1, 2010. The selected papers will be published in book form in the Springer Lecture Notes in Computer Science (LNCS) series. The best paper from all submissions will be determined by the PC and a Best Paper Award will be presented at the Symposium. All submissions must be made electronically. Review Process ====================================================== The submitted papers will pass the blind review process. At least three members of the Program Committee will review each submission. Proceedings and Post-proceedings ====================================================== The accepted papers of RuleML 2010 will be published in book form in the Springer Lecture Notes in Computer Science (LNCS) series. Selected papers from the conference will be published in a special issue of the International Journal of Cooperative Information Systems (World Scientific). Selected papers from the track "Rules and Norms" will be published in a special issue of the journal Artificial Intelligence and Law (Springer). Important Dates: ====================================================== RuleML-2010 Symposium ---------------------------------------------- Abstract submission deadline: May 25, 2010 Paper Submission deadline: June 1, 2010 Notification of acceptance: July 7, 2010 Camera ready due: July 28, 2010 Symposium dates: October 21-23, 2010 RuleML-2010 Challenge ---------------------------------------------- Submission deadline for demo papers and demo systems: August 20th, 2010 Notification of accepted demo papers and demo systems: September 3rd, 2010 Submission deadline for demo systems only: September 17th, 2010 (contingent on availability of demo slots) Notification for demo systems only: October 1st, 2010 Conference Venue ====================================================== RuleML-2010 will take place at the Hilton Alexandria Mark Center, which is located 5 miles southwest of Washington, DC. The symposium is co-located with the 13th Business Rules Forum. Hilton Alexandria Mark Center 5000 Seminary Road, Alexandria, Virginia, United States 22311 Internet: http://www1.hilton.com/en_US/hi/hotel/DCAAHHF-Hilton-Alexandria-Mark-Center-Virginia/index.do Registration ====================================================== Registration details will be published in due course in the RuleML-2010 website. All members of partner organizations (W3C, OASIS, OMG, ECCAI, and EPTS) will receive a 15% discount. Program Committee ====================================================== General Chairs -------------------- Mike Dean, Raytheon BBN Technologies, USA Said Tabet, RuleML Initiative, USA Program Chairs -------------------- John Hall, Model Systems, UK Antonino Rotolo, University of Bologna, Italy Liaison Chair -------------------- Mark Proctor, Red Hat, UK Publicity Chair -------------------- Patrick Hung, University of Waterloo, Canada Rule Responder Chairs -------------------- Efstratios Kontopoulos, Aristotle University of Thessaloniki, Greece Kalliopi Kravari, Aristotle University of Thessaloniki, Greece Track Chairs -------------------- Rules, Semantic Technology, and Cross-Industry Standards Tracy Bost, Valocity, USA Robert Golan, DBMind, USA Rule Transformation and Extraction Mark Linehan, IBM, USA Rules and Uncertainty Davide Sottara, University Bologna, Italy Nikolaus Wulff, University of Muenster, Germany Rules and Norms Thomas Gordon, Fraunhofer FOKUS, Germany Guido Governatori, NICTA, Australia Rules and Inferencing Grigoris Antoniou, Information Systems Laboratory, FORTH, Greece Antonis Bikakis, Information Systems Laboratory, FORTH, Greece Rule-based Event Processing and Reaction Rules Alex Kozlenkov, Betfair Ltd., UK Adrian Paschke, Free Univ. Berlin, Germany Rule-Based Distributed/Multi-Agent Systems Nick Bassiliades, Aristotle University of Thessaloniki, Greece Costin Badica, University of Craiova, Romania RuleML-2010 Challenge Chairs ------------------------------ Enrico Francesconi, ITTIG-CNR, Italy Monica Palmirani, University of Bologna, Italy Fabio Vitali, University of Bologna, Italy RuleML-2010 Program Committee ------------------------------ Hassan Ait-Kaci, IBM, Canada Sidney Bailin, Knowledge Evolution, USA Matteo Baldoni, University of Turin, Italy Claudio Bartolini, HP Labs, USA Bernhard Bauer, University of Augsburg, Germany Mikael Berndtsson, University of Skövde, Sweden Pedro Bizarro, University of Coimbra, Portugal Jonathan Bnayahu, IBM Haifa Research Lab, Israel Guido Boella, University of Turin, Italy Peter Bollen, University of Maastricht, Netherlands Jordi Cabot, Universitat Oberta de Catalunya, Spain Carlos Castro, Unversidad Técnica Federico Santa María, Chile Donald Chapin, Business Semantics Ltd, UK Federico Chesani, University of Bologna, Italy Horatiu Cirstea, Loria, France Matteo Cristani, University of Verona, Italy Claudia d'Amato, University of Bari, Italy Juergen Dix, Technische Universitaet Clausthal, Germany Weichang Du, University of New Brunswick, Canada Schahram Dustdar, Vienna University of Technology, Austria Andreas Eberhart, Fluid Operations, Germany Jenny Eriksson Lundström, Uppsala University, Sweden Opher Etzion, IBM, Israel Todd Everett, Nationwide, USA Maribel Fernandez, King’s College London, UK Enrico Francesconi, ITTIG-CNR, Italy Dragan Gasevic, Athabasca University, Canada Adrian Giurca, Brandenburg University of Technology at Cottbus, Germany Ioannis Hatzilygeroudis, University of Patras, Greece Stijn Heymans, Technischen Universität Wien, Austria Minsu Jang, E&T Research Institute, Korea Yuh-Jong Hu, National Chengchi University, Taiwan Yiannis Kompatsiaris, Informatics and Telematics Institute, Greece Manolis Koubarakis, National and Kapodistrian University of Athens, Greece Wolfgang Laun, Thales Rail Signalling Solutions GesmbH, Austria Ching Long Yeh, Tatung University, Taiwan Ian Mackie, Ecole Polytechnique, France Christopher Matheus, Vistology, USA Craig McKenzie, SAIC Ltd., UK Jing Mei, IBM, China Zoran Milosevic, Deontik, Australia Anamaria Moreira, URFN, Brazil Leora Morgenstern, IBM, USA Joerg Mueller, Technische Universitaet Clausthal, Germany Chieko Nakabasami, Toyo University, Japan Monica Palmirani, University of Bologna, Italy Alun Preece, Cardiff University, UK Maher Rahmouni, HP Labs, UK Dave Reynolds, HP Labs, UK Graham Rong, MIT Sloan School of Management, USA Giovanni Sartor, University of Bologna, Italy Marco Seiriö, RuleCore, Sweden Silvie Spreeuwenberg, LibRT, The Netherlands Giorgos Stamou, National Technical University of Athens, Greece Giorgos Stoilos, National Technical University of Athens, Greece Nenad Stojanovic, University of Karlsruhe, Germany Umberto Straccia, ISTI-CNR, Italy Terrance Swift, XSB Inc., USA Leon Van der Torre, University of Luxembourg, Luxembourg Jan Vanthienen, Katholieke Universiteit Leuven, Belgium Fabio Vitali, University of Bologna, Italy George Vouros, University of the Aegean, Greece Kewen Wang, Griffith University, Australia Segev Wasserkrug, IBM, Israel RuleML-2010 Partners ====================================================== Partner Organizations --------------------------------------------------- W3C: The World Wide Web Consortium OASIS: Advancing Open Standards for the Information Society OMG: Object Management Group ECCAI: European Coordinating Committee for Artificial Intelligence EPTS: Event Processing Technical Society Media Partners --------------------------------------------------- Springer LNCS -- LA RICERCA C’È E SI VEDE: 5 per mille all'Università di Bologna - C.F.: 80007010376 http://www.unibo.it/Vademecum5permille.htm Questa informativa è inserita in automatico dal sistema al fine esclusivo della realizzazione dei fini istituzionali dell’ente.
A Prediction on where CEP is heading.
May 18th, 2010
After listening for two days on incrediblty smart people, here’s what I think (at least at the moment) will happen with event processing in, say, 5-10 years…
Elements from sensor fusion, machine learning and other areas will creep into event processing and will start to get more and more specialized event processing solutions.
Most interesting problems in event processing will require that you can do things like "find all event patterns that deviate from the norm" from the inbound streams. That sort off implies that you need to elements from other fields.
The good news is that this will be rather easy, just take existing stuff and adjust it to the event processing context. No need to do re-invent things here…
Workshop on Event-Driven Business Process Management
May 17th, 2010
4th International Workshop on Event-Driven Business Process Management
collocated with BPM 2010
Hoboken, NJ, USA from September 13-16, 2010
http://www.bpm2010.org/conference-events/workshops/edbpm10/
Introduction
————————–
The recently coined term "Event-Driven Business Process Management" (EDBPM)
is nowadays an enhancement of BPM by new concepts of Service Oriented
Architecture (SOA), Event Driven Architecture (EDA), Software as a Service
(SaaS), Business Activity Monitoring (BAM) and Complex Event Processing
(CEP). In this context BPM means a software platform which provides
companies the ability to model, manage, and optimize these processes for
significant gain. As an independent system, CEP is a parallel running
platform that analyses and processes events. The BPM- and the CEP-platform
correspond via events which are produced by the BPM-workflow engine and by
the – if distributed – IT services which are associated with the business
process steps. Also events coming from different event sources in different
forms can trigger a business process or influence the execution of a process
or a service, which can result in another event. Even more, the correlation
of these events in a particular context can be treated as a complex,
business level event, relevant for the execution of other business processes
or services. A business process – arbitrarily fine or coarse grained – can
be seen as a service again and can be "choreographed" with other business
processes or services, even between different enterprises and organizations.
Loosely coupled event-driven architecture for BPM provides significant
benefits:
* Responsiveness. Events can occur at any time from any source and
processes respond to them immediately, whenever they happen and wherever
they happen.
* Agility. New processes can be modeled, implemented, deployed, and
optimized more quickly in response to changing business requirements.
* Flexibility. Processes can span heterogeneous platforms and
programming languages. Participating applications can be upgraded or changed
without breaking the process model.
Workshop Themes
————————–
Authors are invited to submit novel contributions in the prior described
problem domain.
* Event-driven BPM: Concepts
o Role of event processing in BPM
o Business Events: types and representation
o Event stream processing in business processes
o Data- and event-driven business processes
o Evaluation/ROI of event-driven BPM
o Event-driven SOA
o EDA and BPM
o Real/time awareness in BPM
o Context in BPM
* Design-time CEP and BPM
o Modelling languages, notations and methods for event-driven BPM
o Event Patterns: Definition / Creation / Representation /
Learning
o BPMN and event processing
o Modelling unknown/similar events in business processes
o Modelling events in human-oriented tasks
o Semantics/Ontologies for event-driven BPM
o Publish/subscription mechanism and process modelling
* Run-time CEP and BPM
o Event pattern detection
o BPEL and event processing
o Reasoning about unknown/similar events
o Distributed event processing
o Dynamic workflows
o Ad-hoc workflows
* Applications/Use cases for event-driven BPM
o Event-driven monitoring/BAM
o Event-driven SLA monitoring
o Domains: Logistics, Automotive, …
o Event processing and Internet of Services
Workshop Format
————————–
The Workshop is planned as a full-day event, including a keynote, paper
presentations, lightning talks, demos, posters, and a moderated, open
discussion with the clear goal of agreeing upon a research roadmap for
event-driven Business Process Management research, by taking into account
new challenges, described earlier.
A possible agenda:
* 9:00 – 9:30 Opening and Keynote
* 9:30 – 13:00 Paper Presentations
* 13:00 – 14:00 Lunch
* 14:00 – 15:00 Lightning Talks
* 15:30 – 17:00 Moderated Community Discussion: A Roadmap for
Event-driven Business Process Management Research
* 18:00 – 21:00 Poster and Demo Presentations and Get-Together
For the keynote, we aim at a high-profile speaker, who will give a rather
visionary view on the role of Future Internet for BPM and vice versa. For
the moderated community discussion, we will have senior experts from our
Program Committee and Experts from an industrial background. A clear
objective of that discussion is to yield a first draft of a respective
research agenda.
Important Dates
————————–
Deadline paper submissions: 21 May 2010
Notification of acceptance: 30 June 2010
Camera-ready papers: 25 July 2010
Workshops: 13 September 2010
Submission
————————–
The following types of submission are solicited:
* Long paper submissions, describing substantial contributions of novel
ongoing work. Long papers should be at most 12 pages long.
* Short paper submissions, describing work in progress. These papers
should be at most 6 pages long.
* Use case submission, describing results from an edBPM use case. These
papers should be at most 4 pages long.
Papers should be submitted in the new LNBIP format
(http://www.springer.com/computer/lncs?SGWID=0-164-7-487211-0). Papers have
to present original research contributions not concurrently submitted
elsewhere. The title page must contain a short abstract, a classification of
the topics covered, preferably using the list of topics above, and an
indication of the submission category (Long Paper/ Short Paper).
For submission, please visit
http://www.easychair.org/conferences/?conf=edbpm10.
Organizing Committee
————————–
Nenad Stojanovic
FZI – Research Center for Information Technologies at the University of
Karlsruhe, Germany.
Haid-und-Neu-Str. 10-14
D-76131 Karlsruhe, Germany
nstojano (at) fzi.de
URI: http://www.fzi.de/ipe/mitarbeiter.php?id=483
Rainer von Ammon
CITT Regensburg/Germany
Konrad-Adenauerallee 30
D-93051 Regensburg, Germany
rainer.ammon (at) citt-online.com
Opher Etzion
IBM Research Lab in Haifa
OPHER (at) il.ibm.com
Adrian Paschke
Corporate Semantic Web, Free University Berlin, Germany and RuleML Inc.,
Canada
AG-CSW (Corporate Semantic Web)
Institute for Computer Sciences
Free University Berlin
Königin-Luise-Str 24/26
14195 Berlin, Germany
paschke (at) inf-fu-berlin.de
Program Committee
————————–
(see Website)
Coral8, now at SAP
May 13th, 2010
Given that big companies buy other big companies left and right, this might not even be big news even if it is a big deal. For us event processing aficionados this now means that SAP owns Coral8 and Aleri code base. I would be surprised if this is not the end of those products as standalone CEP products…
StreamBase Update
May 3rd, 2010
I have not looked at what StreamBase do for a while now. But as you might read in my previous blog entry I found some financial information about StreamBase and thought I’d take a quick look to see what they are up to. Or, what can you do with $48,000,000 USD?
A new idea they have is the component exchange. I like. This is something all infrastructure platforms need. A place where you can find plug-in components to your favorite tool. For example the Eclipse platform have thousands of plug-ins available on their marketplace.
So I really like the idea. But 18 components? That’s sadly not much of an exchange. It seems that there’s no third party components and StreamBase don’t seem to put much into their own exchange either. But still a good idea.
Another good place to check what’s going on is in the documentation section of the StreamBase site. They have good documentation and a quick check reveals that there are updates published with regular intervals. While browsing the "what’s new" section I smell Financial Markets all over the place. If I would be looking at this for the first time I would put this into the "tools for traders and other people on Wall St." category and not into the ‘generic CEP tool’
I did not bother to download, which I think you still can (a gold star to StreamBase for that) , as there were not enough interesting stuff to look at. Maybe I would be more excited if I were working in the capital markets.
$6M not enough for StreamBase (more like $48M)
April 29th, 2010
It seems that StreamBase needs more money: www.sec.gov/Archives/edgar/data/1315537/000131553710000003/xslFormDX01/primary_doc.xml
I understand that there’s 5.5M involved. Rest is sort of MBA talk, anyone care to translate this to us mere mortals?
Last year they got $6M if I remember correctly. Does this mean that StreamBase burns about 5-6M more than they earn per year?
Edit:
Following the link posted by Marc in the comments section I think (noticing that I don’t really understand what I’m looking at) StreamBase have got about $48M from investors since 05. Is that lots of money compared to what they have achieved?
But having raised $48M (daydreaming on how much we could do with that sum 
), do that mean that they are spending more on development than customers are willing to pay for the software? Or do software companies have other reasons to take money from investors?
At least some investors think that a CEP company is a good place to put a $48M in, that must be a good sign!
CEP Updates
April 27th, 2010
Comments Off
As you might have noticed, I have not had any inspiration to write in my blog lately. This is partly due to a high work load over here at ruleCore HQ. Lots of small things to fix before a new release and the GUI stuff just sucks up all my time.
Also, there seems not much interesting happening in the world of CEP. StreamBase have turned into a financial solutions provider just as I predicted. IBM, Tibco and Progress seems to work on their offerings but there’s not much excitement but the steady flow of gradual improvements to their products. Which is a good thing. But not much to write about.
One inspiring news is the the new Event Processing in Action book by Opher and Peter is more or less ready. I knew it was a good idea to suggest Opher as a author to Manning. I did some non-rocket science analysis and decided rather quickly that writing a book will steal way too much time from my other work. Reading what Opher writes, I was right… And these two guys are way better authors than me anyway. Maybe I’ll return with a more practical hands on companion books some day which can be read after you got the theory in their book. Or hopefully someone else beats me to it, saving me the trouble…
So if anyone have some more inspiring news about the state of the event processing world, please post them!!
Another good news is that I’m going to Dagstuhl next month (anyone wanna share a cab from Turkismuhle?) , that ought to be great fun. The last time I did not have the guts to give a presentation. But this time I think I will do, at least a short one. With that kind of audience, there’s not much that I can say that they don’t already know. But I realized that there’s at least one topic that I’m better at than they are. [insert cliffhanger]. As Opher would say, more later (in Dagstuhl)…
The Reakt CEP Rule Language
April 18th, 2010
Comments Off
Reakt – The ruleCore CEP language
A White Paper from ruleCore
DRAFT
Introduction
The Rulecore CEP Language – Reakt! – When you need to react immediately.
Reakt is a language for complex event processing, or CEP for short. It is designed to provide a powerful and productive tool for developers allowing them to quickly create solutions for detection oriented CEP scenarios.
Reakt is not a traditional programming language. Reakt is a business situation description language. By using a declarative approach it allows the developer to concentrate on what kind of situations to detect and not how
The design goal of Reakt is to provide a way for organisations to react immediately to critical situations. It achieves this goal by finding related events from streams of inbound events. What events a critical situation consists of, and how the events are related, are defined by the user using Reakt. Reakt allows the user to specify conditions and temporal relationships which must hold in order for the events to be considered a noteworthy situation. When the situation is detected by Reakt, an user defined real-time report is created which provides actionable information about the situation.
Allowing the enterprise to react immediately.
This short white paper gives an informal description of the language and explains the different language elements using examples instead of formal definitions.
Overview
Rulecore Reakt is a small domain specific complex event processing language designed for detecting situations by observing streams of events. A situation is a formal description of a combination, a pattern, of events as they happen over time along with optional restrictions among the events. A simple example of situations that you might want to detect could be
"A vehicle has visited zone A more than 10 times during the last hour"
or
"Event A comes before event B, unless event C has been seen"
Main Reakt language features:
- Continuous Evaluation – Designed to operate on live streams of events as they happen.
- Declarative – Rules and other items are specified using a high-level declarative approach.
- Modular – Rules are defined using modular and re-usable building blocks.
- Temporal – Handling of time is an integral part of Reakt. Timing restrictions and other temporal conditions can be stated using a number of declarations.
- Location Aware – Reakt includes built in support to understand location information and other geographical objects such as zones.
- Implementation Independent – The elements of Reakt are defined using standard XML and are implementation independent. Different implementations can be created in order to provide specialized execution environments.
- Automatic Life Cycle Management – Rule instances are created and deleted automatically in response to events. Rule instances are analyzed and deleted if their situation is determined to be undetectable.
- Dynamic Context Management – Each rule instance maintains its own virtual view into the incoming event stream. The view provides a window into the incoming event stream in order to create a dynamic context for rule evaluation.
- Separation of Concerns – Reakt uses reusable blocks for;
- Selection of events from the inbound stream
- Rule life cycle management
- Situation definition
- Rule trigger reporting
The design goals of Reakt is to provide a language which allows the user to describe complex situations consisting of multiple events with non trivial temporal and geospatial relationships. The language is declarative. Lacking any procedural elements provides a base for formal verification and automated optimization and analysis of the language.
A design assumption made early in the language design was that is is likely that machines will use Reakt to create and modify rules more frequently than humans. The rate at which new rules need to be created and old ones modified is assumed to be high. Thus the language was designed to be easily understandable by machines and at the same time being reasonably human readable for system developers.
The current version of Reakt uses XML for its syntax. XML was chosen to take advantage as much as possible of existing knowledge among the developers, known standards and existing tools for generating and parsing. It would also be possible to create alternative syntaxes for Reakt. For example to create a more compact human friendly representation for business users.
The Reakt language was designed to support execution environments which fits into an architectural style called Event Driven Architecture (EDA). In order be a natural building block in an EDA, the ruleCore Reakt language and its supporting execution environment, the ruleCore CEP Server, was designed to be completely event driven. This purely event driven approach allows all systems in a distributed messaging architecture to integrate with ruleCore using simple events as their only communication mechanism.
The System Information Model
The System Information Model, or SIM for short, is both the name of the information model used by ruleCore CEP Server where all information about different items (rules, situations etc.) can be found and the name of the run-time data structure modelled after the familiar DOM model, commonly used to represent information originating from XML. The SIM is part of the ruleCore CEP Server implementation and not the language definition. But as all the language elements declared are executed against the SIM it is described here briefly. Different implementations can implement varying information models.
The SIM is used for various purposes, for example:
- Listing available definitions – All definitions of items such as rules, views, zones, entities, situations and actions can be found in the SIM. Definitions are created using the systems events, for example AddRuleDef, AddSituationDef and AddActionDef.
- Listing user defined events – Before an event can be sent into ruleCore CEP Server, it must be defined using the AddEventDef system event. The SIM contains all user defined events
- List of rule instances – When a rule instance is created, it’s state and internal data structures can be monitored by querying the SIM. The ruleCore CEP Server provides a query mechanism using XPath. But other implementations might query the SIM using other methods.
- Monitoring situation progress – As a situation develops, its progress can be monitored by querying the SIM.
- Examine view contents – Each rule instance is executed in an unique context of events called the event stream view. The view contains a window into the inbound stream of events and contains only semantically related events.
- Referencing items – When an item such as a rule needs to reference other items it is done using references through the SIM.
- Component Specific Data – Each plug-in component that extends the engine creates an entry in the SIM and may add any component specific information.
The SIM consists internally of dynamic data structures which are updated as needed when the internal state of the ruleCore CEP Server changes in response to inbound events. These data structures have an external representation which is the user’s view of the SIM.
The external representation of the SIM can be treated as a DOM document which can be queried using XPath and transformed using XSLT. XPath queries can be executed against the SIM using the SysInfoQuery system event.
If is also useful to note that the ruleCore CEP Server uses events for all tasks. Special pre-defined system events are used to create rules and other items in ruleCore CEP Server in addition to monitor the run-time state of the rule evaluation.
Structure of a Rule
The main language element in Reakt is the rule. Reakt uses a class of rules called reactive rules to provide its situation detection capabilities. Reactive rules should not to be confused with production rules or other types of rules which must be explicitly evaluated. A rule in Reakt is never explicitly evaluated, its evaluation must be automatically managed by the execution environment in response to inbound events.
A rule consists of a number of elements, the main ones being:
- Initialization directives - Specifies which events can lead to the creation of a rule instance
- Termination directives – Specifies what causes a rule instance to be deleted.
- The rule level – Specifies on what level in the rule hierarchy instances should be created.
- Trigger Directives – Controls how and when to report Rule triggering.
- The View – A reference to a view, providing execution context for the rule.
- The Situation – A reference to a situation to track and detect situations.
- The Action – A reference to an action, which will generate an event when
- the situation is detected, or
- if the situation is determined to be undetectable.
General Rule Attributes
Each rule has a name and a attribute which can be used limit the number of rule instances that can be created of a particular rule. A common usage of the limit is to create a singleton rule, of which only one single instance can exist.
Rules at every level can publish their trigger events externally so that they are visible outside of Reakt.
All rule declarations specifies a level. Every rule instance is created on a specific level in a hierarchy of rules. Level zero is the level on which inbound events arrive. Thus the lowest level for a rule is 1 (one). A rule will only see events from a level directly below it.
For example, three levels of rules where the rule at level one sees the inbound events:
<Rule name="New Location" limit="1000" level="1"> … </Rule>
<Rule name="Zone Entry" limit="1000" level="2"> … </Rule>
<Rule name="Zone Exit" limit="1000" level="2"> … </Rule>
<Rule name="Zone Visit" limit="1000" level="3"> … </Rule>
Rule Life Cycle Management
Rule instances are automatically created and deleted based on directives in the rule declaration section.
The <Initialize> declaration controls when a new rule instance can be created. All rule instances are created in response to events. The <Initialize> section lists those events which can lead to new rule instances being created.
Create a rule instance when an event of type GPSPosition arrives.
<Initialize>
<Assert>
<Event>
<base:XPath>sim:EventDef[@eventType="GPSPosition"]</base:XPath>
</Event>
</Assert>
</Initialize>
The <Key> declaration is the unique key for each rule instance. The key consists of one or multiple properties of the init events.
<Key>
<Property name="Vehicle"/>
<Property name="Zone"/>
</Key>
The key declaration allows for easy detection of situations per something
For example, the above key declaration creates rule instances which are per vehicle and zone. So there would be rule instances created for each unique combination of vehicle and zone.
The <Terminate> section determines when a rule instance is deleted. The <Terminate> section consists of a number of directives which allows for creation of flexible termination strategies. A rule could for example be terminated when it has triggered three times, when it’s view is empty, on a specific date and time or when a particular event arrives.
<Terminate>
<Triggered count="3"/>
<ViewEmpty name="ZoneEntry"/>
<Time>2010-01-31T00:00:00Z</Time>
<Assert>
<Event>
<base:XPath>sim:EventDef[@eventType="Location"]</base:XPath>
</Event>
</Assert>
</Terminate>
Rule Triggering
A rule may trigger when one of two things happen; the situation is detected or it is automatically determined that the situation is undetectable in the future. Depending on the terminate declarations a rule can fire one or multiple times.
<Trigger>
<SituationDetected/>
<UndetectableSituation/>
</Trigger>
When a rule triggers, its action part is executed. The action part creates an event and thus the visible result when a rule triggers is a new event, created according to your specification.
Trigger Reporting
Related to rule triggering is the reporting of the rule triggering. The triggering of the rule and reporting that triggering is separated in two stages. Thus it is possible to postpone the report of a rule triggering. For example, it is possible to report rule triggering, or multiple triggerings, each minute or when the rule has triggered five times or whichever comes first. It is also possible to limit the trigger reports so that only a specified number of reports are created. The most common case is to create rules whose triggering is reported only once.
<Report>
<Every trigCount="10" timeframe="00:10:00">
<Max count="234"/>
<AtTermination/>
</Report>
References
The rule also contains a reference to the rule’s view, situation and action. All references in Reakt are specified using XPath and executed against the SIM.
The view reference contains a reference to a view declaration. The view provides the context in which the rule is executed
<Views>
<ViewRef name="ZoneEntry">
<base:XPath>sim:ViewDef[@name='ZoneEntry']</base:XPath>
</ViewRef>
</Views>
The situation reference contains a reference to the situation which is evaluated in the context of the view.
<Situations>
<SituationRef name="ZoneEntry">
<base:XPath>sim:SituationDef[@name="ZoneEntry"]</base:XPath>
</SituationRef>
</Situations>
The action reference contains a reference to the action part which generates a new event when the rule triggers.
<Actions>
<SituationDetected situationName="ZoneEntry">
<ActionRef name="ZoneEntry" eventVisibility="external">
<base:XPath>sim:ActionDef[@name="ZoneEntry"]</base:XPath>
</ActionRef>
</SituationDetected>
</Actions>
Complete Rule Example
Here is a complete example of a rule.
<Rule name="ZoneEntry" limit="1000" level="1">
<!– Create rule instance per Vehicle and Zone –>
<Key>
<Property name="Vehicle"/>
<Property name="Zone"/>
</Key>
<!– Create a rule instance then the Location event arrives,
if one does not exists per the key above –>
<Initialize>
<Assert>
<Event>
<XPath>EventDef[@eventType="Location"]</XPath>
</Event>
</Assert>
</Initialize>
<!– Delete rule instance when rule has triggered 300 times, the view is
empty of at the specified time or when the Reset event arrives –>
<Terminate>
<Triggered count="300"/>
<ViewEmpty name="ZoneEntry"/>
<Time>2010-01-31T00:00:00Z</Time>
<Assert>
<Event>
<XPath>EventDef[@eventType="Reset"]</XPath>
</Event>
</Assert>
</Terminate>
<!– Report rule triggerings every 10th triggering of every tenth minute
or at rule termination, report a maximum of 234 triggerings –>
<Report>
<Every trigCount="10" timeframe="00:10:00">
<Max count="234"/>
<AtTermination/>
</Report>
<!– Trigger rule when situation is detected –>
<Trigger>
<SituationDetected/>
</Trigger>
<Views>
<ViewRef name="ZoneEntry">
<XPath>ViewDef[@name='ZoneEntry']</XPath>
</ViewRef>
</Views>
<Situations>
<SituationRef name="ZoneEntry">
<XPath>SituationDef[@name="ZoneEntry"]</XPath>
</SituationRef>
</Situations>
<Actions>
<SituationDetected situationName="ZoneEntry">
<ActionRef name="ZoneEntry" eventVisibility="external">
<XPath>ActionDef[@name="ZoneEntry"]</XPath>
</ActionRef>
</SituationDetected>
</Actions>
</Rule>
The View
The rule declaration contains a reference to a view. The view provides the context in which the rule is evaluated. The view keeps related events together and selects only interesting events from the inbound stream of events.
Each rule instance contains its own private view instance. The view contains a collection of events. The view is kept updated based the stream of inbound events. The view provides a dynamically changing execution environment for the rule as each rule instance is executed in the context of its view.
The view consists of a number of declarations, each specifying a property of an event that must be true in order for the event to be present in the view.
All these declarations are evaluated to determine which events are present in the view of each rule instance.
Example of properties available:
- The type view property is used to include events into the view only if they are of the specified type.
- The maxage view property defines the maximum age for each event in the view. You could for example use the age property to let each rule instance see only events from the past 10 minutes.
- The maxcount property defines the maximum number of events in the view. You could for example use this property to limit the view to contain only the last 100 received events.
- The match property defines which event properties must match in order for them to be included in the same view. You could for example require that the property CustomerID matches on all events in the view, thereby creating a view which contains events about the same customer.
- The assert property defines an XPath expression that must evaluate to true for all instances of a specified event type. The assert property could for example be used to compare a certain element in the event body with a constant.
<ViewDef name="ZoneEntry">
<Properties>
<!– All events in the view must be of types InsideZone or OutsideZone –>
<Type>
<Event>
<XPath>EventDef[@eventType="InsideZone"]</XPath>
</Event>
<Event>
<XPath>EventDef[@eventType="OutsideZone"]</XPath>
</Event>
</Type>
<!– All events in the view must be from the same vehicle –>
<Match name="vehicle">
<Value>
<Event>
<XPath>EventDef[@eventType="OutsideZone"]</XPath>
</Event>
<Property name="Vehicle"/>
</Value>
<Value>
<Event>
<XPath>EventDef[@eventType="InsideZone"]</XPath>
</Event>
<Property name="Vehicle"/>
</Value>
</Match>
<!– All events in the view must be from the same zone –>
<Match name="zone">
<Value>
<Event>
<XPath>EventDef[@eventType="InsideZone"]</XPath>
</Event>
<Property name="Zone"/>
</Value>
<Value>
<Event>
<XPath>EventDef[@eventType="OutsideZone"]</XPath>
</Event>
<Property name="Zone"/>
</Value>
</Match>
<!– Keep only the last 100 events in the view –>
<MaxCount>100</MaxCount>
</Properties>
</ViewDef>
Situation
A situation describes an interesting combination of multiple events as they occur over time.
The situation detection capability of Reakt have the following features.
- Declarative – Complex situations are described using a high level declarative approach, avoiding complex procedural solutions using traditional programming languages.
- Temporal support – Support for describing temporal relationships between events. Timing restrictions such as deadlines for detection can easily be added.
- Extensible – The situation detector is designed to be extended with new task specific detector nodes.
- Monitoring – The detector is designed to be monitored and thus the progress of situation detection can easily be tracked.
As a situation describes an interesting combination of multiple events as they occur over time, it must start somewhere. A situation starts developing when a specific initiator event occurs. This is the first event in what could possibly be a fully developed situation consisting of multiple events. This also means that a situation has an exact point in time where is starts developing, namely the timestamp of the initiator event.
The situation develops over time and can either be detected at some point in time or automatically determined to be undetectable. A situation can develop over long periods of time and consist of a large number of contributing events. The ruleCore CEP Server provides a persistent rule state allowing rule instances and their situations to survive restarts and unexpected crashes.
The progress of a situation, as it develops, can be followed by submitting a query event to the ruleCore CEP Server about the state of a particular rule’s situation detector. The degree of completion of a situation can be monitored to provide an early warning about a situation which could possibly be detected soon.
For example, detect when a vehicle has been outside a zone for at least five seconds:
<SituationDef name="OutsideZone5s">
<Detector>
<After timeframe="00:00:05">
<Not>
<EventPickup>
<XPath>Views/View[@default='true']/Events/Event[@eventType="InsideZone"]<XPath>
</EventPickup>
</Not>
</After>
</Detector>
</SituationDef>
The above situation example is detected when the innermost event pickup does not pick up any events from the view five seconds after the rule instance was created. If InsideZone events have arrived into the view during those five seconds the situation is not detected.
A situation is defined using a tree with detector nodes. The <Detector> node is the root of the situation detector. Each node can be either true or false depending on its semantics.
One of the central detector nodes is the <EventPickup> node which selects events from the view. It selects events by executing a XPath query against the view. If the query returns any events from the view, the <EventPickup> node becomes true.
The detector is evaluated each time it is possible that changes in the view or temporal conditions in detector nodes can affect the state of the detector.
Action
Actions are executed as the last part of a rule’s evaluation in response to a detected situation, or optionally when it is determined that a partially detected situation can’t be detected at any point in the future.
An action is used to create a new event. The new event is called the reaction event and is used to notify external systems about a detected situation. Different events can be created for a detected situation and for undetectable situations.
The outbound event created by the action is created using a XSLT stylesheet. The stylesheet is executed in the context of the rule and can access all the events that contributed to the detection of the situation. The stylesheet can also access the events present in the view at the time the action is evaluated.
<ActionDef name="OutsideZone">
<Event>
<EventDef>
<XPath>EventDef[@eventType="OutsideZone" and @eventClass="user"]</XPath>
</EventDef>
<Body>
<XsltBuilder>
<Stylesheet><![CDATA[
<xsl:stylesheet>
<xsl:template match="child::*">
<base:EventBody>
<xsl:for-each select="user:Views/user:View[@default='true']/user:Properties">
<user:VehicleId><xsl:value-of select="descendant::user:MatchedValue[@name='vehicle']"/></user:VehicleId>
<user:ZoneId><xsl:value-of select="descendant::user:MatchedZone/child::text()"/></user:ZoneId>
</xsl:for-each>
</base:EventBody>
</xsl:template>
</xsl:stylesheet>]]>
</Stylesheet>
</Stylesheet>
</XsltBuilder>
</Body>
</Event>
</ActionDef>
Reakt is a trademark of Rulecore KB of Sweden.
DataMentors – First commercial…
February 23rd, 2010
DataMentors is a company I was not aware of previously. Apparently they are branding some of their products as CEP, with some interesting claims:
GEMS technology is the first commercial event pattern language that includes an operator to express causality in the moment. This new real-time CEP technology enables companies to simultaneously examine historical, current, and predictive data.
Not sure what causality in the moment is. But I’m pretty sure that several other vendors would not agree them being first commercial to do anything with causality.
Furthermore:
GEMS ability to instantly compile data from thousands of disparate data-streams and conduct nesting pattern evaluations quickly and accurately is dependent on DataMentors’ software. DataMentors’ processing speeds are necessary to enable the GEMS technology to deliver insights and models instantly to end users, which result in higher levels of accurately predicting future events.
which a description one would expect from a company doing CEP. Once again interesting to see that another company chooses to position their products as CEP-something. It’s probably not done on random, so they must think that there is something to gain from calling themselves CEP and not something else. Which I like, as it shows that CEP is, slowly, gaining some traction.
See rest here
Why did Sybase get Aleri?
February 19th, 2010
I have tried to google around for something about the Sybase/Aleri deal. There is lots of the normal "information" which is corporate blahblabla without any substance. But I found one piece of info that actually have something interesting in it about what Sybase might have in mind for Aleri:
"In addition to slashing latency, CEP can also help to reduce warehouse storage requirements and help tame problems that were heretofore too costly to solve. By conducting analysis of a stream as it arrives, much of needed analysis and aggregation can be done without persisting details to the warehouse at all. If your data has no retention requirements, other than your analysis needs, you may be able to use CEP to greatly reduce your warehouse costs." – http://www.beyeblogs.com/sybaseiq/archive/2010/02/inastreamofgood.php
Analytic BI + nifty real-time front-end for the datawarehousing guys would seem like a pretty good use for the Coral8 engine, I just assume that they will use the Coral8 engine and throw away the rest.
Apart from that, I don’t think this deal will turn into anything significant for the CEP community. Sybase will add some real-time features and give some nice shiny toys for their capital market customers that can be used to game the markets even further. But I don’t think they will have a generic CEP engine to offer in the future.
Not CEP – Yet…
February 17th, 2010
Comments Off
"SenSage 4 includes a highly scalable real-time correlation engine, the Scalable Alert Server (SAS). Correlation is based on the application of threshold and scenario-based rules against multi-source, real-time event streams" – http://www.sensage.com/products/sensage-40.php
I wonder how long it takes before this product carries the CEP label…
The new CEP Book – EPIA
February 12th, 2010
Comments Off
At last I had some time to read the new CEP book from Opher Etzion and Peter Niblett. The book, Event Processing in Action, has an eta of april 2010. But there is a preview from the publisher Manning Publications. Allowing us to read an unfinished book is a brave move by Manning. A bit like introducing agile methods into the book writing process; release early and often.
I have a really good feeling about this book. Power of Events by David Luckham was the book that sort of defined the field. It got everything started and more importantly introduced the idea that event processing is its own discipline.
What is needed is a book that summarizes the ideas that have evolved around event processing since David’s book. I think that Event Processing in Action will be that book. My gut feeling is that this book will define the event processing field for the years to come. It does a great job of putting everything into context and explaining how all the bits and pieces in this new field are related.
The early access edition can be found here if you are interested in a preview: http://www.manning.com/etzion/
Sybase did buy Aleri
February 4th, 2010
Now it is official: http://www.sybase.com/detail?id=1067153
Unfortunately there is no new information in the press release apart from the fact that they bought something from Aleri for an unknown sum.
Is Sybase Buying Aleri?
January 29th, 2010
Does anyone know if this is true? Is Sybase buying Aleri? No surprise if they were, see my earlier post on predictions for 2010…
Free CEP Tool for Trading
January 25th, 2010
Here’s a free tool that claims it provides CEP for building trading apps.
Here: project.xceptra.com/
Download from here: http://sourceforge.net/projects/xceptra
Sounds interesting, don’t have time to look into it now…. (Note to self: must check later)
Summer Internships in CEP
January 19th, 2010
Comments Off
Complex event analysis specialist: Duties include evaluating existing methods and developing new approaches for complex event processing, identifying meaningful events within a large pool of events, event pattern recognition and evolution of event patterns over time. The candidate must be currently enrolled in computer science or similar graduate program and should have prior experience in machine learning, data mining and text mining. Programming will be required and working knowledge of programming/scripting languages such as Java, Perl, Python and Matlab, as well as sufficient familiarity with database systems are highly desirable.
This might be interesting to some of my readers. Full text here
License Cost of Sybase (Coral8) RAP is…
January 18th, 2010
I just found an interesting interview with John Chen, CEO of Sybase, the interesting part:
I’ve heard about your real-time analytics apps for Wall Street. Yes, our complex event processing software for risk platforms. The financial vertical is so excited about it, they are turning into design partners. They want to handle both structured and unstructured data in a more real-time basis.
Was this how you were able to grow despite all of the bank failures we’ve heard about in the past year and a half? The funny thing is that although there were bank and brokerage failures, trading volumes never subsided. The algorithms behind the trades are getting more sophisticated. We sold lots of [Sybase] IQ systems to do counterparty risk analysis, not only in the U.S. and Western Europe, but in China, where the firms tell me they don’t want to repeat the mistakes the U.S. made.
Are you worried that the SEC will rule that real-time trading systems like these give big firms an unfair advantage over smaller investors and should be outlawed? No, because everybody can buy one.
Well, anyone with a certain amount of money. It’s not that much money. And pretty soon, we will be releasing a hosted solution. Even small hedge funds will be able to come in and rent time. We are releasing it by June. It will cost about $37,000 a year, or 25% of the license cost.
The last question is the interesting one. I suppose it’s Sybase RAP he is talking about. The one that is actually Coral8 if I got is right. So they charge a nice 37k*4 for it. Much more than Coral8 ever did. $148k for a license is pretty expensive for anything than firms toying around with money on Wall St. The point being, in my predictions a couple of posts ago I said that I don’t think that CEP is going to spread to mainstream use this year. With a price tag of $148.000 it still a solution to very few problems…
But its nice to see that CEP technology is worth a lot in the right places. I hope Sybase sells and rents a lot of these.
ruleCore Internals – Our GUI Approach
January 13th, 2010
As some of you already know we have spent lots of time on user interfaces lately. It’s a complete nightmare to decide how to create a gui these days. There are just too many options to choose from. Traditional client or web client? Flash or Silverlight? .Net or Java? Which Java framwork and so on ad infinitum…
For user interface work, we follow some guiding principles:
- All user interfaces as web based. Runnable in your favorite browser. The reason for doing only web based, as opposed to traditional fat clients, is that we like to have some flexibility on what kind of client the application runs on.
- The output should be plain html and javascript. This opens up creative mashups where our user interface is a part of a greater whole in ways we can not anticipate. Users (and children) tend to use everything in the most creative ways that we ourselves can not think of. So, we just assume that we are stupid and our users are smart.
- A system integrator using our GUI should be able to change layout and the overall look and feel of the user interface.
- User interfaces should provide some real benefits. That is, they must be easy to use. No stupid things like popping up a editing window where the user have to write a script or enter some XML.
This lead us to the following approach for user interfaces:
We have a rather traditional three tier architecture. The data tier with a normal sql database sits and the bottom, providing data to the business logic tier which in turn response to request from the web tier. The database have other clients too. So its design looks like one would expect from a traditional relational model.
The business logic tier uses JPA with Hibernate to provide a nice object oriented model of the database. As there can be other clients to this data we need to be careful in keeping JPA in sync with all the other clients. This is an area where JPA needs to do more work. JPA seems to assume that it is the only client that accesses the database. So you need to be careful when letting other clients to access the database. But overall we a pleased with the JPA+Hibernate combo. We use a couple of Hibernate specific extensions as plain JPA lacks some critical features.
We use Glassfish+PostgreSQL plus some own additions to provide a multi tenant database environment. All user’s data is protected at database level, so there is no risk of two users seeing others data as the user needs to be able log in into PostgreSQL in order for JPA to get access to the data. This is done using some home brewed connection management in Glassfish. The normal approach is to use one single db login from the app server and handle all the security in app server code. This is something I think is really a bad idea as a single bug can allow users to see each others data. That’s something we can’t risk.
In order to stich everything together we use SEAM from JBoss. It provides us with smart components both in the business and web tier. Seam provides us with server side state management, which is invaluable in a web application. In the logic tier there are a number EJB components that provides an API to get business data.
The visible parts of the web tier is done using JSF and UI components from Icefaces. The Icefaces components are usable as they are in most cases, but for some special cases we need to add other components. Mostly JQuery ui elements. For example, there is no Icefaces data picker component which allows you to pick more than one date. Very annoying. Overall Icefaces is a good choice for GUI work, but I would like to see a bit more mature components. I think Delphi had more mature components back in 1995.
Icefaces uses a lot of magic javascript to provide its smooth server driven updates and it takes some tricks to get it to work with other javascript frameworks such as JQuery. To get different frameworks to work together is the biggest problem with our setup. There are a number of things that can go wrong when you need to get JSF, Icefaces, JQuery, Hibernate, SEAM, JPA, JSP and a couple of other libraries to work together in one project. But when everything is in place, it gives us a very flexible environment to work in.
We rely heavily on CSS to do the layout and look and feel for our user interface. The idea is to give the css to our clients and allow them to move around different elements in the gui and changes colors and other layout elements. Normally all layouts start with a paper and pencil, from there its straight to a prototype in code. We do this back and forth a couple of times before the design starts to feel right. We don’t use any gui tools for doing the layout of the web pages. It’s all hand built using JSP, Icefaces and CSS. It would be nice to have a tool for this but we have not found anything that works with the setup we have.
Here’s an example on how it might look, this is a page from FleetNotice, the GPS real-time alert management system for fleet managers that we built ontop of ruleCore CEP Server:
The above example uses our standard Black&White theme. It’s the standard CSS that we base everything on. The idea is to first have something that is very clean and then start adding colors and other fancy graphics.
All components except from the date picker is Icefaces. In this example they look just like plain html input boxes. The benefits from Icefaces are apparent first when actually using the interface. For example the "Output" box is validated immediately when the user types something into it. Not when submitting the page. The data picker is a JQuery component. We prefer to use pre-built components and avoid creating our own gui components at all cost. We try to support most major browsers. But as we use Icefaces and JQuery we are basically supporting whatever they support.
Nothing to See Here, Move Along
January 12th, 2010
Here’s an interesting article about CEP:
http://architects.dzone.com/news/complex-event-processing-cep
It touches an interesting and perhaps a bit sensitive topic; CEP is nothing new. The author also makes an assumption that CEP equals streaming sql. Which is the view many people have on CEP. I think it’s StreamBase and Coral8 that did a good job on marketing CEP in the early days as a streaming extension to SQL.
There are of course other approaches: declarative rules, prolog style factoid things and scripting languages specially designed for event processing. Just to mention a few.
One other, perhaps also sensitive issue, that the author touches is that many CEP vendors don’t do CEP. They simply perform calculations on streams of data. This is also true. If you were following the market that became CEP a couple years ago you remember that most vendors that push "CEP" started as streaming data processing vendors. At first they did not use the word event at all. Then at some point many vendors discovered that the CEP label started to stick. A quick re-branding and voilà, everyone had this thing called CEP.
.jpg)
This does not make these data stream processing tools better or worse technically. But it confuses the users. They read about event processing in theory and then when they try out the tools they just get confused as the package does not contain what the label says.
Just to not get the reputation of that guy that hates streaming data processing and streaming sql: The SQL based tools are great! Every single of them. But are they CEP? Well, not sure about that…
But the fun part is that nobody agrees on what CEP is anyway. So whom am I to say that some vendor does CEP or not. The CEP label seems to live its own life and ultimately its up to the (paying) clients to decide if they appreciate a product which is CEP-something.
Interesting Vendor using CEP
January 11th, 2010
Just stumbled across this:
http://www.starviewtechnology.com/index.html
Seems like an interesting use of CEP. It looks like they are heavy users of CEP, but not positioning themselves as a CEP company. Using CEP "merely" as a tool to build their solutions. This kind of companies are good news for CEP. Its soon time to talk about an "active tier" in systems architecture amonside the data, integration,logic and presentation tiers.
CEP Predictions for 2010
January 7th, 2010
Comments Off
In 1989 GM bought Saab Automobile (The car company part of Saab, not the fighter jet part). One would assume that the people set to lead GM are among the best leaders and smartest corporate leaders available. Now in January 2009 GM is shutting Saab down and from what I understand they never made any profits from Saab at all.
Does this sound like something smart people would do? Given the power of GM and 10 years, how come they totally failed with Saab?
Who’s Getting Who?
In 2010 there will be a number of smart people buying CEP companies. A post on predictions would not be complete with some guesses on who will acquire the most interesting companies.
In 2010 I predict that CEP will find its way to people outside the core event processing community. As a consequence people with no background in CEP will find out what we CEP companies have been doing the past years. Those that will find these companies or their technology the most interesting are probably those that are drowning in real-time data. I would not be surprised if a major stock exchange, something like Reuters, SCADA company or an integration company buys one of the CEP vendors in 2010. Actually my best guess is that a CEP company is bought to be used as a base for SCADA solutions. SCADA is almost never mentioned when talking about CEP, but that’s where there will be some action in 2010.
If would need to name a company that I think will be acquired first, it would be Aleri. Maybe really soon? Who knows. The open source project Esper is also a good candidate to be bought or assimilated by another open source company. Maybe RedHat or Sprinsource are in the mood for some shopping?
The New Generation
Considering the small amount of pure play CEP vendors available, will there be anyone left after the wave of acquisitions that I think will happen in 2010?
This brings me to my next prediction. New CEP vendors will emerge in 2010! I think we will see several vendors with interesting ideas on how to do CEP. These vendors are in a good position as they can learn from the current vendors and take the ideas and develop them further without any concerns for existing users.
I would guess that at least a couple of them will fail, but maybe one or two will do OK and it will be among those that the new cool ideas will come from. I don’t think open source will lead the way here. From what I see, the open source community is good at taking existing mature ideas and creating good implementations of them. Not inventing new concepts and ways of doing things. Something that is required for the next wave of CEP. I have the distinct feeling that the current CEP tools are just the first wave. There will be at least a couple of waves before things settle and we get some kind of collective agreement on how CEP should be done. Compare to SQL in the database world.
The Great Escape from Wall Street
To many people CEP equals algo trading and other kinds (legal) tricks for gaming the markets. Several event processing companies talk a lot about capital markets and less about other application areas. It’s not that there’s a total lack of efforts in trying to capture other markets. But still, capital markets are where many vendors have done OK.
In 2010 I think that many of the leading CEP vendors are trying to figure out how to break out of the world of finance. I think they would love to find new markets and have started to probe into various fields in 2009. In 2009 many companies at Wall St. took some beating and I bet this made the life of the sales reps at the CEP vendors a bit more, well, interesting. My prediction is that they won’t come that far with the great escape in 2010. I think the vendors will find that the capital markets will suck up all their resources and moving beyond selling very expensive solutions doing all kinds of financial magic and trickery is not at easy as is sounds.
Driving in Snow
This year I have been driving in snow for longer than most winters. And the winter is not over yet. We have had temperatures around -10 C for a long time here in south Sweden where I live. Just a couple of hours to the north, they had -40.4 C (about the same in F) this night.
Here my idea was to have some clever transition from snow driving to CEP, some authors are really great at doing that. But it seems that I’m not one of them. So I’ll continue just a little bit more about driving in snow before getting back to CEP.
The above tire is actually from my very own Saab. I read in the news that today is the last day for Saab. So now I’m driving a car which is not produced anymore. It will be interesting to see what that does to prices and availability of spare parts.
I think my transition had to do something with Saab being discontinued or that Saab have always had a performance profile. The Turbo Saab is somewhat of an icon here in Sweden.
The Cult of Performance
Whenever you talk to with other vendors, system integrators, clients, prospects that have heard about CEP they constantly focus on performance. It seems that the main perceived benefit of CEP is performance, and performance alone.
It’s simply seen as a way to do things faster than competing offerings. This is both good and bad. It’s good for CEP to be viewed as something that is needed when high performance is required. But it’s bad because Moore’s law will outrun us. If performance is the only selling point, then you can be sure that a normal desktop PC will replace you within a couple of years. Also, compared custom built code in, say C, any generic engine will most likely be slower.
But in 2010 I predict this will change. In 2010 CEP will find uses in other areas where speed in the not main selling point. I think that agility is a word that will be used more and more when talking about CEP in 2010. CEP solves many problems in a natural way, even if they could be solved using other tools. This is something that will be discovered by users outside the CEP community. We, the CEP community, know this already so it’s not a very bold prediction.
Show Me The Money
2010 will be the year when all kinds of psychedelic business ideas with no chance of ever generating profits will fail. Now it’s time to start making profits. Now more playing around. This is good news for CEP and bad news for Twitter and others. CEP is what I call a old school business idea. That is, there is a plan how to make money by providing something of value to a business. Our clients make money, we make money; everyone is happy.
Still, 2010 will not be an easy ride. Considering the amount of investments made in many of the CEP vendors, there’s bound to be some pressure to increase revenue and start making considerable profits. For example, a wild guess would be that the StreamBase’s investors would like to get back at least $600M. So how much profit do you need to make each year in order to justify that valuation? There’s lots of CEP licenses that need to be sold. The dependence on the capital markets can be a problem here. A guess would be that at least one of the current vendors will see some revenue related problems in 2010 in the CEP area. For the smaller vendors mainly doing CEP this can be disastrous. For bigger vendors where CEP products is just one part of the portfolio you might expect re-branding and other kinds of creative moves when they discover that selling CEP takes more time than they thought.
It sounds like I think that there’s no market for CEP. There is. But viewed from a customer point of view the CEP market is in a tricky phase. No standards, no agreed upon way to do CEP, we don’t even agree on what CEP is. This is confusing for the customers and I think many of them will build instead of buying during 2010. But the demand will increase, CEP solves real problems providing real benefits.
A Major Event
In 2010 I think something critical will happen that puts CEP in the spotlight. A customer success story or something that the press will write about. This will be the starting point of spreading CEP. Unfortunately I think that the news about CEP that everyone will talk about can be rather negative. Considering the greed culture of Wall St. and the focus of CEP on capital markets, I would not be surprised that some clever uses of CEP will play a major rule in another PR disaster at Wall Street.
Another, not very bold, prediction is that there will be at least two CEP books published in 2010.
Who Cares About CEP?
January 6th, 2010
Comments Off
Some (un)interesting facts about you, the reader of my blog:
- There’s been more of you during the year. No dramatic increase, but some 30% more.
- You guys eat lot of bandwidth! 30% more of you consume 4 times more! It must be the images.
- On average, you come back five times.
- You read about five blog posts, strage correlation to above fact.
- You visit the blog evenly throughout the week, but Wednesday is the most popular "Marco on CEP" reading day.
- It seems that reading the blog is something you do just before going home from work, 4pm-5pm is the most popular reading time.
- Top countries you come from: USA, UK, France, Mexico, China, EU, Canada, Switzerland, Germany, Hong Kong, Denmark, Russia, Netherlands, Israel, Japan and Austria.
- Top bandwidth wasting spiders: Yahoo Slurp and MSNBot.
- Most of you do Windows (42%)…
- But suprisingly most of you have the good taste to avoid IE, only 20% come here with IE.
- If you found your way here by searching you most likely came from Google (57%) or billg’s search engine (38%). The old time search favorite AltaVista (if you are old enough to remember) reports eight (yes, 8!) hits.
- You made about 200 comments and 2000 spammers tried get spam comments through the filters. If I remember correctly, no spam comments managed to get into the blog.
During the past year you have also seen some random photos in many of the blog posts. These are all my own. Most taken with my cell phone, so they are not as good as you would expect if you hire me to shoot your wedding or for other work. I have received a couple of mails asking if there is any connection between the pictures and the blog posts? The answer is; no. No connection at all. Just random images to break up the text. For those that don’t know it, I’m also a photographer. The above photo is actually from my back yard. It’s cold and there’s lots of snow here in Sweden.
A Must Read
January 4th, 2010
New Open Source CEP Platform
December 28th, 2009
I have now added ActiveInsight into the vendors list.
So now there’s five open source projects to choose from when you something really cheap* (= zero dollars in license fee). I have not yet looked into their product in detail, but their web page says all the right things, so it seems that they understand this CEP thing at least on presentation level. If someone have used their product, please share your expericeces. What would be most interesting to learn is how they compare to the other open source products (Pion, Esper, Drools Fusion and IEP) .
It’s a good start for the next year to see another open source product in the CEP space, I hope there will be a couple of more in 2010. At least I heard rumours about a major fork to Esper that is being developed with a rather large number of developers on the project.
2010 is soon upon us. Maybe it’s time to be a Gartner wannabe for a while and do a bunch of predictions on CEP… Maybe in next post, so stay tuned
* If you still prefer to pay for your CEP software, there’s Progress Apama, Tibco Business Events, IBM BusinessEvents, Rulecore, Aleri, StreamBase to choose from.
All is Good
December 23rd, 2009
Comments Off
Soon it’s Christmas, at least in some parts of the world. The holiday that once had some religious meaning. Can’t even remember what it was all about, says a bit about my religion skills…
A couple of weeks ago I took a look at StreamBase, who are kind enough to provide free evaluation downloads, and they are certainly starting to get all bits and pieces in place. If you need to do real-time calculation on a data stream, this is probably one of the best ways to go about it. So stop all that custom development that you first thought of. Aleri/Coral8 is in the same camp so is might be a matter of taste which one to go for.
So if you have a continuous stream of data, perhaps some financial information that changes all the time like prices, then stop developing yourself and look at these tools instead.
Next I tried to see if I could find more about Apama, and Progress is going in the right direction with Apama. Not only that Mr. Apama is now CTO of Progress, but I like their approach with rules (working at ruleCore that would make sense, would it not? ;) ) Expect lots of interesting developments from the Apama team next year. Maybe integration with other technologies like BPM engines?
A friend I know at IBM said that we could expect that it takes a couple of years to IBMify anything that they buy. The start is slow but when they get the speed up they can at best be unstoppable. So next year should be the year of AptSoft, er, IBM Business Events (with or without space? I can’t seems to never remember if it’s IBM or Tibco that uses the space there)
Speaking of Tibco, I think Tibco’s interest in selling CEP software is a good sign that there’s money to be made in the CEP field. Being the masters of building wealth by selling infrastructure software.
The presence of these big players in the CEP field is what actually makes "CEP Software" its own segment. Now we see that customers are actually asking for "solution based on CEP", not that our beloved customer always know that they are asking for. But at least they are asking for it. I would say that this formation of a CEP market is thanks to the collective marketing efforts of all these big vendors. We smaller vendors can’t create a new field within IT. But we can work in one, created by the big ones.
In addition to the bigger ones, there are a number of small vendors too. Providing parts of the solution which sometimes might be a better solution, for many reasons, than going with the bigger vendors. There’s of course our very own ruleCore, there’s the free and gratis open source Esper, we have EventZero, SQLStream, RTM, Pion, WestGlobal, Truviso and other small vendors all building companies around CEP technology. All with their own twist to it.
What about Microsoft? Did I forget them? No, not at all. There’s StreamInsight. But it is way too early to say in what form it will be available in the future. But at least someone at Microsoft thinks that CEP is worth putting developer dollars into. Somebody did convince his manager to give him a budget to build this CEP thingy and actually got an approved budget to work with. Another very good sign. But I would expect that, following Microsoft tradition, that their offering will be mostly usable in a pure Microsoft world. Don’t expect something that fits smoothly into your brand new EDA/SOA master piece. I would guess that the same is true for Oracle’s CQL. CQL adds interesting capabilities into their RDBMS, which is good. Provided that you use it. For others, it might not be that interesting. I don’t think anyone buys a Oracle license just for CQL.
That sort of sums it up. If we can avoid abandoned open source projects, companies that run out of cash or products being dropped we can look forward for a very exciting year for CEP in 2010.
Difference Between Events and Data
December 14th, 2009
With Christmas approaching rapidly I struggle to get a ton of small tasks done before it’s time to take a couple of days off and eat way too much food and candy. This year I actually hope for some snow. Here it starts to get dark at 3pm or so, with some snow it would not at least be completely dark. And of course everything looks prettier with some snow and the sounds of the city are much more softer with snow on the ground.
Lately I have been reading Event Processing : Designing IT Systems for Agile Companies by Mani Chandy and Roy Schulte. They present a good list of principles an event driven application should follow in order to actually implement EDA:
- Report current events. Things that happen right now. Not (old) historical data
- Pushes event - A producer creates and event and publishes it. Not the other way around. In EDA, a system never queries for event. It receives them.
- Process events immediately – Don’t store events and query them later, that’s a task better left for databases
- Know nothing of event destinations – Just fire away events and assume someone else will route them to proper destinations.
- Events are not commands – You don’t know anything about event destinations, per above bullet, so you can’t command them to do anything.
So if you look at a CEP system, which should naturally fit into an EDA. Then the CEP system should obviously follow the principles above. (advertisement: ruleCore CEP Server does, it was built to be a component in an EDA)
With an EDA and lots of available event processing systems that you can use as high level components, we have a very smart way to assemble enterprise wide systems. This is something SOA is not very well suited for. Everything is too coupled in SOA. But with a nicely designed EDA and a bunch of CEP, or other event processing agents, you are a step closer to actually creating an EA which your CEO will love.
So to my original question. What is the difference between data and events? Data is something that is your primitive entity that holds business information in a tightly coupled SOA or database centric architecture. Events are something carries around your business data in an EDA.
When moving from databases and SOA to EDA, you need to stop having a data centric view of the world and start thinking of events instead. Otherwise your EDA project is doomed.
If you look at the bullets above. You can learn a couple of things:
Report current events: If multiple systems receive the same report (event), they must somehow be able to know that it is the same report they are looking at. So events must have globally unique id.
Pushes events: If some other systems are to push an event to me, I need to know how the event is defined in order to make sense out of it. So everyone will need to share event meta data
Process events immediately: Later is too late. So I can’t use a query to process the event I receive as I need to do it now. Not query later.
Know nothing of event destinations: The implication of this is that I really don’t know who is listening and on what. So I need to publish events whenever I do something some other entity might be interested in. Now or sometime in the future. So basically you need to publish every potentially interesting event from every system.
Events are not commands: SOA style you know lots of things about other systems and know what you can ask them to do. In EDA, you just sprinkle around events and hope for something interesting to happen. This scales way better and does not cost a fortune to manage.
So before you can ever dream of having a nice EDA in place with CEP components in place doing all kinds of interesting processing you need to start thinking about events and not data. Programmers are used from kindergarten to think in query/respond patterns so this is something that will take some time to get used to. So I still think EDA and CEP in the mainstream is at least a couple of years away.
New CEP Player?
December 3rd, 2009
So, What is This CEP Good For?
November 26th, 2009
When talking to other developers I need to explain, at least once a week, what this CEP stuff that I’m working on is good for.
Developers, with a slightly simplified view of the world, falls into surprisingly few categories.
- There’s the database guys, they solve everything with a database in the middle.
- There’s the Java crowd, everything is built around a application server. They don’t like the database guys.
- Then there’s the poor souls (a joke, I promise) stuck in .net lalalaa land. They use at least two products from Microsoft in each solution. They fear the Java crowd.
- Then there’s the free spirit Linux coder. He (She…) codes in php, mysql, python, perl, bash or whatever it takes to make it work. They never understood why the others complicate things.
That’s it. Ok, there’s the wannabe coders too. They write macros in every application that supports it. And they proudly display their achievements to other non-coders.
So, for this crowd (drove, sloth, wake, train, murder, charm, crash or whatever a collection of programmers is called) I often need to explain why we need just another technology…. easy. yeees…
From now and I’ll try to explain things using pictures and non-technology. Here’s my first attempt:
This is a screen dump of FleetNotice (think real-time temporal geofence and much more), an application built ontop of ruleCore.
Basically what is does is to tell a poor guy who is sitting at his console looking at 348 couriers moving downtown manhattan if all his couriers are doing the right thing. Or to be more precise, it tells him when someone is doing something they are not supposed to, otherwise it shuts up. Looking at 348 moving icons and trying to figure out if somebody is doing something wrong is mildly fun. So he calls in help in the form of FleetNotice.
So, back to explanation. What if you would be required to code this application without CEP? Pick your favorite tool – Application server, database or just plain code.
How long would it take? How complex would the resulting application be?
With a CEP server like ruleCore it’s just a matter of a couple of rules. About 20 or so to do more or less all types of monitoring that a sane person can think of.
FleetNotice can produce alerts like this without breaking a sweat:
"Alert me if Couriers Ben, Jerry and Mr Smith diverts from their designated route for more than five minutes and they take more than 20 minutes between the waypoints (40 minutes is ok if it’s between 4pm and 6pm on fridays, except last friday of month), but only on working days. Also alert me if the courier does not arrive before 6pm to it’s destination and alert me also if he does not leave the starting point before 1pm"
And to add to misery you must deliver the alert within one minute to the operator and if the server reboots in the middle you really can’t loose track of things. And yes, there’s not only Ben, Jerry and Mr Smith in reality. I made those up. In reality they are thousands of moving objects.
So, there’s the case for CEP that most developers understand. Or at least they slowly start to appreciate the beauty of CEP. It’s not a replacement for their beloved databases or application servers. But for the right type of problems, it’s feels so much more natural.
Get it?
State of the CEP
November 15th, 2009
I realized that I have not had any time to do some creative googling on CEP for long.
I tried to get a feeling on what kind of progress is made in the world of CEP.
My first stops didn’t require any googling at all.
Old timer Aleri which bought (merged with or whatever…) Coral8 seems to be doing alright with customers in the capital markets. Sort of what I predicted, that Aleri would be marketed more or less as a provider of tools for the capital makets and not a generic CEP platform. But no news on what they are doing with the Coral8 they bought. Some googling reveals that many of the Coral8 team left and are now on other positions. So Aleri will indeed have a interesting exercise in bringing Aleri+Coral8 into a coherent product.
StreamBase seems to follow Aleri and talk mostly about different kind of things in the capital markets. The language of capital markets is just too strange "Dispersed liquidity amongst multiple FX venues leads to an increase in trades, a decrease in order sizes, and makes it extremely difficult to gain complete market visibility. This results in inefficient executions and higher execution costs", say again? So I stopped after two clicks on the StreamBase web site.
I tried to figure out how long the assimilation of AptSoft into IBM have gone and it looks like ongoing progress. Lots of things are in place but large companies work slowly. I tried to do some googling and asking around and it seems that IBM don’t have a single customer on BusinessEvents in Sweden. Yet, I hope.
For Generation Gratis, there’s Esper which are continuing development and pusing out new releases now and then. Gratis don’t pay the bills so there’s an traditional license for the "pro" version of Esper. Latest news is that the HA (High-Availability) version is now finished, in the pro edition. Good work by the Esper team, I know how painful these things are to add to an existing product. I have not tried it, but the marketing material contained some impressive features.
The EPTS have had some additional meetings and there seems to be some progress, but too little and too slow in my opinion. Although I can’t really complain as I’m partly (very small part though) responsible for that as I have not had time to do any EPTS activities at all. Sadly.
Progress Apama seems to be one vendor that are trying actively to reach outside the capital markets. There’s mention of work done in transportation and logistics (good choice I must say, this is our (ruleCore) area of interest too) , telco and energy. With the size of Progress, not too big and not too small, my best guess is that Apama will be one of the most interesting solutions in the future. (alongside ruleCore ofcourse )
One of the most secret and perhaps best CEP tools, Agent Logic, was acquired by an integration company a while ago and it is hard to say what they are going to do with it. But now there’s one less pure play CEP company around. This seems to be a trend which I’m not sure if it is good or bad.
Truviso seems to have some really nice things going on that I need to look at some day. If you need less capital markets and more CEP then my guess is that Truviso is a good candidate to include in your evaluations.
I think progress in CEP world is going in the pace I anticipated. CEP requires that developers learn a new way of thinking and that will take some time. So my 2012 mainstream guess a few years back is still valid.
Meanwhile, we have lots of interesting things to work on. Back to some fancy GUI coding here at ruleCore HQ…
A Business Case for CEP
November 11th, 2009
Comments Off
Ever wondered why you need CEP:
Traditional tool at t+30 minutes: Here’s a report that hereby notifies you that you just lost $9M.
CEP Tool at t+1 minutes: I just detected a situation which is an indication of ongoing fraud.
"Over 130 different ATM machines in 49 cities worldwide were accessed in a 30-minute period on November 8," Agents Rice said. "So you can get an idea of the number of people involved in this and the scope of the operation."
Here is the amazing part: With these cashers ready to do their dirty work around the world, the hacker somehow had the ability to lift those limits we all have on our ATM cards. For example, I’m only allowed to take out $500 a day, but the cashers were able to cash once, twice, three times over and over again. When it was all over, they only used 100 cards but they ripped off $9 million." – from here
Who’s Time is it Anyway?
November 6th, 2009
So… if you find two times laying around in a database, say 9pm and 6am, how would you (and the amazing CEP software that you are writing) calculate the number of hours between 9pm and 6am the next day?
Sounds simple?
Well, as we all know assumptions are the mother of all fuckups.
The first assumption that is normally made is that these two times are times in our current time zone.
The next is that these are times both from the same time zone.
These times are not usable is you don’t know for sure what time zone they refer to. Your 9pm does not happen at the same time as mine.
Normally, if you come this far, the next assumption seems to be that is enough to have an offset from CET for each of these times. So you would assume that 9pm CET-10 and 6am CET+4 would give you the correct answer.
Wrong again.
You need to consider the evil of daylight saving time. If some countries and regions time is shifted forward and backward twice a year. And the beauty of it is that the rules for when this is done is specific to regions and coutries. So countries lying in timezone CET+1 could have different rules for when daylight saving is done.
Another assumption is that the days on which daylight saving is done never changes. Wrong too. They don’t change too often. But the do change. So basically you need to look this up from some reference data which need to be updated every day in order to stay on top of things.
So if you have come this far you might get it right.
First, you need to know in what time zones each of these times are assumed to be. You need the specific version of the time zone. For example if the first time is a departure time from Cocos Islands you need to know that they might be at GMT+6:30. Yes, another complication. Not all time zones are at an even hour boundary. If the other time is an arrival time to Australia and their Central Western Time, they might be at GMT+8:45.
So by now you might have realized that a time like 9pm is not usable if you don’t know in what location it is assumed to be in and what day it is.
You need the exact date also when calculating the difference between 9pm and 6am the next day as it might be that night daylight saving kicks into effect, and this would mean that you either skip on hour or add one magic hour.
Basically this is not rocket science. But there’s lots of small things to consider and most of the times developers will get this wrong.
Try testing your CEP software. Specify a time period which spans over a night when the daylight saving adjustment is done and see if it does the right thing.
Wintertime – Did your CEP fail?
November 1st, 2009
Comments Off
A couple of days ago we changed to wintertime here in Sweden. This is done by adjusting the clock one hour backwards.
Later on we adjust back to summer time again. So two times a year the clock is adjusted on specific days.
Sounds simple? Right… For example, here’s a simple thing that most get wrong:
How many hours are there between 9pm and 6am the next morning?
answer and explanation in next post ;)
A New CEP/EDA Book
October 23rd, 2009
Comments Off
Past week I have been reading a new book on CEP/EDA:
"Event Processing – Designing IT Systems for Agile Companies" by K. Mani Chandy and Roy Schulte
It approaches both EDA and CEP from the conceptual and business angle. Although I have just read a couple of chapters I can say that it’s a good read for anyone interested in CEP.
There seems to be no or very little technical detail in the book. It’s more about the ideas behind CEP and tries to explain the business case for CEP. The first chapters gives and answer to "why event processing?" and "why now?"
.jpg)
As there are no books at all about the new era CEP this is a welcome addition. Plus it’s written by two people that have a place in the CEP Hall of Fame. Even though Luckham’s Power of Events kicked off the CEP area, a lot have happened since that book. This neo-CEP needs new books as the ideas have matured and now weunderstand better where CEP fits into the big picture.
So get it and read it. It’s easy to read as there is not technical details. For more technical book you’ll need to wait for Opher’s book.
Cloud – What Deployment Options?
October 19th, 2009
Comments Off
I did some googling to see what deployment option various Cloud/SaaS products offer.
I’m doing some thinking on what route our future cloud service strategy should follow. As of now, I see three major ways of delivering ruleCore as a cloud service:
1) Build data center yourself (Well, not yourself as in you personally, It’s more about buying someone to do the work). Get a number of servers and some storage and own everything yourself. It’s will cost less (yes, google/amazon are not that cheap) if you’re going to do lots of processing. The drawback is that we will be sitting with lots of hardware which needs some life cycle planning and possible rather large investments now and then.
2) Build support for deployment into a well known cloud. Like google or amazon. The benefits are you are getting some brand recognition and an execution environment with a high level of confidence built into it. There’s little internal selling to do when deploying into these as people tend to trust big players more that small. The drawback is that third party cloud services will be very expensive when you need lot of continious processing power.
3) Build a vmware (or other virtual server) kit that our customers can deploy onto which ever hardware they want. It’s more job for us and updates, patching and that kind of stuff requires lots of work to be completely automated. In 1) and 2) we don’t need to automate every single task. Some tasks are not worth automating. But if our clients are running on their own without us accessing the servers everything need to be automatic.
A fourth would be to sell an appliance, but I don’t like those so that’s not an option.
The major question is what our customer will prefer in a couple of years…
ruleCore Internals – The Sequencer
October 1st, 2009
Comments Off
Rulecore is built to be used as a component in a distributed environment, the buzz-word compliant description would include some use of SOA and perhaps EDA with some mention of ESB. Our definition of component is a rather large box. It’s not a Java bean or .net assembly or anything of that size. Think more of it as a stand alone server, which you talk with using events.
As a component of a distributed environment we must assume a heterogeneous environment. That is, lots of different systems with no central authority controling them all.
This usually (almost always) implies the use of several unsynchronized clocks and varying event transmission times. As a result of this, events arriving to ruleCore will in nearly all cases not be picked up in the same order as they were originally emitted.
This is important, so I’ll repeat:
- Events can be sent from multiple sources which do not co-ordinate the event generation process in any way.
To make the rule authors life a bit simpler, the ruleCore CEP Server creates one single unified event stream of all these inbound events. This unified event stream have two nice properties:
- All events in the stream are arranged in order by ascending time.
- All events have unique timestamps, that is you don’t find two events with the same timestamp.
So, the rule author does not have to care about where the events come from and can assume that they are all neatly ordered.
To allow for receiving events from un-synchronized event sources, the ruleCore CEP Server conains a sub-systems which we call The Event Sequencer.
The Event Sequencer makes sure the requirement of an ordered event stream is fulfilled at all times by putting the incoming events on hold for a certain amount of time before releasing them to the ruleCore engine. The engine is where the actual rule evaluation is done. The maximum amount of time for which the event is put on hold is a configurable parameter, making it easy to adapt the sequencing to the particular operating environment.
Incoming events whose timestamp is older than the maximum age are instantly rejected. Incoming events with a timestamp in the future may or may not be kept for sequencing, this is a configurable option.
The fact that two events can’t have the same timestamp might feel a bit strange. Can’t two independent sources generate an event with the exact same timestamp? Well, yes they can.
But life would be much, much more simpler if they could not.
So we do some tiny tweaking of the timestamps to make sure that no two events happen at the same time. If the have the same timestamp, we create an synthetic ordering. This might feel a bit uncomfortable from a theorethical viewpoint. But in practice it works great and simplifies things a lot.
For example, when creating a rule which should react to a sequence of events, things would get a bit complicated if we would could not assume that any two events are always after each other, in some order. The the semantics of a sequence, for example, would be overly complicated without gaining too much real benefits.
As events are (in the ruleCore event model anyway) notifications about some real-world activity, it’s very rare that two events actually happen at exactly the same point in time given enough timer resolution.
Also remember that ruleCore is build to run in "business time". That is, it deals with sequences and time periods which are on a scale that makes sense in a business environment where humans are involved in the processes on which the rules of ruleCore reacts to. Weeks, days, hours minutes are the common scale and it’s very rare that seconds are used in reality. Sub-second timings are seldom of interest when dealing with some kind of business process involving actual humans.
ruleCore Internals – Our dev env
September 27th, 2009
Comments Off
Continuing my series of ruleCore Internals today It’s time to look at our development environment… So if you’re not a geek. This is not interesting to you.
Work here is done completely on Linux. Everything runs Linux. The servers on which the ruleCore ‘cloud’ is running are Linux and so are the developer’s workstations. We use Linux as it’s the best tool for developers. It gets the work done and allows us to be flexible as it’s easy to find tools for all parts of the development cycle.
But lately we have starting to see Mac as an alternative. There are a number of small problems with Linux which might lead us to Mac some day. We love tools that just work, so that we can spend more time creating ruleCore code. With Linux, we can spend hours with seemingly simple tasks.. Like trying to figure out why printing suddenly stopped working? Mac OS X seems to be a bit more polished and "just works", so we’ll certainly keep it on our radar screen as an alternative for developing on. I would love to see see a more polished Linux distribution which works in a commercial software development setting where there’s always too little time to get the next set of features ready.
Under Linux we write code (obviously). Most of the code is written using Netbeans. Why not Eclipse? Our experience shows us that we have much less problems with Netbeans. We prefer to spend time coding and not fighting with our IDE. Basically we don’t need any advanced features from our IDE. Basic editing and debugging is enough.
For source code management we have our trusty subversion. Normally we check out and commit from within inside Netbeans. But sometimes the subversion plugin breaks down and we need to use the good’ol command line.That’s one of the strengths of Linux, you can most of the times track down problems and find workarounds. Compared to a Microsoft environment where finding problems can be totally impossible and you have to spend days with reinstalling everything.
Most of the code is written in Java and Python. With some additional Unix shell (bash) scripts to glue everything together. Both Java (offcourse) and Python is well supported in Netbeans. Java support just great but sometimes the Pyhon mode behaves strangely. That’s basically our biggest complaint against netbeans. It’s bad support for Python. The other thing that could improve is how to share projects among developers. It’s a pain to check in everything required to run a project. But overall netbeans is a good tools and gets the work done (Well, not in the same class as visual studio from Microsoft, but it’s enough for our needs…)
When coding in Java we use Glassfish as our app server and PostgreSQL as database. For GUI work we are in the framework hell using a clever combination of JSF, Seam, JPA and IceFaces. Java and all it’s frameworks are great in one way, but takes some time and skill to set up everything.
In the Python world we tend to use just plain Python, no frameworks or few other dependencies. So all Python coding feels very simple as there is not multiple frameworks to learn, just Python.
Almost forgot. There’s actually another database besides PostgreSQL used; BerkeleyDB. It’s used for internal purposes by the engine to provide the basis on which we build transactional crash recovery.
When everything is done, it’s deployed on Linux too. We don’t run those servers ourselves. There’s other guys that are awesome in building vmware clusters with all kind of features for scalability and uptime so we really don’t want to do that ourselves. So we just buy processing power and storage from these guys.
We also use the Mule ESB as a way to transport and distribute events. We don’t want to spend time in implementing transport protocols for event transport or spending time in creating integration infrastructure. Mule gives us that more or less free. We’ll not completely free. We had to extend Mule to read it’s configuration from a database (postgres) and do the configuration programmatically instead of using the standard way of XML. Mule comes with so many transport protocols so we can easily get events in and out of ruleCore to just about every system out there.
The event sequencer is running as a component inside Mule. It could run in an app server of as a stand-alone service but it fits nicely into Mule for now.
The most common question I get about this setup is why we use Python? Is not Python very slow? Well, it’s slower that Java most of the times. But sometimes developer productivity is more important than raw speed. My guess is that we do five times more in Python than in Java. With the event-driven architecture we have, we don’t base our performance on running fast on one single server anyway. Our scalability strategy is instead to add lots of cheap PCs. And sometimes a slow system is better than a non-existing one. Python allows us to create things that would be too expensive (taking too much time) otherwise.
Go to RuleML 2009
September 25th, 2009
Comments Off
Want to learn more about rule technologies, applications and standards?
Want to discuss latest research results and developments?
Want to get in touch with the industry and research experts in rule-based systems and technologies?
Don't miss RuleML-2009 - early bird registration deadline ends on September 25th
http://2009.ruleml.org/registration
---------------------------------------------------------------------------
RuleML 2009
3rd International Symposium on Rules, Applications and Interoperability
November 5-7 2009, Las Vegas, Nevada, USA
http://2009.ruleml.org/
co-located with the 12th Business Rules Forum
----------------------------
We cordially invite you to participate in the 3rd International Symposium on
Rules, Applications and Interoperability (RuleML-2009) - the premier event
addressing research, applications and standards for rule technologies and
rule-based event processing technologies. It takes place in Las Vegas,
Nevada, USA, Nov. 5-7, collocated with the 12th Business Rules Forum, the
world's largest Business Rules event.
RuleML-2009 has evolved from an annual series of international workshops
since 2002, international conferences in 2005 and 2006, and international
RuleML Symposia since 2007, to the premier event on rule-based systems
addressing industry developments, standards and academic research at the
same time. RuleML-2009 is in the top 100 venues for impact factor (71):
http://citeseerx.ist.psu.edu/stats/venues. It is supported by W3C, OMG, ACM,
EPTS, AAAI, ECCAI, IAAIL, ACM, Belgian Business Rules Forum, MIT Sloan CIO
Symposium, BPM Forum Belgium and a listed EU ICT technology conference.
We are building an excellent program with renowned experts as keynote
speakers, tutorials, standards panel sessions, Open Source Day, W3C RIF
workshop, Int. Rule Challenge with prestigious prizes, etc.
MAKE RULEML-2009 YOUR SHOWCASE!
Submit to the 3rd Int. Rule Challenge at http://2009.ruleml.org/challenge
*******************************************************************
RuleML-2009 highlights and news:
* 3rd Int. Rule Challenge
"Demonstration of benchmarks/evaluations, demos, case studies / use
cases, experience reports, best practice solutions, rule-based
implementations/ tools/ applications, demonstrations engineering methods,
implementations of rule standards"
* Rule Open Source Day
"Discover how Open Source Rules solutions are being deployed in the
marketplace and how they compare to proprietary solutions"
* Lunch Panel on Standards for Business Rules
"Want to learn more about latest business rules standard in W3C, OMG, OASIS,
ISO, RuleML and how these apply in the enterprise business arena"
* W3C Rule Interchange Format Standard Workshop
* Keynotes and special talks by
Sandro Hawke (W3C) about W3C RIF
Donald Chapin (OMG) about OMG SBVR and ISO Terminology standards
Paul Vincent (TIBCO) about CEP and Rules
Jans Aasman (FranzInc. CEO)
* Tutorials by
Mark Proctor (Drools) - Drools Tutorial
Christian Saint Marie (ILog/IBM) - W3C RIF Tutorial
Larry Goldberg and Babara von Halle - Decision Management
Guido Governatori (NICTA) - Business Process Compliance
* Tracks on
Rule-based Event Processing
Rule Systems on the Web
Rule Transformation and Extraction
Process and Data
Rules and Norms
Rules and Uncertainty
* RuleML-2009 Springer LNCS proceedings published
* 3rd Rule Challenge CEUR proceedings to appear
* IEEE TKDE special issue on Rule technologies to appear Dec. 2009
* Forthcoming RuleML-2009 special journal issue
*******************************************************************
Supported by
===================================================================
W3C, OMG, ACM, AAAI, ECCAI, IAAIL, ACM, Belgian Business Rules Forum
MIT Sloan CIO Symposium, EPTS, BPM Forum Belgium
Sponsored by
===================================================================
BBN Technologies
Franz Inc
NICTA (National ICT Australia) Ltd
Corporate Semantic Web
Logic Programming Associated Ltd
ruleCore
JBoss RedHat
Modelsystems Ltd
(sponsoring opportunities: http://2009.ruleml.org/sponsors)
===================================================================
What About Time?
September 11th, 2009
In my ruleCore Internal series (well, there’s only one post this far so it’s not much of a series yet) I thought of writing about time. There’s actually lots to say about this simple concept.
Meanwhile there’s these:
- http://www.aleri.com/developers/blogs/aspects-time
- http://epthinking.blogspot.com/2009/09/on-temporal-aspects-of-event-processing.html
A really good summary of the different aspects of time. Maybe I’ll write my "ruleCore Internals - Time" article by answering these questions…
Recent Comments