Why does 'complex event processing' have to be so complex?

I've always felt that "complex event processing" is a term that scares people away. Perhaps there needs to be a softer way to describe what this thing is -- perhaps just plain old "event processing" will do? Or perhaps "sense and respond, times 100,000"?

You don't need to be a state-of-the-art, completely online organization to move to event processing. David Olson, director of CEP product marketing for Progress Software's Apama unit, says from a technical standpoint, the fundamental system requirement for event processing is having a server to capture and correlate events, apply appropriate rules to an event stream, and send the information to a dashboard or portal.

But the success of an event processing implementation is a management challenge, he says.  "One of the more complicated endeavors that people have to figure out is what events make sense to them, and what types of patterns do they need to look for?"

I recently had the opportunity to join David, along with Brenda Michelson, principal with Elemental Links, in a rousing roundtable discussion on the impacts of event processing in what has been a very eventful year for businesses. (Full transcript available here.)

For example, financial services has been at the center of the economic storm over the past 18 months. If these companies had more advanced capabilities to process and analyze key events, they might have been able to head off many of the slips and slides that led to the financial crisis of 2008, David and Brenda agreed. Still, there are many lessons being gleaned from the whole experience that will benefit all industries for years to come.

"I think [Complex Event Processing] could have been used in some situations that got us into this financial crisis," says Olson."Perhaps we should have employed CEP a few more years ago in order to help stem the tide of what has happened. But there's certainly a lot of learning that's been going on as people go back and research what has happened, to figure out what kind of rules they should put in place in the future to make it happen again."

Brenda pointed out that millions of events occur across enterprises every day -- from stock prices bouncing to soda cans shifting on a pallet in a warehouse. The challenge is being able to capture and process the events that have the greatest impact on the business. "Event processing is what discerns is if that thing that happened is notable," she explained:

"Is it important to me? Is it important to my business? Do I need to act on it?  And is that notability by itself, or perhaps its notable because of a couple of other events that are going on? Is it just my business that's bad, or is it the entire credit market that's tanked? And then once you discern that notability.. you decide what am I going to do with that event next?  Am I going to forward it along into an event channel? am I going to trigger some kind of downstream action?"

David elaborated on that theme and pointed out that "its not sufficient just to be able to capture the events and perhaps store them for future use, since the volume and the velocity can be quite great in certain circumstances."

There is great potential being demonstrated in financial services in terms of unearthing fraud, David points out. "One of the exciting areas in CEP, especially in capital markets is the whole notion of surveillance, where there are a fair amount of rules that look for abnormal patterns of activity," he says.

"For example, one of our customers uses it to monitor actions that could indicate insider trading, or joint trading opportunities where two traders trade large volumes of a symbol, but not large enough that it tips anybody's radar to abnormal activity.But when these joint trading activities occur either ahead or behind a news feed, that could indicate to us there's unusual activity going on here, and we should do something with this particular symbol that's being traded. In the past, trying to catch those types of activities could take months of auditing in order to figure out that something bad happened. And in the capital markets space, minutes of something bad happening could be billions of dollars."