Is anyone ready to process a trillion events per day?

What is event processing and how could it make a difference? Here is one example of where it could take us: Scientists are talking about the possibility of an in-body network that could detect heart attacks or diabetic collapses and alert emergency services.

A typical company deals with millions, if not billions, of events in a single day, and most are still handled manually, by people

Interesting idea. But then the article goes on to suggest that the same technology could be used to monitor whether patients are taking their medicines. "A pill dispenser would send an automatic reminder and, if the pills were not taken within a certain time, an alarm would sound and a message would be sent to the patient’s family or caregivers."

So we have the need-to-have, lifesaving value of event processing, and the intrusive, nagging aspect. Now, apply this kind of scenario with a business. That is, if the ongoing health of the business is monitored and maintained with intelligent technology, serious issues could be headed off either by alerting decision makers before things happened, or even in an automated fashion. But is it possible to have too much processing of events that aren't as critical, and merely drive staff nuts?

Of course, the challenge is that businesses require more than taking a few pills each day to keep problems in check. And note that the pill-nagging system still requires human intervention at the end of the process. A typical company deals with millions, if not billions, of events in a single day, and most, if not all, of these events are still handled manually, by people. It's obviously not an efficient and competitive way to do things, and the more you can automate, the better. But how badly is it needed?

Gartner VP Roy Schulte recently made a compelling case for increased complex event processing in an online presentation delivered as part of ebizQ's recent Event Processing virtual conference (registration required). "At any one second, a large company has on its network anywhere from 10,000 to 10 billion business events," Roy explained. "At the low end, that's almost a billion events per day -- at the high end, that’s almost a trillion events per day."

The question is, which of these billions of events need to be captured and automated?

A little aside here: The ante on all these enterprise events in play is definitely rising. In my last post, I quoted Tibco's CEO Vivek Ranadivé, who discussed a typical bank having 100 million events a month to be managed. One reader made this observation about this number, which could be way higher, and therefore really drive up scalability requirements:

"100 million a month is hardly a large bank. Besides, most of these events will be transactions that are typically still processed in batch, resulting in peak loads of millions transactions per hour. If all these transactions can be considered events, that will result in peaks up to 1000 events per second. Considering the idea in your post that in the event driven era the main goal is to share the information, one should expect that one event will result in a few processing actions per event. So the hardware you will need for this will be impressive."

Schulte also stressed that a lot of work -- and perhaps a lot of purchases -- lay ahead before organizations can embark on an effective event processing path. Most of these events are not captured or automated in enterprises -- the challenge is that most of the stovepiped and legacy applications that power enterprises are not yet event driven, Schulte said.

The question is, then, does everything need to be tied into complex event processing? As with real time, there may be some functions and some industries that need it more than others. Many organizations can plod along with manual responses where there isn't an impact on the competitiveness of the business. But we're gradually getting there. The growing array of sensors, such as RFID tags, combined with front-end systems such as business activity monitoring (BAM) dashboards, with service-oriented middleware in between, make complex event processing a reality with today's technology, Schulte points out.

However, where complex event driven processing is occurring on a well-integrated enterprise scale, the results have been demonstrable and promising, Schulte said.

• In airline operations, "information from hundreds of sources, including sensors on board the aircraft, information coming in from the FAA, and information coming in from standard systems is sent to the 'enterprise nervous system,' and is temporarily stored in event databases. It helps to create the data, the outgoing alerts and notifications that is sent to hundreds of applications on the consuming side.... Information helps the fueling and maintenance management applications to change their schedules and so forth. By using an event based system, the turnaround time of each plane can be shortened… Fewer airplanes are needed to handle the same schedule."
• In supply chain management systems, embedded complex event processing logic "consolidates event data that is coming in through various channels such as through email and Web services. Supply chain management system can answers questions like, 'where are the goods,' 'when are they are going to arrive,' and so forth. The events are persisted in an event database made available to customers and suppliers."

There's a good case to be made for completely automating the ability to capture and make decisions on multiple event streams coming into the core business systems, Schulte says. "For example, you can have a complex event that says, ‘this mornings sales were 30% above our daily average.' That of course is much easier to digest and act on than sending a person 500 detailed sales records, and making the person compute what happened that day manually."

"The value of complex event processing, overall, can be summarized as improving situation awareness," Schulte said. "Simply put, that is just knowing what is going on, so you can figure out what to do." The benefits of complex event processing, Schulte said, include better decision quality, faster response times, reduced information glut, and reduced costs."

Complex event processing is the ability to act, in real time, on multiple streams of event data flowing in from different parts of the enterprise. Schulte defined a business event as a "meaningful change in a state that is something that is relevant to the business. Examples include depositing or withdrawing money from a bank, submitting a purchase order, or hiring an employee."

There is also a second term, "event object," that describes how the event is packaged for processing, typically as an XML document these days. "We have to record events using event objects so computers can receive them and do computations on those events," Schulte said.