Sense and respond -- should humans be part of complex event processing?

Sense and respond -- should humans be part of complex event processing?

Summary: A war is a complex event, so is a business customer transaction


This week, a report out of the Financial Times (cited here in the LA Times) said that a bug used in the computer models used by analysts at Moody’s Investors Service caused Moody’s to award "incorrect triple-A ratings to billions of dollars worth of a type of complex debt product." The report alleges that the errors were not corrected even after the code was fixed a year ago.

The implication is that millions, if not hundreds of millions, of dollars in investment decisions may have been made on faulty data generated by the system. If you've been following the financial markets lately, it was not a good time for that.

This once again raises the question of how risky is it to take humans out of the loop of event processing? Of course, humans are fallible, and usually create the messes we then look to technology to fix.

Dr. David Luckham, considered the "father of complex event processing," provided a sterling example in a recent ebizQ panel discussion of a very complex event that needed a lot of processing: World War I.

World War I -- which was sparked by a chain events from an assassination in the Balkans, and was followed by a chain of events that exploded into World War II -- was probably too complex for any human or machine to process the various inputs and outcomes. The event was "so complex that many books have been written about how it all happened," said Luckham. "We can't really pin down the functional relationship between the First World War and all the things that came out of it, all the events that were its members. It was more than a composite event."

This is typical of a complex event, which is something composed of multiple events. A complex event is "something that has many members, and there’s some doubt as to what the members are," he explained.

Fortunately, while organizations have their own sets of complex events to be understood and managed, most are not on the scale or with the consequences of a world war, Luckham added:

"Processing complex events can in fact be very simple. An example can be filtering events based on their creation time. Or the processing could be more complicated. You need techniques for organizing the processing. That’s what complex event processing is about, its about techniques for processing events."

Luckham had joined Dr. K. Mani Chandy, Rodney Morrison, and Beth Gold-Bernstein for an informative panel discussion on the relationship between EDA (Event Driven Architecture) and SOA. (Registration required for this ebizQ Webcast.)

The architecture created to organize the processing of multiple events, or events springing from other events, is Event Driven Architecture, which Chandy defined as "the architecture for acquiring information that’s relevant to the enterprise, senseing everything that's going on, and detecting a significant state change to the event. This whole cycle of detecting, analyzing and responding is supported by EDA."

As noted in a previous post at this blogsite, it is estimated that some organizations may be seeing up to a trillion events a day. It stands to reason that to process and act on this kind of volume, a lot of complex event processing will have to be automated.

However, when is human intervention called for? While the panelists were in agreement over the necessity of moving to event-driven processing to maintain competitive, they were divided over how much human interaction should remain in event-driven processes.

Chandy, for example, said that human engagement is essential at some point in the complex event processing cycle. "I’ve seen quite a few applications, but almost none in which there is no human involved" For example, Chandy said, "in military applications, when there is a gun or device is fired to kill somebody. It’s always done with a human being responsible for their final action."

As Chandy pointed out:

"I think its absolutely critical for human beings to be part of this sense and respond system. It's important how the application supports the human being if you're looking at trading or fraud detection. In all these cases, its really important to have a human being involved. Fraud is one case where you might have an application that informs a credit card user that something inappropriate may be going on, without having a human being first check that."

Luckham, however, pointed out that there are many events are processed independent of human intervention. "There are many examples of event driven architectures where there are absolutely no humans whatever," he said. "The CPU on your computer is an event driven architecture, believe me. And its entirely event driven, clocked, without a human in the loop.

Whether human will be involved in complex event processing "just depends on what kinds of architectures you’re looking at," Luckham concluded. "In the systems Mani was talking about, yep, you've got humans in the loops. In standard event driven architectures, however, there aren’t any humans in those loops."

Panelists also connected the dots between EDA and SOA. "If SOA means loosely coupled subsystems with very clean interfaces, so that new systems could be coupled into the substrate, then EDA events fit within that framework, because EDA is also based on a loose framework, and is extensible," Chandy said. In a request-reply SOA scenario, "then EDA can still be coupled. There will be layering between the push and the pull parts."Plus, SOA can play a role in helping to manage EDA systems, Chandy said:

"One way that I’ve seen EDA used in conjunction with SOA is for service management. Many SOA vendors are exposing metrics that can give you information like end to end process time and activity times. Those metrics can be provided to a CEP system to help control and manage those services. I can, for example, do dynamic provisioning for a service that's getting maxed out."

Morrison provided examples of EDA in action, noting that it has been standard practice in some industries for a couple of decades already, including the manufacturing process control and defense industries. Now information-intensive industries such as finance, telecom, supply chain and transportation have been adopting EDA methodologies and technologies. They have led the way in "standardization of such things as pub-sub oriented messaging oriented middleware, and putting together processes that really do event driven processing."

Examples of EDA and CEP applications range from "very intense, highly specialized applications such as algorithmic trading in the energy or finance markets, to applications that need to integrate well with a large number of IT services, like supply chain applications," Morrison said.

It's the applications requiring integration that are the best fit for SOA environments, he continued. "They obviously can gain some great benefits, and agility be able to integrate those event driven applications into a SOA framework."

Topics: Browser, Apps, Enterprise Software, Software, Software Development

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • The problem as I see it is that

    computers have no sense of magnitude of a problem and human does. Say something as simple as running off payroll checks. To a computer being off a dollar or a thousand dollars is the same thing. To a human a thousand dollar mistake tiggers a reaction.
    • Exactly...

      And that raises the question of how much of a process to entrust to rules automation. There's a chasm of trust in automation that needs to be crossed, but its a wide chasm. In some instances of event processing, event triggers and subsequent processes may occur in subseconds. For routine processes that may, for example, adjust the performance of a server, automation is desired. But processes that direct the business itself require very careful planning and oversight.
      • MRP and MRP II are good examples.

        MRP (Manufacturing Resources Planning) was to be the best thing ever to hit manufatures and the supply chain.

        Based on Inventory On Hand and orders it was to plan purchasing or restocking of raw materials of sub assemnblies and communicate that to your vendors and order materials/parts and in kind their MRP system contacted their vendors and ordered materials, and on and on in a long and very complex chain.

        Of course the more complex the supply and delivery chain, the more "events" there were to throw a wrench in things. What we seen is that beyond one or two steps in the vendor chain the events became to many and too varied for the system to work effectively without stringent human supervision.

        I am not saying MRP is without its uses, far from it. But its not something companies can simply "turn on" and allow it to run and act independently on events.
  • RE: Sense and respond -- should humans be part of complex event processing?

    Humans are unavoidably part of complex event processsing, but their roles can be, and too often are, ignored. A business is a social system, which means that people are intrinsically involved--even in an entirely automated process, such as cash dispensing from an ATM.

    This is so because only humans, not technologies, can athentically assume accountability for an outcome. If the ATM spits out the wrong amount, or is out of service, or is too user-unfriendly for a client, it is a human --usually a teller or branch manager --who is accountable for seeing that the right amount is dispensed.

    On top of that, whenever a new variable becomes part of the complex system, event processors working from models without that variable are inadequate, and the default is/should be to an accountable individual to deal with the situation. The Maytag serviceman may not DO anything, but he is, in theory at least, accountable for keeping your Maytag Dryer working.

    Sense & Respond is fundamentally a managerial, rather than a technological innovation.It is necessary in order to fully exploit adaptive technologies. See for information on the history and nature of S&R, and access to the Intellectual Capital on S&R that is in the public domain.