Computer 'mystery outages' plague Australian Emergency Services

A series of three "mystery outages" have forced the Department of Emergency Services (DES) in Queensland, Australia to suspend rolling out a new, $6 million computer-aided dispatch system.

Responding to the outages, Queensland Emergency Services Minister Neil Roberts commented:

The one just yesterday is unexplained at this stage, however we're confident we're going to be able to find out the reason for that.

We're seeking advice from the United States suppliers and until we get a clear indication of what the actual issue was, I've instructed the Department to put on hold the further roll-out of this system....

After an outage last June, a Department press release cut the system some slack:

Late yesterday the system experienced an outage for approximately 90 minutes where it was necessary for operators to resort back to a manual system. The cause of this outage was related to a maintenance issue, not the system.

Emergency response personnel are dissatisfied with the problems, as demonstrated by this quote in AustralianIT:

Fire officers have been highly critical of the new system, which they say has only been "half implemented" by the department making it ineffective.

"The system is meant to locate the closest vehicle to an incident and dispatch that vehicle. But they're yet to install the automatic vehicle locaters in our trucks so it can do this," an officer said.

The Department of Emergency Services has investigated all three outages, according to a spokesman:

Thorough investigations have been conducted as to the cause of each, with the following outcomes:

• Two of the outages have been attributed directly to human error. The first was an SQL update which had unexpected results, the second was a server which was inadvertently shut down.
• The third outage has been traced back to a fault with the replication software 'Replistor', which resulted in the primary database server rebooting.

The United Supplier referred to in the DES media release was in fact Stratus, as the primary database server had rebooted during the 3rd outage, and we had asked Stratus to investigate the logs and provide a report.

Based on these facts, it appears the DES should evaluate whatever procedures are required to ensure that similar human errors do not cause additional failures. At the same time, the Department is to be applauded for its openness in presenting facts to the public.