Westpac killed its own services after fault

Westpac killed its own services after fault

Summary: It has been revealed that Westpac staff yesterday made an executive decision to turn off the bank's online banking, EFTPOS and ATM network rather than failover to a disaster recovery site in order to "save time".

SHARE:

It has been revealed that Westpac staff yesterday made an executive decision to turn off the bank's online banking, EFTPOS and ATM network rather than failover to a disaster recovery site in order to "save time".

It was originally thought that an air conditioning fault in Westpac's datacentre environment had forced the services offline, with Westpac scrambling to restore service throughout the day. While there was an air conditioning fault, it is now apparent that Westpac made the decision itself to switch off systems to prevent damage to the infrastructure from high temperatures. An alternative would have been to switch systems to a disaster recovery site, but the bank said it had decided that this would take too long.

"We did take the decision to focus on restoring services from the impacted datacentre rather than switch to our backup systems, as it would have taken more time to do so," said Paul Marriage, head of media relations for Westpac.

A full review has been promised by Westpac executives; however, Marriage said that Westpac isn't looking to point fingers over the fault in the air conditioning, which is managed by an external maintenance provider.

"As to who's to blame … it's a Westpac responsibility and through our review we'll get to the bottom of it and make sure it gets fixed and processes get fixed around it but at this stage, for our customers, it doesn't help them if Westpac gets into a blame game."

Marriage wouldn't be drawn on whether Westpac would be looking to switch maintenance providers following the outage.

The system meltdown struck customers early yesterday morning, forcing ATMs, online banking and website services offline.

Westpac gradually restored services throughout the day, with headaches flowing down to customers such as CityRail, which was unable sell tickets via EFTPOS.

Westpac's card customers also experienced issues booking tickets for popular music festival Splendour in the Grass. Ticket sales were suspended overnight after some music fans missed out, with sales going live again at 9am this morning.

The outage came after Westpac announced a record half-yearly profit figure of $3.96 billion this week.

According the Marriage, the review is "ongoing".

Topics: Outage, Banking, Data Centers

Luke Hopewell

About Luke Hopewell

A fresh recruit onto the tech journalism battlefield, Luke Hopewell is eager to see some action. After a tour of duty in the belly of the Telstra beast, he is keen to report big stories on the enterprise beat. Drawing on past experience in radio, print and magazine, he plans to ask all the tough questions you want answered.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

5 comments
Log in or register to join the discussion
  • Good DR... Wonder what their RTO is as defined by the BIA.

    It is interesting that the RTO should really vary by time of day - an outage around peak hour 8-9:30am has a much greater impact than 2-3am.
    gumb0r
  • It may be true that Wespac decided to shut the computer systems down, but it was an AC fault that caused the issue. It is the very final step to decide to shut down the operations due to the system recovery required, the inconvenance to customers and the bad publicity, and the very very very large cost of processing all the paper vouchers form the point of sale retail customers with the loss of electronic processing.
    If it is not the service providers management of the site "United Group Services" why are they in crisis meeting yesterday afternoon and the senior staff had to fly up from melbourne. You just dont turn things off if you dont have to.
    Gottobesaid
  • Looks like the airconditioning was just a scapegoat and they had to retract after the Data Centre vendor complained and then Westpac said they shutdown the system themself. I find this a strange explanation given that it is not easy to make a call like shutting down major banking systems without the highest level of approval. Westpac Board should be looking hard at how this happened and who authorised it.
    thinkinghat1
  • I was thinking how this can happen - if they have taken there IT and its physical infrastructure through a system such as ISO. IF they have conducted Risk Analysis, then the Analysis will indicate - major disaster due to Thermal runway - which is highly unlikely due to temperature ratings on device itself and there are precautions in place to prevent it.
    Anyway, if they have conducted Risk analysis properly then ISO will pick up issues relating to Physical infrastructure and impacts flowing down the track creating chain reaction. It seems like they have not taken the business through standards such as ISO 27001 -x (or Specifically Australian Standards AS). These types of failures are not possible if the controls are identified and Risk mitigation is in Place. Standards are designed to identify and prevent RISK TO BUSINESS – also this indicates that their disaster recovery is not adequate enough to handle the load.
    -Prasanna
    abey1
  • Instead of implementing some good technologies, management will engage in a religious talks about ITIL, project management, gouvernance and risks.
    If they had implemented 100% VMware and Site Recovery Manager as we did, DR would be just a 10 min experience.
    No respect to infrastructure tells me a lot about Westpac management.
    vladimir_popov9