DBS points outage finger at IBM
SINGAPORE--DBS Bank says a failure on IBM's part to correctly fix an identified instability within the bank's storage system led to the seven-hour service outage last week.
In a statement posted on its Web site addressing its customers, DBS CEO Piyush Gupta said IBM was carrying out "a routine repair job" to address alert messages that indicated a problem with a component in the disk storage subsystem, which was connected to the bank's mainframe.
The IT vendor's Asia-Pacific team was scheduled to replace the component during off-peak hours, specifically 3 a.m., but an error in how the fix was applied triggered a malfunction in the bank's redundancy systems, leading to the outage.
Noting that investigations are still ongoing and also involve IBM's lab engineers in the U.S., Gupta said: "So far, we understand from IBM that an outdated procedure was used to carry out the repair. In short, a procedural error in what was to have been a routine maintenance operation subsequently caused a complete system outage."
The bank's network was brought to a standstill on Jul. 5 when DBS and POSB customers across the island-state were unable to access branch, ATM (auto teller machine), Internet and mobile banking services. Its services remained down for seven hours before they were fully restored at 10 that same morning.
IBM released its own statement here Tuesday evening acknowledging its failure to correctly deploy the fix to address a detected "instability in a communications link within DBS' storage system".
Offering its apology in the statement, Big Blue said: "IBM has also taken steps to enhance training of our personnel related to current procedures and brought in experts from our global team to provide further assistance. Working with DBS, additional actions are underway to increase the resiliency and redundancy of this part of DBS' infrastructure."
Gupta further admitted that the bank's response to the outage could have been better managed and more immediate. "We could also have done more to mobilize broadcast channels to inform customers of the disruption in services first thing in the morning," he said. "Please accept my apologies and know that we take full responsibility for this incident. There have been valuable lessons learned and I assure you that this matter will continue to remain a top priority for me."
According to the CEO, all payments and transactions made on the day of the outage were completed and "full data integrity was maintained".
The Monetary Authority of Singapore also in a statement Monday said it would decide on the appropriate course of action after evaluating the incident and the extent to which DBS had failed to meet recommended procedures stated in the Internet Banking and Technology Risk Management Guidelines.
DBS has over 900 ATMs in Singapore and 1.35 million online banking customers as of March 2010, with an average 3 million logins a month.