BlackBerry services now fully restored, says RIM chief
"Nobody's gone home since Monday," says RIM co-CEO Jim Balsillie...
BlackBerry is back in business, according to RIM, after the largest outage the company has experiencedPhoto: Dushaun
RIM says it has restored normal service to the BlackBerry network following this week's outage, the largest the company has experienced.
"We have now restored full services," said RIM co-CEO Mike Lazaridis adding that the BlackBerry network has had a "99.97 per cent service level" over the last 18 months.
Lazaridis was speaking during a webcast this afternoon to give an update on a situation that has seen BlackBerry users in EMEA, India, South America and the US disconnected from email and the mobile web, and confirmed that the outage was the largest RIM has experienced.
Any residual delays still being experienced by BlackBerry users are down to the backlog of data generated by the outage. "We're just flushing through the queue now," noted co-CEO Jim Balsillie, who was also on the call.
Lazaridis also said some BlackBerrys may have lost synchronisation because of the network outage - saying this can be resolved by removing and replacing the BlackBerry battery to reset the handset.
Lazaridis gave further details of the cause of the service outage - saying that on Monday, when the outage began affecting BlackBerry users in EMEA, a switch failed which in turn caused other problems in the system.
He explained: "On Monday we had a hardware failure that caused a ripple effect in our system. A dual-redundant high-capacity core switch designed to protect the infrastructure failed and caused outages and delays for some customers.
"This caused a cascade failure in our system. There was a back-up switch but the back-up didn't function as intended and this led to a backlog of data in the system. The failure in Europe in turn overloaded systems elsewhere. When we restarted the system based in Europe the data queue processing took much longer than we had expected to restore to our standard service levels. This backlog impaired service levels."
"The back-up didn't function as intended and overloaded systems elsewhere," Lazaridis added. RIM would be performing "root cause analysis" to determine why the back-up didn't perform as expected, said Lazaridis.
"We don't know why the switch failed in the particular way it did - and why it did not fail-over [to the back-up]," he said.
Lazaridis said RIM will be working with vendors to fully understand and correct the switch failure which led to the outage in the first place.
The BlackBerry maker will also be auditing its infrastructure to understand why it took the system so long to recover from the hardware failure.
Lazaridis would not single out any particular switch vendor or vendors at this point saying it is too soon to pin any blame for the hardware failure.
He again apologised to BlackBerry users for the outage, and said: "Our inability to fix this quickly has been frustrating" adding that RIM is "taking immediate and aggressive steps to minimise the risk of this happening again".
"Nobody's gone home since Monday," added Balsillie.
Lazaridis said RIM would also be working very hard to win back the trust of BlackBerry users. "We've worked for 12 years since the launch of BlackBerry to win the trust of our 70 million subscribers - and we're going to work fully to win the trust back," he said.
The company said it is too soon to address the question of compensation for BlackBerry users or mobile operators affected by the outage.
The company added that it will continue to provide additional service updates throughout the day.