BlackBerry's outage post-mortem: Where did it all go wrong?

Just as the BlackBerry maker will be looking for answers to its four-day global outage, so are users. Why was the situation handled so badly?
Written by Zack Whittaker, Contributor

As BlackBerry service around the world returns to normal, online tension has calmed and many are seeing normal service resume on their smartphones.

But many are still angry, not least Research in Motion, a company on the cliff-edge of the mobile market.

Even as the company scrambles to discover exactly what went wrong, the outages are but one of  the many problems facing the Research in Motion. Also, the manner in which the company treated its customers was contemptible at very least, and this will no doubt serve as a very painful learning curve for Research in Motion.

Whether it can recover from it in the long run is another story completely.

Source: Associated Press

BlackBerry suffered a four-day outage, which spread from Europe and the Middle East, through to Africa, Latin America, and finally the U.S. and Canada.

Starting as a server outage at a BlackBerry datacenter in Slough, near London, the knock-on effect spread to almost every continent, affecting over three-quarters of the 70 million BlackBerry population.

Users took to social networking sites, and Twitter erupted with trending topics of anger, frustration and disappointment in the service.

But the outage itself was eclipsed by the overriding issues of Research in Motion's stonewall of silence, followed by confusing messages and then a hardly-humble apology from executives which added to the press frenzy. Almost nothing the BlackBerry maker did was right; and now, only a few days after the issues started, Research in Motion will be performing its own post-mortem on "what went wrong".

Communication's company didn't communicate

A great deal of focus for BlackBerry to take note of, as part of its post-mortem process, is the awful communication on the part of the company during the outage. In many cases, the silence from the Ontario-based company left the talking to the mobile networks, like T-Mobile and Vodafone.

Particular focus was on a "core switch failure", which -- to the vast majority of people -- will mean nothing. And here lies the problem.

During the initial wall of silence, the company finally spoke up via Twitter to explain that there was a "core switch failure." A number of these tweets included confusing jargon like 'EMEA' and even 'RIM', the acronym for the company that owns the BlackBerry brand.

Many had turned to Twitter in fact to discover what was going wrong with the service. Though some cellular networks had taken it upon themselves to send out text messages -- unaffected by the outage -- to inform customers of the ongoing difficulties, many BlackBerry users couldn't even use Twitter from their handsets due to the global data outage.

From a reputation standpoint, only delivering  a handful of tweets during a global outage of its services leaves the impression that the company is denying the existence of the problem -- hoping that the issue will resolve before the press notice.

They did, and it didn't, forcing the hand of Twitter users with the #DearBlackBerry hashtag, promoting it to worldwide trends, as many scrambled to find answers to why their services had stopped.

Only in the final day of the worldwide disruption did you have the two executives, Stephen Bates as managing director of the UK branch of BlackBerry, and Mike Lazaridis, the company's co-chief executive, come out and apologise for the extended downtime.

For many, this came too late. Utter silence gave way to verbal assurances peppered with images of further disruption ahead. It was a public relations disaster of epic proportions, and many will not forgive the company for handling the outage so badly.

Key is the network: Disruption was inevitable

BlackBerrys run on Research in Motion's infrastructure of email delivery, data provision and browsing capability, which piggy-back on the cellular networks. Adding another link to the chain of the ecosystem allows for secure email and messaging, but also increases the chance of something in that chain breaking down.

In this day and age, though smartphones are still 'phones' and can send text messages almost by standard, data is the killer feature to any smartphone on the market.

But as BlackBerrys rely on data to function, its smartphones rely on that infrastructure to run. With backup systems and backup systems from those backup systems, when an infrastructure is edited or changed, it has to be fail-safed. In this case, it wasn't.

In any case, infrastructures are delicate 'organism-like' technologies, and require maintenance and upkeep. But because of the BlackBerry data centralisation, something was inevitably going to go wrong, whether it was this week, next month or in a year from now.

Competition wise: 'If it ain't broke...'

The outage hit at the worst possible time for the Ontario-based company. Embarrassingly, the Canadian company faced questions on its own home turf when the issues spread from Europe, the Middle East and Africa, through to South and Latin America, ending up on the doorstep in the United States and Canada itself.

But others were quick to explain why Android and iPhones "won't go down like BlackBerry" did over those painful four-days where service was heavily disrupted.

Having said that, with Apple's new iOS 5-based iMessage service, there is reason to believe that the BlackBerry Messenger competing product could also face difficulties, along with iCloud email. But because Apple barely invests in critical email infrastructure of its own, unlike Research in Motion, it allows Apple and other phone manufacturers to blame the networks, rather than the phones.

As BlackBerrys are 'slaves' to the infrastructure, iPhones and Androids can still run critical services -- even without direct data connections to the 3G or 4G cellular network.

The nail in BlackBerry's coffin?

BlackBerry customers are surprisingly loyal, through personal preference by the choice of their employers. Though enterprise customers were not nearly as widely affected as end-user consumers were, it amounts to further pressure for businesses to consider whether the BlackBerry is a stable communications provider for their email and other services.

But a massive revolt of the service is unlikely to change how Research in Motion moves on. If anything, the worldwide struggle to access data services will be reflected by investors and shareholders, who will want a change in management, or to sell off the company altogether.

The outage itself will not be the nail in the BlackBerry coffin, but it's certainly cutting it close to the company's downfall.


Across the CBS Interactive network:

Editorial standards