BlackBerry's outage post-mortem: Where did it all go wrong?

BlackBerry's outage post-mortem: Where did it all go wrong?

Summary: Just as the BlackBerry maker will be looking for answers to its four-day global outage, so are users. Why was the situation handled so badly?


As BlackBerry service around the world returns to normal, online tension has calmed and many are seeing normal service resume on their smartphones.

But many are still angry, not least Research in Motion, a company on the cliff-edge of the mobile market.

Even as the company scrambles to discover exactly what went wrong, the outages are but one of  the many problems facing the Research in Motion. Also, the manner in which the company treated its customers was contemptible at very least, and this will no doubt serve as a very painful learning curve for Research in Motion.

Whether it can recover from it in the long run is another story completely.

Source: Associated Press

BlackBerry suffered a four-day outage, which spread from Europe and the Middle East, through to Africa, Latin America, and finally the U.S. and Canada.

Starting as a server outage at a BlackBerry datacenter in Slough, near London, the knock-on effect spread to almost every continent, affecting over three-quarters of the 70 million BlackBerry population.

Users took to social networking sites, and Twitter erupted with trending topics of anger, frustration and disappointment in the service.

But the outage itself was eclipsed by the overriding issues of Research in Motion's stonewall of silence, followed by confusing messages and then a hardly-humble apology from executives which added to the press frenzy. Almost nothing the BlackBerry maker did was right; and now, only a few days after the issues started, Research in Motion will be performing its own post-mortem on "what went wrong".

Communication's company didn't communicate

A great deal of focus for BlackBerry to take note of, as part of its post-mortem process, is the awful communication on the part of the company during the outage. In many cases, the silence from the Ontario-based company left the talking to the mobile networks, like T-Mobile and Vodafone.

Particular focus was on a "core switch failure", which -- to the vast majority of people -- will mean nothing. And here lies the problem.

During the initial wall of silence, the company finally spoke up via Twitter to explain that there was a "core switch failure." A number of these tweets included confusing jargon like 'EMEA' and even 'RIM', the acronym for the company that owns the BlackBerry brand.

Many had turned to Twitter in fact to discover what was going wrong with the service. Though some cellular networks had taken it upon themselves to send out text messages -- unaffected by the outage -- to inform customers of the ongoing difficulties, many BlackBerry users couldn't even use Twitter from their handsets due to the global data outage.

From a reputation standpoint, only delivering  a handful of tweets during a global outage of its services leaves the impression that the company is denying the existence of the problem -- hoping that the issue will resolve before the press notice.

They did, and it didn't, forcing the hand of Twitter users with the #DearBlackBerry hashtag, promoting it to worldwide trends, as many scrambled to find answers to why their services had stopped.

Only in the final day of the worldwide disruption did you have the two executives, Stephen Bates as managing director of the UK branch of BlackBerry, and Mike Lazaridis, the company's co-chief executive, come out and apologise for the extended downtime.

For many, this came too late. Utter silence gave way to verbal assurances peppered with images of further disruption ahead. It was a public relations disaster of epic proportions, and many will not forgive the company for handling the outage so badly.

Key is the network: Disruption was inevitable

BlackBerrys run on Research in Motion's infrastructure of email delivery, data provision and browsing capability, which piggy-back on the cellular networks. Adding another link to the chain of the ecosystem allows for secure email and messaging, but also increases the chance of something in that chain breaking down.

In this day and age, though smartphones are still 'phones' and can send text messages almost by standard, data is the killer feature to any smartphone on the market.

But as BlackBerrys rely on data to function, its smartphones rely on that infrastructure to run. With backup systems and backup systems from those backup systems, when an infrastructure is edited or changed, it has to be fail-safed. In this case, it wasn't.

In any case, infrastructures are delicate 'organism-like' technologies, and require maintenance and upkeep. But because of the BlackBerry data centralisation, something was inevitably going to go wrong, whether it was this week, next month or in a year from now.

Competition wise: 'If it ain't broke...'

The outage hit at the worst possible time for the Ontario-based company. Embarrassingly, the Canadian company faced questions on its own home turf when the issues spread from Europe, the Middle East and Africa, through to South and Latin America, ending up on the doorstep in the United States and Canada itself.

But others were quick to explain why Android and iPhones "won't go down like BlackBerry" did over those painful four-days where service was heavily disrupted.

Having said that, with Apple's new iOS 5-based iMessage service, there is reason to believe that the BlackBerry Messenger competing product could also face difficulties, along with iCloud email. But because Apple barely invests in critical email infrastructure of its own, unlike Research in Motion, it allows Apple and other phone manufacturers to blame the networks, rather than the phones.

As BlackBerrys are 'slaves' to the infrastructure, iPhones and Androids can still run critical services -- even without direct data connections to the 3G or 4G cellular network.

The nail in BlackBerry's coffin?

BlackBerry customers are surprisingly loyal, through personal preference by the choice of their employers. Though enterprise customers were not nearly as widely affected as end-user consumers were, it amounts to further pressure for businesses to consider whether the BlackBerry is a stable communications provider for their email and other services.

But a massive revolt of the service is unlikely to change how Research in Motion moves on. If anything, the worldwide struggle to access data services will be reflected by investors and shareholders, who will want a change in management, or to sell off the company altogether.

The outage itself will not be the nail in the BlackBerry coffin, but it's certainly cutting it close to the company's downfall.


Across the CBS Interactive network:

Topics: Mobile OS, Hardware, Mobility, BlackBerry

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • R.I.P. RIM

    There have been so many nails driven into that coffin lately that it's only a matter of time before we hit the "last nail." It started with their arrogance at the introduction of the iPhone. Their attitude was that they were untouchable. That's what Lotus, Netscape, Yahoo, AOL, and so many others thought about their products, too. RIM thought touchscreens would never catch on. A couple years later, they were scrambling to create their own. My wife told me virtually everyone in her company has requested that the company switch to the iPhone over the past few days. The reality is, there are a ton of better products out there now and RIM just can't compete because they started trying to evolve FAR too late. The laurels they've been sitting on for so long have become poisonous thorn-covered weeds.
    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

      Please define "better" than Blackberry.
      Faster? NO
      More efficient? NO
      Bigger Screens? Yes, sometimes
      Sync Memo and Tasks wireless via free BES? NO
      Push Email, lower data usage? NO
      LED indicating missed call or Email? NO
      Custom Sound Profiles easy to use? NO
      One Button Dialing? NO
      Fastest email? NO
      Vendor Lock-in? Yes (Apple)
      Encryption and security? NO
      Fast, efficient, productive? NO
      Better Browser? NO
      Cheaper Data usage? NO
      Better World Phones? NO
      Better Keyboard? NO
      More mangled auto correction? Yes
      Easier for simple minds? Yes
      Arrogant, A*hole Company founders? Equal
      Form over Function? Yes, a little bit
      Better for wasting time? Yes
      Better Camera? Yes, but a big deal only to some.
      Models available without Camera? No
      Support Free Tethering with BB Playbook? NO
      Nickle and Dime you for everything? Yes
      Better Marketing? Yes
      Has NFC? No
      Supports Removable SD Card? NO
      Has Removable Battery? Not Apple
      Need more examples?
      John Hanks
      • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

        @John Hanks The fact that the bulk of your answers are incorrect show that you have no idea what is going on in the smart phone world beyond RIM. You don't even know about something as basic as push email, which has been commonly available on non-Blackberry devices since Exchange 2003. I administer an Enterprise BES server and support other phones as well, so I have a good case for comparison. Shall we go down the list?
        Faster? Yes, quite a few models are
        More efficient? NO - I'd say no one has as good of battery life as Blackberry, true
        Bigger Screens? Yes, usually
        Sync Memo and Tasks wireless via free BES? NO, not via BES, since that is Blackberry specific, so you use other products instead. (Honestly, though, I think the BES does it quite well)
        Push Email, lower data usage? YES
        LED indicating missed call or Email? YES (model specific, and on Android you can use the LED for all sorts of other indicators if you want)
        Custom Sound Profiles easy to use? YES
        One Button Dialing? YES (I use this every day, extremely common feature)
        Fastest email? YES - my blackberry phones sync at the same rate as my other push phones
        Vendor Lock-in? No more so than with Blackberry. Android phones are much easier to transfer data to another phone, than moving from Blackberry to another platform.
        Encryption and security? YES
        Fast, efficient, productive? YES
        Better Browser? By far
        Cheaper Data usage? YES - no extra Blackberry network fee
        Better World Phones? Not the iPhone, but plenty of Androids are comparable
        Better Keyboard? NO - I think I'll grant you this one. There are some Android phones with great physical keyboards, but I still prefer the Blackberry physical keyboard. However, their onscreen keyboard is terible compared to just about anyone else's.
        More mangled auto correction? NO - Never seen a problem with this, especially since there are any number of third party keyboards. No one, and I'm including Blackberry, has anything nearly as good as Swiftkey's predictive keyboard.
        Easier for simple minds? No - I give Blackberries to the users who need a "simple" phone with email.
        Arrogant, A*hole Company founders? Equal - Yeah, that depends on which company you are talking about, since you seem to be ignoring Samsung, HTC, Motorola, etc. I'm sure they're all pretty equal here, though.
        Form over Function? Yes, a little bit
        Better for wasting time? Yes
        Better Camera? Yes, but a big deal only to some.
        Models available without Camera? Yes - Also, you can disable the ability to access or use the camera on Android and iOS.
        Support Free Tethering with BB Playbook? NO - Of course not - that's a Blackberry only product that is pretty worthless to someone without a Blackberry, since they haven't enabled an independent email client.
        Nickle and Dime you for everything? Who are we talking about here?
        Better Marketing? Yes
        Has NFC? This came out on Android well before RIM had anything.
        Supports Removable SD Card? YES - not iPhones, but pretty much every Android device.
        Has Removable Battery? Not Apple - but pretty much every other smart phone manufacturer
        Need more examples? Yes
    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

      @BillDem Despite it's flaws, Blackberry is still the most secure smartphone platform. No other system can match it's End-to-End encryption and it's reliability. Nor do they even get close as being as secure as RIM's systems. To me it al comes down to this simple fact: as severe as this outage has been, RIM, despite having problems involving main backbone still has not lost a single email. That is enough proof I need to know they care and that their service is equal to none.
      That's why I think you're mistaken, it's not arrogance. It's confidence. Cheers.
  • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

    Another brutal article over reacting to the magnitude of what actually happened. Having 2/3 of the 70 million BB users on the most secure, encrypted communication system in the world being affected wasn't good for RIM, but keep it in perspective. It's technology and it's back up and running as it was. Lessons will be learned. Thousands of people worked on this problem and you can be guaranteed, steps will be taken to prevent it from happening again. If they don't, they will go over the precipice. On the other hand, with $2-3 Billion in the bank, I'm pretty sure they have management running on all cylinders now. The old adage of learning from your mistakes should be taking place. Being a Canadian company and being Canadian myself, we can poke fun at ourselves. On the other hand, if you piss us off, we take things in our own hands. Hope you're pissed off enough RIM. Now get back to work and get that update out for the Playbook, come out with a 10 inch model, get your new BBX phones out and work on your ecosystem.
    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

      AOL use to think the same way.
  • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

    The real problem is that tech sites like your own continue to cry out for Rims downfall solely to drive traffic to your sites. My contacts at Rim told me that they worked feverishly to solve the issue. The "lets pile on Rim" mentality is reaching a feeding frenzy and its frustrating. As far as Apple goes- it got hacked recently and it barely got a paragraph of coverage in the media. Rim remains unhackable. This Imessage feature is not as robust as BBM or push email but we HEAR NOTHING of imessage shortfalls just that its going to put another nail in Rims coffin.
    Also even if I wanted to use an iphone as well as my blackberry- I CAN"T because iphones use micro sim cards. If BB introduced micro sim cards first there would have been an UPROAR. Android devices are popular as ENTERTAINMENT devices only. (they are fun to use) There is nothing that my BB won't do to suit my communication or entertainment needs in a smartphone. As a world business traveller BBM is irreplacebale to me as it doesn't matter what sim card I use.
    I don't have to pull email. I receive it INSTANTLY. I can surf the web the same as any other smartphone. The time has come to give UNBIASED fair reporting across the board for all smartphone devices. That means when Rim does something good or bad it gets fair coverage. It also means taking a shot at apple (gasp) when it needs it.
    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?


    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

      @dennycranium I'm not sure you have an accurate perspective on other platforms. My business does not have a need for 100% secure, unhackable messaging, and this is putting the final nail in the coffin. What our users do need is access to their email, all the time. We are moving off BBs to Android and iOS devices, which also do not have to pull messages, but instead get them instantly via Activesync push, similar to a blackberry.

      Why would you say an Android device is only an entertainment device? I'm not sure they are that good for entertainment only, but they work great as business devices. They (and iOS devices) are easily managed, with enforceable passwords and remote wipe capabilities out of the box, and if you want to spend some money (like with a BES), you can have fine control over what data is wiped, what apps can be installed, push apps, etc. Lots of that is available via free apps for an end user, though you have to pay for enterprise management.

      And plenty of Android devices have swappable sim cards, though that is a downfall of iPhones.
      • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

        @chadness BES Express is absolutely free and includes everything most enterprises would need. Only if you need even more security and managebility then you pay for the complete BES.
      • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

        @SinfoCOMAR True, but the free version is unsupported. We pay for Enterprise BES for the extra features and the support. The price really isn't too bad, and has been worth the upgrade (for us) from just buying support for the free BES. I'm not totally against Blackberry, we just need to expand beyond because of user demand and extra features available on non-Blackerry devices.
  • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

    Gee, as I was following the reports of RIM's network problems I thought they explained things pretty well. They indicated where the problem was and they repeatedly said it would take some time to clear the backlog, no information was lost. They did have backup and fail safe systems but unfortunately they didn't work as planned. It is usually really difficult to fully test your back up systems to see if they will work as intended until something actually happens.
    It will be interesting to see if ZDNET issues similar poorly based articles when non-RIM Cloud based services crap out from time to time and I am sure they will.
    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?


      It's just so blatantly obvious, isn't it?
    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?


      In fact I thought that RIM has actually been more forthcoming with this outage than in the past. All that's mentioned on here are the tweets and Lazaridis' apology video (a landmark in itself). In reality there were updates on the website, 2 conference press calls and a letter from one of their ladies in top management as well.
    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

      @1950graybeard But they have covered similar outages... from the top of my head:
      Google Mail outage: 150.000 email accounts lost
      MS Messengar/Live/Hotmail/360: can't remember how many they've had so far and how much data they've lost.
      Apple/MobileMe: Sept 30th 2011. Two day outage. %75 users unable to access services.
  • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

    Anything for a headline, huh, ZDNET? Please lose the trash pile news and get on with the real tech news. Geez...
    • RE: BlackBerry's outage post-mortem: Where did it all go wrong?


      Trep Ford
  • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

    Nothing to see here. Move along people.

    Nothing is infallible. RIM is not going to die because of a technical glitch.

    In my opinion, RIM's biggest mistake was to market Blackberries to consumers. However, at one time, if you weren't on AT&T, there wasn't much of a choice if you wanted a smart phone. We just got the iPhone in Canada about 18 months ago. Android had just come out at that time. The iPhone has only recently spread to almost all of the US carriers. So, you had a choice of Blackerry or Symbian until that point.

    Blackberry is and always has been an enterprise platform. Of course consumers are going to leave in droves now that there are dozens of Android phones out there and the iPhone is more widely available. These phones are much better suited to the needs of consumers.

    The Playbook is the only tablet that the US government will allow on their networks due to its high security. Again, this is not a consumer tablet.

    The large corporations that want fine control over what is in their network will always choose Blackberry. The workers will always ask for Android and iPhone because of the non-work related apps that are available. Very cheap companies will allow bring your own devices, which is a mistake and a huge security risk as far as I am concerned.

    Blackberry is going to lose sales, but they are just dropping back to their core business and leaving the consumer business to Android and Apple.

    Give it up, ZD Net. You're becoming the National Enquirer of the tech world.
  • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

    The power goes out, the cable craps out, the satellite signal dies when it storms outside, the internet takes a dump and the list of things that quit regularly is long. So why you so called tech writers obsess over something like this. As far as I can recall this is about RIM's only outage in 10 years of service. Do you all go wacko every time something quits or goes offline? I think its poor reporting and lame opinions. I have a playbook. Its works great, and I can barely pry it out of my daughters hands to use it myself. In the real world you defectives would be selling pencils on the corner. GET OVER IT!!
    Home Grown IT
  • RE: BlackBerry's outage post-mortem: Where did it all go wrong?

    Blackberry and its Enterprise Server is lost its luster and became superfluous after Native support became available for Microsoft Exchange and Lotus Notes on the iPhone. The BES is just point of failure, I am glad its not needed anymore. I do can EVERYTHING on my iPhone without it.