Why Twitter's failing

Right now, Twitter is a mess. Most of today, European users have experienced outages of varying length and even hard core Twitter fans are saying they've had enough. Some have talked of jumping ship to Pownce or Jaiku. Louis van Proosdij, in answer to TechCrunch France's Ouriel Ohayon offered:

@OurielOhayon This is where Twitter may fail fast, and Wordpress new Prologue become THE solution with an effective distributed twitter like

No-one seems to know what's going on yet if you read DataCenter Knowledge, you'd think everything in the garden is rosy. That's not the case. The Twitter blog told us they'd had a rough night and elicited an as expected amount of sympathy from the mostly US based commenters. All great PR but if you're in business, absolutely useless.

You can argue that Twitter hardly qualifies as something that's business critical and therefore any outages don't matter. Yet TechCrunch's Mike Butcher and ReadWriteWeb's Marshall Kirkpatrick have enough faith in Twitter to agree that it helps them in their daily work. Marshall goes so far as to say it's paying his rent. If he'd been in Europe today, it would have been costing Marshall money. Is it Joyent's responsibility? They're providing the infrastructure aren't they? Well sort of.

If you check out Adonomics, you'll see the number of calls to Facebook applications. 12% of those run through Joyent. That's 3.8 million a day. I know this because I spoke with another Irregular, Rod Boothby who works at Joyent. Now check the graphs for Facebook, Wordpress and Twitter over at Compete. There's an order of magnitude in difference between the traffic Twitter moves and that of Wordpress and Facebook. It's reasonable to conclude then that if Twitter is running on Joyent then it isn't down to Joyent's infrastructure. I say 'if' because anecdotal rumors have suggested otherwise.

Running highly scalable applications is hard. It is rocket science but the problems have been solved. Facebook and, to a lesser extent Wordpress have had their share of problems but nothing like Twitter's regular stream of outages. I asked Rod if he'd care to comment on the apparent disconnect between what is being said at Data Knowledge Center on the very day that so many Europeans were expressing annoyance. Understandably, he chose not to comment but we did reprise what it takes to run large scale operations.

This is something with which both of us are familiar, Rod as an ex-bank trader and me as someone who spent years understanding what companies like TIBCO do to keep NASDAQ ticking along while responding to demand spikes. Rod's an IBM MQ Series fan, I'm a TIBCO messaging fan. It matters not because the principles are the same. It's about moving relatively small packets of data as fast as you can through fat pipes in a controllable manner.

The fact that Joyent is successfully running large amounts of data for Facebook developers tells me it's a stretch to assume they have a significant role to play in Twitter's current woes. I may be deluded and I'm sure commenters with direct knowledge will tell me if I am wrong. But from this distance, Twitter's problems can only mean one thing.

Twitter is trying to reinvent the scaling wheel when companies like IBM and TIBCO worked this out years ago. And it's not going well. The fact Twitter went down when Steve jobs took to the stage at MacWorld implies they've not worked out how to shed load effectively and tune the system to cope for anticipated demand. There could well be other issues.

Much as I am a fan of Twitter the outages are becoming tiresome. Today was a one when I saw for the first time not a few, but many people bemoaning its performance. Whatever the real problems they need to solve them and quickly. Goodwill only takes you so far and it seems to me that tank of fuel is being rapidly drained. Patience is wearing thin.

UPDATE: Joyeur has announced that Twitter is officially OFF Joyent technology as of 10pm last night.