BBC websites recover from massive outage

The BBC has apologised for a massive outage that took all of its websites and iPlayer offline for an hour on Tuesday evening.

The BBC's online output was taken offline for around an hour on Tuesday evening. Photo credit: Phil Parker on Flickr

The problem, caused by a router failure, happened at around 11pm and affected the whole of BBC Online, according to the broadcaster. The disruption meant that high-profile websites such as the BBC News and Sport pages were completely inaccessible, as were BBC iPlayer on-demand radio and TV streaming services on a range of devices.

"It's not often we get a message from the BBC's technical support teams saying, 'Total outage of all BBC websites'. But for getting on for an hour this evening, until just before midnight, that's what happened," Steve Herrmann, editor of the BBC News website, wrote on the BBC News blog in the early hours of Wednesday morning.

The blackout began at around 22:40pm and lasted for around an hour, with most services up and running again by midnight. It took more than five hours to get the systems up to full resilience again, at 4am on Wednesday, Herrmann said in a later blog post.

The root of the problem was the "unlikely" failure of its routing systems, according to Richard Cooper, controller of digital distribution for BBC Future Media. The failure affected two functions, he said.

It's not often we get a message from the BBC's technical support teams saying, 'Total outage of all BBC websites'.

– Steve Herrmann, BBC News website editor

"The first is the aggregation of network traffic from the BBC's hosting centres to the internet. The second is the announcement of 'routes' onto the internet that allows BBC Online to be 'found'," Cooper said on the BBC Internet Blog. "With both of these having failed, we really were down."

While the broadcaster has backup systems, these did not kick in as expected. "In an unusual turn of events, these [backups] also failed meaning that the whole of BBC Online became unavailable," Herrmann said. "A number of internal services were also affected."

Several users posted on Twitter complaining that they were unable to access BBC web pages; the issue was tracked using the hashtag #bbcblackout.

The BBC apologised for the outage and reassured users that it is investigating what needs to be done to ensure the situation does not happen again.

Siemens row

However, a row has broken out between the broadcaster and its IT contractor Siemens in the aftermath of the outage, according to a report in The Guardian. In a story on the issue, BBC News posted extracts from an internal email sent by Siemens to corporation staff — something that is said to have infuriated the IT service provider.

According to the report, the email said: "Cause of issue: Faulty Switch ... Services Impacted: Everything."

It adds that in order to rectify the fault, "Siemens network engineers remotely powered down equipment at a second internet connection at Telehouse Docklands. This got things back up and running again".

"They then isolated the core router in Telehouse Docklands, and restored power to it," the story continued. "Once power was restored and the router was running in a satisfactory way, they reconnected to the internet and BBC networks in a controlled manner."

The BBC News story now has no references to the Siemens email.

In January, BBC Online announced it will be axing one-fifth of its staff — 360 jobs from its 1,700-strong workforce — by 2013 as a result of budget cuts. It also plans to close 180 of its 400 websites before the end of 2011.