Telecity datacentre power outage hits customers

One of Telecity's Docklands-based datacentres took a hit on Wednesday when the mains power supply to the seventh floor of the building lost power, resulting in downed servers and equipment failure for some customers.

Telecity told customers "we are currently experiencing issues with the mains power supplying the 7th floor infrastructure at Harbour Exchange 7/6", according to connectivity specialist Entanet, which received the statement.

"Power has not been transferred to the UPS [Uninterruptible Power Supply] and generator systems due to a fault that is currently being investigated and this has resulted in a loss of power to 7th floor customer equipment at the site," Telecity added.

Entanet reported that power went off to its equipment at Harbour Exchange at around 11:30am. It was restored to their rack by 14:05pm, it confirmed.

In the intervening period, the floor's UPS exhausted its backup batteries and shut down, bringing down all Entanet services from the location and causing one of its flash cards to fail.

Besides Entanet, other customers suffered problems from the power outage. Spitfire, a provider of hosted telephony services, said in a recorded voicemail for customers that their calls could not be answered because it was seeing a high volume, as a consequence of "a power outage in one of our datacentres".

AQL, a provider of telecommunications services ranging from SMS to voice communications, was hit as well. "There were huge problems in a datacentre managed by a third party called Telecity," a customer service representative from the company told ZDNet UK. "We suffered a mains power failure, so some services went down, but we believe they are back now."

ZDNet UK understands the Telecity backup power for AQL failed as well, so its servers running off Telecity's power and UPS went down.

"We have resolved a power outage which affected a zone in one of our datacentres in Harbour Exchange earlier today," Telecity said in a statement on Wednesday afternoon. "It took our engineers just over an hour to fix the problem and restore power to those customers affected. We kept all affected customers informed throughout the process and have apologised for the disruption."

Update: 9:40am Thursday

AQL equipment was damaged by the power outage. "Several server devices have 'died' and have had to be removed and replaced," Adam Beaumont, AQL's managing director, told ZDNet UK. "A lot of carrier grade equipment is not that tolerant to a sudden loss of power."

"Although a lot of our wholesale services were unaffected, it caused outage to some colocated customers on that site, some of our SMS API users were tied to particular IP addresses. It's tied up our entire technical team all day."