Lightning strikes Amazon's European cloud

Amazon Web Services's European cloud suffered an outage over the weekend after lightning struck a transformer at one of its power suppliers.

The outage, which Amazon Web Services (AWS) acknowledged on Sunday evening, affected its Dublin-based Elastic Compute Cloud (EC2) and Relational Database Service (RDS) cloud services, among others. The damage to the electricity infrastructure may have affected Microsoft's Business Productivity Online Services (BPOS) cloud as well, Microsoft said in a separate statement.

"A lightning strike hit a transformer from a utility provider to one of our availability zones in Dublin, sparking an explosion and fire," AWS wrote in an update to its status page at 11pm on Sunday. "The transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronises the backup generator plant, disabling some of [the generators]."

"Power sources must be phase synchronised before they can be brought online to load. Bringing these generators online required manual synchronisation," it added.

Amazon acquired 240,000 square feet of Dublin datacentre space in February to help it expand its AWS cloud infrastructure. The Dublin datacentres provide services to customers in Europe who want to access low-latency cloud services.

Service impaired

The AWS disruption affected a single "availability zone" within the European EU-WEST-1 region, the company said. An availability zone is a set of hardware that supports cloud services and is independent from other zones.

A lightning strike hit a transformer from a utility provider to one of our availability zones in Dublin, sparking an explosion and fire.

– AWS

As a consequence of the transformer problem, services were "impaired" from 7pm on Sunday, according to AWS. At the time of writing, the Amazon unit said it expected problems to continue for at least the next 24 hours.

The power shutdown took out a swathe of AWS's servers, which led to the outage in cloud services such as EC2, RDS and Elastic Block Storage (EBS). Because the cloud services are made up of complex pieces of software, the company needed to allocate further hardware to resurrect services when power was restored. This drew the recovery period out further.

"Restoring [EBS] volumes requires that we make an extra copy of all data, which has consumed most spare [hardware] capacity and slowed our recovery process," AWS wrote at 7am on Monday.

BPOS disruption

Microsoft also reported disruption in its Dublin-based BPOS cloud service on Sunday. The problems arose from "a widespread power outage in Dublin," the company said in a statement. It did not respond to request for comment as to whether the lightning-hit transformer was at the root of it.

The disruption in BPOS provision ran from around 7pm on Sunday through to 1:30am on Monday, according to Microsoft. "Throughout the incident, we updated our customers regularly on the issue via our normal communication channels," it said.