AWS disrupted by US east coast failure
Amazon Web Services' cloud has taken a hit from problems in its US East Coast region, downing multiple sites that depend upon the service.
The Elastic Compute Cloud (EC2) service began reporting problems in its North Virginia region at 1:41am PDT (8.41am GMT), followed by the Relational Database Service (RDS) at 8.48am. At around 9:18am Amazon confirmed the EC2 and RDS downtime on its AWS status microsite. Both EC2 and RDS are key components of Amazon Web Services' (AWS) global infrastructure-as-a-service-cloud, providing compute and storage (EC2) and database (RDS) services.
"We can confirm connectivity errors impacting EC2 instances and increased latencies impacting [Elastic Block Storage] volumes in multiple availability zones in the US-EAST-1 region," Amazon wrote.
Amazon Web Services' global infrastructure-as-a-service product is composed of five distinct regions — North California, North Virginia, Ireland, Singapore and just-opened Tokyo — which each contain multiple availability zones. The availability zones are geographically distinct locations within a region and are believed to be individual datacentres operated by Amazon.
Service interruptions in AWS's North Virginia region have downed multiple sites, including Reddit. Credit: ZDNet UK
Problems are continuing in the North Virgina region at the time of writing. As of around 12.02pm Amazon said "latency issues have recovered" for both the RDS and EC2 Elastic Block Storage (EBS) elements in some of their availability zones within the region, but problems were ongoing.
The CloudFormation, CloudWatch and Elastic Beanstalk services operating out of the region were also affected, Amazon said, but to a lesser extent.
The downtime has taken down a number of AWS-dependent sites that have key infrastructure in the North Virginia region. As of 2pm major social networking site Reddit was reporting downtime due to EC2 problems along with a host of other services, such as timeline service Dipity, social question and answer site Quora and media sharing service mobypicture.
The North Virginia region has been affected by service interruptions before. In March Reddit was downed by a similar outage. At the time, Reddit mulled migrating its core services away from EBS services located in the region, due to worries over reliability.
Update: AWS has updated the status page with further information. As of around 2pm the problems have led to difficulties in launching instances — the rentable hardware that is fundamental to EC2 — within the North Virginia region.
In addition to causing difficulties in launching instances, EBS-backed instances in the US-EAST-1 region have begun to fail, Amazon said at 3:40pm.
At around 4pm Amazon also said that it has made no headway with the RDS service problem. "Despite the continued effort from the team to resolve the issue we have not made any meaningful progress for the affected database instances since the last update," Amazon said.
Update: 5pm: At 4:54pm Amazon Web Services gave greater detail on the outage that has plagued the North Virginia zone since Thursday morning.
A "networking event" early in the day caused a large amount of re-mirroring of EBS volumes in the region, which created a shortage of capacity in one of the region's availability zones, Amazon said on its AWS status page. This affected the creation of new EBS storage volumes and hindered the pace at which AWS could alleviate the problems.
Alongside this, an internal tool used by AWS for administering EBS "has become inundated such that it's difficult to create new EBS volumes and EBS backed instances", Amazon said.
"We are working as quickly as possible to add capacity to that one availability zone to speed up the re-mirroring, and working to restore the control plane issue," Amazon wrote. "We're starting to see progress on these efforts, but are not there yet. We will continue to provide updates when we have them."