Amazon EC2 Cloud Data Center Loses Power

Users of the Amazon EC2 cloud with workloads in Amazon's Northern Virginia data center experienced problems early Wednesday morning, with some operations in a part of the data center interrupted during a five-hour period.

Amazon didn't name the location of the facility, but Amazon Web Services is known to operate a data center near McLean, Va. Amazon posted a set of one-line status reports on the incident on its Service Health Dashboard as it sought to restore customer workloads after an apparent power outage.

The status reports first stated that Amazon was "investigating connectivity issues" in its East-1 region at 4:08 a.m. Eastern. "We are experiencing power issues for a subset of instances" it reported at 4:26 a.m. Then at 4:51 a.m. it reported that "the underlying power issue has been addressed. Instances have begun to recover."

The reports continued. At 5:11 a.m. Eastern it reported that "most affected workloads have regained power and operating normally. We are working to recover a small number of instances that have not fully recovered."

Again at 6:33 a.m. and 8:31 a.m. Eastern, it reported it was continuing to recover a small number of workloads. All jobs were reported recovered at 9:41 a.m.

Amazon public relations representatives did not immediately respond to a request for information on the cause of the outage, referring only to the dashboard listings.

Jim Melvin, president of Apparent Networks said his firm's Cloud Performance Center monitoring system caught sight of an outage at the Amazon data center at 3:34 a.m. Eastern. He said a single availability zone (Amazon data centers are broken up in multiple availability zones) was affected for 44 minutes until 4:19 a.m.

Apparent Networks issued an advisory to its customers on the outage, stating that "during that time, access to systems in the northern Virginia Data Center was unavailable. Businesses utilizing Amazon Elastic Compute Cloud services from this data center were affected," the advisory stated. Amazon spokesmen couldn't be reached for comment.

The Cloud Performance Center monitors network performance at junction points leading to cloud data centers hosting EC2, Google, GoGrid, and Salesforce.com cloud operations. Melvin said the monitoring service can't see what's going on inside the cloud site but can see where its periodic pings were stopped at a router feeding traffic into the cloud site.

For a month, Apparent Networks has operated the Cloud Performance Center as a way to show the capabilities of its PathView Cloud software, which tests the performance of the network immediately outside a cloud service provider. The free service gives cloud users an independent means of determining whether a performance problem was due to network latencies outside the data center or performance problems inside the data center itself.

Amazon offers a CloudWatch service for a fee to monitor the activity of specific workloads in EC2. Open source systems manager Hyperic, now part of VMware, also monitors workload performance in the cloud.