Amazon sheds light on Dynamo

Werner Vogels, CTO of Amazon, is outlining details of "Dynamo," an internal operating system that runs the e-commerce giant's storage system.

In a post, Vogels said Dynamo will be detailed at the Operating Systems conference in a paper (PDF). Here's his description:

Dynamo is internal technology developed at Amazon to address the need for an incrementally scalable, highly-available key-value storage system. The technology is designed to give its users the ability to trade-off cost, consistency, durability and performance, while maintaining high-availability.

And Dynamo won't be for sale. Vogels wrote:

Dynamo is an internal-only tool. It is not a public web-service and will not be offered through that channel. If there are any questions then they should not go to the AWS forums, the AWS evangelists or Amazon PR, as they will not be able to help you.

The paper is a bit out of reach for the average bear (notably me), but software engineers will be able to follow along. Techmeme has more.

The key takeaways:

Dynamo's big goal is to deliver reliability even as components in a large system fail continuously. Dynamo will sacrifice consistency to provide an always on experience. The paper then details the nuts and bolts as well as the key trade-offs.

Does storage get short shrift in services oriented architecture? Amazon's paper details how storage technology is critical to managing SOA--a point I haven't heard in many places.

From the paper:

One of the lessons our organization has learned from operating Amazon’s platform is that the reliability and scalability of a system is dependent on how its application state is managed. Amazon uses a highly decentralized, loosely coupled, service oriented architecture consisting of hundreds of services. In this environment there is a particular need for storage technologies that are always available. For example, customers should be able to view and add items to their shopping cart even if disks are failing, network routes are flapping, or data centers are being destroyed by tornados. Therefore, the service responsible for managing shopping carts requires that it can always write to and read from its data store, and that its data needs to be available across multiple data centers.

Each Web service has a service level agreement. In technology service level agreements (SLAs) are critical to keep things running. Web services are no different, according to Amazon. Each component has a defined SLA.

In Amazon’s decentralized service oriented infrastructure, SLAs play an important role. For example a page request to one of the e-commerce sites typically requires the rendering engine to construct its response by sending requests to over 150 services. These services often have multiple dependencies, which frequently are other services, and as such it is not uncommon for the call graph of an application to have more than one level. To ensure that the page rendering engine can maintain a clear bound on page delivery each service within the call chain must obey its performance contract.

Here's how it all fits together.

There is a lot more in the paper that's worth checking out.