How Amazon exposed its guts: The History of AWS's EC2

How Amazon exposed its guts: The History of AWS's EC2

Summary: One of Amazon Web Service's key components, EC2, was developed by a small team in a satellite development office in South Africa. We trace the history of the EC2 cloud, and talk to the person who came up with the idea

SHARE:
TOPICS: Cloud
0

In 2006 Amazon's chief Jeff Bezos took the stage at MIT's Emerging Technologies conference to talk about two cloud computing products the company had announced and the ambitions the company had for them.

Three years earlier, one of these products — EC2 — was only an idea.

"There's a hidden Amazon, just under the epidermis, the guts of Amazon, this is all the stuff we have to do on the back-end to make this work," Bezos said [MP3], before noting that the combined technologies of S3, EC2 and Mechanical Turk represented the culmination of "11 years of web-scale computing" and, cumulatively, "multiple billions of dollars in technology and content investment".

EC2 was developed first and foremost for Amazon's internal infrastructure. It started out as an idea in the head of Chris Pinkham, who worked as an engineer in charge of Amazon's global infrastructure in the early 2000s.

"It struck us in the infrastructure engineering organisation that we really needed to decentralise the infrastructure by providing services to development teams," Pinkham says. "That was a big motivating factor."

An 'infrastructure service for the world'

Pinkham thought about this issue and, in 2003, started trying to build an "infrastructure service for the world". His hope was that he could develop a service that would not only deal with Amazon's infrastructure, but also help developers.

"From experience we knew that the cost of maintaining a reliable, scalable infrastructure in a traditional multi-datacentre model could be as high as 70 percent, both in time and effort, and requires significant investment of intellectual capital to sustain over a longer period of time," Werner Vogels, Amazon's CTO, wrote on Quora. "The initial thinking was to deliver services that could reduce that cost to 30 percent or less (we now know it can be much less)."

Pinkham and Benjamin Black, another Amazon engineer, wrote a short paper outlining the ideas for Amazon's chief Jeff Bezos, who liked it and followed up by asking for more details on a virtual cloud-provisionable server.

But Pinkham had a baby on the way and, after talking with other people at Amazon, left to set up a satellite development office in South Africa — Amazon's first in the region — where he and some other engineers, including Christopher Brown and Wiljem Van Biljon, worked on designing the EC2 service.

Aggressive approach

By 2005 Amazon was offering the technology to some customers under non-disclosure agreements. The service would officially launch in the summer of 2006.

"Amazon is a very strong believer in moving very aggressively," Pinkham says.

The company's aggression manifested itself in two ways: by scaling up the service very quickly, and opting for relatively low margins as a way to "lock the door for competitors", Pinkham said.

When EC2 and S3 launched, analysts peppered Amazon's chief financial officer, Tom Szkutak, with questions about the service: What would the operating profits be? How would growth in AWS feed the growth of Amazon? What would the investment plan be?

"The reason we're doing the web services that we are doing is because they are things that we've gotten good at over the last 11 years in terms of building out this web-scale application called Amazon.com," Szkutak said in 2006, according to a transcript on SeekingAlpha. "So as we go about exposing the guts of Amazon, there are other developers out there who require those same sorts of web-scale services... what we're doing here is exposing those and, over time, building that into a meaningful business."

Pinkham eventually left Amazon to co-found Nimbula, a company that tries to take the technology popularised by Amazon in EC2 and bring it to private, on-premise clouds. "I'm astonished, coming out of the Amazon world, [at] the effort it takes to stand up small amounts of infrastructure," he says.

Since its official launch in 2006, EC2's value has grown and it has become the cornerstone of Amazon's ecosystem of cloud services. A customer interested in any of Amazon's advanced products — DynamoDB or Elastic Beanstalk, for example — has a huge incentive to provision one of the EC2 virtual compute instances on offer. Not bad for a technology designed to make an online bookstore better.

Also in this series: The rise of AWS in pictures and Amazon Web Services: Rise of the utility cloud


Get the latest technology news and analysis, blogs and reviews delivered directly to your inbox with ZDNet UK's newsletters.

Topic: Cloud

Jack Clark

About Jack Clark

Currently a reporter for ZDNet UK, I previously worked as a technology researcher and reporter for a London-based news agency.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

0 comments
Log in or register to start the discussion