Amazon puts Hadoop data-crunching in the cloud

Summary:Customers can use the new Amazon MapReduce to pay only for the capacity they use as they perform tasks such as indexing the web, mining data or conducting financial analysis

Amazon on Thursday announced a new cloud-computing service that uses Hadoop, an open-source software framework, to crunch large amounts of data.

The service, called Amazon Elastic MapReduce, is designed for businesses, researchers and analysts trying to conduct data-intensive number crunching. Hadoop, an Apache-run distributed-computing technology used by companies such as Yahoo, is being promoted for the enterprise datacentre by startups such as Cloudera.

Amazon's Hadoop framework runs on the company's Elastic Compute Cloud (EC2) and Simple Storage Service (S3). Customers that use MapReduce will be able to pay only for the capacity they use, as they do things such as index the web, data mine or conduct financial analysis, simulation and bioinformatics research.

Amazon Elastic MapReduce works by creating data-processing jobs that are carried out by Hadoop software on EC2, the company said in its announcement. The service automatically launches and configures EC2 instances according to the customer's specifications. Next, it uses Hadoop to load large amounts of data from S3, and that data is then divided up for parallel processing using EC2. Once that is done, the data is recombined and the results are put back into S3.

Topics: Tech Industry


Larry Dignan is Editor in Chief of ZDNet and SmartPlanet as well as Editorial Director of ZDNet's sister site TechRepublic. He was most recently Executive Editor of News and Blogs at ZDNet. Prior to that he was executive news editor at eWeek and news editor at Baseline. He also served as the East Coast news editor and finance editor at CN... Full Bio

zdnet_core.socialButton.googleLabel Contact Disclosure

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.