Google's warehouse-size power problem

Google's warehouse-size power problem

Summary: Power to the internet!Beginning 5 years ago, Google took the lead in making a power consumption an issue for IT vendors.


Power to the internet! Beginning 5 years ago, Google took the lead in making a power consumption an issue for IT vendors. No one cared that much before that because no one else was building 100,000+ server data centers using free software and cheap PC hardware. Google wasn't the only factor, but their use of free software, cheap hardware and massive scale meant that energy consumption became one of the few places they could cut costs.

Now, 5 years later, Intel's power hungry NetBurst architecture is dead and power-efficient kit - multi-core CPUs, LED backlights and even disk drives - are coming on strong across the industry. Google's timing was excellent, true, but they also used their clout to get vendors moving. Bravo!

But Google isn't done. Using the data-intensive methodologies based on their massive scale they've now published their analysis of computer power consumption. The paper offers an interesting peek into the problems of running the world's largest internet data centers as well as some pointers to what we may be seeing in five years.

Googlers take a hard look at power How accurate are the power requirements we see in specs? Is CPU power scaling really helpful? What determines computer power consumption? Google now gives us one large-scale data point, looking at groups of up to 15,000 servers. That's a tiny fraction of the server population of a recent Google data center, but large enough to be useful.

Power Provisioning for a Warehouse-sized Computer (an 11 page PDF) by Xiaobo Fan, Wolf-Dietrich Weber and Luiz André Barroso is worth a read if you're willing to interpret cumulative distribution function graphs. The paper begins by noting that a datacenter costs $10-$20 per deployed watt of peak computer power, excluding cooling and other loads. This is more than 10 years of power costs, so getting the numbers right pays big dividends.

Google doesn't buy power the way you and I do Google's newest data center in Oregon has taken the place of decommissioned aluminum smelters as a major power consumer. The cheap Columbia River hydro-power costs roughly 25 cents per watt/year, or $2.50 per watt for 10 years. Not only that, but Google - like other big users - gets charged based on their peak watt/hour power consumption. If they do one hour at 100 MW and the rest of the month at 25 MW, they get charged for consuming a 100 MW for a month.

So keeping an even strain on a data center's capacity is important. Ideally they want to build a data center that uses, say, a steady 50 megawatts so they can build a so they can build an efficient data centers and they don't get billed for power spikes. That isn't the average homeowner's problem. Google really needs to understand power consumption.

So what did they figure out? Well, a whole heck of a lot. Here's some key findings.

  • The gap between aggregate and spec power can be as great as 40% for a datacenter, though Google's applications are better behaved. That is a lot of wasted distribution and cooling capacity.
  • User-operated systems can be more efficient. This is true of home systems and of Google, and not true in the average enterprise data center. because the system can be run at close to its rated performance.
  • Power management is more effective at the datacenter level than at the rack level. There is no comment on whether that applies to an individual PC.

How do you measure power on 15,000 systems? Google uses the cheapest possible - but no cheaper - PC hardware for their servers, which is one thing they have in common with many SOHO users. Even under heavy load a server would use less than 60% of its nameplate power.

Google couldn't measure 15,000 servers directly, so they looked for a reliable and easy-to-monitor indicator. After testing they determined that CPU utilization predicted power usage to within 1% of measured power use. If the CPU is busy, everything else tends to be busy too: memory, disks, fans and - in the home case - video cards. Also, Google measures large numbers of machines, so individual variations get smoothed out.

So how can this help us save power? Google evaluated two power saving techniques. CPU voltage/frequency scaling has been implemented on some AMD and Intel chips. The idea behind scaling is that when the processor is less busy, it can reduce its input voltage and clock frequency to save power.

Modeling CPU scaling under data center loads, Google found that datacenters could see savings of 15-25%, depending on how aggressively it was used. They also found that I/O bound servers benefitted less than compute-intensive workloads.

Another option is to improve non-peak power efficiency. Most "efficiency per watt" metrics are based on peak loads, but Google found that most systems spent little time at "peak". Google found that idle systems power never dropped below 50% of peak load, while ideally an idle system's consumption would drop to zero.

If idle power were only 10% of peak power, an enterprise data center could save 50% on its power. Even Google's well-behaved apps would see savings in the 30%+ range. We can expect to see Google push the non-peak power efficiency issue pretty hard, and since they spend some $500 million a year on servers, I expect vendors will listen.

For us home and office users the savings would probably be even greater, since typical office tasks rarely stress a system, let alone take it to a peak load. Our systems spend all their time in "non-peak" performance - gamers excepted.

The Storage Bits take As computing systems weave their way deeper into the fabric of industrial societies - and I believe we are still in the early stages of the process - the energy demand becomes a greater issue. Whatever you believe about global warming, more efficient computers are a Good Thing, just as more efficient automobiles and trucks are. I applaud Google for the leadership role they've played in getting energy consumption into vendor's roadmaps and I look forward to much more efficient PCs and servers 5 years from now.

Comments welcome, of course. Has anyone played with those wattmeter gadgets to see what their system really does?

Topics: Servers, Data Centers, Google, Processors

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Google turned Iowa in the Horrors for electricity.

    And yes, horrors is misspelled to get past the censor bot.

    Google may not be able to curb their power appetite but they managed to turn Iowa in a state of *horrors* by making a deal where they pay no taxes on electricity. Yes sir, shove it up the tax payers backside once again to save a multi-billion dollar a year corporation some bucks.

    Google, "do not evil" my DONKEY...
    • All corporations try to do that.

      Many cities/states give companies tax breaks if they move or open operations to that region. What else is new? That is just part of doing business. Anyone who has/operates a business (or has a job) understands this. The tax payers always pick up the slack, that is what they are there for!
      • mabye in your world.

        But I manage to run businesses without being on the dole. In fact I actaully pay taxes. Imagine that...
        • Just says that the state didn't value your janitorial services company.

          High tech companies tend to get tax breaks if they set up shop in some states (cities or countries), janitorial companies like yours are a dime a dozen (if that). So what?
        • Corporations work a bit like this...

          <Big Corp> Hey, we have this large business, that could employ a few hundred employees, the problem is that we don't want to spend so much money in setup and we don't want to pay all those taxes, and we want a few other perks, since we will bring lots of income into your area by providing jobs and selling goods at a global level.

          <Small Town Leadership> Provide us with jobs which we can tax? Where do I sign up?

          This is one of those win-win situations. Areas where corporations don't worry about collecting taxes so much as they get population growth and employment growth. They can make up the taxes by getting a big fish in the pond, they can clean up off of all the small fish that hover around it for shelter.

          I know this isn't the best way of doing business, but some places need bargaining chips to get businesses to move in.
        • In many worlds . . . . .

          Those [b]tax breaks[/b] are incentives to bring high paying jobs into areas that could use them. As the cost to do business rises, business will look for someplace [i]cheaper[/i]. After all, they are not NON PROFITS!!!!
    • Electrical taxes bad all over!

      Ax, there are power/taxation issues all over the country. No need to single Google out. Many power companies and coops have very heavy lobby efforts, and many times more than not large businesses usually get power tax free. So don't stop at Google on this one, there are many many fish in this tax-free ocean.
  • Sun is going Green.

    I am not a Sun employee nor have I any financial interest in Sun.
    I am writing this to straighten the record.

    There is nothing in your article mentioned about Sun's contribution and partnership on the Google's power savings program. Sun has been very active in this area - ready about their contribution at and you can read more from

  • Another Good Article on Power

    Tom's Hardware recently did a interesting review of power consumption. I followed the advice and reduced my CPU and MB heat considerably. I've not bothered to attach an amp meter but I think the reduction of heat indicates a reduction of power use -- it certainly means that I'm spending less on air conditioning. I see no performance change in my SOHO environment.
  • Power usage of a few home systems

    I got one of those Kill A Watt power measurement units. Just for fun:

    (Idle= no apps running, CPU usage near 0, Full= doing whatever I could to get CPU at 100%)

    Dell Dimension4500 P4 2GHz, 1MB, 2xHD, 32MB VideoRAM WinXP Pro
    Off - 1W
    Stby - 2W
    Idle - 60W
    Full - 90W

    Homebuilt PIII 850MHz, 512MB, 1xHD, 32MB VideoRAM WinXP Pro
    Off - 3W
    Stby - 40W
    Idle - 60W
    Full - 80W

    Compaq V5305 Laptop Sempron 3300 (2GHz) 2GB, 1XHD, shared video RAM Vista Home Premium
    Off - 1W
    Hibernate - 1W
    Sleep - 2W
    Idle - 25W
    Full - 43W

    Shuttle XPC 3.06GHz HT, 1GB, 1xHD, 128MB VideoRAM WinXP Pro
    Off - 4W
    Stby - 55W
    Idle - 72W
    Full - 135W