Google reveals secret server hardware

Google reveals secret server hardware

Summary: Google for the first time on Wednesday revealed the hardware at the core of its Internet operations at a conference about the increasingly prominent issue of data center efficiency, reports CNET's Stephen Shankland.Instead of buying hardware from companies such as Dell, Hewlett-Packard, IBM, or Sun Microsystems, Google designs and builds its own.

SHARE:

Google for the first time on Wednesday revealed the hardware at the core of its Internet operations at a conference about the increasingly prominent issue of data center efficiency, reports CNET's Stephen Shankland.

Instead of buying hardware from companies such as Dell, Hewlett-Packard, IBM, or Sun Microsystems, Google designs and builds its own. (The company has hundreds of thousands of servers.)

Ben Jai, who designed many of Google's servers, unveiled the server hardware. The first surprise: each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity.

Shankland writes:

Why is the battery approach significant? Money.

Typical data centers rely on large, centralized machines called uninterruptible power supplies (UPS)--essentially giant batteries that kick in when the main supply fails and before generators have time to kick in. Building the power supply into the server is cheaper and means costs are matched directly to the number of servers, Jai said.

"This is much cheaper than huge centralized UPS," he said. "Therefore no wasted capacity."

Efficiency is another financial factor. Large UPSs can reach 92 to 95 percent efficiency, meaning that a large amount of power is squandered. The server-mounted batteries do better, Jai said: "We were able to measure our actual usage to greater than 99.9 percent efficiency."

Since 2005, Google's data centers have been composed of standard shipping containers -- each with 1,160 servers and a power consumption that can reach 250 kilowatts, the company said.

Google has been using the design since 2005 and now is in its sixth or seventh generation of design.

"It was our Manhattan Project," Jai said of the design.

Energy efficiency, power distribution, cooling, and ensuring hot and cool air don't intermingle are all on the top of Google's list, the company said.

As for the actual unit, the server was 3.5 inches thick (2U) and had two processors, two hard drives, and eight memory slots mounted on a Gigabyte motherboard. Google uses x86 processors from both AMD and Intel. The battery design is used on its network equipment as well, Jai said in Shankland's article.

What's fascinating about all this is that Google operates servers on such an immense scale that every decision it must make in turn has a large affect (and potential cost/savings).

Take the power supply design, for example: Google's designs supply only 12-volt power, with the necessary conversions taking place on the motherboard. That adds $1 or $2 to the cost of the motherboard, Shankland writes, "but it's worth it not just because the power supply is cheaper, but because the power supply can be run closer to its peak capacity, which means it runs much more efficiently." Google even pays attention to the greater efficiency of transmitting power over copper wires at 12 volts compared to 5 volts, Shankland writes.

That kind of attention can translate to big savings in power or cost -- or both.

Topics: Google, Data Centers, Hardware, Servers, Storage

Andrew Nusca

About Andrew Nusca

Andrew Nusca is a former writer-editor for ZDNet and contributor to CNET. During his tenure, he was the editor of SmartPlanet, ZDNet's sister site about innovation.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

39 comments
Log in or register to join the discussion
  • How green is that?

    Every battery goes bad eventually, and having THOUSANDS of them leads me to ask - what happens to them when they die?
    Roger Ramjet
    • It's actually very green.

      When the battery goes bad, it can be recycled. There are businesses built around doing just that.
      Letophoro
  • RE: Google reveals secret server hardware

    I am interested in how they hook that 12 volt battery up to the mobo and whether the average user could perform that task.
    SysAdminII
    • 12V Battery

      Normal PC's use several voltages from the power supply, so a 12V battery could not be easily added to an existing mobo that was not designed to be supplied with a single voltage.
      tomjhen
      • Power Supply

        I think that the 12 volts is the 'main voltage' which is still being transformed to lower voltages such as 5v etc. Not directly feeding 12v to the mobo. Just like the 120v AC being transformed to lower volts DC.
        adelacuesta
        • Power Supply

          From the article - "Take the power supply design, for example: Google?s designs supply only 12-volt power, with the necessary conversions taking place on the motherboard. That adds $1 or $2 to the cost of the motherboard, Shankland writes, ?but it?s worth it not just because the power supply is cheaper, but because the power supply can be run closer to its peak capacity, which means it runs much more efficiently.? Google even pays attention to the greater efficiency of transmitting power over copper wires at 12 volts compared to 5 volts, Shankland writes."

          So, the mobo converts the single 12V supply to the other voltages needed. Very smart design...
          tomjhen
    • 12V Battery Backup requires special

      12V Battery Backup requires special Motherboard power supply. The Power Supply is also designed to charge the 12V battery as well as switch over to the battery when AC-power fails.

      This is a great design:
      it is extremely fault tolerant (100s of servers can go down and not affect the whole).
      Very power efficient, you only use the power you need for each server. Because the load on each power-supply is know, the efficiency can be tailored to the load. Much better than a large single supply that has to maintain efficiency over varying loads.

      The modular design makes adding or moving capacity so easy a union worker could do it.
      Cisco-SA
  • RE: Google reveals secret server hardware

    So then what would happen when the big UPS dies? Same problem.

    It *would* be interesting to compare life cycles of the two approaches. Daily operating costs are one thing, but replacement costs are different.
    RealGem
    • Agreed, but they've thought of that

      I'm pretty sure that they would incorporate the fully life cycle of the machine into the base cost calculations. If they're worried about a $2 motherboard mod then they've been detailed enough to cover that.
      mford66215
    • ? They're not using a "big" UPS. nt

      nt
      T1Oracle
  • Wonder about the acid and fire risk of batteries with each server.

    any insight on the above topic?
    cmm@...
    • Distributed batteries/Power also distributes

      Distributed batteries/Power also distributes the risk.

      Whether it is a huge and possibly oversized single UPS with a single battery or many smaller batteries, the risk is the same.

      Only on a widely distributed battery scenario, the heat dissipation and therefor the risk of fire is greatly reduced.
      Cisco-SA
    • Gelcel batteries are the answer`

      Gelcel batteries don't have the problem traditional lead acid batteries have. They can be oriented in any direction, including shipping, and if they are overcharged, they will vent or swell up.
      LarryPTL
  • What about virtualization and consolidation?

    Using thousands of 2 CPU boxes seems to run counter to the prevailing attitude of virtualization. Everyone else seems to be pushing multi-CPU/multi-core big boxes - that get carved up using virtualization. Perhaps one side knows what they're doing and the other side . . .
    Roger Ramjet
    • Depends on utilization

      If all the servers are running at high utilization, virtualizing makes no sense.

      If you have a bunch of servers running at low utilization, then they are candidates for virtualization.

      N.
      Norm_z
    • The machines they are using are rather old.

      Google uses the infrastructure and hardware that they have because it is met their needs. The machines are rather old (I know, as I had a contract there and discussed this with employees) and specialized. If a company wants to purchase their appliances, they will get a machine running VMWare ESX and the Google software on top of that, capable of running 4 to 8 hosts (depending on how Google implements their software on top of ESX). They are a big VMWare shop, though it is not used for the on-line service(s).

      Other major name companies that use VMWare ESX include: Verisign.
      B.O.F.H.
  • Low maintenance car battery

    Yeah. A low maintenance car battery plus a 12v & 5V regulator can easily accomplish that. you just install the batteries in arrays (parallel) so that when you remove the other battery for maintenance, the other batteries in array will take over. Typically, the battery will last for 2-3 yrs in its life time. then afterwards it can be recycled or disposed.
    m4rk.gm4il@...
  • RE: Google reveals secret server hardware

    It would seem that the overall design objective here is bare-bones least-cost solution without regard to the engineering or the history of such implementation topologies that involve increasing the number of failure points. They are depending on never having more than a small percentage of the batteries/servers fail at the same time for whatever the reason. And they are depending on using fail-over techniques to insulate single server failures.

    It has been my experience that increasing the number of hardware units to mitigate a failure will at some point reach the point of diminishing returns.

    Recall the laptop battery fires? The Google server batteries, probably from the cheapest vendor, are not immune from such outcomes.
    jdoe2000
    • I know, if only you thought of it first. :\ nt

      nt
      T1Oracle
  • RE: Google reveals secret server hardware

    Wasn't Wednesday 1st April???
    NotesGuru