The ToyBox

Ricardo Bilton & Gloria Sin

Google reveals secret server hardware

By | April 3, 2009, 5:14am PDT

Summary: Google for the first time on Wednesday revealed the hardware at the core of its Internet operations at a conference about the increasingly prominent issue of data center efficiency, reports CNET’s Stephen Shankland. Instead of buying hardware from companies such as Dell, Hewlett-Packard, IBM, or Sun Microsystems, Google designs and builds its own. (The company has [...]

Google for the first time on Wednesday revealed the hardware at the core of its Internet operations at a conference about the increasingly prominent issue of data center efficiency, reports CNET’s Stephen Shankland.

Instead of buying hardware from companies such as Dell, Hewlett-Packard, IBM, or Sun Microsystems, Google designs and builds its own. (The company has hundreds of thousands of servers.)

Ben Jai, who designed many of Google’s servers, unveiled the server hardware. The first surprise: each server has its own 12-volt battery to supply power if there’s a problem with the main source of electricity.

Shankland writes:

Why is the battery approach significant? Money.

Typical data centers rely on large, centralized machines called uninterruptible power supplies (UPS)–essentially giant batteries that kick in when the main supply fails and before generators have time to kick in. Building the power supply into the server is cheaper and means costs are matched directly to the number of servers, Jai said.

“This is much cheaper than huge centralized UPS,” he said. “Therefore no wasted capacity.”

Efficiency is another financial factor. Large UPSs can reach 92 to 95 percent efficiency, meaning that a large amount of power is squandered. The server-mounted batteries do better, Jai said: “We were able to measure our actual usage to greater than 99.9 percent efficiency.”

Since 2005, Google’s data centers have been composed of standard shipping containers — each with 1,160 servers and a power consumption that can reach 250 kilowatts, the company said.

Google has been using the design since 2005 and now is in its sixth or seventh generation of design.

“It was our Manhattan Project,” Jai said of the design.

Energy efficiency, power distribution, cooling, and ensuring hot and cool air don’t intermingle are all on the top of Google’s list, the company said.

As for the actual unit, the server was 3.5 inches thick (2U) and had two processors, two hard drives, and eight memory slots mounted on a Gigabyte motherboard. Google uses x86 processors from both AMD and Intel. The battery design is used on its network equipment as well, Jai said in Shankland’s article.

What’s fascinating about all this is that Google operates servers on such an immense scale that every decision it must make in turn has a large affect (and potential cost/savings).

Take the power supply design, for example: Google’s designs supply only 12-volt power, with the necessary conversions taking place on the motherboard. That adds $1 or $2 to the cost of the motherboard, Shankland writes, “but it’s worth it not just because the power supply is cheaper, but because the power supply can be run closer to its peak capacity, which means it runs much more efficiently.” Google even pays attention to the greater efficiency of transmitting power over copper wires at 12 volts compared to 5 volts, Shankland writes.

That kind of attention can translate to big savings in power or cost — or both.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Topics

Andrew J. Nusca is editor of ZDNet and SmartPlanet.

Disclosure

Andrew Nusca

Andrew J. Nusca does not hold any investments in the technology companies he covers.

Biography

Andrew Nusca

Editor

Andrew J. Nusca is an editor for ZDNet and SmartPlanet. As a journalist based in New York City, he has written for Popular Mechanics and Men's Vogue and his byline has appeared in New York magazine, The Huffington Post, New York Daily News, Editor & Publisher, New York Press and many others. He also writes The Editorialiste, a media criticism blog.

He is a New York University graduate and former news editor and columnist of the Washington Square News. He is a graduate of the Columbia University Graduate School of Journalism. He has been named "Howard Kurtz, Jr." by film critic John Lichman despite having no relation to him. He lives in his native Philadelphia with his wife, cat and Boston Terrier.

Follow him on Twitter.

Related Discussions on TechRepublic

Did you know you can take part in these discussions with your ZDNet membership?
39
Comments

Join the conversation!

Just In

RE: Google reveals secret server hardware
arunabhdas 12th Apr 2009
What OS do they run on the servers? - Arunabh Das
0 Votes
+ -
How green is that?
Roger Ramjet 3rd Apr 2009
Every battery goes bad eventually, and having THOUSANDS of them leads me to ask - what happens to them when they die?
0 Votes
+ -
It's actually very green.
Letophoro 3rd Apr 2009
When the battery goes bad, it can be recycled. There are businesses built around doing just that.
0 Votes
+ -
I am interested in how they hook that 12 volt battery up to the mobo and whether the average user could perform that task.
0 Votes
+ -
12V Battery
tomjhen 3rd Apr 2009
Normal PC's use several voltages from the power supply, so a 12V battery could not be easily added to an existing mobo that was not designed to be supplied with a single voltage.
0 Votes
+ -
Power Supply
adelacuesta 3rd Apr 2009
I think that the 12 volts is the 'main voltage' which is still being transformed to lower voltages such as 5v etc. Not directly feeding 12v to the mobo. Just like the 120v AC being transformed to lower volts DC.
0 Votes
+ -
Power Supply
tomjhen 3rd Apr 2009
From the article - "Take the power supply design, for example: Google?s designs supply only 12-volt power, with the necessary conversions taking place on the motherboard. That adds $1 or $2 to the cost of the motherboard, Shankland writes, ?but it?s worth it not just because the power supply is cheaper, but because the power supply can be run closer to its peak capacity, which means it runs much more efficiently.? Google even pays attention to the greater efficiency of transmitting power over copper wires at 12 volts compared to 5 volts, Shankland writes."

So, the mobo converts the single 12V supply to the other voltages needed. Very smart design...
0 Votes
+ -
12V Battery Backup requires special
Cisco-SA 3rd Apr 2009
12V Battery Backup requires special Motherboard power supply. The Power Supply is also designed to charge the 12V battery as well as switch over to the battery when AC-power fails.

This is a great design:
it is extremely fault tolerant (100s of servers can go down and not affect the whole).
Very power efficient, you only use the power you need for each server. Because the load on each power-supply is know, the efficiency can be tailored to the load. Much better than a large single supply that has to maintain efficiency over varying loads.

The modular design makes adding or moving capacity so easy a union worker could do it.
0 Votes
+ -
So then what would happen when the big UPS dies? Same problem.

It *would* be interesting to compare life cycles of the two approaches. Daily operating costs are one thing, but replacement costs are different.
0 Votes
+ -
Agreed, but they've thought of that
mford66215 3rd Apr 2009
I'm pretty sure that they would incorporate the fully life cycle of the machine into the base cost calculations. If they're worried about a $2 motherboard mod then they've been detailed enough to cover that.
0 Votes
+ -
? They're not using a "big" UPS. nt
T1Oracle 3rd Apr 2009
nt
any insight on the above topic?
0 Votes
+ -
Distributed batteries/Power also distributes the risk.

Whether it is a huge and possibly oversized single UPS with a single battery or many smaller batteries, the risk is the same.

Only on a widely distributed battery scenario, the heat dissipation and therefor the risk of fire is greatly reduced.
0 Votes
+ -
Gelcel batteries are the answer`
LarryPTL 6th Apr 2009
Gelcel batteries don't have the problem traditional lead acid batteries have. They can be oriented in any direction, including shipping, and if they are overcharged, they will vent or swell up.
0 Votes
+ -
Using thousands of 2 CPU boxes seems to run counter to the prevailing attitude of virtualization. Everyone else seems to be pushing multi-CPU/multi-core big boxes - that get carved up using virtualization. Perhaps one side knows what they're doing and the other side . . .
0 Votes
+ -
Depends on utilization
Norm_z 3rd Apr 2009
If all the servers are running at high utilization, virtualizing makes no sense.

If you have a bunch of servers running at low utilization, then they are candidates for virtualization.

N.
0 Votes
+ -
The machines they are using are rather old.
B.O.F.H. Updated - 5th Apr 2009
Google uses the infrastructure and hardware that they have because it is met their needs. The machines are rather old (I know, as I had a contract there and discussed this with employees) and specialized. If a company wants to purchase their appliances, they will get a machine running VMWare ESX and the Google software on top of that, capable of running 4 to 8 hosts (depending on how Google implements their software on top of ESX). They are a big VMWare shop, though it is not used for the on-line service(s).

Other major name companies that use VMWare ESX include: Verisign.
0 Votes
+ -
Low maintenance car battery
m4rk.gm4il@... 3rd Apr 2009
Yeah. A low maintenance car battery plus a 12v & 5V regulator can easily accomplish that. you just install the batteries in arrays (parallel) so that when you remove the other battery for maintenance, the other batteries in array will take over. Typically, the battery will last for 2-3 yrs in its life time. then afterwards it can be recycled or disposed.
0 Votes
+ -
It would seem that the overall design objective here is bare-bones least-cost solution without regard to the engineering or the history of such implementation topologies that involve increasing the number of failure points. They are depending on never having more than a small percentage of the batteries/servers fail at the same time for whatever the reason. And they are depending on using fail-over techniques to insulate single server failures.

It has been my experience that increasing the number of hardware units to mitigate a failure will at some point reach the point of diminishing returns.

Recall the laptop battery fires? The Google server batteries, probably from the cheapest vendor, are not immune from such outcomes.
0 Votes
+ -
nt
0 Votes
+ -
Wasn't Wednesday 1st April???
0 Votes
+ -
It's all about the date
dstitt@... 3rd Apr 2009
I can't believe none of these pundits noticed the date this thing was released!!
0 Votes
+ -
It's a JOKE kids!
IT_Guy_z 3rd Apr 2009
It was published on APRIL 1st...you know...April Fools Day?

Geesshh!
0 Votes
+ -
Remember last year? A space colony? Or was
that two years ago?
0 Votes
+ -
Uhmm...
Stuka 3rd Apr 2009
So I suppose all the people that were *AT* the conferance were not actually there?
0 Votes
+ -
Mmm... says April 3rd in the header
javarunner 3rd Apr 2009
unless I'm mistaken.
0 Votes
+ -
NA
0 Votes
+ -
Your worst nightmare
Earthling2 3rd Apr 2009
This is the beginning of the Skynet. You want to turn the network off, but all the servers continue to run on their own individual batteries. Oh, no!..
Everybody, this is not revolutionary. All Google is saying is that they racked their server rooms with a bunch of laptops!

Have a Fun Weekend!!
0 Votes
+ -
wow. Google could be competing with IBM and other server manufacturers! These guys are creating history!

Bhaktavatsalam Nallanthighal
0 Votes
+ -
Google is a remarkable company that thrives on
thinking outside the box. I am worried about other PC
hardware manufactures,like Dell, IBM, and HP. They may
fell threatened in the race for server evolution with
Google creating their own servers with integrated
batteries.
0 Votes
+ -
This design possibly has another more incredible cost saving!

A redundate power supply is not necessary

Dell HP IBM all sell their business server with redundate power supply option that most companies are compelled to purchase because not even a bacup UPS can provide power to a server that blows power supply.

Because this is a 12v system a 12v batter may be able to run the system when the power supply blows. A very common hardware issue to be concerned with. I have not seen critical server without redundant power supplies. Think of the saving on a massive scale.
0 Votes
+ -
This design possibly has another more incredible cost saving!

A redundate power supply is not necessary

Dell HP IBM all sell their business server with redundate power supply option that most companies are compelled to purchase because not even a bacup UPS can provide power to a server that blows power supply.

Because this is a 12v system a 12v batter may be able to run the system when the power supply blows. A very common hardware issue to be concerned with. I have not seen critical server without redundant power supplies. Think of the saving on a massive scale.
0 Votes
+ -
re; power supply
bart001fr 6th Apr 2009
A single power supply could distribute power to two or three such mobos and still keep the batteries of each topped up.

After all, how much power do those servers actually use? Count maybe 120 watts per board and just distribute the output of one 500W dual power supply to 3 boards, and you still have wattage to spare, so you're not overloading the power supply. And you make it a dual power supply so if one fails the other kicks in immediately (These are called redundant power supplies, I think.). In the fraction of a second interval during switchover, the attached battery has kept the level of 12V and whatever amperage is used a constant so the memory and CPUs never even see a fluctuation.

This design is actually the bee's knees, and was the basis of a small add-on board in PC's back in the late 80's. A small battery was attached to a card and the card plugged into the mobo at the connector. Power was supplied through the bus, in case of failure of the main PS, but the battery was of such small capacity, that they were there to providt you with a graceful shutdown instead of a sudden crash due to either a brownout or a total power loss.

I think we're going to see a resurgence of this design, though I wouldn't use it in a big, power-hungry games computer, or something that does graphics intensive work.

But in a home server, whose only job is to serve up data and interface with the web, well, that's another ball game where it could easily handle the job.

Or how about using it in a NAS? Even if it is connected to the web and downloading stuff 24/7, such as a torrent box. In case of power failure, the battery, being connected to the mobo would respond faster than a UPS which is upstream of the regular PS, and must feed 120V off a 12V battery.


0 Votes
+ -
Gelcell batteries don't have the problems traditional lead-acid batteries have with fires and acid spills. If overcharged, they eventually 'vent'. I've seen overcharged gelcells and all they do is either vent or swell up.
0 Votes
+ -
Secret wiring
FiOS-Dave 6th Apr 2009
Google was also very clever in replacing all the copper wiring with gold wiring.
This reduction in resistance has saved them a fortune, along with the fact that as the price of gold skyrockets, they have increased intrinsic value...
WTF?!
0 Votes
+ -
RE: Google reveals secret server hardware
JOHN_TUOHY Updated - 7th Apr 2009
Gold wiring paying for itself in energy savings and multiple small batteries operating at optimum performance are textbook examples of Sustainable Whole System Design.
Kudus to the real engineers at Google.
0 Votes
+ -
Gold wiring maybe gives the game away ...
Ray berry Updated - 7th Apr 2009
Gold has a higher resistivity than both copper and silver (so it is a fallacy to say that cable losses are less with gold) - it is used for 'plating' contacts because it does not oxidise - it stays clean where Cu and Ag both oxidise with time. Copper is used for wiring because it has the lowest resisitivity (lowest loss for the wire diameter) per cost and is easily soldered. It is also less prone to fatigue failures than Aluminium which is the next best choice (and it is more difficult to solder). See http://en.wikipedia.org/wiki/Resistivity#Table_of_resistivities
0 Votes
+ -
Gold wiring?
bart001fr Updated - 8th Apr 2009
Where does it say that?

I read Mr Shankland's report and he said nothing about gold wiring. One would think that he would have mentioned such a (costly) point.

What he did point out is that the power supplies are 12-volt only. Any needed 5 volt is done by the modification on the motherboard.


0 Votes
+ -
What OS do they run on the servers? - Arunabh Das

Join the conversation!

Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]
ie8 fix

The best of ZDNet, delivered

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Facebook Activity

White Papers, Webcasts, & Resources
ie8 fix