Google's 650,000-core warehouse-size computer

Google's 650,000-core warehouse-size computer

Summary: My quad-core tower suddenly feels wimpyWhat does it take to power the world's most popular search engine? Lots of CPU cycles.

SHARE:

My quad-core tower suddenly feels wimpy What does it take to power the world's most popular search engine? Lots of CPU cycles. Which is just what Google's new data centers provide. No one is talking, thanks to Google's tight NDA policy, but with satellite imagery and some deft estimation we can figure it out.

Powering a warehouse-sized computer? OK, how big is a warehouse-sized computer? Google wrote a paper about it (see Google’s warehouse-size power problem), but they were vague on details like the number of processors.

Take Google's new data center on the banks of the Columbia river in The Dalles, Oregon. The area had been hurting since the aluminum smelters shut down after power went over $30 per megawatt. Pricing is complex, but it looks to be about $45/mw today.

If that sounds cheaper than what you pay, it is. A lot cheaper. That's why Google, Microsoft and Yahoo are building data centers in the hydropower-rich Columbia river basin. The world's best windsurfing is just a bonus.

Sizing the Google data centers I can't republish copyrighted photos, but using aerial and satellite photos, Google Maps (of course!) and some mapping software, I determined that each of the two identical buildings contain about 100,000 square feet of Class A data center floor space, not including the covered loading docks. Here's a picture of the cooling end of one data center.

Google’s 650,000-core warehouse-size computer

Counting racks Each 19" rack is about 24" wide and 30" deep (I saw a picture of one at the Seattle Conference on Scalability) or 5 square feet. Fire codes require a 3 foot hallway in front of each rack so the total footprint is 11 square feet - just over 1 square meter in the metric system. That means each data center has room for 100,000/11 or 9,090 racks.

There are also requirements for gigE fabric racks, power conversion and conditioning equipment, cooling and air-handling equipment. Let's say that 10% of the racks are used for non-server equipment. That leaves 8,180 racks.

According to published reports, each rack contains 80 cores in 40 dual-core Intel chips. 8,180 x 80 = 654,400 processor cores. And don't forget there are 2 of those buildings on the Oregon site. Over 1.3 MILLION cores in that one facility alone.

The Storage Bits take As I noted in Google's three rules Google embraces cheap computers. They optimize their workloads for parallel computing, so the number of processors is more important than their speed.

And they certainly have the numbers!

Comments welcome, of course.

Topics: Hardware, Data Centers, Google, Processors, Storage

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

36 comments
Log in or register to join the discussion
  • And, that is the problem for Microsoft. Google can scale faster and cheaper

    than anybody else on the planet, and has a huge head start building data centers. One huge advantage that will be difficult for Microsoft to overcome. Really, with the cost of Windows Server, and the fact that it is closed, NOBODY can compete against Google using Windows, other than Microsoft itself. And, even they are fighting an uphill battle.
    DonnieBoy
    • You really should grow up, DonnieBoy

      [i]And, that is the problem for Microsoft. Google can scale faster and cheaper than anybody else on the planet[/i]

      Once again, you know this how?

      You do not. You think, heck you [b]want[/b] to beleive that you have an idea, but you do not. You know nothing of Windows Server, Google's servers, nor anyone's for that matter, yet you are allways the first to praise Google or to dismiss Microsoft (or anyone who chooses MS over Google) to the point that you have to fabricate many of your facts in the past..

      This is yet just another example of while not nknowing the inner workings of Google (the term NDA used in the article does have a meaning) you quickly praise them for their abilities.

      Would you care for some Soylent Green, sir?
      GuidingLight
      • It is widely believed that Google can scale faster, cheaper, and better

        than anybody. Of course Google (and others) do not publish information on the costs of building data centers.

        Do you have any evidence that would disprove the common knowledge?
        DonnieBoy
        • Google's infrastructure IS cheaper than Microsoft's

          I ran the numbers on Google vs Yahoo, which, like Msoft, uses commercial boxes
          with a free operating system. Google has a 5-8x cost advantage per user I/O over
          Yahoo. <br>
          <br>
          Check out <a href="http://storagemojo.com/2007/07/05/how-yahoo-can-beat-
          google/" target="_blank">How Yahoo can beat Google</a> for more info and a link
          back to last year's article on Google vs Yahoo costs.

          Robin
          R Harris
      • Do you live to belittle others .

        You truly are a joke . Donnieboy brings thought here to ponder upon , you on the
        other hand are a pessimist when it does not suit Microsoft's or your needs . Get over
        yourself , will you .
        Intellihence
  • Can You IMAGINE If It Was Running Vista?

    The heat generated alone by the bloated use of memory and processors by Vista would be enough to vaporize everything on the Pacific Coast....

    Sad..very sad..
    itanalyst
    • no.

      Vista doesn't have a web server, can only take 10 connections at a time, is designed with user interaction in mind and not service processing. Desktop heap is scaled out for multiple windows and many services are designed for workstation type of users.

      I don't think Microsoft positions Vista for use in the manner
      LinuxHippie
      • They use Windows Server (silly)

        Why would they use Windows Vista when they have custom inhouse server software? Silly people.
        evildictaitor
        • Actually, no.

          They use a custom version of Linux.
          http://en.wikipedia.org/wiki/Google_platform#Current_hardware
          dayjm
    • re:Can You IMAGINE If It Was Running Vista ?

      Good lord , let's just say that if it was possible itanalyst . Could you imagine the
      repercussions worldwide if that scrap of heap got hacked . Good lord help us all that
      it's not true . Whew ,,,
      Intellihence
  • All of you...

    I don't get it.. microsoft has its own data centers y'know.. i mean.. the article just stated that.. so.. i'm guessing windows is not the problem there.. and i'm also guessing that their data center has not blown out of the face of the earth because of the heat that windows generates..

    The fact is that linux gets hacked and has a lot of bugs too.. if you havent read the news.. then i urge you to "google" it.
    salilbhattarai@...
  • Another idea ...

    ... which I have my fingers crossed for.
    After a battle of, say 5 years, with M$ over search and SAAS (Ballmer having learnt to play basketball with 12 year olds as he put it) ... wouldn't it be nice if Google released an energy efficient, high performance, scalable,non-RAID secure storage NETWORK OPERATING SYSTEM FOR HOME SUPERCOMPUTERS.
    Deep joy.
    jacksonjohn
    • Now .....

      that would be heavenly bliss. Just thinking about all the downloading i could do makes me drool .......
      goldenpirate@...
    • Robin...and another idea...

      This seems like 'overkill' unless you place more load on the network other than short bursts of html queries followed by long periods of idleness.

      But if you put it together with [url=http://en.wikipedia.org/wiki/Vps]Virtual Private Servers[/url], well then you go into another realm....Robin?

      I see Google offering managed and unmanaged VPSs in the near future with low LOW cost thin clients.
      D T Schmitz
    • People are working on that storage model

      I think we'll see some announcements early next year.

      Not from Google, since they don't have a dog in that fight, but from other companies
      that see the need.
      R Harris
  • Hello, Skynet!

    After spending the weekend watching all three "Terminator" movies with my daughter, I wonder how long it is before Google employees come to work to hear their computers greet them in a deep Austrian accent.

    No fate but what we make.
    muzhik
    • hmmm, interesting but..

      but i was hoping for something more along the lines of an A.I. like Cortana from the Halo games. that would be cool.
      jetman36
      • Yes, interesting but...

        ... I'm sorry Dave. I can't let you do that.

        I'm frightened, Dave. I can feel my mind going. I'm frightened. I'm ...

        d-a-a-i-s-y, d-a-a-i-s-y, give me your answer do ...........
        muzhik
        • Why imagine it....

          They already DO that... "We're sorry dave..."

          Google
          Error

          We're sorry...

          ... but your query looks similar to automated requests from a computer virus or spyware application. To protect our users, we can't process your request right now.

          We'll restore your access as quickly as possible, so try again soon. In the meantime, if you suspect that your computer or network has been infected, you might want to run a virus checker or spyware remover to make sure that your systems are free of viruses and other spurious software.

          We apologize for the inconvenience, and hope we'll see you again on Google.
          stonehenge@...
  • RE: Google's 650,000-core warehouse-size computer

    I'm sure you mean kWh, kilo-Watt-hour, the unit of energy, not kilo-Watt, the unit of power.
    atwood@...