When it’s raining inside your datacenter, you have a problem…

When it’s raining inside your datacenter, you have a problem…

Summary: A sudden shower inside Facebook’s first datacenter highlights the issue.

TOPICS: Cloud, Data Centers

As reported in The Register this week, Facebook had a unique manifestation of the problems with free air cooling early in the life of their Prineville datacenter. A failure in their building management system combined with ideal weather conditions created a miniature perfect storm that brought clouds and rain to the interior of the building.

Managing humidity in the datacenter is a traditional piece of the datacenter cooling puzzle, but Facebook’s experience is a good indicator of how much more complex the environmental management issues can become when you are relying on Mother Nature to provide the bulk of your cooling. Since you can’t control the weather outside, you need to have tight control on the internal environment and be ready for a broader range of challenges than would be found in a traditional hardware cooling datacenter infrastructure.

Basically, due to a problem in the environmental management systems, the process of recirculating air through the evaporative cooling system resulted in the cold aisle supply temperature exceeding 80°F and relative humidity exceeding 95%. This effectively meant that it was almost as if they were using a garden hose to mist their servers, an environment that few servers are designed to operate in.

As you would expect, this meant that more than a few power supplies fried themselves and servers shut themselves down. The Register quoted Jay Parikah, Vice President of Infrastructure Engineering at Facebook, as saying, “For a few minutes, you could stand in Facebook's data center and hear the pop and fizzle of Facebook's ultra-lean servers obeying the ultra-uncompromising laws of physics.”

Facebook has made changes to prevent this from happening in the future, from weatherproofing their Open Compute server’s power supplies to beefing up their environmental management systems, but beyond the humorous nature of this problem, it is important to learn the lesson it teaches.

While free air cooling is much less energy-expensive than using powered chillers to control air temperature, the overall effort needed to control the environment in your datacenter is not reduced, and, depending on the geographical area that your facility resides, you may find yourself needing to write the book as you go along to assure the most efficient and effective implementation of a free air cooling datacenter infrastructure.


Topics: Cloud, Data Centers

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • your kidding right?

    evaporative cooling system in this day and age for any kind of electronics is just retarded. Hell learnt that over 40 years ago in the military unless you have a secondary system to take care of nothing the humidity
    • sorry

      need more coffee you have to have a secondary system to take care of nothing but the humidity separate from your cooling system.
  • Best friend's aunt my Aunt Fanny

    I want juia54's IP blocked, and in fact I would pay good money for a private raid on whatever site is hosting this spam. If they can get into TechRepublic then they're a threat and I would end that threat with whatever force seemed necessary. More than necessary, in fact.
  • Joke is on them

    No, they're not kidding. I've heard it myself from a CIO of another large consumer of IT resources - they want IT equipment manufacturers to enable them install equipment in an environment largely reminiscent of a swamp - 'cold air' aisles at 80-ish F, no regard for humidity. Blending outside air with hot air to control the temperature of the 'cold air'. Heat removal, and humidity removal as a consequence, is a significant cost to these guys and they want to largely eliminate it.

    Being in a position where I can see the abuse that some customers put their equipment through, they are not alone. Some of these people could break a solid steel ball, have no idea how they did so, and argue that the ball was inferior somehow.
  • Floundering

    Did they have an engineer design the system, a accountant or neither? The design must accommodate the worst possible conditions.
  • Prineville, not Pineville

    Just in case anybody is looking for this on Google Maps.
  • Please Join the 21st Century

    Old news from 2011. The state of the art has advanced. Please note that Facebook didn't stop using air cooling.

    Adiabatic cooling is here, and it works.

    I'm running a 500 kw data center as I type this with the cold aisle at 83 degrees dry bulb and 90% relative humidity. PUE of 1.038 And no rain.

    Let me know if you need any information.

    Dave Speed
    • Adiabatic Process? Typo?

      An Adiabatic Process is a process occurring without exchange of heat of a system with its environment.

      Ref: Wikipedia: http://en.wikipedia.org/wiki/Adiabatic_process

      Possibly, you made an inadvertent typo.
      I tried to type what I thought was the correct word; but, ZDNet gave me a misspelling warning.
  • You Mean Clouds Had Formed In Their Cloud?

    ...I'll get me coat.