When it’s raining inside your datacenter, you have a problem…

As reported in The Register this week, Facebook had a unique manifestation of the problems with free air cooling early in the life of their Prineville datacenter. A failure in their building management system combined with ideal weather conditions created a miniature perfect storm that brought clouds and rain to the interior of the building.

Managing humidity in the datacenter is a traditional piece of the datacenter cooling puzzle, but Facebook’s experience is a good indicator of how much more complex the environmental management issues can become when you are relying on Mother Nature to provide the bulk of your cooling. Since you can’t control the weather outside, you need to have tight control on the internal environment and be ready for a broader range of challenges than would be found in a traditional hardware cooling datacenter infrastructure.

Basically, due to a problem in the environmental management systems, the process of recirculating air through the evaporative cooling system resulted in the cold aisle supply temperature exceeding 80°F and relative humidity exceeding 95%. This effectively meant that it was almost as if they were using a garden hose to mist their servers, an environment that few servers are designed to operate in.

As you would expect, this meant that more than a few power supplies fried themselves and servers shut themselves down. The Register quoted Jay Parikah, Vice President of Infrastructure Engineering at Facebook, as saying, “For a few minutes, you could stand in Facebook's data center and hear the pop and fizzle of Facebook's ultra-lean servers obeying the ultra-uncompromising laws of physics.”

Facebook has made changes to prevent this from happening in the future, from weatherproofing their Open Compute server’s power supplies to beefing up their environmental management systems, but beyond the humorous nature of this problem, it is important to learn the lesson it teaches.

While free air cooling is much less energy-expensive than using powered chillers to control air temperature, the overall effort needed to control the environment in your datacenter is not reduced, and, depending on the geographical area that your facility resides, you may find yourself needing to write the book as you go along to assure the most efficient and effective implementation of a free air cooling datacenter infrastructure.