How prepared are you for the perfect storm?

Sometimes you think that you've managed to squeak past the surprises that Mother Nature plans on throwing at you. When I left for a vacation in Alaska last month I landed to a flurry of emails and text messages asking if I was OK after the surprising earthquake that was felt through my part of the country. While I was gone hurricanes ravaged the East Coast, and friends who checked up on my homes let me know that they were both OK (I split my time between a house near Philadelphia and a place in the mountains of PA).

On arrival back home this week I had planned to settle in then head back to my mountain home to work on a few projects and enjoy the semi-solitude it offers. Unfortunately, Mother Nature had other plans for me. It seemed that my home in rural NE Pennsylvania was basically inaccessible. Every major access route was flooded out, and most of the back roads were closed.  The surrounding counties were in declared emergency mode, with only government and emergency personnel allowed to be driving on the few streets that were still usable. Phone lines were up, but there have been rolling brownouts and blackouts as the local power companies struggle to deal with downed power lines that were unreachable or submerged.

For most of my neighbors, this is more of an annoyance than anything else. Being a mile from the river and about 500 feet higher meant that our homes weren't flooding (not that there are that many homes in our little valley) and the creek that runs through the hollow hadn't flooded any more than normal, leaving the two roads intact. Of course, once you were off those roads, there weren't too many places to go. One of my neighbors, with a lifted Jeep he uses as an off-road toy, spent six hours driving the 20 miles to pick up his wife from work in the County seat a few nights back. And it's gotten worse since then.

Yes, that tractor-trailer is floating down the road

I had a long phone call with that neighbor last night, going over what was happening in the area, and letting him know that he was welcome to any supplies he knew that I had in storage if they were stuck there long.  Being of similar mindsets, we both have homes that are well-stocked for situations like these, with generators, fuel, food, and the basic necessities of life (just like most of my rural neighbors).Our conversation basically led to one conclusion; they were going to wait it out, and, if I was lucky, I should be able to access my property in as little as three or four days, depending on how much the rivers continued to rise and how much additional precipitation fell. If the projections held for rainfall, however, it could be as long as a week until I will be able to get there.

So what does this have to do with datacenters?         

I've always thought that my rural area was a good candidate for a datacenter location. Like many rural areas getting new datacenter projects, there is an existing viable infrastructure, a few cities within a 90 minute drive that can provide a strong recruitment base for staff, good cool weather climate for natural cooling, tectonically stable, and plenty of locations suitable for installing a state-of-the-art facility.

As I mentioned, Internet services have remained up, as well as telco, and intermittent power issues would be easily dealt with by standard datacenter emergency power generation facilities. But the accessibility issue brought to mind something that I had not previously considered; how do you staff your datacenter when you can't get there?  And how do you support the staff that is in-place?

Due to the rural nature of the areas where many new datacenter facilities are being built, many of the people that choose to work in these facilities don't live in the immediate area, but instead have relatively short commutes from slightly less rural locations. If your staff is stuck at your operational datacenter, what are your plans for providing them with the basics needed to survive?  They have shelter and bathroom facilities, but how about places to sleep and food supplies? Even if they are very local to the datacenter, if there is a state of emergency declared and they can't drive to or from work what arrangements have been made?

If your planning considers this a complete disaster and rolls your workload over to a backup facility, what are you doing for those personnel trapped at the primary?

Any plan is only as strong as its weakest point; how well does your disaster planning take into account the well-being of your IT staff?