The Disaster Recovery Items Few Consider

The Disaster Recovery Items Few Consider

Summary: How Too Much of a Disaster Screws Up the Best Plans I spent a lot of time one night this week with the CFO of a coastal Texas organization that’s based between Houston and Galveston. That phone interview yielded some surprising observations about disaster recovery and how things might or might not work according to your firm’s disaster recovery plan.

SHARE:

How Too Much of a Disaster Screws Up the Best Plans

I spent a lot of time one night this week with the CFO of a coastal Texas organization that’s based between Houston and Galveston. That phone interview yielded some surprising observations about disaster recovery and how things might or might not work according to your firm’s disaster recovery plan.

This is definitely a post you’ll want to discuss with your disaster preparedness team.

In no particular order, here are some observations this CFO made:

-Disasters that last for more than 72 hours present some significant issues. For example, suppose your firm needs to contact workers but they do not have electric power at their homes. Their cell phones lose their charge in a day or two. Their modern telephones at home do not work unless they get power from the electric grid. How do you reach employees who cannot be reached?

-How can your workers read your critical email updates if they cannot recharge their laptops? How would they laptops receive your messages if these employees cannot get an active internet connection or find a cell tower that’s operational?

-How does your firm communicate with others if its offices are either non-existent or inaccessible? What do you do if local authorities bar you from accessing your perfectly good facility for days, weeks or months?

-Don’t keep your master phone directory online only. One of the key lessons this CFO shared with me was that a master list of all employees, building engineers, elevator repair personnel, security personnel, window board up services, electric company contacts, etc. was printed out and shared with several key executives. That printed version was invaluable in a business environment most unfriendly to online systems. Knowing who to call for any contingency was a godsend.

-Know who your friends are. This CFO recounted how two sister organizations were seriously impacted by this disaster. One had their offices completely destroyed – totally obliterated. The other had 90+% of their windows blown out. The second structure will be uninhabitable for at least 4-5 months. If your employees need a place to work after a disaster, where will they go? Don’t just assume a disaster will only affect one building. When a natural disaster strikes, it can take out massive amounts of infrastructure across a huge geography. Employees can’t commute to a new work location 100 miles away. Be sure to strike some sort of temporary office space deals with a key customer, supplier, related entity, etc. because you will never find additional office after the disaster strikes.

-Not all disasters are the same. Most disaster recovery plans assume a very isolated outage or problem has occurred (e.g., a localized fire). If your plan doesn’t have a scenario where hundreds or thousands of businesses in a large area around your facility are destroyed, then your plan is flawed and inadequate.

This CFO told me that he learned that:

-Relying on laptop or cellular connections is not much of a solution. -Access to power after 48 hours is a real problem. He found numerous people sitting in cars in his firm’s parking lot using their car batteries to power laptops while trying to siphon bandwidth off of open wi-fi connections in that building. -Paying employees during a disaster has some real challenges if your local bank was in the disaster area. He strongly recommended direct deposit as an employment requirement and that businesses use regional or national banks for payroll deposits so that employees can access their monies. -A backup plan that requires two T-1 lines talking directly to each other is not optimal. While I wanted to learn more on this, we ran out of time.

As the call wound down, this CFO shared a few more observations. Power was such a key commodity that he witnessed lines of people at whatever stores had electricity. These people were bringing their cell phones and laptops just to have the opportunity to plug-in for a few minutes. Even wilder, he reported seeing a man in a folding chair camped out in front of an abandoned gas station. This people went to the shuttered store on the hope that it might still have electricity. It did and this enterprising soul had setup their impromptu office in front of this closed building.

Finally, my Finance friend gave a big thumbs-up to a couple of retailers. Lowes opened a store at 7 am the morning after the eye passed over even though the store had no electricity. This store was fully stocked with approximately 500 portable generators that arrived hours beforehand. I was surprised to hear this as I can’t find clerks anymore who can make change let alone calculate prices without a fully powered POS register. Way to go, Lowes.

Topics: Laptops, Data Centers, Hardware, Mobility, Telcos

About

Brian is currently CEO of TechVentive, a strategy consultancy serving technology providers and other firms. He is also a research analyst with Vital Analysis.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

2 comments
Log in or register to join the discussion
  • More! More!

    Great article. I would love to read more about what the folks in the trenches dealt with.

    Bruce
    bruce@...
  • some small-scale disaster recovery items for smaller businesses

    As owner/operator of a small ASP (Application Service Provider), I thought I would share a few tidbits. We don't have a full-scale enterprise to worry about, but there are plenty of things that we do that minimize our exposure.

    1. We have our own (locked) cabinet at a colo, with our own KVM, various tools, a worklight, and our own UPS (despite the fact that the colo has plenty). This means that we can have guaranteed access to our servers, etc. -- we don't rely on a few shared "crash carts", which will be in HEAVY DEMAND when stuff hits the fan.

    2. Everything in our office is on UPS (except for laser printers; we can still print on inkjets that use less power).

    3. All employees and contractors have good UPS setups for working from home. Several of these folks can realistically host a small workforce at their location, as they have generator backup. The only real exposure is Internet access, which is somewhat mitigated through use of different ISPs.

    4. In addition to KVM access at the colo, we have KVM-over-IP for monitoring boot sequences, etc. Again, for when stuff hits the fan.

    5. We mirror data at a *geographically distant* location.

    6. Watch for single points of failure, and get rid of them.

    7. Use 3rd party services for hosting DNS, email, your corporate website, etc. For a modest monthly fee, they will provide highly redundant systems that are resistant to denial of service, weather-related outages, etc.

    8. By using 3rd parties for email, corporate website, and DNS, your company will still appear mostly "alive" even if your application servers are offline. In fact, you can use your regular website to explain to your customers what is going on. So obvious, but so many firms host their website on the same equipment as their application servers.

    9. Keep print-outs of ALL information needed to access critical resources. For example, our instructions for colo access tell you where to park, which colo techs are most helpful, what cellphone services work while in the colo, etc. These documents are under "code control", and current print-outs are kept in a couple of well-known places. Obviously, passwords are not printed, but oblique references/clues are provided so that the right people know what to use.

    Hope this helps.
    johnradams