When the weather turns nasty, your business doesn't have to shut down
Texas' failure to manage its electrical grid led to tens of millions of Texans not having power for their homes, never mind their business. It didn't have to be that way. The Electric Reliability Council of Texas (ERCOT) knew that major electrical failures were possible, but refused to address the issue. One Texas business -- Dallas-based Evoque Data Center Solutions with over two dozen data centers around the world -- took the right precautions and its Allen- and Dallas-based data centers weren't down for even a second. Indeed, Evoque ran on its Allen generators for about 15-hours not because it needed to, but because it could and it shared megawatts of power to the local grid.
Why? A company spokesperson explained in an exclusive interview that "Evoque has a standard winter weather preparedness process, using a pre-planned checklist, at our Allen and Dallas data centers. We implemented, and are carefully following, Evoque's emergency preparedness checklist to help keep our customers and employees safe, our facilities secure, and our communications consistent. Generator fuel tanks and make-up water tanks (HVAC) have been topped off and each center has multiple days of fuel on-site."
The result? "We had had no loss of power, and our clients have had consistent access to their corporate-critical data, applications, and workloads."
How did they do it? It starts by realizing that it's not "if" a major power outage will happen, it's "when" it will happen. Officials in Texas who claim they never saw a cold-event knocking out their power have short memories -- or perhaps they don't actually live in the state. Ten years ago, in February 2011, about 200 major generators faltered because of the cold -- knocking out power to 3.2 million customers. Think of it as a dress rehearsal for the killer freeze of February 2021.
Besides, if the cold doesn't get your power, another natural disaster will. Tornadoes, hurricanes, wild-fires, earthquakes, you name it, your power's vulnerable to it.
Here's the checklist Evoque uses for all its data centers.
Make sure the exterior of your data center is clean, to avoid any debris coming loose.
Check your oil and fuel levels; how many days of fuel do you have on hand, and how many hours of uptime can it be used to provide, if needed?
Check your emergency water supply, for HVAC purposes.
Make sure your UPS systems are fully functional.
Make sure you have enough food, water, even cots on hand if your staff is required to stay on-site for several days.
Test your emergency preparedness plan regularly.
Do you have an emergency phone list in place? When did you last check it to make sure all of the contacts and numbers are current?
You'd be wise to adopt these simple suggestions yourself. There's nothing fancy or expensive about them. It's just a matter of making sure your disaster preparations are in place and that they actually work.
In short, you must have a plan. Alas, according to the Disaster Recovery Preparedness Council 2020 report, only 38 percent of respondents feel very prepared they could recover their IT services in the event of a site failure or disaster. Even those numbers may be optimistic. While most companies say improving disaster recovery is a critical priority, the reality is disaster recovery's monetary, personnel, and time investments have actually been declining.
Evoque points out if you can't trust your internal business continuity plans, you always try to save your business by moving some of your data and services in the cloud. Don't trust the public cloud? Consider a colocation (colo) service. With a colo, you still own your hardware and run your own applications, but the hardware can be miles away from potential trouble. Then, you just need to make sure your colo partner can handle the disasters.
That means, an Evoque representative said, you need to know the answers to the following questions:
- Do they have an emergency plan in place to proactively prepare for incidents?
- How often do they test their emergency plans, to ensure they're fully functional?
- Do they have adequate stocks of fuel in place, to run your hardware if needed? How many days, and what plans are in place to secure additional fuel?
- Do they have interconnectivity between their data centers?
- Do they recommend that you locate your information across multiple data centers so that if a disaster occurs, no matter where, you can stay in business?
- If you have a problem, how quickly can you contact them?
These suggestions are appropriate regardless of the disaster -- cold snaps, tornadoes, fires, earthquakes, as well as man-made problems such as severed cables. If you and your cloud or colo partner have tested plans in place to deal with all of these, it will still matter when something awful happens, but at least you can keep your servers on and your customers happy. And, as many Texas-based businesses can tell you, this is no small thing.
Related Stories:
- How to build a successful disaster recovery plan using multicloud technology
- How to protect your IT power from deep-freeze disasters
- Extreme weather forecast? Essential gear for when the power goes out
- Jackery Solar Generator review: A 1000-watt power station for your mobile office
- Bluetti AC50S power station review: Perfect for power cuts