Datacenter Business Continuity Requires Rigorous Testing

In more than 20 years of designing backup, disaster recovery, and business continuity systems for clients, there is factor which I have continually stressed. Testing, testing, and more testing; test often, and test regularly.  Make sure that you test a variety of scenarios; it's not enough just to test the connections between sites. And remember that the worst case scenario isn't a meteor strike leveling your datacenter; it's a series of cascading failures that stop your disaster recovery or business continuity plans from working.

This isn't a technology issue; it's simply one of common sense. There is little point in spending the money necessary for a full-fledged business continuity solution without some significant assurance that it actually works. And as companies move to cloud solutions, the failure of the solution provider doesn't affect a single business, but instead dozens, if not hundreds or thousands of businesses.

As it did with last week's series of failures with PayPal; not one, but two separate failures took the credit card processing capabilities for thousands of merchant's offline for almost three hours. On the plus side; all PayPal customers were affected, so a potential purchase wouldn't have shifted from one PayPal vendor to another as the buyer fought to spend their money. On the extra negative side, PayPal didn't acknowledge the initial failure until after it had been resolved.

So from the prospective of the potential cloud customer, this vendor suffered not a single failure, but effectively three failures:

• The network hardware failure that was the original problem
• A failover failure which caused a second outage
• A communications failure where PayPal didn't acknowledge the problem until after the first issue had been resolved.

Frankly, at this point in time, I would want to not only see the business continuity plan of any cloud vendor I was planning on entrusting with a business critical process, but also their policy and actual practices for testing their own business continuity process.