MUST READ: Congress demands answers from Google over Glass privacy concerns

Topic: Data Centers

Datacenter Business Continuity Requires Rigorous Testing

Summary: PayPal's failure highlights the need for cloud service consumers to be comfortable with their providers business continuity capabilities.

David Chernicoff

By for Five Nines: The Next Gen Datacenter |

In more than 20 years of designing backup, disaster recovery, and business continuity systems for clients, there is factor which I have continually stressed. Testing, testing, and more testing; test often, and test regularly.  Make sure that you test a variety of scenarios; it's not enough just to test the connections between sites. And remember that the worst case scenario isn't a meteor strike leveling your datacenter; it's a series of cascading failures that stop your disaster recovery or business continuity plans from working.

This isn't a technology issue; it's simply one of common sense. There is little point in spending the money necessary for a full-fledged business continuity solution without some significant assurance that it actually works. And as companies move to cloud solutions, the failure of the solution provider doesn't affect a single business, but instead dozens, if not hundreds or thousands of businesses.

As it did with last week's series of failures with PayPal; not one, but two separate failures took the credit card processing capabilities for thousands of merchant's offline for almost three hours. On the plus side; all PayPal customers were affected, so a potential purchase wouldn't have shifted from one PayPal vendor to another as the buyer fought to spend their money. On the extra negative side, PayPal didn't acknowledge the initial failure until after it had been resolved.

So from the prospective of the potential cloud customer, this vendor suffered not a single failure, but effectively three failures:

  • The network hardware failure that was the original problem
  • A failover failure which caused a second outage
  • A communications failure where PayPal didn't acknowledge the problem until after the first issue had been resolved.

Frankly, at this point in time, I would want to not only see the business continuity plan of any cloud vendor I was planning on entrusting with a business critical process, but also their policy and actual practices for testing their own business continuity process.

Topics: Data Centers, Enterprise Software

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

9 comments
Log in or register to join the discussion

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    David, you are soooooo RIGHT ON! But, I think the problem with Cloud Providers is a little more systemic, and can be summed up with "Not taking care of the basics." Anyone who has run an enterprise data center takes your point about testing for granted. It is just part of data center life. Another section of the enterprise data center "creed" is to notify your users of problems BEFORE they notify you. Having watched the Cloud community over the last 12-18 months, I see other disturbing actions that could and will be extremely detribmental to the adoption of Cloud Computing by enteprises. In 2009, we saw outage after outage with the big cloud players: Google, Amazon, Microsoft, Rackspace, Salesforce. One Microsoft incident struck me in particular. It was an elongated mid-week, mid-day outage, caused by a change they made that they could't back-out. In an enterprise environment, you do not make changes to production environments during peek hours. Even if it is an emergency, you ALWAYS have a crisp back-out or bypass plan. Another incident was a facility outage....in a highly touted "Tier 4" data center. They gave a great explanation even showing facility schematics. The problem was that their schematics clearly showed that the data center was not Tier 4....oops!

    Look back through the cloud provider outages and you will see a pattern of what I call a "lack of maturity" in core data center management discipline. They need to hire some top notch Enterprise class IT Infrastructure executives and then give them the authority to lay down the laws.
    Ken Cameron
    Reply Vote

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    I also desire to signal in your RSS feeds. Thank you as soon as once again and maintain up the great operate!<a href="http://nccma.com">nccma</a> <a href="http://coolerkings.com">cooler</a>
    MACKENZI
    Reply Vote

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    I used to be more than happy to seek out this internet-site.I wanted to thanks in your time for this glorious read!! I positively enjoying each little bit of it and I have you bookmarked to check out new stuff you weblog post. this thread is amazing i like your work and i appreciate you that you have share a useful stuff thanks for sharing <a href="http://the-ishop.com">the i shop</a> <a href="http://abatwa.com">abatwa</a>
    PEARLINEI
    Reply Vote

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    I used to be more than happy to seek out this internet-site.I wanted to thanks in your time for this glorious read!! I positively enjoying each little bit of it and I have you bookmarked to check out new stuff you weblog post.Bookmarking now thanks please consider a follow up post.<a href="http://power28.com">power</a> <a href="http://sagesinc.com">sa</a> <a href="http://iloveshoping.net">shop</a>
    RHIANNONA
    Reply Vote

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    I think the representation of this article is actually superb one. This is my first visit to your site. Thanks a lot and keep sharing the information. Keep updating the information for all of us. Thanks ZDNet Government was launched as the brand's first industry vertical, with a mission to cater to IT professionals in the public secto I agree with your post. However, do you have any sources I can cite for my paper <a href="http://easy-wheels.com/">wheel</a> <a href="http://pbcars.com/">car</a> <a href="http://com69.net">com</a> <a href="http://cadburry.com">bury</a>
    SATURNINA
    Reply Vote

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    Well welcome, hopefully you can become a vital member of the community and really help to push far ahead of google. Which Im sure the development team would love. This will of course earn you alot points too and get you on the leaders board.<a href="http://vintagesnapbackhatsfan.com">z</a><a href="http://bestsolidstatedrive.net">d</a><a href="http://b2days.com/">n</a><a href="http://b2wp.com/">e</a><a href="http://buy-sell-cheap.com/">t</a> <a href="http://sellcheap.net/">t</a><a href="http://newsoftwarepc.com/">h</a><a href="http://bestlaptoppcreviews.com/">a</a><a href="http://buyfurniturefreeshipping.com/">n</a><a href="http://cheapclothingstoresonline.com/">k</a> Im not sure i come to an agreement with you on every level, howevor it absolutely was a good posting, many thanks for taking the time to put up your ideas.
    TOCCAR
    Reply Vote

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    Thanks nice info <a href="http://buyboxinggloves.net/">z</a><a href="http://buygemicrowave.com/">d</a><a href="http://cheapweldingsupplies.com/">n</a><a href="http://cheapcarcareproducts.com/">e</a><a href="http://cheapluggageforsale.com/">t</a> I really liked your current article write more..let me add you to its favorite The articles you have on zdnet <a href="http://mlbshopgiants.com/">s</a><a href="http://best3dtvavailable.com/">i</a><a href="http://lampsplusstorelocator.com/">t</a><a href="http://discountperfumewebsites.com/">e</a> are always so enjoyable to read. Good work and I bookmarked it.
    MCKNIGH
    Reply Vote

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    Fantastic news about the new release.I positively enjoying each little bit of it and I have you <a href="http://www.youtube.com/watch?v=JQ3nfbubdTA">b</a><a href="http://www.youtube.com/watch?v=thkxMxMihl0">o</a><a href="http://www.youtube.com/watch?v=VG9Uw27cJH0">o</a><a href="http://www.squidoo.com/kindlefirereviews">k</a><a href="http://www.squidoo.com/bestkindlefire">m</a><a href="http://www.squidoo.com/kindlefireforsale">a</a><a href="http://www.squidoo.com/kindlefireprice">r</a><a href="http://www.squidoo.com/cheapkindlefire">k</a><a href="http://the-ishop.com/kindlefireprice/">e</a><a href="http://power28.com/kindlefire/">d</a> to check out new stuff you weblog post.Im not sure i come to an agreement with you on every level, howevor it absolutely was a good posting, many thanks for taking the time to put up your ideas
    RICHMONFT
    Reply Vote

  • RE: Datacenter Business Continuity Requires Rigorous Testing

    Good day to confirm this comment I would appreciate <a href="http://golfcarttops.com/">T</a><a href="http://snowgum.net/">h</a><a href="http://gatesbydesign.com/">e</a> <a href="http://ashleighblinds.com/">b</a><a href="http://yjsound.com/">e</a><a href="http://dry-fruit.net/">s</a><a href="http://xweddings.com/">t</a> <a href="http://netrail.net/">o</a><a href="http://7ey.net/">f</a> <a href="http://birdellis.com/">Z</a><a href="http://airtechinc.com/">D</a><a href="http://medicalwholesalers.net/">N</a><a href="http://kustombike.com/">e</a><a href="http://beelinebicycles.com/">t</a> <a href="http://dippitydog.com/">d</a><a href="http://infinityskate.com/">e</a><a href="http://jeansjournal.com/">l</a><a href="http://vwebcams.com/">i</a><a href="http://compactlightbulb.com/">v</a><a href="http://expressionphotography.net/">e</a><a href="http://cavehicles.com/">r</a><a href="http://falconconcrete.com/">e</a><a href="http://fedson.com/">d</a> your website very nice to everyone Yes, Oracle is the only one with shared-disk architecture, but that is there advantage. It means you can add or remove nodes and the database lives on. In a shared nothing architecture, if you lose a node, you lose the system. I'm sure Oracle appreciates EMC highlighting their advantage.I also desire to signal in your RSS feeds. Thank you as soon as once again and maintain up the great operate Awesome post! Thank you very much || thanks for nice content this is really benefit to me.
    JACOBSONR
    Reply Vote
CNET Join | Log In | Privacy | Cookies Close