Preparing for the worst: Read a banker's post-9/11 playbook
"It's kind of ironic," the State of Georgia's commissioner of banking and finance David Sorrell told me. "We put all of these contingency plans in place in the event of a Y2K disaster, and then nothing happened. But if it hadn't been for all that Y2K preparation, most banks wouldn't have been nearly as prepared for 9/11. The investment paid off after all."
Sorrell's experience with disaster planning for the banking and finance system could provide some lessons for just about any business--even if it's not integral to national security in the way banks are.
Georgia's recipe for disaster endurance
| ||||||||
"For example," said Sorrell, "what if the people who are supposed to be making sure that all the corporate systems are working during an emergency aren't doing their jobs? That could disrupt all of the other systems that depend on those corporate systems to work, and then you get a domino effect."
Today, the U.S. dollar-with neither gold nor silver backing it--depends solely on public confidence and the U.S. government's ability to satisfy its creditors. "The banking and finance system is so ingrained into our country and our culture that it needs to be a priority in times of an emergency," Sorrell says.
A key component of that priority status is what Sorrell calls the system's "telephonic preference" on the nation's telecommunications infrastructure. Sorrell and his counterparts in 49 other states and four territories, as well as those at federal agencies like the Federal Reserve and the FDIC, will never get that "all circuits are busy" message when attempting to complete a phone call during a national emergency .
While ordinary businesses may not rate highly enough in the scheme of national security to get telephonic preference, this emphasis on communications is worth considering for any business. For any business to endure a disaster, those individuals who are critical to that endurance must be able to communicate. What contingency plans do you have in place, down to physical rendezvous locations and times, to guarantee that those critical to your business' ability to endure an emergency are in contact with each other? Something as simple as widely distributed contact information makes a lot of sense.
Says Sorrell: "In the event of an emergency, we know how to get in contact with all of the other state and federal banking agencies. We have a full list of emergency contact information for all the agencies and state banking departments and I can get in touch with my counterparts immediately."
People at other agencies aren't the only ones that Sorrell needs to get a hold of. On Sept. 11, for example, several Georgia banks wanted to close. According law, it's within the rights of a bank's leadership to close. But, on that day, bank CEOs wanted the close decision to come from Sorrell. He turned them down. "Georgia was not under direct attack, we had control of the skies, and we did not want to spread panic within the state by declaring a financial emergency. It was important for us to remain calm. I told them that they should be there to help to maintain the customer's confidence and that the cash was on the way to make sure they could meet any obligation they may have."
"All of the banks have to be prepared in case there's a run on deposits. Part of our contingency plan--which requires good communications, resilient information systems, and contracts with armored car companies--is to get cash from the local Federal Reserve out to all the banks as quickly as possible. If we think a lot of people will want their money, then the banks have to be open and the cash absolutely has to be there in order to maintain confidence."
If Sept. 11 proved one thing, Sorrell believes, it is that the nation's banking systems are ready for most anything, and that the integrity of everything from balances to transactions-in-progress (i.e., electronic funds transfer) are not in jeopardy. Says Sorrell: "If we had every large American city struck with this kind of attack, the potential for large-scale panic would definitely be there. But now, the banking systems are designed for just such an occurrence. They're designed to deliver liquidity, to keep the system up, and ultimately to maintain confidence."
To this resilience, Sorrell attributes several factors that could serve as lessons for other businesses. Sorrell sees a big trend towards decentralizing and spreading business' and agencies' physical facilities across several geographic locations, and coming up with detailed contingency plans that involve people working from their homes.
"The NY attack and results of that really did attack IT and systems. Not only did those systems stand up to the test, but with the Federal Reserves being distributed around the country, you have the ability to make sure the cash and systems are decentralized, which is in itself a contingency plan," advises Sorrell. Given his agency's importance in the disaster response process, Sorrell says Georgia's Banking and Finance Department is both decentralized and located in areas that are far away from busy city centers. "We have five locations other than the main office and two have been tapped as backup locations. We have full backups at those locations, and there are other state agencies that have similar systems that we could turn to if need be. One thing that has helped is that we don't run any mainframe or legacy systems. We're completely in a networked server-based environment."
Sorrell acknowledges two other factors critical to the success of any contingency plan: testing and people.
Leaving a contingency plan untested is begging for trouble. Sorrell says that as soon as some new component of his agency's disaster recovery plan is in place, it gets tested. But, in the same breath, he also acknowledges the difficulty od conducting one massive test of all a plan's components. "Running a test like that could take months to prepare for and just preparing for it is a luxury you don't have when there's a real emergency. So, we're always testing it and always thinking about the plan and looking for changes to make, but we also have to realize that whatever we have right now is the best we have."
Of course, it doesn't matter how good your disaster endurance plan is if your people don't have the composure or presence to execute it. In addition, make sure there are no single points of failure-- technological or human, Sorrell advises, "You need to have backup provisions and roles. Look at a person's ability to handle things from their home location. How do you handle a role if your primary person can't get to the backup location? You need to have backup people for those critical roles. You shouldn't have any technological or human single points of failure. You're committing yourself to failure if you do."
Is the banking system's disaster recovery plan impregnable? Probably not. Sorrell admits there's work to be done. "We have to improve our tests. The more we learn about new vulnerabilities, the more we must be sure to test for them. Viruses and worms are a vulnerability and could very easily be involved in a cyberattack. We've done a lot of intrusion testing" says Sorrell.
One way Sorrell stays ahead of the game is to not run traditional Microsoft-based applications. "We're not an Outlook shop. That's been a major vehicle for worms and viruses, so we're sticking with something else for now."
While Sorrell shared his concern about using Microsoft products, he also said that going with non-Microsoft products may not be the answer. "I read an article that Apple's operating system has had only one virus attack since 2001. That's very interesting to me. But then again, you're probably just creating a false sense of security to say no to Microsoft."
Where does Sorrell go for the latest information on improving the resilience of Georgia's banking system? "The Gartner Group, and then our counterparts in New York," says Sorrell. "The Gartner Group has been a great source of information and recommendations when it comes to disaster planning and recovery. But the model that Georgia and most other states are working from comes from the one that's currently deployed in New York. They've been through it, so no one knows what to look out for better then them."
The New York State Banking Department's ability to rebound from 9/11, as well as learn from it is what earned it that honor (not that it's an honor anyone really wants). Via email, the organization's office and information technology director Connie VanDecker told me, "I don't think anyone could have ever been 'ready' for 9/11. In spite of the most horrendous circumstances we were very prepared for situations that we had not anticipated. We have had a detailed disaster recovery plan in place for the last three years, fully tested and operational. Our readiness was heightened during the planning for Y2k, specifically with regard to remediation of application systems and accessibility to our network. During and after 9/11, we were prepared to execute the plan in real life."
Getting people online at alternative locations was one of the major issues VanDecker and team faced on 9/11. "The biggest challenge was that about 300 people had to be placed in several locations we had not anticipated using," Van Decker said. "Our IT staff worked to identify analog lines to connect to the hotsite and sought out established local area networks with targeted user data. Our disaster recovery plan was successful because we were able to carry out the banking department's critical functions by putting our hotsite into operation and by having dial up capability to access essential information."
Despite her department handling 9/11 better than most, VanDecker says the plan has evolved since that time. "One of the things we didn't anticipate was having to temporarily relocate our entire staff. Once we evacuated our building downtown, there was no going back because we were in the frozen zone. I would say this was the biggest challenge. We had to have live systems in place at four different locations, which proved to be a new challenge that caused us to stretch our thinking, particularly with respect to consolidating it all at the end. Since that time, we have made few adjustments to our backup plan, mostly related to connectivity. We continue to have teams in place to review and test the plan, procedures (including low-tech phone trees), laptops and BlackBerry PIN connections on an ongoing basis."
Later this year, the Conference of State's Banks Supervisors (CSBS) will be publishing disaster recovery models for use by all states that are largely based on the one created by New York in the wake of Sept. 11.
Are you ready for anything? Share your disaster endurance do's and don'ts with your fellow readers using ZDNet's Use TalkBack. Or write to me at david.berlind@cnet.com. If you're looking for my commentaries on other IT topics, check the archives.