Tech planning for a minor disaster
While many organisations have business-continuity plans, it's fair to assume that most of them were drawn up with the kind of disaster in mind that hit the Buncefield oil depot, in Hertfordshire in December 2005.
But the disaster that leads a company to put its carefully crafted plan into action doesn't have to be so extreme. As this summer's unexpected floods showed, low-level disruption can be just as difficult to deal with.
Take the experience of law firm Irwin Mitchell this summer. Faced with a car park full of filthy water rapidly seeping into its Sheffield city-centre offices, managers needed to have a way to continue the firm's operations in such a way that its customers wouldn't know there had been any issues at all.
Unseasonably heavy rain this summer led to localised flooding across large parts of the North and Midlands, often in places with low exposure to such phenomena historically. Sheffield was just one city whose public services, businesses and community found themselves thrown into difficulties and disrupted due to overflowing rivers and canals.
One of the UK's largest law firms, with roots stretching back to 1912, Irwin Mitchell has a major call-centre operation and other back-office staff working out of its two new buildings in Mill Sands in the city centre. In fact, 1,000 of the firm's 4,500 total headcount work at the site.
Early on the morning of 25 June staff turning up for work — those that could get in, at any rate — found themselves literally wading into the office. Though not as severely affected as other parts of the city, water was rising and the ground floor was quickly becoming a less than attractive working environment.
Managers could have been forgiven for being caught totally unawares. "Sheffield hasn't had a major flood in 150 years," pointed out business improvement manager Peter Sloane. "When we had these purpose-built buildings made ready that just wasn't part of the risk assessment."
By 9am, a foot of water was sitting in the ground floors of both buildings. "IT is a crucial element in what we do here," said Gary Thomas, the company's head of IT operations. "As a full-service law firm, we deal with a large volume of transactions — 6,000 calls can come into the call centre on any given day. But we also have very tight service-level agreements with many of our clients, in insurance especially. We need to be very responsive. We even have one-hour agreement clauses in some contracts — as in, we need to respond to a client in that period or face financial penalties. So we cannot afford to be down at all."
Patently, for health and safety reasons alone, Irwin Mitchell staff couldn't sit in that water though, no matter how important those service-level agreements.
The good news is that the company had plans in place for disruptions to business. The challenge was seeing if they were more than a set of documents in a binder or if they would actually work.
Part of the response was organisational. The majority of the call-centre team (30 out of 45) obeyed already laid-down instructions to relocate to the company's fifth-floor boardroom, which was immediately transformed into a shadow call centre. The rest — 100 staff in all — immediately started relocating to a workplace recovery site some 35 miles away, in Elland, West Yorkshire — a process that meant the company was ready for operation in its new "home" at the start of the next business day.
However, a very significant part of the response was technology-based. Moving the call centre upstairs, for instance, was made a lot easier by the company's integrated communications network. "The voice over IP [VoIP] has always been more about functionality for us, not cost saving," said Thomas. "We had already had frame relay, for instance, so transfer of calls had been easy. But, by using the ACD [automatic call distributor] functionality, we could now move staff to a new location and they immediately had the same phone functionality and extension numbers — that was very useful." The same applied to staff logging in next morning at the Elland remote site, he added.
VoIP played a part in the overall success of Irwin Mitchell's response to the June flooding problem but...
...other communications and storage technologies also contributed significantly. "We have an MPLS-based wide area network managed by Cable & Wireless and also use technology from Network Appliance that means we can easily 'snap-mirror' all our data from our regional sites to one node," said Taylor.
That node is physically located in the main London data centre of business-continuity specialist SunGard, which also acts as the firm's main disaster-recovery partner. Staff on that June day were being sent to a SunGard facility, just one, in fact, of a national network of sites available in the event that trouble strikes one of Irwin Mitchell's operations. For example, there are provisions which come into play in a Leicester SunGard office if trouble strikes the Birmingham centre, and so on.
So, although it wasn't until the afternoon that the formal invocation of the company's business-continuity resource took place, the foundations were there already. Undoubtedly, this was a good thing, as, ultimately, some £2m worth of damage was sustained by the building, and the waterlogged ground floor is still being made good months later.
So technology was a big factor in the company's robust response to the disruption of its physical environment. By 1am the next morning, two recovery suites had been made available, with 100 PCs and 30 IP phones. All the desktops of Irwin Mitchell's call-centre staff had been mirrored so they had exactly the same information on screen when they came in. By 4am, Irwin Mitchell had access to 100 call-centre positions, each of which was identical to those in its own offices.
But, as Thomas's colleague, Sloane, said, in many ways the overall business plan was just as important as technology for dealing with the crisis. "People issues were a definite factor. We had the communications but we needed to reassure people, get them back on track, physically get them moved and deal with the logistics — get people fed and accommodated at the new site, for example. The lesson I think is that it has to be a partnership between the IT side and the people side, and team spirit was very central to getting us back to work as quickly and efficiently as we did."
The immediate crisis, then, had been dealt with. Irwin Mitchell claimed it did not miss a single one of its customer service-level agreements, thanks to its business-continuity plan. There were power issues at the company for another week and a half, said Thomas, but things settled back down and, while repairs are not complete, the company's business was — in the way all good business-continuity managers like — not materially affected by the summer floods.
But, being a good business-continuity manager, Sloane wasn't completely satisfied with the response to the emergency. "We were pleased things worked but we also saw shortcomings and we want to fix them, as well as keeping things up-to-date. You must never be complacent." To this end, the company is committing to working towards the emerging BS 25999 British Standard for business-continuity management. "This not only provides a best-practice structure, it has some very good tests and exercises guidance."
Sloane has heard that the document outlining the standard is the most downloaded document in British Standards' history — and he was not surprised to hear it. "Clients are increasingly asking for this and we think it will quickly become an expected way of doing business to be accredited to this level," he said.
Dovetailing with the organisational interest in BS 25999 is Thomas's continuing work on getting more out of technology. "We are always looking at ways of improving things and I see virtualisation as probably our next big focus area. This will have two benefits: we will be moving our environmental policy forward by having a more concentrated server operation, but it will also have business-continuity benefits. By having a more portable server environment, we can reduce not just our recovery point objective but recovery time can be improved too."
While the environment continues to be unpredictable, responsible IT leaders undeniably need to be as clued up as possible on business continuity. However, getting the right technology base in place also seems to be key to a — forgive the pun — watertight policy.