Community banking grows storage bank at Bendigo Bank

Constructing a new, AU$100 million head office was a natural step in the growth of Victoria's Bendigo Bank. However, with rapidly-increasing storage demands and a requirement for 24x7 availability, the bank's IT team was forced to do some creative thinking in figuring out how to upgrade and move its 20 terabyte storage area network (SAN) to the new data centre at the bank's new headquarters.

The need for an upgrade came after more than a year of heady growth for the group, which has significantly expanded its operations in recent years with the addition of 190 locally-managed Community Bank branches across the state.

This growth had pushed the bank's total number of branches to 350 within 12 months; storage requirements grew commensurately, from 7.5TB to current levels of 20TB. That storage, built around an existing Hitachi Data Systems 9960 storage array, supported a total of more than 100 physical and 160 virtual servers spanning IBM AIX-based midrange, z-Series mainframe, Microsoft Windows NT/2000/2003, and Sun Microsystems Solaris environments running SQL Server 2000, Oracle, and B2/UDB databases.

"As a result of this growth, the infrastructure was starting to creak a bit," says storage manager Greg Smith. "We had a look at our midrange and mainframe infrastructure, and the storage infrastructure that supports both those environments, and decided we'd better make some significant changes in order for us to be able to continue supporting the business."

Those changes would be made in the context of the bank's move to brand-new headquarters at a new site on the other side of Bendigo. This meant the new storage infrastructure would not only be set up within the data centre at the new property, but would have to have the full 20TB of data moved between sites without interrupting service.

Bigger, better storage
One of the major goals of the storage upgrade would be to allow for more flexible provisioning of storage. In the past, use of strictly defined logical unit number (LUN)-based SANs meant, for example, that many bank-standard 16GB partitions -- carved out from the SAN as an approximation of future requirements -- might only have a few gigabytes of data; the rest was "just sitting there, absolutely unusable," says Smith.

To allow for more flexible storage allocation, Smith's team set out to build the new SAN with new equipment including a more flexible IBM SAN Volume Controller (SVC) that would enable easy reallocation of data storage as requirements changed.

SVC's MetroMirror functionality, which enables the smart mirroring of data between two storage systems, was also to prove useful during the premises move. "We needed some way to get all the 20TB of data, and all the servers supporting that, to the new site in tiny little bits," said Smith.

Before the data was pushed cross-town, however, the team first built a second SAN within its existing data centre. This SAN was built around an IBM DS8100 storage array, fed by a pair of Cisco Systems 9509 Fibre Channel switches, and the IBM SVC.

Transferring data from the old SAN to the new SAN took several steps. Using the SVC's VDisk Migration feature, Smith's team mounted the existing storage LUNs on the SVC as "virtual disks", or VDisks. The SVC was also pointed at the new storage array, with similar VDisks created on that device. SVC then duplicated each VDisk to its counterpart on the new system, working overnight to move the huge volume of data. The SVC was then set to discover the data storage partitions on the new device, adding them and resuming business as usual for the bank.

By using live disks and careful application of SVC's capabilities rather than a conventional backup-based migration, the team was able to move the data quite efficiently. "The whole process was very straightforward," says Smith. "There was no time consumed doing a tape backup and restore, and there was no risk of tape failure. The server wasn't offline because we took a point-in-time backup, and [risk was minimised because] any time throughout the process we could confidently go back to that point."

Down to business
With its data moved to the modernised SAN, Bendigo Bank's job was only half over; still awaiting the team was the task of moving the data -- and the physical hardware it was living on -- to the company's new datacentre. Given that the systems had to stay operational throughout the process, this process was complex and demanding; all told, the team ended up taking eight months planning, testing, and migrating the data.

A major part of this effort was planning how to physically move the equipment from one site to the other. Taking the gear offline to truck it to the new site would introduce unacceptable downtime, so Smith's team approached the business with a bold proposal: build a second, identical -- and equally expensive -- SAN at the new data centre.

"Our advice to the business was that it was almost suicidal to think that we would turn off some of those key processes, stick them on a truck and move them up the road," says Smith. "Thankfully, our wise people took this onboard and supported us by allowing us to replicate this very expensive infrastructure at our new site. We had somewhere for the data to go to, which was key to the whole thing."

To minimise risk, the migration was planned over the course of four weeks, beginning right after the New Year's break. Every night, a certain number of storage volumes were set to replicate, using SVC's MetroMirror feature to create a mirror image between the two sites -- and keep them mirrored. Data flowed over two pairs of 2Gbps fibre-optic connections between the sites, which provided redundant data paths to keep the datacentres linked at all times.

Actually moving the data proved relatively straightforward using the MetroMirror relationships, since data was continually copied from the primary to the secondary site. Because the SAN had been seamlessly extended between the two sites, with SVC keeping track of which data was where, servers were none the wiser: as long as they were connected to the SAN, they could find the data they were looking for. Once all the data had been migrated and the new datacentre was ready to go live, Smith instructed SVC to designate the new site -- previously treated as the secondary, backup site -- as the primary site.

After eight months of planning and a painstaking data migration, Bendigo Bank's new, roomier datacentre was up and running with 20TB of historical data and plenty of room for future storage requirements.

Because the data was moved separately from the equipment it was stored on, the bank was able to execute a systems upgrade and data migration at the same time. "The fact that the server guys were taking us down the virtual server path made the whole migration much easier simply because there's a lot less physical infrastructure to move," says Smith.

Under the new storage management paradigm, Bendigo Bank's IT team has been able to break away from the strict LUN-based storage partitioning of the past; using virtual storage spaces, business users can now grow their allocated disk space as they need it.

"Expanding virtual disk happens all the time because people can't make up their mind as to how much storage they want," says Smith. "That's one of the things that provides responsiveness [for the IT organisation] -- the ability to be granular and give a server as much disk as it needs, and when."