SAN backup buffer saves Kiwi college

When one of Waiariki Institute of Technology's servers went down a month ago, it was an opportunity -- unwanted, no doubt -- to test the restore capabilities of the institute's new backup system, implemented after years of problems with backups.

Located on New Zealand's North Island, Waiariki has more than 9,000 students and 450 staff spread across the main campus in Rotorua, and satellite sites in Waipa, Tokoroa, Taupo and Whakatane. As at any educational institution, backup at Waiariki had become extremely complex as data volumes grew and the sheer variety of its operating systems -- NetWare, Windows 2000 Server, Windows Server 2003, Linux, and Unix -- complicated regular backups.

The institute's previous ARCServe backup solution had easily met its requirements eight years ago, but growing storage complexity and data volumes -- Waiariki now runs 47 different servers supporting over 800 desktops, nearly 500 thin-client terminals and several dozen laptops -- had changed the environment in ways that the legacy solution couldn't accommodate.

These problems were compounded by the limitations in backing up data from Waiariki's four remote campuses, as opposed to doing a separate backup at each site. Attempts to back up all of the sites onto a centralised tape array had produced nothing but frustration for network administrator Blake and the rest of the IT team.

Waiariki's IT staff had spent the better part of five years trying to get reliable backups from the previous solution before finally accepting that a more economically viable approach was to simply ditch the whole thing and start over.

"The amount of data we were handling had skyrocketed, but all this time we kept using the same backup solution," says Blake. "Even our incremental backups were very slow, and we were paying IT companies to try and get our backup system working at about $100 an hour. Over five years, I can't even guess how much we spent on trying to get our backups going properly. We just wanted a solution that was either quick enough that we could have it all done between 9pm and 6am, or one that wouldn't impact the network."

One major issue was Waiariki's continuing use of the NetWare network operating system, which has fallen out of use in most corporations but continues to have a strong following in education and government bodies typically with thousands of users to support. This was one of the major problems with the institute's previous backup solution, and for this reason Blake was naturally sceptical at the claims made by vendors as the team evaluated potential options.

In the end, Waiariki settled on CommVault Systems' Galaxy backup and recovery software, which is now used to back up all sites' new data onto a central storage area network (SAN) that was expanded to 3TB to meet the organisation's increased data storage requirements.

Server images are initially stored directly onto the SAN, and kept there for a few days until being automatically dumped to tape. Buffering the 1TB nightly backups on disk has proved much more effective than previous approaches, which relied on pushing backups straight to tape and offered little recourse in the event of a backup failure. Even verifying the data sent to tape was difficult, since rereading all the data blew out the backup window to unacceptable lengths.

The new system was up and running after a three-day implementation. Despite the inevitable hiccups -- the need to apply only the NetWare service packs already supported by Galaxy is one small issue -- it has nonetheless proved to be far more robust than the previous solution.

For now, only Windows and NetWare servers -- infrastructure servers hosting key applications like Citrix MetaFrame and NetWare eDirectory -- are being backed up; more research-focused Linux servers aren't yet part of this backup environment. This proved to be enough, however, when an eDirectory server crashed and Blake's team faced the challenge of recovering its image from the Galaxy environment. Despite the inevitable jitters, however, the recovery went off smoothly: "it all just restored and it was fine," says Blake.

Because the ability to recover backup images is as critical to an effective solution as the ability to back up the data, this test was a real confidence builder for Waiariki. Overall, says Blake, the easier administration from the new solution means the team is only spending around an hour a week administering the backup environment -- compared with the more than 15 hours a week the previous solution was demanding.

This alone has saved so much money that Blake believes the new environment will have paid for itself well within a year. Added to that, however, are the intangible benefits of such a solution: reliability, flexibility, and the confidence to know that the data is actually getting backed up by a system that Blake says is 99.9 percent reliable.

Galaxy is working so well that the team is now considering new uses for its two-tiered backup environment. Future applications under consideration include strategic archiving of e-mail and better disaster recovery plans.

"In the past, we often just weren't getting backups," says Blake. "Now, if somebody wants something backed up, or if a server crashes and burns, we're confident that we can do it."