Data migration tough but essential for government dept

The Department of Families, Community Services and Indigenous Affairs (FaCSIA) is no stranger to electronic record keeping, but in 2002 it realised that some of its most crucial information might soon become inaccessible.

The Federal Government department develops policy and procedures for welfare and other income support payments to Australians, working in partnership with Centrelink, which provides the actual service delivery.

One of the most important tools in delivering those services is the "Guide To Social Security Law", which has existed in one form or another since the first social security laws were passed in Australia in 1947.

"It was important for staff to be able to access the most current version to assess applications for income support," Melanie Randall, knowledge training manager for the library and information services division of FaCSIA, explained at the Enterprise Content Management 2006 seminar in Sydney this week.

Historically, "the Guide" (as staff refer to it) was delivered to service offices in ring binders, leaving staff with the "unenviable" task of adding and subtracting pages as policies changed.

"In the paper-based days, in each office there would be multiple versions," Randall explained. While front-line staff were largely concerned with ensuring they accessed the current version, policy researchers also wanted access to archival versions -- a task that became even more visible following the division of the Department of Social Security into FaCSIA and Centrelink. Archive versions were also important for use as evidence in the case of legal challenges to decisions.

Electronic versions of the guide were produced as far back as 1991, and by 1994 electronic publication had become the norm, albeit via an unwieldy system using a proprietary publishing system, Epublish, and with floppy discs or CD-ROMs sent to individual offices. Each update was sent out as a complete new set of discs, which could be issued as frequently as once a fortnight.

"It was quite a sophisticated product, and it was heavily customised to the department's needs," Randall said. "The conundrum was none of this was accessible via the intranet." Archive research was also fiddly, requiring separate installation and removal of each version that needed comparing.

Long-term accessibility was a major concern. "The library became increasingly worried that the age of the discs could render the storage medium unusable," Randall said.

The problem unexpectedly came to a head in late 2002, when Epublish announced that support for its produce would cease in March 2003. FaCSIA was left with no choice but to develop a procedure for migrating all that data, ideally into a format that wasn't tied to a particular product.

"Data migration can be a complex task and one that struggles for priority in government and corporate project budgets," Randall said. "But unless something was done, there would be a loss of essential corporate records."

The library division eventually scored a migration budget by highlighting the potential dangers of not having access to older versions of the guide. "It was the risks rather than the return on investment that was the business driver for this particular project," Randall said.

An immediate challenge was that in 2003, procedures for archiving documents in a long-term format for government use were still in their infancy. "For us there was no benchmark in terms of previous experience. No-one had had done what we had intended to do. The standards were still evolving," Randall said.

FaCSIA sought advice from the National Archives of Australia, which encouraged the department to conduct focus groups to identify the essential features of the current documents before conversion into new formats. Those groups identified the use of hyperlinking and the retention of current document formats as key considerations. Separately, the department decided to utilise the then-new VERS (Victorian Electronic Records Strategy) format as the basis for its storage architecture.

Finding the right staff to manage the project was equally important. "We were able to retain a contractor who had worked extensively with the various releases of the guide to create the prototypes we needed," Randall said. An additional IT complication was the need for the resources to be accessed by two different departments, each with a distinct IT management strategy.

The process used by FaCSIA exported documents from the old Epublish system into RTF files, and then converted them into HTML for use on the department intranet, and into XML for storage in a long-term archive. Users can now access both current and archival versions online, with new versions being produced in HTML and XML formats.

For now, the XML versions exist purely to ensure a long-term archive, but if FaCSIA switches any software to XML-based systems -- an increasingly common choice by government agencies -- then they might be utilised more directly, Randall said.