A computer tech at the Alaska Department of Revenue was performing routine maintenance, reformatting the tax agency's hard disks when he mistakenly wiped out the data backup drive, as well. Oops! Well, not to worry, there's always the backup backup, magtapes updates nightly. Those, it turns out were unreadable, the Associated Press reports.
"Nobody panicked, but we instantly went into planning for the worst-case scenario," said Permanent Fund Dividend Division Director Amy Skow.
But it soon became clear as the department, its consultants, Microsoft and Dell scurried about to resurrect the data that the worst case was at hand.
Nine months worth of data was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter and supporting documentation such as birth certificates and proof of residence. And the only backup was the paperwork itself stored in more than 300 cardboard boxes.Seventy people working overtime and on weekends re-entered all the data by the end of August. By October and November, the department was able to get out people's dividends. The department didn't fire or blame the tech.
"We had to bring that paper back to the scanning room and send it through again, and quality-control it, and then you have to have a way to link that paper to that person's file," said Skow.
"Everybody felt very bad about it and we all learned a lesson. There was no witch hunt," said Corbus.
According to department staff, they now have a proven and regularly tested backup and restore procedure. Skow said the failure of the backup tapes was attributed to an error in the instructions.
"Had a full scale backup restore test been run prior to the July incident, the error in the recovery directions would have been discovered and rectified," Skow said.