"Most, if not all" Sidekick data recovered

The Sidekick data wipe debacle rolls on. There is, however, good news on the horizon for Sidekick users.

Roz Ho, corporate vice president for Microsoft's Premium Mobile Experiences division apologizes for the recent upheaval experienced by Sidekick users and also offers the good news that "most, if not all" of the lost Sidekick data has been recovered.

We are pleased to report that we have recovered most, if not all, customer data for those  Sidekick customers whose data was affected by the recent outage.  We plan to begin restoring users’ personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.

We also get the first indication as to what went wrong, and it points to something being terribly wrong with the way Danger/Microsoft was handling Sidekick data:

We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way.  This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data. [emphasis added]

So a single system failure took out the main database and the backup. Seriously, what bone-headed backup system had to be in place to allow that to happen? Was the data just copies to another folder on the same drive? Likely not that simple, but something equally grossly incompetent had to have happened.

So, what's stopping the same thing from happening again?

We are eager to deliver the level of reliable service that our incredibly loyal customers have become accustomed to, and we are taking immediate steps to help ensure this does not happen again.  Specifically, we have made changes to improve the overall stability of the Sidekick Service and initiated a more resilient backup process to ensure that the integrity of our database backups is maintained. [emphasis added]

Maybe store the backup data on a different machine perhaps? Maybe have an off-site backup scheme?