Data retention will require massive storage re-architecture

Storage vendor Hitachi Data Systems has said that whatever data the government may decide to force ISPs to retain, it will require a massive overhaul of storage systems.

While it is not certain what type of data the Australian government may require internet service providers (ISPs) to retain for up to two years under new proposals, any change would require a costly overhaul of storage systems, according to Hitachi Data Systems' Australian Chief Technology Officer Adrian de Luca.

The Federal government has proposed , as part of a suit of suggested modifications to telecommunications interception legislation, that ISPs be required to retain (as yet-undefined) customer "metadata" for up to two years. The telecommunications industry has previously warned that it will cost somewhere up to AU$700 million for the industry to set up the systems and processes in order to retain customer data — and iiNet estimated that it could cost up to AU$60 million for it to set up the systems required for data retention . This would amount to an AU$5 per month additional tax for customers, the company said last week.

De Luca said that, based on the way telecommunications companies currently store data, this would be a "conservative estimate" on the cost.

"There's no doubt that there is going to be huge logs that need to be maintained. The ability to find an email that was around two years ago from, potentially, a customer that no longer is with them, how do you go and do that easily?"

"High levels of automation and understanding the context of information is going to be critical."

He said that telecommunications companies will need to look at how best to keep the data without compromising the current level of service it provides to customers.

"You don't want these to actually impact the day-to-day user experience of the telco's customers, [but] there may actually be a re-architecture of current information," he said.

"You may actually need to upgrade your actual storage capacity, and move what is currently sitting on low speed disks onto high speed disks. In some cases, you may need to invest in solid state storage so that it doesn't impact performance levels."

The ability to then classify that information and store it on different tiers, depending on relevance, will also be critical.

"You can start to think of log files, where you're doing lots of rapid succession of writing. You're going to potentially need to write that onto a flash technology or high speed media, so you can classify the data that is most frequently written and recalled, versus data that may lie dormant."

Depending on the sorts of information required, ISPs may need to keep data from applications long after they have stopped using that application, and that will require the ability to search through data independent of the application itself, de Luca said.

"Being independent of the application means [that a telco] can search consistently across all types of application and data," he said. "If [the telco] decommissions the application midway through that data life policy, it means they don't necessarily need to keep the application around."

If automation, search and tiering are in place, de Luca suggested that this could keep the cost of retaining data lower.

"Having good integration at the storage layer, having good integration at the application layer, and automating a lot of that management, which also includes disposal, means the cost of compliance is going to be a lot lower."