Firms facing data storage mountain

Organisations will have to rethink the way they manage data as storage density levels rise towards their ceiling, according to a leading expert on the topic. That could involve IT departments being called upon to reconsider how much data needs to be kept and a redrawing of lines between online, near-line and offline storage.

George Purrio, European technical manager for Imation, said that it was feasible that storage requirements by 2010 will have reached yottabytes, the technical term for one trillion terabytes. That demand will be driven by the need to store clickstream data from e-commerce sites, a generation shift towards dependence on digital storage rather than physical output, and the expected rise in numbers of computer users, particularly in China.

Beyond that time, data density (the ability to store optimum amounts of data in the smallest physical space) will reach its physical limit at some point. "Everybody knows that there is a physical limit but nobody knows exactly where it is," Purrio said.

To cope with growing storage demands, storage developers will have to continue to increase data density but organisations and governments will eventually need to examine strategies for handling information.

One short-term answer could be a move away from reliance on local storage towards more use of centralised offsite data through virtual tape servers that let managers and users see the data as if it were held locally. Many experts expect that client devices will become caches for data and that storage will move offsite. But beyond that, organisations will need to change their mentality on data storage.

Even for startups storage demands are spiralling. "It's very important to manage the data and eventually it will be a problem," said Chris Ledgard, head of corporate marketing at e-commerce hosting services company Shopcreator. "We have 300GB in online storage alone and I wouldn't even like to guess how much offline." "I really doubt that we need to store everything locally when a lot of what we want is on the Internet, and the other point is how much we archive," said Purrio. "How much of what we have is essential? Will all the records of the last 50 years make a difference to history? If not we need to think about how much we need to keep."

That could require lobbying of governments to change laws on digital archiving. However, in many cases, such laws are becoming stricter. For example, US government organisations are being made to keep all email correspondence.

In the shorter term, Imation expects to be able to deliver holographic storage from as soon as 2002. Initial media will be CD-like disks capable of storing up to 125GB of data, with a 50ms access time. IBM recently released hard disk drives with a 75GB capacity.