Battling the giant data monster

Cheap storage means it's easy to save nearly everything: but beware the hidden costs and complexities.
Written by Steve Ranger, Global News Director

Don't let the claws of the data monster damage your business.
Image: Lukiyanova Natalia

Companies are generating huge amounts of data - from their own internal systems and from contacts with customers and social media, to name just two sources, and then storing it for as long as possible. While some of it is certainly useful and can provide the raw material for nascent big data projects, plenty of the stuff that gets squirreled away is just junk.

So is this sudden trend for data hoarding causing problems for the tech team tasked with keeping it all safe and secure?

When asked, "Is the need to store more and more data causing a headache for CIOs?", the ZDNet/TechRepublic panel of technology chiefs was evenly split. While retaining more data is not necessarily a big problem thanks to an array of cheap storage options, managing and utilising that information remains a bigger challenge, according to CIOs.

Gavin Whatrup, Group IT Director at Creston, said: "Storage costs have plummeted on traditional media; enterprise SSD is making eyes water, not bleed now; and tiering and connectivity allow for fitness and resilience respectively. The problem is in making all of this work together, presenting the data to the right people, in the right way, at the right time."

And as Florentin Albu CIO Rothamsted Research, said: "Storing data is relatively cheap once the foundation of storage and backup infrastructure is in place. The headache comes from the side of data management, ensuring the data can be retrieved with a high degree of relevance and - specifically for large data sets - that its accuracy and integrity is maintained during processing."

Michael Hanken, VP of IT for Multiquip, echoed this, noting: "It's not the physical storage rather than the surrounding efforts (retrieval, tagging, compliance, analysis... )."

Shawn Beighle, CIO at the International Republican Institute, made a similar point: that storage is so affordable that there isn't much of an issue with retaining everything that people wish to keep. But he added: "That being said, I do not believe that everything is worth keeping, and organizations need to better set the parameters for which things get purged."

Putting rules in place can be hard, he said, because different individuals have different requirements, plus one must consider the additional time it will take staff to sift through and decide what needs to be kept and what can be destroyed.

"In this age of 'everything now', are staff going to take the time to look back to determine what is/isn't relevant anymore when they're constantly required to focus on not just today, but the three, five, ten year plan?" he said.

Beighle said the best option is to forget about the past and focus on the future: "Set your parameters for retention and implement. Everything before that date you keep and set a realistic plan, as in, make that part of your five year plan, to go back and classify. You have to start somewhere though, or else the data monster will just keep growing, and today is as good a day as any other."

Michael Spears, CIO at NCCI Holdings, is also the organisation's chief data officer, and oversees two separate units: IT with a remit to support the business and data resources as a business unit overseeing the collection, validation, and governance of data.

Spears said: "It's more of an opportunity than a threat, if you have the right governance in place. If you understand what data you need and how it might need to be accessed, the technology strategy to store the data isn't too difficult. Governance isn't where it needs to be if you find people saying 'storage is cheap, let's just keep it to be safe'."

As well as the need to manage stored data, some tech chiefs pointed out some of the more day-to-day complexities of dealing with storage issues in the enterprise.

Mike McGavock, CIO at NeoHealth, said: "In a non-profit like I serve in, the cost of storage is a constant struggle, and as systems collect more data and more space is needed, balancing space for production versus backups, etc etc."

Meanwhile Dirk De Busser, IT manager at Fashion Club 70, said: "We have one colleague with two 20GB PST files. He doesn't want to lose a single email. This is definitely a storage headache. We're lucky that not all our colleagues handle email this way."

He adds: "I'm sure we're holding too much data. And there is private data; work/live balance also has its percussions on enterprise storage."

De Busser added: "The business owners of the data should make rules about how long and where data is stored. But only the owners of the data can set these rules. In the meantime, CIOs try to keep backing up on fast deduplication storage."

However, one CIO took a different view. John Gracyalny, VP of IT at SafeAmerica Credit Union, said: "It is not a 'headache'. It's just another part of life as a CIO." Meanwhile Delano Gordon, CIO at Roofing Supply Group, added: "IT is full of headaches. This is just the latest."

This week's CIO Jury was

  • Mike McGavock, CIO, NeoHealth
  • Florentin Albu CIO Rothamsted Research
  • Rocky Goforth, Director, IT Operations and Infrastructure, Thoratec Corporation
  • Delano Gordon, CIO at Roofing Supply Group
  • Mike Klaus, Information Systems Manager, City of Kearney
  • Joel Robertson, director of IT, King College
  • Gavin Whatrup, Group IT Director at Creston
  • Kurt Schmidt, VP of IT, Capital Credit Union
  • John Gracyalny, VP of IT, SafeAmerica Credit Union
  • Shawn Beighle, CIO, International Republican Institute
  • Dirk De Busser, IT-Manager, Fashion Club 70
  • John Gracyalny, VP of IT, SafeAmerica Credit Union

Want to be part of TechRepublic's CIO Jury and have your say on the hot issues for IT decision-makers? If you are a CIO, CTO, IT director or equivalent at a large or small company, working in the private sector or in government, and you want to join TechRepublic's CIO Jury pool, or you know an IT chief who should, then get in contact. Either click the Contact link below or email me, steve dot ranger at techrepublic dot com, and send your name, title, company, location, and email address.

Editorial standards