Dealing with the data deluge, and putting the 'information' back into CIO
Read this...
We are generating huge amounts of data every day: analyst firm IDC has calculated that between 2005 to 2020 the scale of all the digital data created, replicated, and consumed in a single year will grow by a factor of 300: from 130 exabytes to 40,000 exabytes, or 40 trillion gigabytes.
Looked at another way, the amount of data we generate effectively doubles each year.
And businesses are busily storing as much information as they can. Whether they are storing more data because it is demanded by regulators or because they want to sift through it for useful business insights, or because of new technologies such as mobile and the internet of things, volumes continue to increase.
But as they salt away more and more data, companies are increasingly wondering whether they are making the best use of this information: are they simply piling it up and creating a storage and analytics headache? We might be doubling the amount of data every year, but are we anywhere near twice as smart as a result?
Information overload?
When asked "Are businesses at risk of suffering from information overload?" members of the ZDNet/TechRepublic CIO Jury said the risk of corporate information overload was real, but one they were addressing.
Richard Storey, head of IT at Guy's and St Thomas' NHS Foundation Trust, is sold on the value of analysis: "Those who succeed use and make sense of all data; those who fail do not," he said.
However, the increasing scale of data is certainly creating additional costs for tech chiefs. John Gracyalny, Vice President of IT with SafeAmerica Credit Union, said: "The expense of accumulating, storing, and analyzing the data is getting out of hand. Almost half my IT personnel budget is now going to DBA-types."
Dealing with data overload is an appropriate challenge for the CIO. As Gavin Megnauth, Group CIO at Impellam, pointed out: "We are called Chief Information Officers — but taking that title literally, it's worth asking ourselves whether we are spending the appropriate amount of time managing unstructured company information in particular."
Megnauth said that while big data is this year's hot topic, with much focus around business intelligence, data warehousing and analytics, an overlooked area is ensuring that unstructured data can be used for business advantage.
Kevin Leypoldt, IS director at Structural Integrity Associates, said he is actively working to put more structure around the company's unstructured data. "Without a doubt unstructured data is growing at an enormous rate, and without the necessary policies, procedures and infrastructure to create a formal "data lifecycle" I would say we would suffer from information overload."
According to Leypoldt, the proliferation of mobile devices and fragmentation of this data will make this lifecycle more difficult to govern and therefore structure.
"Without a doubt unstructured data is growing at an enormous rate, and without the necessary policies, procedures and infrastructure to create a formal 'data lifecycle' I would say we would suffer from information overload."
— Kevin Leypoldt, IS director at Structural Integrity Associates
Michael Spears, CIO at NCCI Holdings, said that when it comes to information overload businesses are at risk, but it can be solved fairly easy with solid data governance and creating 'data steward' roles within the organisation.
"I head up IT and a business function responsible for data collection. Our IT process is very solid, but the reality is that business users need direct access to raw data."
"This is the nature of the beast for an information company such as ours," continued Spears. "The data steward concept has worked great for internal QA on information that goes external, expanding the concept of metadata to include a far more robust set of knowledge on the history of a data element, and what are the pitfalls of using it. It's also resulted in a conscious and logical approach to data retention."
Duncan James, infrastructure manager at Clarion Solicitors, said that while the information businesses are storing is vast and growing, "businesses are being engulfed by an expanding array of persistent information communication channels." However, there's a danger it can be obsessively over-analysed.
"Businesses are constantly looking to consolidate the number of information sources into the nirvana that is a single pane of glass; the trouble is, this is a mirage that most businesses will never see," warned James.
Different industries face different challenges when it comes to data, as Brian Wells, associate CIO at Penn Medicine, noted: "The healthcare industry is still playing catch-up with respect to electronic data capture. But even if we were all at the optimum level of electronic data, we are required by law to retain patient data for 7 years for adults and 25 years for children. The challenge is not retention. It is finding ways to mine the data securely for new insights and discoveries."