In a hypercompetitive world where companies struggle with slimmer and slimmer margins, businesses are looking to big data to provide them with an edge to survive. Professional services firm Deloitte has predicted that by the end of this year, over 90 per cent of the Fortune 500 companies will have at least some big-data initiatives on the boil. So what is big data, and why should you care?
What is big data?
As with cloud, what one person means when they talk about big data might not necessarily match up with the next person's understanding.
The easy definition
Just by looking at the term, one might presume that big data simply refers to the handling and analysis of large volumes of data.
According to the McKinsey Institute's report "Big data: The next frontier for innovation, competition and productivity", big data refers to datasets where the size is beyond the ability of typical database software tools to capture, store, manage and analyse. And the world's data repositories have certainly been growing.
In IDC's mid-year 2011 Digital Universe Study (sponsored by EMC), it was predicted that 1.8 zettabytes (1.8 trillion gigabytes) of data would be created and replicated in 2011 — a ninefold increase over what was produced in 2006.
The more complicated definition
Yet, big data is more than just analysing large amounts of data. Not only are organisations creating a lot of data, but much of this data isn't in a format that sits well in traditional, structured databases — weblogs, videos, text documents, machine-to-machine data or geospatial data, for example.
This data also resides in a number of different silos (sometimes even outside of the organisation), which means that although businesses might have access to an enormous amount of information, they probably don't have the tools to link the data together and draw conclusions from it.
Add to that the fact that data is being updated at shorter and shorter intervals (giving it high velocity), and you've got a situation where traditional data-analysis methods cannot keep up with the large volumes of constantly updated data, paving the way for big-data technologies.
The best definition
In essence, big data is about liberating data that is large in volume, broad in variety and high in velocity from multiple sources in order to create efficiencies, develop new products and be more competitive. Forrester puts it succinctly in saying that big data encompasses "techniques and technologies that make capturing value from data at an extreme scale economical".