What makes Big Data so 'big'

A new study estimates that more than $4.3 billion (or €3.38 billion) was spent in 2011 on 'Big Data' technologies, and this spending will be increasing at a rate of 36% a year over the next four years.

The Experton Group projects that global spending on Big Data technologies -- databases, analytical tools, network equipment and cloud services -- will reach more than $20 billion (or €15.73 billion) by 2016. "The market for Big Data is therefore one of the fastest-growing segments of the overall IT market and is becoming the workhorse of the IT economy,"the study's authors conclude.

Another multi-billion-dollar/euro market emerges. What does this mean to enterprises?

For one thing, it means a lot of venture capital and startup resources are now pouring into this sector. Experton points out that in the last 24 months, more than 30 new Big Data startups have been launched with significant degrees of funding. These are companies specializing in new database and analytics technologies. These specialized providers now account for around 8% of global IT spending and investment, says Dr. Carlo Velten, advisor with the Experton Group. "Big Data is a new market segment and not a recycled [business intelligence], as is sometimes suggested."

But if Big Data is not BI redux, what is it, and why is it different than the BI and analytics that have been a continuing priority in enterprises for more than a decade?

While the essence Big Data is its bigness -- data volumes are now scaling into the petabytes for many organizations -- this is a relative measure, and doesn't suggest anything unique for the current times. Some people have always had and will always have "Big Data" in terms of volume. For example, I remember back around 2000 hearing about telecom's record-shattering, gigantic database, which was topping a terabyte in size. Now that's what some of us carry around in our laptops.

What really makes Big Data big data this time around is the amount of "unstructured data" now flowing into organizations -- increasing volumes of machine-generated data such as sensor and weblogs, along with user-generated data such as graphics, video and documents. That's what enterprises need so much help with covering.

Organizations with their relational database systems -- built up and tweaked and perfected over the past two decades -- would have no issues amping up the amount of transactional or structured data, even if it did go into the petabye or even exabyte range. But today's data environments are simply not ready for the deluge of unstructured data -- all those tweets, all those log files, all those videos. And it's unstructured data that will drive new capabilities, new collaboration and new innovation.

Hence, a multi-billion-dollar/euro market in the making.

(Photo: Joe McKendrick)

This post was originally published on Smartplanet.com