While big data technologies like Hadoop can deal with over-hyped new data like mobile and social, the broad demand is simply not there yet. Instead, the big deal about big data is the potential for getting more value more quickly from more data, at a lower cost and with greater agility.
Big data noise has reached the point where most are reaching for the ear plugs. And with any good hype bubble, the naysayers are now grabbing attention with contrarian positions.
For example, The New York Times expressed doubt about the economic viability of big data in "Is Big Data an Economic Big Dud?" This post grabbed a lot of attention, but, like many others I read, it fundamentally misses the point of what big data is all about and why it's important. The article compares the productivity boom associated with the first wave of the Internet to the lack of growth experienced since the inception of "big data"; it implies that big data’s expected economic impact may not happen.
Furthermore, the article implies that big data is something that firms will do or implement. Thinking about big data this way or differentiating between data sets as big, medium, or small is dangerous. It leads to chasing rabbits down holes.
Here is what I mean: We all know that big data is getting bigger — I just read another NYT post that projects 40 trillion gigabytes of global data production by 2020. Awesome, but so what? Most firms I talk to have only a vague notion of what could be done with all this data, yet they are ramping up big investments in big data hoping to capture insight from new social and mobile data sets. After six months to a year of dinking around and being disappointed with their "build big data and they will come" strategy, the business will stop funding things and IT will get another black eye. Nobody needs that.
This brings me to the myth about big data that I debunk the most — that it's all about social and mobile data gone out of control. In Forrester's Forrsights Strategy Spotlight: Business Intelligence And Big Data, Q4 2012 survey of 634 BI users and planners, we found that social and mobile data is actually a pretty low priority:
While big data technologies like Hadoop can definitely deal with over-hyped new data like mobile and social, the broad demand is simply not there yet. Instead we find leaders recognize that the big deal about big data is the potential for getting more value more quickly from more data, at a lower cost and with greater agility; and it is a whole range of technologies and new techniques that make more possible.
But... just because you can doesn't mean you should. Tomorrow's winners understand this, and they are making selective investments aimed at specific opportunities with tangible benefits where big data offers a more economical solution to meet a need. For example, we compared firms that were growing 15 percent or more year-on-year to their peers in our Forrsights Strategy Spotlight: Business Intelligence And Big Data, Q4 2012, and found that:
High-growth firms are more interested in what's possible tomorrow than what's painful today. For example, high-growth firms were 49 percent more likely to be concerned about data velocity and 23 percent more likely to be concerned about data access limitations of older-generation technology. The pack was concerned about data cost and not being able to react fast enough.
High-growth firms are investing more aggressively than their peers. For example, high-growth firms were 53 percent more likely to be expanding or upgrading their big data investments this year than the rest of the pack.
High-growth firms are much more likely to be investing in next-generation technologies. For example, high-growth firms were only 6 percent more likely to be investing in relational database technology than their peers. But these same firms were 54 percent more likely to be investing in Hadoop and a whopping 156 percent more likely to be investing in other NoSQL databases.
Furthermore, I've found, in my direct interactions with firms, that leaders know the real value of big data is lower cost and greater agility. When I interviewed 11 firms with production-class big data implementations, all of them told me the same thing — they got into big data when they couldn't figure out how to accomplish what their business wanted in an affordable way with their existing technology. Further, none of these firms started with external social, mobile, or other exotic forms of data. They began with data they had and understood but could not afford to capture and analyze at scale.
The point is that high-performance firms seem to be on to something in big data that many miss. It's not about the hype, it's about the best solution to meet needs most effectively. They are opportunistically finding ways to exploit new technology and analytic techniques to get past limitations, lower cost, and create greater agility; and they are focusing on the data they have while looking opportunistically toward the potential for adding newer internal and external data sets into the mix.
This is the big deal about big data, but the story is only getting started.
In the meantime, big data will continue to cannibalize older-generation technology approaches, and vendors will keep at the "me too" marketing message to sustain their revenue stream. Eventually, once firms stop trying to fit square pegs into round holes, we will see big data deliver economic benefit as firms become much more responsive to the opportunities that digital information opens to them. We won't be able to say that big data was the only reason, as there are other factors like cloud that are creating tomorrow's ferocious, and nimble, competitors.
While all this works out, the best advice I can give firms is to dump their isolated, IT-led "big data strategy" aspiration, and embrace the potential that the big data concept implies. Rather than separate big data or even data strategies, I encourage firms to ensure that a plan for data is a part of their business strategy, and this plan must account for the new reality — we have the tools and techniques to enable agility while we affordably operate a larger scale. At the end of the day, that's what big data is all about.