A Big Data survey smorgasbord
A number of organizations have been conducting Big Data-related surveys recently, and last week, things hit a crescendo. On Tuesday, open source BI vendor Jaspersoft published results of its Big Data Survey; and on Wednesday Big Data vendor RainStor did likewise. Additionally, in my interview with IBM’s Vice President for Business Analytics Products and Solutions, Deepak Advani last week, he pointed out to me a study (but I'm going to call it a survey) that IBM conducted last year, involving interviews with 1734 Chief Marketing Officers, and containing a number of analytics and Big Data-focused questions. And in late June, SAP contacted me to let me know about a survey on Big Data it commissioned Harris Interactive to carry out.
With three surveys and one study now gathered up, I wanted to see if I could suss out some common findings. Although the questions varied widely across the surveys, certain consistent themes did emerge.
Bullish on Big Data
All the surveys indicate strong optimism around big data and buy-in on its effectiveness and/or priority. IBM’s survey showed that 81% of CMOs plan to deploy new technologies to “grapple with Big Data.” The SAP survey reports that a whopping 94% of “C-suite executives” believe there are areas of their businesses that would benefit from the leveraging of Big Data. Jaspersoft’s survey found that 62% of its respondents will be deploying a Big Data solution within the next year, or have already done so. And 75% of RainStor’s Big Data Survey participants said that “Big Data helps their organization make better business decisions.”
Some worry too
That’s a pretty enthusiastic showing. But there are definitely blockers to Big Data too. The complexity around Big Data and its management was a recurring theme. In total, more than 62% of the RainStor survey respondents cited the velocity, variety or increased infrastructure costs of Big Data as challenge to managing it. 71% of the CMOs IBM spoke to say they are under-prepared to deal with the "data explosion" that they face in the marketing arena, and 61% cited “lack of ROI certainty” as a barrier to adopting new tools and technologies. The Jaspersoft and SAP surveys have no such cautionary data, but they also lacked questions that made it reasonably feasible for participants to express such misgivings.
Getting definitions straight
Another caveat: when survey respondents talk about Big Data, they might not mean what you do. Not everyone classifies Big Data as petabyte-scale work with unstructured data in Hadoop. Just how dominant is Hadoop, really? Jaspersoft found that 60% of its survey respondents are using relational databases as their primary Big Data store! MongoDB claims another 18%, and Hadoop only garnered the same share. The RainStor survey found that a bit more than half of its participants use Hadoop to augment, rather than replace, their data warehouses.
More than half of the SAP survey’s respondents identified an average Big Data project data set size of 100TB or more (but less than 500TB) and 24% said the average size was only 10TB. Perhaps more revealing is that a full 73% of respondents in the SAP poll said they prioritized use of existing data sets, rather than social/external data sets. And while the poll didn’t say this, I would think social/external data sets tend to be larger and interaction-based and that existing data sets are smaller and transactional in granularity. And here's more evidence of this phenomenon: use of Enterprise applications as the primary Big Data source scored 79% on Jaspersoft’s survey.
Analytics, baby
79% of the CMO’s who spoke with IBM say that customer analytics influences strategy decisions, 81% say they plan to increase the use of the technology and 92% plan to increase the use of customer and data analytics partnerships over the next 3-5 years. Four out of the top five Big Data use case categories in Jaspersoft’s survey contain the word “analysis” or “analytics.” And in SAP’s poll, “instant access to data in mobile BI/real-time analytics” scored 57% as being “absolutely essential/very important.”
Meanwhile, analytics is a tough nut to crack. RainStor found that 85.7% of its survey participants believe standard SQL query and statements are still “an important tool for query and analysis.” On the IBM side, 61% of CMOs interviewed plan to increase the use of IT skills partnerships over the next 3-5 years. Given the explicit citing of internal Big Data skills deficits in two of the surveys and the majority use of relational databases for Big Data projects in a third survey, it would seem that analytics is the Big Data holy grail. But that many organizations lack the talent to embark on a bold quest for it, at least without outside help.
Big Data: Mainstream or aspirational?
The overarching theme from these four surveys is that there’s huge enthusiasm and big hopes around Big Data, but that the hopes are ahead of real adoption and readiness. Organizations believe they can gain big advantage and competitive edge from Big Data, but they haven’t seen it, at least not to the degree they think is possible. And while many orgs report that they have embarked on Big Data project, many of them are still using relational database technology with data sets far below the petabyte-scale work that Big Data cool kids like to brag about. Worse yet, progress on Big Data’s big driver -- analytics -- is hampered hard-to-use technology and skill set shortages.
The bottom line? For Big Data, lots of businesses are talking a big game, but starting small. Despite the rhetoric, true Big Data will probably take a while to go truly mainstream. But that’s probably just fine. If companies are going to do Big Data right, then they should do it purposefully and carefully to. And surveys like the four we’ve discussed here should help companies taking the slow and steady route know that they are not alone.