Predicting Big Data's 2014

There has been no shortage of new year's Big Data and Business Intelligence predictions in my inbox in the past month.  As the predictions have trickled in, I was unsure of the value of running any one slate of predictions as an article in itself.  But once they collected, I realized there's real value in reviewing a compilation of these predictions, seeing where they overlap, and appending a few predictions of my own.

The predictions come from a set of companies in the analytics world ranging from an Enterprise software company (TIBCO, makers of Spotfire); a publicly-traded Business Intelligence company (Tableau); an analytics applications platform start-up (Alteryx); a major NoSQL vendor (Basho); and a customer analytics company (Gainsight).  Together, these companies paint an interesting picture of what the analytics state-of-the-art and market will look like in 2014.

The overarching themes in most of the predictions are: Big Data technologies going mainstream; highly specialized areas of analytics becoming more accessible; an increased influence from cloud and mobile; the continued explosion of data volumes, driven by device- and machine-borne data; and disruption to the incumbent megavendors' hold on the database market.

Mainstream or bust
In the Big Data goes mainstream department, Alteryx predicts 2014 will be when "Hadoop Moves From Curiosity to Critical."  Gainsight imagines a world in 2014 where "People Stop Saying Big Data and Start Meaning It," adding "next year it's table stakes."  TIBCO's CTO, Matt Quinn, says "Big Data and all of its tools and technologies will need to move away from being science experiments, and more into the day-to-day, second-by-second operational decision making."

Another spin on analytics technology becoming more mainstream is that it becomes more applied as well.  Alteryx predicts that in 2014 "Big Data Brings Its 'A Game' in Marketing" and that "Predictive Analytics Will No Longer Be A Specialist Subject." Tableau agrees, surmising that "Predictive Analytics, once the realm of advanced and specialized systems, will move into the mainstream as businesses seek forward-looking rather than backward looking insight from data."

Gainsight says "Businesses Get Proactive by Leveraging Customer Data" and TIBCO's Quinn says "Big Data will focus more on value," explaining that "The challenge is no longer in storing the data...but how to extract value from it."

We're all data scientists now
A couple of companies on our panel are predicting that the high priesthood of data scientists will start to become less crucial as analytics tools become more accessible by business users.  Tableau goes so far as to predict "The end of data scientists," adding "Familiarity with data analysis becomes part of the skill set of ordinary business users, not experts."

Alteryx echos this sentiment with its own "Modern Analysts Matter More Than Data Scientists" prediction, saying "Empowering analysts in business departments with Big Data and analytics will become more important than filling the perceived need for millions of data scientists."  That sounds like a good answer to a problem that I've asserted previously, that "Data scientists don't scale."

Cloud and mobile
Tableau predicts that "Cloud business intelligence goes mainstream" in 2014, which is a bit ironic since the company only launched its paid cloud solution this year.  It also predicts "Big data finally goes to the sky."  Gainsight says "SaaS Will Become Table Stakes," and although that sentiment refers to the industry as a whole, we can pretty much bet that it applies to the world of Big Data in a prominent fashion.  Basho goes so far as to say "CIOs become cloud operators," expounding that this "will result in many organizations deploying a range of public and private cloud solutions."

TIBCO's view of Big Data and the cloud is encapsulated in its "Big Data will be used for security/cloud" prediction. Quinn expands on that thought, saying "Big Data will be stored and analyzed in the cloud and I expect the bulk of the 2014 data to be generated from cloud services across the stack from PaaS on up."

Moving from cloud to mobile, Tableau says "mobile business intelligence becomes the primary experience, not an occasional experience."  Interestingly, none of the other vendors addressed the question of mobile.  Although Alteryx calls out mobile devices as an important source of data in volume.

Data explosion
The volume and velocity of data shows no sign of letting up, according to most of our predictors.  As I said, Alteryx calls out mobile devices and location data in general as a big source.  Basho has two relevant predictions here, conjecturing that the "'Internet of Things' hits prime time, accelerating the data explosion" and "Your customers' experience will be measured in milliseconds, not seconds."

Measuring customer and market sentiment at such fine granularity has implications for both velocity and volume of data.  Tableau certainly doesn't disagree with Basho's assertion, predicting that "Organizations begin to analyze social data in earnest."

Disruption complex
Not surprisingly, Alteryx, a start-up company, reasons that "A new data and analytics stack emerges with new solutions for databases, analytics and visualization, all disrupting the traditional mega-vendors."  Building on the new stack idea, Tableau sees a world in 2014 where "NoSQL technologies become more popular as companies seek ways to assimilate this kind of data."

This sword has two edges though: Basho thinks that the incursion of the new stack into IT will cause IT to demand some help, predicting "Strained Enterprise IT will demand operational simplicity from vendors."  Gainsight might agree, saying 2014 will be a "Big Year for Enterprise Tech IPOs" (emphasis mine).

IMHO, ho, ho
I think the consensus in the predictions we've just gone over serves as a reasonable validation of their efficacy.  I also think our collection of tech companies may have missed a few important potential developments for next year: Hadoop will become more integrated, embedded and less discrete; conventional databases will accommodate NoSQL/semi-structured data workloads, and the industry will consolidate.

With the release of Hadoop 2.0 and its YARN management layer, Hadoop is no longer tied to the MapReduce algorithm and batch processing; as such, it becomes more of a platform and an engine for distributed processing of data, and less of "product."  Additionally, the ability to process key/value data, JSON object/documents, sparse columns and other NoSQL data has already shown up in relational products like DB2 and Vertica, and that trend will continue.

With those opportunities for integration and consolidation of products, I believe it follows that companies will consolidate too.  I expect the Big Data world is teed up for the same kind of M&A activity that the BI world underwent, circa 2007.  

So don't count the enterprise companies out, because they are the likely buyers.  But don't despair, as this is in no way a zero-sum game.  Without all of the innovation of the last few years in the analytics world, the megavendors would likely be giving us another round of relatively minor, evolutionary upgrades in their database platforms.  As things now stand, in 2014, that would be a road that would lead to the industry wilderness.