Big data steps closer to mainstream

Big data projects haven't hit an inflection point yet, but the signs are beginning to add up. Here's what to look for to indicate big data has it business prime time.

Big data pilots are turning to implementations and meaningful adoption of technologies such as Hadoop and NoSQL-based databases are likely to follow. In other words, big data is coming close to the starting line of mainstream adoption.

That's the gist of a research note from Cowen & Co. analyst Peter Goldmacher. He was recapping the vibe from Hadoop World last week. The most notable item during that big data powwow was Cloudera's public beta of its Impala technology, which is designed to speed up Hadoop for real-time analysis.

Goldmacher's take meshes well with the chats among CIOs at the Gartner Symposium last week. Big data pilots were plentiful in 2012, will ramp more in 2013 and mainstream adoption quickly follows.


However, big data technologies, notably Hadoop, aren't mainstream yet, but as Goldmacher noted "this is clearly changing." Goldmacher added:

With the introduction of newly funded Big Data startups like Platfora that offers in memory BI for Hadoop, Continuity and Mortar Data selling custom development frameworks on top of Hadoop, or Hadapt that offers users a familiar SQL interface to leverage Hadoop, the focus now is to make it easier to use Hadoop. The challenge is all these vendors are still forcing their customers to: 1. Figure out where and when to use Hadoop, and then, 2. Mix and match a variety of technologies to accomplish their goals.

Also:  How big data is transforming health care and predictive analytics


How do we know when we're at this big data utopia?

  • Goldmacher reckons that there will be meaningful and repeatable projects. These projects are being spearheaded by early adopters and serve as reference case studies for others.
  • Mergers and acquisitions. There are multiple big data startups and these companies will have to consolidate because the market won't support them. Cloudera clearly is building a big data stack.
  • Apps companies will emerge built on big data infrastructure. Goldmacher noted that the database battle in the early 1990s included Oracle, Sybase, IBM and Informix. Oracle became dominant due to its relationships with independent software vendors and application companies.
  • As big data goes mainstream, Goldmacher said that a few obvious winners are infrastructure and application providers. But the biggest winners will be "big data practitioners," also known as companies using the technology.

Goldmacher concludes:

Past as prologue, if we look at the history of ERP, there were over 200 companies created to capitalize on the automation of standard business processes. This means that investors in 1990 had less than a 0.5% chance of picking either SAP or Oracle as the ultimate winners in the space. However, if an investor had purchased stock in the 30 components of the Dow in 1990 that were all deploying ERP, that investor would have benefitted from a 35% decline in General and Administrative costs as a percentage of revenues, a five-fold increase in revenues as automation enabled massive scale, and an almost eight-fold increase in market cap.

That last point shouldn't be underestimated. Talking Hadoop and big data technologies can be fun, but it's far more interesting hearing about the use cases and connective tissue between tech and real business.