Big data: How the revolution may play out

If 2012 was the year of big data hype, interest and pilot projects, 2013 will bring production deployments, early returns on investment and a bit of disruption. By 2014, big data projects and systems are likely to be commonplace.

This year, big data became a tech term on par with cloud computing. The term means a lot yet is becoming used so much it loses its definition. By the way that definition typically revolves around velocity (data is moving fast), volume (there's too much of it) and variety (unstructured and structured information).

Does big data live up to the hype? Yes. To me, big data means technology and business alignment---that Holy Grail endlessly pursued by CIOs---becomes a no brainer. Big data projects by nature are about revenue, risk and profits. In other words, IT and the business can't help but be aligned.

Clearly, we're in a big data hype cycle that I put on par with the Linux and open source software craze in the late 1990s and early 2000s. Back then, Linux was going to change the world, kill Microsoft and other things. In many respects, Linux and open source software (Android for instance) did change everything. But a funny thing happened on the way to revolution---open source software became commonplace in every data center and now is take for granted. The revolution happened, but we just stopped talking about it as much. Cloud computing is playing out in a similar fashion.

Big data will follow this cycle too. Sure, millions of jobs will be created. And yes, talent pools will be stretched for a bit. Companies will also reinvent their industries. The vendor pecking order will be altered as startups like Cloudera become the new Red Hats. There will probably be a big data backlash of some sort (see cloud, sustainability etc).

Here's how I see the big data progression as we look ahead.

2013: Those 2012 pilots become production systems. Every vertical will have a big data success story. Oddly enough, success stories will be everywhere. Why? The big data projects are initiated by the business---CEOs, CFOs, CMOs---and IT is seen as an enabler not a cost center.

2014: Based on 2013 success stories and customer case studies, the fast followers will enter the big data game. Industries will all follow a big data playbook. Initially, these early returns will look good. Companies will primarily focus on internal data because there's a lot to mine there. Incorporating external data will be a nice to have, but nothing more at this stage.

2015: Companies will begin to look at external data in their big data plans. Before 2015, consumer facing companies spent the most time with external information and using it. Every analytics and data warehousing stack will have a Hadoop cluster and big data layer. Technologies like Hadoop cease to be a focus because they remain important, but fade into the software stack as a given. Big data mergers and acquisitions pick up steam.

2016: By this point, big data is seen as a utopia of sorts and companies become cocky---they always do. Data driven decisions replace gut feel and common sense. Early wins and common business cases are played out. Now companies have to start really thinking about the data and avoiding errors and correlations that aren't meaningful. There will be spectacular errors as companies incorrectly reject hypothesis, adopt other ones and mistakenly conclude that there are relationships between data that are meaningful.

2017: Cloud combines with big data and data warehousing as a service, analytics as a service and data as a service become the norm. Few companies actually think of building their own Hadoop clusters doing the integration work. Big data infrastructure is just there. Note: 2017 is a guess on when these big data as a service efforts will be common to the masses. The big data as a service game is starting now, but will hit critical mass later.

How does big data play out for the IT buying cycle? By its very nature, big data projects require more C-level types in the ball game. CIOs are still important---and arguably the center of the technology decisions---but there's a gaggle of execs at the table. Here's breakdown:

• CIO: Big data projects allow CIOs to finally break past that "are we aligned?" phase. 
• CFO: All of this information flow is utopia for CFOs who rally behind the cause as a way to control costs and maximize revenue. One risk is that companies lose that human element that inspires big bets. 
• Chief Marketing Officer: In 2012, CMOs became the belles of the IT spending ball. That focus is likely to be premature. Why? CMOs will primarily rely on external data and signals for their projects. Companies just aren't there yet unless they're consumer facing. CMOs have budget though. Also: Can big data engineer marketing influence?
• Chief Operating Officers, Procurement officers: Big data will allow inventory, supplies and manufacturing processes to be tracked from beginning to end. Efficiency will improve once the analysis is figured out.
• Data scientists: These folks will increasingly be seen as C-level material. Career wise, data wonks can write their own tickets.

More:

• Big data: brainstorming the possibilities
• Big data steps closer to mainstream
• Big data projects: Is the hardware infrastructure overlooked?
• 30 big data project takeaways
• Big data, analytics as a service: Likely boom on deck