Big data: all you need to know

Summary:Big data's the big buzz word of 2012. So what's behind the hype?

Real trend or just hype?


1. What is big data?

2. Real trend or just hype?

3. How can we harness big data?

4. What are the pitfalls?

5. Steps to big data

The doubters

Not everyone in the IT industry is convinced that big data is really as "big" as the hype that it has created. Some experts say that just because you have access to piles of data and the ability to analyse it doesn't mean that you'll do it well.

A report, called "Big data: Harnessing a game-changing asset" (PDF) by the Economist Intelligence Unit and sponsored by SAS, quotes Peter Fader, professor of marketing at the University of Pennsylvania's Wharton School, as saying that the big-data trend is not a boon to businesses right now, as the volume and velocity of the data reduces the time we spend analysing it.

"In some ways, we are going in the wrong direction," he said. "Back in the old days, companies like Nielsen would put together these big, syndicated reports. They would look at market share, wallet share and all that good stuff. But there used to be time to digest the information between data dumps. Companies would spend time thinking about the numbers, looking at benchmarks and making thoughtful decisions. But that idea of forecasting and diagnosing is getting lost today, because the data are coming so rapidly. In some ways we are processing the data less thoughtfully."

One might argue that there's limited competitive advantage to spending hours mulling over the ramifications of data that everyone's got, and that big data is about using new data and creating insights that no one else has. Even so, it's important to assign meaning and context to data quickly, and in some cases this might be difficult.

Henry Sedden, VP of global field marketing for Qlikview, a company that specialises in business intelligence (BI) products, calls the masses of data that organisations are hoping to pull in to their big-data analyses "exhaust data". He said that in his experience, companies aren't even managing to extract information from their enterprise resource-planning systems, and are therefore not ready for more complex data analysis.

"I think it's a very popular conversation for vendors to have," he said, "but most companies, they are struggling to deal with the normal data in their business rather than what I call the exhaust data."

Deloitte director Greg Szwartz agrees.

"Sure, if we could crack the code on big data, we'd all be swimming in game-changing insights. Sounds great. But in my day-to-day work with clients, I know better. They're already waging a war to make sense of the growing pile of data that's right under their noses. Forget big data — those more immediate insights alone could be game changers, and most companies still aren't even there yet. Even worse, all this noise about big data threatens to throw them off the trail at exactly the wrong moment."

However, Gartner analyst Mark Beyer believes there can be no such thing as data overload, because big data is a fundamental change in the way that data is seen. If firms don't grapple with the masses of information that big data enables them to, they will miss out on an opportunity that will see them outperform their peers by 20 per cent in 2015.

A recent O'Reilly Strata Conference survey of 100 conference attendees found that:

  • 18 per cent already had a big-data solution

  • 28 per cent had no plans at the time

  • 22 per cent planned to have a big-data solution in six months

  • 17 per cent planned to have a big-data solution in 12 months

  • 15 per cent planned to have a big-data solution in two years.

A US survey by Techaisle of 800 small to medium businesses (SMBs) showed that despite their size, one third of the companies that responded were interested in introducing big data. A lack of expertise was their main problem.

Seeing these numbers, can companies afford not to jump on the bandwagon?

Is data being created too fast for us to process?
(Pipe stream image by Prophet6, royalty free)

Is there a time when it's not appropriate?

Szwartz doesn't think that companies should dive in to big data if they don't think it will deliver the answers they're looking for. This is something that Jill Dyché, vice president of Thought Leadership for DataFlux Corporation, agrees with.

"Business leaders must be able to provide guidance on the problem they want big data to solve, whether you're trying to speed up existing processes (like fraud detection) or introduce new ones that have heretofore been expensive or impractical (like streaming data from "smart meters" or tracking weather spikes that affect sales). If you can't define the goal of a big-data effort, don't pursue it," she said in a Harvard Business Review post.

This process requires understanding as to which data will provide the best decision support. If the data that is best analysed using big-data technologies will provide the best decision support, then it's likely time to go down that path. If the data that is best analysed using regular BI technologies will provide the best decision support, then perhaps it's better to give big data a miss.

How is big data different to BI?

Fujitsu Australia executive general manager of marketing and chief technology officer Craig Baty said that while BI is descriptive, by looking at what the business has done in a certain period of time, the velocity of big data allows it to be predictive, providing information on what the business will do. Big data can also analyse more types of data than BI, which moves it on from the structured data warehouse, Baty said.

Matt Slocum from O'Reilly Radar said that while big data and BI both have the same aim — answering questions — big data is different to BI in three ways:

1. It's about more data than BI, and this is certainly a traditional definition of big data

2. It's about faster data than BI, which means exploration and interactivity, and in some cases delivering results in less time than it takes to load a web page

3. It's about unstructured data, which we only decide how to use after we've collected it, and [we] need algorithms and interactivity in order to find the patterns it contains.

According to an Oracle whitepaper titled "Oracle Information Architecture: An Architect's Guide to Big Data" (PDF), we also treat data differently in big data than we do in BI.

Big data is unlike conventional business intelligence, where the simple summing of a known value reveals a result, such as order sales becoming year-to-date sales. With big data, the value is discovered through a refining modelling process: make a hypothesis, create statistical, visual or semantic models, validate, then make a new hypothesis. It either takes a person interpreting visualisations or making interactive knowledge-based queries, or by developing "machine-learning" adaptive algorithms that can discover meaning. And, in the end, the algorithm may be short lived.

Topics: Big Data, Australia, TechLines


Suzanne Tindal cut her teeth at as the site's telecommunications reporter, a role that saw her break some of the biggest stories associated with the National Broadband Network process. She then turned her attention to all matters in government and corporate ICT circles. Now she's taking on the whole gamut as news editor for t... Full Bio

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.