Big data: Why most businesses just don't get it
Despite the buzz around big data, take-up among businesses remains low — because most organisations don't understand how to exploit it.
Many are looking at big data — large datasets from multiple sources — and trying to figure out what it is, according to Gartner vice president and distinguished analyst Debra Logan.
"What is it that a massive set of data tells you about a particular problem that a more reasonable set of data doesn't tell you?" — Debra Logan, Gartner
"People have that sense that once things get hyped like this they are somehow behind the curve," she told a London roundtable debate.
"But I would say that 95 to 97 percent of the organisations that I know — outside research organisations, people who crunch weather data, that kind of thing — are in fact only in the exploratory phase right now," she said. "Of that 97 percent, how many are actually going to have big-data issues or big-data benefits?"
According to Logan: "It's almost as if this is a solution looking for a problem."
Questions about implementation
Her views on take-up by business conflict with research released on Monday by Microsoft, which found that more than 75 percent of medium to large companies are implementing big data-related projects in the next 12 months.
However, Logan was sceptical about the motives of vendors when it comes to data.
"Software companies in general have no interest in helping you make anything smaller because they make their money from more data and the more disorganised that data is, the more money they make," she said.
Logan agreed that some organisations are making progress with big data, particularly retailers. "The most advanced industry in terms of big data is retail. It's the stuff they do with all the RFID, the supply chain, with loyalty cards. Those are real big-data problems," she said.
Big web companies and broadcasters are also advanced users. While Logan cited the BBC as a prime example, she said banks are not really pioneering because their data is well organised and in mainframes.
"Even banks are not doing big data in a production sense. Certainly, they're investigating it, they're wondering what it means," she said.
"[They] want to learn about Hadoop. They want to learn about that kind of technical architecture and the kinds of methods you use to program and do analytics. But again, it's still quite in the early phases."
Big-data infrastructure investments
Gartner's own figures project spending on big data will reach $34bn by the end of 2013. However, Logan cautioned against large big-data infrastructure investments.
"Nobody should be making this a central part of their business proposition, with the exception of retail and possibly media. Let's hope the investment is in people rather than in infrastructure. If anybody's investing in this infrastructure, I would absolutely discourage them from doing so," she said.
"Some data is not an asset. It's just costing you money"
On the other hand, acquiring big-data services from a third party could make sense. "If it is cheap, if big data turns out to be something you can get from someone else, you can rent the infrastructure, you can ship a bunch of your data and you can just see what happens, then why not? Why wouldn't you do that?" she said.
According to Logan, many organisations are struggling to identify the point at which the investment in big data gives a distinct business benefit and how can you prove that. "I don't have a hard time proving that too much information that is uncontrolled is in fact a liability — that's a completely straightforward case," she said.
"People can't throw anything away. There is just this weird mentality that businesses have about deleting data. You have to make decisions about what could not possibly be useful. Some data is not an asset. It's just costing you money," she added.
"Again what question are you asking? What is it that a massive set of data tells you about a particular problem that a more reasonable set of data about a problem doesn't tell you?"
Barriers to big data
One of the main barriers to asking the right questions of big data is a lack of expertise and a shortage of data scientists.
Logan said some of her London clients recently tried to find consulting help and expertise in the vendor community and consulting companies. "They said they couldn't find anybody. So they basically had to learn it all themselves and do it themselves," she said.
"The people who understand this stuff are currently working in [places such as] CERN — in astrophysics and those kind of places where there's masses of data and people have mathematical and statistical modelling capabilities."
She said developers are not being trained in those skills. "So that's a big opportunity as well as right now a huge gap in the market and also a word of warning," Logan said.
"Anybody who's waving a card around that says data scientist — because everybody is — you really have to dig into that and see if they actually know what they're doing," she added.