Big data: Why most businesses just don't get it

Big data: Why most businesses just don't get it

Summary: Big data could be the answer to all their problems but the trouble is most businesses may be failing to ask the right questions, according to analyst firm Gartner.


Despite the buzz around big data, take-up among businesses remains low — because most organisations don't understand how to exploit it.

Many are looking at big data — large datasets from multiple sources — and trying to figure out what it is, according to Gartner vice president and distinguished analyst Debra Logan.

"What is it that a massive set of data tells you about a particular problem that a more reasonable set of data doesn't tell you?" — Debra Logan, Gartner

"People have that sense that once things get hyped like this they are somehow behind the curve," she told a London roundtable debate.

"But I would say that 95 to 97 percent of the organisations that I know — outside research organisations, people who crunch weather data, that kind of thing — are in fact only in the exploratory phase right now," she said. "Of that 97 percent, how many are actually going to have big-data issues or big-data benefits?"

According to Logan: "It's almost as if this is a solution looking for a problem."

Questions about implementation

Her views on take-up by business conflict with research released on Monday by Microsoft, which found that more than 75 percent of medium to large companies are implementing big data-related projects in the next 12 months.

However, Logan was sceptical about the motives of vendors when it comes to data.

"Software companies in general have no interest in helping you make anything smaller because they make their money from more data and the more disorganised that data is, the more money they make," she said.

Logan agreed that some organisations are making progress with big data, particularly retailers. "The most advanced industry in terms of big data is retail. It's the stuff they do with all the RFID, the supply chain, with loyalty cards. Those are real big-data problems," she said.

Big web companies and broadcasters are also advanced users. While Logan cited the BBC as a prime example, she said banks are not really pioneering because their data is well organised and in mainframes.

"Even banks are not doing big data in a production sense. Certainly, they're investigating it, they're wondering what it means," she said.

"[They] want to learn about Hadoop. They want to learn about that kind of technical architecture and the kinds of methods you use to program and do analytics. But again, it's still quite in the early phases."

Big-data infrastructure investments

Gartner's own figures project spending on big data will reach $34bn by the end of 2013. However, Logan cautioned against large big-data infrastructure investments.

"Nobody should be making this a central part of their business proposition, with the exception of retail and possibly media. Let's hope the investment is in people rather than in infrastructure. If anybody's investing in this infrastructure, I would absolutely discourage them from doing so," she said.

"Some data is not an asset. It's just costing you money"

On the other hand, acquiring big-data services from a third party could make sense. "If it is cheap, if big data turns out to be something you can get from someone else, you can rent the infrastructure, you can ship a bunch of your data and you can just see what happens, then why not? Why wouldn't you do that?" she said.

According to Logan, many organisations are struggling to identify the point at which the investment in big data gives a distinct business benefit and how can you prove that. "I don't have a hard time proving that too much information that is uncontrolled is in fact a liability — that's a completely straightforward case," she said.

"People can't throw anything away. There is just this weird mentality that businesses have about deleting data. You have to make decisions about what could not possibly be useful. Some data is not an asset. It's just costing you money," she added.

"Again what question are you asking? What is it that a massive set of data tells you about a particular problem that a more reasonable set of data about a problem doesn't tell you?"

Barriers to big data

One of the main barriers to asking the right questions of big data is a lack of expertise and a shortage of data scientists.

Logan said some of her London clients recently tried to find consulting help and expertise in the vendor community and consulting companies. "They said they couldn't find anybody. So they basically had to learn it all themselves and do it themselves," she said.

"The people who understand this stuff are currently working in [places such as] CERN — in astrophysics and those kind of places where there's masses of data and people have mathematical and statistical modelling capabilities."

She said developers are not being trained in those skills. "So that's a big opportunity as well as right now a huge gap in the market and also a word of warning," Logan said.

"Anybody who's waving a card around that says data scientist — because everybody is — you really have to dig into that and see if they actually know what they're doing," she added.

Topics: Big Data, Data Management, Enterprise Software

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Most of it is big JUNK data

    If you really look into what's being stored in these big data solutions you'd quickly realize it's mostly blogs, images, comments and such that used by social networks. Such data has little business value for the real enterprise as they cannot run efficient analysis over this data to dig value out of it. That's the root reason no one takes it seriously.
    • So give it the IMPRESSION of value and sell it

      That's what facebook, Apple, and others all do - create a perception of value.

      Just like how our economy is not backed by gold, but by faith - whatever that is, apart from faith being generated by - you guessed it - a perception.
      • Have to give you one vote up

        Finally a fellow ZD poster that understand gold rather than paper is money. Would give you ten if I coiuld
        • I agree with you both.

          As for Junk. The article was a prime example of "big data", a lot of nonsense you had to mentality filter out. And when you did, what was left wasn't of much substance. Fools gold, so to speak.
    • I dont completely agree with you

      Hi LBiege - I can understand the reason behind your comment. But there is another side of the Big Data story that I wrote about in my article yesterday, focussing on how it is helping with business decisions and how the social media data is coming in handy:
      • Thanks for the link

        I'll take a look.
    • Strongly disagree

      You can find value in blogs, comments and other stuff still. All you need is good semantic algorithms. With that you can determine how good you're doing with your products or services.

      Sometimes if you consider the fact that you can actually find value later as this technology develops, maybe it makes sense building your own cluster to store a lot of data and making it available for your data scientists. As they get creative, they may find value eventually. As technology develops, there may be something worth of it that can be seen only in the future. If you narrow your mind to small use-cases you may have a lot of ROI, but if you open your mind for big data cases, then maybe you'll have a lot of innovation and a lot of ROI at some point.

      Machine Learning has been around for 30 years now, so is BI and visualization, but now we're seeing a lot of applications that the first torrent of applications on the internet did not, but they figured out they could with the data they saved. Big data has been around for more than 60 years, since the mainframes. What's new is the network technology that now is providing more speed to data processing through parallelization. The next step is what we'll create with it as we create tools that create value in the future. And I think this revolution is just beginning.
  • If every company listened to Gartner...the would all be bankrupt

    Most companys will never realize a return on the "Big Expense" that "Big Data" creates.
  • So when Gartner talks and people jump to do everything they say,

    who stops the buck once Gartner makes a mistake?
  • It's a bit of a gamble. Results aren't guarunteed.

    "According to Logan: 'It's almost as if this is a solution looking for a problem.'"

    It basically is. "Big data" is often akin to throwing darts and seeing what sticks. You may or may not get something useful out of it. There is no guarantee that "big data" will give you anything useful, unless you know ahead of time what you are looking for.

    There are also privacy issues - granted, it's all gonna be done via machine, but it still has to be done transparently and with permission.

    I don't think it's alack of understanding of "big data." I think they understand it. It's just that it's not really all it's cracked up to be. It's a gamble. You may or may not win big. Sometimes the risk is worth it, especially if you know ahead of time what you are looking for, but often it's not.
  • Is "Big Data" Just The New Term For "Data Mining"?

    Inquiring minds want to know.
  • Data skills

    I think with the requisite visualisation skills, and appropriate amount of data from relevant sources, Big Data can be very beneficial. Having those skills in-house is key in reaping the benefits - organisations may need to focus on freeing up time to train developers.
    M R Griff
  • Big Data Correlation

    I agree that there is currently a lot of hype around Big Data. I would however say that this does not mean companies should shy away from getting the most out of the data they collect, be it big or small. @LBiege, I’d have to disagree that all data collected is junk and agree with @CobraA1’s comment that it’s a case of knowing what to look for. So I thought I’d share some insight on how our customers are using their big data.

    Say you are a Telco Operator with 30,000 3G cells in your network. You want to monitor 20 KPIs across all cells e.g. DCR, CSSR, etc. You want to know if any cell breaches a defined threshold for each KPI as soon as it happens. Assume each KPI formula uses 4 raw performance counters. Your correlation engine needs to receive and process 2.4 million relevant counters per 15 minute pm performance management (PM) data collection period and maintain 600,000 discreet monitoring scenarios simultaneously and continuously. The ZenPM real-time streaming correlator is purpose built to handle the massive scope and volumes that the big data revolution necessitates.

    So I’d say there really are companies who are deriving game changing insights from big data and they are taking the next leap into big data correlation.
    Chris at SysMech
  • Let's get "small data" right first

    As several other commentators have noted, Big Data is largely irrelevant - at present - to the vast majority of businesses (telcos aside, @Chris). Few businesses can claim to manage their 'small data' in a reliable and valuable manner, so Big Data just creates Bigger Problems.

    The excitement around Big Data is predicated on several factors; accurate data (usually captured automatically at source), automated machine analytics and sufficient data points to infer something statistically relevant from the results. All of these considerations are rare outside telecommunications, social networking and physical science; for the rest of us, Big Data is off-topic.

    To make the most of existing 'small data', and set ourselves up for Big Data in due course, we need to think about these prerequisites and adapt our businesses to consider data integration, quality and analysis as priorities.