The dismal failure of 'Big Data'

The dismal failure of 'Big Data'

Summary: Economists have no clue despite all the Big Data at their disposal...

TOPICS: Big Data
Data scientist Hilary Mason at a GE Software panel.

The excitement about "Big Data" in tech circles is very optimistic and many companies are rushing to hire "Data Scientists" to profit from the explosion of hype about the reams of data collected inside their own organizations,  and in the world outside.

But having access to Big Data doesn't guarantee that companies, or individuals, will understand or be able to derive much value from it. The very few examples of companies doing that, are very few. And for a good reason – finding insight in all that data is difficult and becomes more difficult the bigger the data sets.

Take for example the field of economics — it's the original Big Data profession. But in all these years, it hasn't been able to do much at all. The profession is well regarded and respected despite its collective failure to understand the economy and predict its behavior.

Surely, a Big Data profession such as the study of economics over the past 150 plus years would by now be refined and almost scientific in its precision, especially since these days we have as much compute power as an economist might need, not to mention even more data to analyze. But it's not even close.

After the financial meltdown in 2008, Alan Greenspan, the former Chairman of the Federal Reserve was asked questions by a Washington committee about how the crisis occurred.

He said all his financial models over the past 40 years were wrong. Yet those models informed his adjustment of interest rates, and if they were based on wrong models, he likely harmed the economy, consistently, decade after decade. It's truly an epic fail for Big Data.

It doesn't require much Googling to discover more examples of the deceit of economists and the failure of Big Data:

How Did Economists Get It So Wrong? -

It’s hard to believe now, but not long ago economists were congratulating themselves over the success of their field. Those successes — or so they believed — were both theoretical and practical, leading to a golden era for the profession.

The puzzling failure of economics | The Economist

The Financial Crisis and the Systemic Failure of Academic Economics

Economics has failed us: but where are the fresh voices?Broken Recovery: Have Economists Failed Us |

Economics is easily the single most important failure of the application of Big Data. And to call economics the "dismal science" is unfair on scientists because there's nothing scientific about it.

How will our new workforce of data scientists fare in their missions? I predict dismally. It's better they avoid Big Data and focus on the small data — just the bits that matter. 

- - -

Talent Shortage Looms Over Big Data -

Become A Data Scientist ... In 12 Weeks? | Big Data


Topic: Big Data

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Today's buzzword

    15 years ago it was Data Warehouses. Big Data is the same thing. The elusive pursuit of correlation, and linear algebra to solve logistics / JIT processes.
    Luke Skywalker
    • Google & the NSA did well with it

      The only two organisations to do really well out of big data were Google (which it used to extend its monopoly in advertising), and the NSA, which now has a profile on every American.

      For everyone else it's an elusive pursuit of correlation. For Google and the NSA, they correlated and they got what they were looking for.
    • Today's buzzword, but still different

      It is correct to say that the "big data" is a natural evolution of what people started trying to do with data warehouses (indeed, a lot of "big data" computing is still done in data warehouses), but there is a really big change from how things were done 15 years ago. The MAD paper ( really highlighted the chang in direction of how data warehouses were done.

      It also ignores the reality that thanks to Moore's law, we've had a massive increase in the amount of data that is being consumed to inform the output. As Google has demonstrated very well (particularly with this:, that actually really *does* change things, with previously ineffective methods achieve impressive results.

      Finally, it *isn't* all about logistics & JIT processes. With the decreasing cost of consuming and analyzing data, there's been a pretty huge explosion in terms of applications. Even the canonical success example: Google search, is not a logistics or JIT process.
  • A needle in a haystack that may not have a needle in it.

    It's basically trying to find a needle in a haystack - when you were never told there was a needle in it to begin with. That's big data in a nutshell.
  • Chaotic systems and feedback loops

    I don't think it's fair to dismiss economics because economists are unable to predict economic crises. The simple fact is that economists are studying extremely complex systems laced with chaotic elements and intrinsically rife with feedback loops. Economics is one of those areas whose subject of study is altered by the mere fact of studying it. It is fashionable to cite the "observer effect" of quantum physics to show a concrete example of such a system, but let me instead engage in a thought experiment. Let's say that after observing a man for many months you have developed a heuristic for predicting what the man will order for breakfast at his favorite restaurant. After taking into account such variables as the weather, the day of the week, the time, whether or not he brewed himself a cup of coffee before leaving home, you find that you can predict, with 95% accuracy whether he will order pancakes or eggs. You've got a surveillance van parked outside the restaurant and you amaze your colleagues by telling them, five minutes before he orders, exactly what he's going to order. So now you decide to take it once step further. Filled with bravado and self confidence, one day you waltz into the restaurant just before the man gives his order to the waitress and you announce: "Sir, I know you so well that I can tell you right now, before you have even ordered, what menu item you are going to order today."
    • Part II

      The man looks at you incredulously. "No you don't" he says.
      "Oh, yes I do!" You insist "Today you are going to order eggs!"
      "Waitress!" The man calls out, "Today I'd like an order of pancakes!"
      Flustered by the experience, you leave. Then come back the next day at the same time to announce "Sir, based on everything I now about you, today you are going to order pancakes! Prove me wrong!"
      And the man orders eggs just to prove you wrong.
      And the same scene is repeated time and time again. Whereas before you were batting 95% in predicting his behavior, now you are batting 0%, you are guessing worse than chance would have it.

      So what happened? Simple: while you were a mere silent observer you were able to predict the man's behavior with great accuracy. But once you began announcing your predictions to the man you inserted information into the system you were studying that created a feedback loop, at which point the system could no longer be predicted (except by an observer who might be extraneous to the system and who happened to notice that the man always orders the opposite from what you tell him he will).

      So the reason why economists will likely never be able to predict economic crises is that the minute economists develop the tools to predict economic crises and make these tools available to economic actors (which they must, because economics is not a secret society) those economics actors will alter their behavior based on the information that they glean from those models.
  • Greenspan biggest Dupee in US History - Economists don't have laws..

    I write about this topic a lot and communicate with some Quants as all quant are in fact data scientists but all data scientists are not quants for sure. I have a page called Algo Duping and it's a collection of videos with people smarter than me that educate. The PBS video, The Warning is on there and you can see where Bob Rubin just pulled Greenspan and Larry Summers around by the nose, he came from Goldman Sachs.

    Also the Quants of Wall Street is there and very good and you get to hear former Wall Street quants tell you how it is with models and ones that lie as subprime could not have happened with those models.

    Economists just guess anymore and they lost a lot of their resource history as it doesn't apply today with one rogue algorithm that changes week of research they to to being useless. They should hang around with more quants.

    US government needs to learn how to model as the banks run models that hide risk and the subsequent Killer Algorithms like crazy and how do you think they got all the money, models and algorithms. Took advantage of a naive and dumb government and pubic in this area. Listen to Mike Osinki who wrote the sub prime software that all the banks modified and abused. "You can do anything with software" and the banks did but he also says how models play out in real life are not always the same.

    Models can also be living in fantasy world at times too. Now we have a school for a price that crank out a data scientist in 12 weeks..yeah for Facebook and Linked in the bs stuff out there but hardly good for much of anything else. Cathy O"Neil who was at one time the quant for Larry Summers at DE Shaw is doing a lot of good work bringing this awareness around too..modeling inequality with segmentation, Weapons of Math Destruction her lecture series name.
  • A Few Got it Right

    Only a handful of economists predicted the great recession. One was the late Professor Wynne Godley and another was Australian economist Professor Steve Keen. There was a couple more who I don't remember.

    The really sad part is that these men are still vilified by the very same mainstream neo-classical economists who spectacularly failed to predict the great recession.
  • It is a question of tools

    Collecting data is the easy part. Developing tools that can effectively analyze the data and transform it into something useful is the difficult part. The other problem is decision makers often have personal, political, and corporate interests to consider and rarely make decisions based solely on the data.
  • Economics

    I remember there was a book about the major problem with economics was that economics used a false model of human behavior and also made the assumption people made had all the information they needed when they needed. Both meant that economic theory was more like some areas of thermodynamics useful to provide a baseline but not always an accurate reflection of reality. The problem is people are seduced by the models and not by the fact they rarely have all the information. Thus, everyone to some degree is operating in a fog of incomplete information.

    The economist who wrote this book was a major economist of the mid 80's.
  • Say what?

    "The profession is well regarded and respected despite its collective failure to understand the economy and predict its behavior."

    It is?

    By whom - bankers, politicians, academics and other economists?
  • Dismal failure doesn't even begin to describe it ...

    Boy how I love to try and drag Foremski into the real world:

    1. The mathematicians to note when dealing with the economy are Benoit Mandelbrot, who showed that they are chaotic systems prone to crash ... and Nassim Taleb who warned that Government officials making financial predictions are basically overpaid clueless charlatans.

    2. The reasons for the last big crash however were nothing to do with chaos, mathematics or big data ... they all lie at the feet of greedy politicians and Finance people. Most especially Americans.

    a. Clinton and previous administrations progressively removed Financial controls ... until there were none. Then in a typical head-up-your-posterior move mandated to counter arguments of racial prejudice, MANDATED, that subprime borrowers be loaned lots of money. (Obama fought such a case when he was a lawyer!)

    b. The banks complied whist leveraging their supporting capital to a dangerously high point ... to increase shareholder returns.

    Borrowers and banks had 'get out of jail free' cards - the borrower could just default untouched (like a limoted company) and the banks knew they would be bailed out. Neither had to consider the downside of ...

    c. The principal facilitating and protective mechanism for large scale borrowing is the process of SECURITISATION ... which was driven to a depth of 20 in some cases because EVERYBODY in the Finance industry recognised the dangers of the subprime loans. And guess what? At each level an institution reaped a 'small handling fee'. By the time this process was over the liabilities were only supported by half the amount in collateral.

    d. The housing market dropped 30%. The few subprime borrowers who could pay stopped paying. Would you when you could bail from negative equity? The highly leveraged banks were bankrupt within a few months. The tidal wave spread abroad to the greedy bwankers over the pond who saw a chance for the same fees. Serves them right.

    Here's the thing. Governments know exactly what they are doing when they invade Iraq. And the Financial world knows exactly what its doing when it leverages toxic loans with a get out of jail free card. All these 'maybe they have weapons of mass destruction' and 'we are very sorry our financial mathematics failed' stories are propaganda.
    Instead of holding the Bolivian presidential jet in France ... I think we should divert Air Force one to somewhere in iraq or Iran with Bush and Blair on it ... and let's how they feel about 'democracy'!

    Foremski - big data- BS.
    And the best you can do for a global collapse is a picture of a female student?
    Where does ZDNET find these bloggers?
    • No need for AF1 after all!

      I see Tony's new transport has been dubbed:
      'Blair Force One'.
      Maybe the NSA will hack the flight computer and set him down in, say, Syria ...
      ... for 'piece negotiations'.
      No, not 'peace' negotiations ... they need more weapons over there.