Big Data: Revolution or evolution?

Moderated by Jason Hiner | April 2, 2012 -- 07:00 GMT (00:00 PDT)

Summary: The technology to collect, process and analyze Big Data has been around for a while. So what's changed?

Andrew Brust

Andrew Brust

Revolution

or

Evolution

Dan Kusnetzky

Dan Kusnetzky

Best Argument: Revolution

Closing Statements

The revolution isn't televised

Andrew Brust

In this debate, we discussed a number of scenarios where Big Data ties into more established database, Data Warehouse, BI and analysis technologies. The tie-ins are numerous indeed, which may make Big Data’s advances seem merely incremental.  After all, if we can continue to use established tools, how can the change be "Big?"


But the revolution isn’t televised through these tools.  It’s happening away from them.

We're taking huge amounts of data, much of it unstructured, using cheap servers and disks.  And then we're on-boarding that sifted data into our traditional systems. We're answering new, bigger questions, and a lot of them.  We're using data we once threw away, because storage was too expensive and processing too slow. And then we're working with it, in familiar ways -- with little re-tooling or disruption.  It's empowering.  It's unprecedented.  And at the same time, it feels intuitive.

That's revolutionary.

An evolutionary step

Dan Kusnetzky

I find that my role is often that of a "systems archeologist.” I have learned a great deal by watching the market grow and evolve over the years. Big data is clearly an evolution rather than something entirely new and different.

Suppliers come forward with new products or services and declare that they are both unique and new. I’m often forced to rain on their parade by telling them of products from the 1970s, 1980s, 1990s, or 2000s that did the same thing.  Often the only thing new is the platform upon which they've built their product.  I see the same thing when suppliers of big data products and services take time to visit me.

Although the tools that big data suppliers are offering make the analytical process easier and allow IT analysts and non-IT analysts to sift through larger mounds of data, the analytical process is still the same.

What’s new is the sources of data, the volume of data, the different formats of that data and how fast the data is coming in -- not the basic process.

Big data is just an evolutionary step rather than something entirely new.

Big data has potential to change the game

Jason Hiner

There was a lot to like about this debate. I think it helped tease out some of the real value of Big Data and the differences between Big Data and Business Intelligence, Data Warehouse, and other types of old school business reports. I share Dan's skepticism with all of the new stuff in tech that gets championed as the next big thing, when in reality it's just a repackaged version of something that's been around for decades. 
 
That said, in this case Big Data has the potential to change the game. By bringing in real-time, unstructured data from the open web and social networks, Big Data is going to provide business reports with a new level of immediacy and much deeper insights into customer behavior and preferences. On the backend, Big Data is also going to give more people in the organization the tools to run reports and tap into these massive data streams. It's no longer going to be limited to Excel experts and programmers running SQL queries. Andrew explained all this quite clearly, and that's why I'm giving him the nod.

Talkback

24 comments
Log in or register to join the discussion
  • Search is not Big Data analytics

    "Big Data", which has to be the worst term yet coined, goes far beyond log management and search. At the moment, it's being abused by those vendors to give relevance to their products.

    Big Data is combination of technologies, like new computing paradigms (e.g. Red Lambda's collaborative grids, or Hadoop), simpler throughput-oriented storage (e.g. lock-free NoSQL, graph databases, etc.) and most importantly, incremental data mining. Big Data is really best defined as the situation in which data is either too large, or too 'continuous' to ever make an analytical pass over the entire dataset. You can't just fire up k-means and cluster your data, by the time you are done, the results are at worst irrelevant, or at best forensic.

    The crown jewel of Big Data is incremental knowledge discovery. This is the act of applying data mining techniques to *any* events as they arrive, without referencing any other data that has been received. The trick is how to cluster, classify and perform anomaly detection on such data for the life of the system's operation. No batch processing method (like Hadoop) can solve this problem. *Any* events has to mean *any*. Binary files, imagery, audio, and UTF-8 logs are all forms of data. Being able to perform basic searches on one of them hardly qualifies for this category.
    conduit242
    Reply Vote I'm for Revolution
  • concept of big data and analytics isn't new

    it's the ease in which the data is captured (mobile devices and apps) and the context. Big data isn't necessarily transactional (e.g. I bought a product)...it's behavioral.
    getrichieb
    Reply Vote I'm for Revolution
  • I've always thought that the term "big data" is not indicative of exactly

    what the tech people really want to convey. The data collected is "voluminous", and hard to tackle or tame for quick analysis. Huge volumes of data should be called exactly what they are, and that's "voluminous collections of data" or perhaps "voluminous data" for short.
    adornoe
    Reply Vote I'm for Evolution
  • Big Data hardly matters

    What matters is what you do with it:
    - Use it to obtain information to improve decision-making
    - Protect what you have in custody

    If you use the data wisely, it is a revolution. But the existence of the data explosion means little if it is not exploited and protected.
    nmarks2
    Reply Vote I'm for Revolution
    • BD Matters, Evolution vs. Revolution Does Not.

      I agree to the extent that what's truly important is "what you do with it."

      The debate as to whether it's evolution or a revolution is, to me, primarily a question of etymology. So the question itself is intrinsically arbitrary. However, it seems that the highest benefits of this argument's conclusion(s) will be yielded from the questions that logically follow. For example, as Kusnetzky point out, "The key is that non-IT analysts can take part." This would be huge. The ability to regularly use experts in the fields directly related to the data at hand, rather than IT analysts, would be nothing short of game changing.
      felipeowen
      Reply Vote I'm Undecided
  • New Technologies of the Age Spell Big Data Revolution

    The techhnologies needed to make big data meaningful are reasonably new in terms of availability: supercomputers. Specifically we're seeing impressive developments in quantum computing which will truly give rise to the big data revolution.
    xamountofwords
    Reply Vote I'm for Revolution
    • Big data, or voluminous data, is happening now, and quantum computing

      is not yet there.

      So, we need practical solutions and practical hardware, and practical applications, NOW!
      adornoe
      Reply Vote I'm Undecided
      • We have practical hardware

        We have practical hardware. That's not really the problem. We can process huge volumes of data easily. The question is, why are you processing the data, and how will the results affect your business?
        CobraA1
        Reply Vote I'm for Evolution
      • CobraA1: Agree with you; the problem was with xamountofwords's quantum

        computer comments, of which I was trying to impress that, we need the practical solutions now, and quantum computers are not there yet. And, yes, you're absolutely right about there already being the hardware and the software to process the huge amounts of data, systematically and organizationally, and with results that matter. It's only a matter of making sure that we continue to keep up with the amounts of data being generated.
        adornoe
        Reply Vote I'm Undecided
  • Big Data has no relevance for operational systems

    Which are the interesting things as far as I am concerned.

    As for analysis there is a whole discipline devoted to interpreting large data sets. It's called statistics. You can't make "big data" intuitive because statistical analysis often reveals counter-intuitive results; that why you need statistics. Our intuitive sense of probability is extremely poor.

    If there had been some huge breakthrough in statistical techniques then we might talk about revolution, but all I see is some optimisation techniques that aren't new and are liable to lead to incorrect results.

    Likewise, if someone had devised a technique to query operational data in real-time with little or no impact on operational systems then I might be impressed. Very useful, but not really revolutionary.
    jorwell
    Reply Vote I'm for Evolution