Andrew Brust
Revolution
Evolution
Dan Kusnetzky
Best Argument: Revolution
Closing Statements
The revolution isn't televised
Andrew Brust
In this debate, we discussed a number of scenarios where Big Data ties into more established database, Data Warehouse, BI and analysis technologies. The tie-ins are numerous indeed, which may make Big Data’s advances seem merely incremental. After all, if we can continue to use established tools, how can the change be "Big?"
But the revolution isn’t televised through these tools. It’s happening away from them.
We're taking huge amounts of data, much of it unstructured, using cheap servers and disks. And then we're on-boarding that sifted data into our traditional systems. We're answering new, bigger questions, and a lot of them. We're using data we once threw away, because storage was too expensive and processing too slow. And then we're working with it, in familiar ways -- with little re-tooling or disruption. It's empowering. It's unprecedented. And at the same time, it feels intuitive.
That's revolutionary.
An evolutionary step
Dan Kusnetzky
I find that my role is often that of a "systems archeologist.” I have learned a great deal by watching the market grow and evolve over the years. Big data is clearly an evolution rather than something entirely new and different.
Suppliers come forward with new products or services and declare that they are both unique and new. I’m often forced to rain on their parade by telling them of products from the 1970s, 1980s, 1990s, or 2000s that did the same thing. Often the only thing new is the platform upon which they've built their product. I see the same thing when suppliers of big data products and services take time to visit me.
Although the tools that big data suppliers are offering make the analytical process easier and allow IT analysts and non-IT analysts to sift through larger mounds of data, the analytical process is still the same.
What’s new is the sources of data, the volume of data, the different formats of that data and how fast the data is coming in -- not the basic process.
Big data is just an evolutionary step rather than something entirely new.
Big data has potential to change the game
Jason Hiner
Talkback
Search is not Big Data analytics
Big Data is combination of technologies, like new computing paradigms (e.g. Red Lambda's collaborative grids, or Hadoop), simpler throughput-oriented storage (e.g. lock-free NoSQL, graph databases, etc.) and most importantly, incremental data mining. Big Data is really best defined as the situation in which data is either too large, or too 'continuous' to ever make an analytical pass over the entire dataset. You can't just fire up k-means and cluster your data, by the time you are done, the results are at worst irrelevant, or at best forensic.
The crown jewel of Big Data is incremental knowledge discovery. This is the act of applying data mining techniques to *any* events as they arrive, without referencing any other data that has been received. The trick is how to cluster, classify and perform anomaly detection on such data for the life of the system's operation. No batch processing method (like Hadoop) can solve this problem. *Any* events has to mean *any*. Binary files, imagery, audio, and UTF-8 logs are all forms of data. Being able to perform basic searches on one of them hardly qualifies for this category.
concept of big data and analytics isn't new
I've always thought that the term "big data" is not indicative of exactly
Big Data hardly matters
- Use it to obtain information to improve decision-making
- Protect what you have in custody
If you use the data wisely, it is a revolution. But the existence of the data explosion means little if it is not exploited and protected.
BD Matters, Evolution vs. Revolution Does Not.
The debate as to whether it's evolution or a revolution is, to me, primarily a question of etymology. So the question itself is intrinsically arbitrary. However, it seems that the highest benefits of this argument's conclusion(s) will be yielded from the questions that logically follow. For example, as Kusnetzky point out, "The key is that non-IT analysts can take part." This would be huge. The ability to regularly use experts in the fields directly related to the data at hand, rather than IT analysts, would be nothing short of game changing.
New Technologies of the Age Spell Big Data Revolution
Big data, or voluminous data, is happening now, and quantum computing
So, we need practical solutions and practical hardware, and practical applications, NOW!
We have practical hardware
CobraA1: Agree with you; the problem was with xamountofwords's quantum
Big Data has no relevance for operational systems
As for analysis there is a whole discipline devoted to interpreting large data sets. It's called statistics. You can't make "big data" intuitive because statistical analysis often reveals counter-intuitive results; that why you need statistics. Our intuitive sense of probability is extremely poor.
If there had been some huge breakthrough in statistical techniques then we might talk about revolution, but all I see is some optimisation techniques that aren't new and are liable to lead to incorrect results.
Likewise, if someone had devised a technique to query operational data in real-time with little or no impact on operational systems then I might be impressed. Very useful, but not really revolutionary.