Lawrence Dignan
Will ramp quickly
or
Not even close
Andrew Brust
Best Argument: Not even close
Audience Favored: Not even close (56%)
The moderator has delivered a final verdict.
Opening Statements
Will ramp quickly
Larry Dignan: Big data is being piloted in large enterprise settings with Hadoop clusters, connections to data warehousing and other plumbing being hooked up. In other words, big data is already happening in the enterprise, but it is admittedly early. I'm betting that adoption will ramp quickly because enterprises---already drowning in data---will need big data to make sense of internal information flows. Everything from fraud detection to network maintenance to sensor data to customer service will be touched. Once real ROI is achieved---and I've seen a handful of business cases up close---every corporation will want on the bandwagon. The only real limitation will be talent.In the end, enterprises will have a data fabric with Hadoop, analytics and data warehousing information. Many of those parts are already in place.
Not even close
Andrew Brust: Big data technology is exciting, innovative and genuinely powerful. It can absolutely take Enterprise analytics to the next level...but not yet.In Global 1000 organizations, and numerous smaller companies, skill sets and best-practices have been building for years around Business Intelligence (BI) technology and for decades around relational database management systems (RDBMSes). The products in these categories have superior tooling, manageability and fault tolerance. They offer user interfaces designed for non-developers. They are repositories for carefully crafted data models, refined over the years, representing unparalleled investment.
Meanwhile, Hadoop is typically used at the command line, controlled by MapReduce code that must be written in Java, using a file system (HDFS) controlled by a single, vulnerable name node. Some browser-based tooling is emerging and technologies like Hive provide a primitive connection layer for BI tools, but we’re still at a 1990s-era level of sophistication. This stuff is not enterprise-ready yet. It’s not even close.
Talkback
The low hanging fruit is driving fast adoption
I see it as more of us starting now so we have a running start as we learn about the pain points we discover as we race down to the valley of disalusionment and start up the next hill.
If it's not simple to understand, simple to implement, and simple to use,
If the tools aren't there, and companies and developers aren't adopting big data in large and small companies, then, it makes it even worse. SQL works are many different levels, and thus, we'll have SQL serving our basic and big needs for a long time to come, without having to adopt big data. SQL itself is capable of doing "big data", and, until somebody can make big data something that companies, big and small, want to adopt, and until a massive number of developers everywhere want to involved, then, big data will remain a curiosity that, those that absolutely need it, will adopt and use.
Perhaps the marketing is all wrong, and "big data" sounds like something useful to only those that generate humongous amounts of data. Most companies don't generate "big" data, and many of those that do, can manage with what's already available, namely, SQL and flat data management.
Slow uptake is an understatement
Not sure I really see the benefits . . .
If you find something special, great. But if you don't - haven't you just thrown out millions of dollars?
Frankly, it sounds to me more like large scale gambling than a real business plan.
Big data needs a strategic approach
Many firms over recent years have gained strategic advantage through customer data and CRM, processing millions of transactional data points with near real-time associated details linked to spending habits and behaviours. Big data of course will continue to become more complex with multi-channel, social media and mobile being added to the mix; but this does not need to be an overwhelming challenge if the board recognises this and implements a robust and transparent data management strategy.
just getting started
For the DBAs above that don't yet get it, your relational SQL database can't scale to terra/petabytes without a lot of pain and $$$. Hadoop, BigQuery, Drill, etc can and do. You can run a very capeable 8 node Hadoop cluster at AWS for < $1K a month (and easily process terrabytes per day).
You might ask who has that data much data? I would counter anyone who has weblogs, text, or large tables across systems. The easy winds are in data we are already collecting but throwing away. We (as technologists) need to get away from the mindset that we don't have the space or processing capacity to interrogate what we are now throwing away. We do.
Even better is the ability to deal with complexity, being able to throw hardware at complex joins that bring DWs to their knees is life changing. Right now I am doing million/million row full table scans in minutes and for spare change (EMR/S3).
And it is not just SQL. Mahout and R give you the ability to create ML algorighms for predicting and driving real time system decisions. And not stuck in some DW somewhere!
When I told our DW/BI team I had every action of our 50K daily users in our application, and I had the last few years of actions mapped to outcomes they just didn't get it. I have that data raw (not stripped down summary data), and can move it in and out of my cluster as needed. I can create models where I look at things happening in real time, and then based on what I have seen in the past try to steer things in the right direction.
That, is where we are headed.
Uhm what?
I didn't learn anything from Lawrence's arguments
Andrew's ones make sense and are consistent with the questions asked. Thanks for the dabate.
Larry is Always Right
You don't have to understand it for it to be true.
Big data might be ready, but enterprises are not
They will have many different data silos, different formats and standards. A strategy will have to be put in place to effectively use this data first, and that will take time, LOTS of time.
So although the technology is there and ready, big corporations are not ready yet.