Big Data: How close to enterprise-ready?

Moderated by Steve Ranger | October 1, 2012 -- 07:00 GMT (00:00 PDT)

Summary: Big Data technology can absolutely take Enterprise analytics to the next level....but when?

Lawrence Dignan

Lawrence Dignan

Will ramp quickly


Not even close

Andrew Brust

Andrew Brust

Best Argument: Not even close


Audience Favored: Not even close (56%)

The moderator has delivered a final verdict.

Opening Statements

Will ramp quickly

Larry Dignan: Big data is being piloted in large enterprise settings with Hadoop clusters, connections to data warehousing and other plumbing being hooked up. In other words, big data is already happening in the enterprise, but it is admittedly early. I'm betting that adoption will ramp quickly because enterprises---already drowning in data---will need big data to make sense of internal information flows. Everything from fraud detection to network maintenance to sensor data to customer service will be touched. Once real ROI is achieved---and I've seen a handful of business cases up close---every corporation will want on the bandwagon. The only real limitation will be talent.

In the end, enterprises will have a data fabric with Hadoop, analytics and data warehousing information. Many of those parts are already in place.

Not even close

Andrew Brust: Big data technology is exciting, innovative and genuinely powerful. It can absolutely take Enterprise analytics to the next level...but not yet.

In Global 1000 organizations, and numerous smaller companies, skill sets and best-practices have been building for years around Business Intelligence (BI) technology and for decades around relational database management systems (RDBMSes). The products in these categories have superior tooling, manageability and fault tolerance. They offer user interfaces designed for non-developers. They are repositories for carefully crafted data models, refined over the years, representing unparalleled investment.

Meanwhile, Hadoop is typically used at the command line, controlled by MapReduce code that must be written in Java, using a file system (HDFS) controlled by a single, vulnerable name node. Some browser-based tooling is emerging and technologies like Hive provide a primitive connection layer for BI tools, but we’re still at a 1990s-era level of sophistication. This stuff is not enterprise-ready yet. It’s not even close.


Log in or register to join the discussion
  • The low hanging fruit is driving fast adoption

    There are some fast wins (projects that are easy to impliment, low cost, small in scope, and will deliver business value greater than TCO) that will allow companies to say "We are also doing BigData" which seems to drive a lot of executive motivation (similar to cloud efforts in 2009). In the trenches I plan to see how we can virtualize the Hadoop products and minimize or replace parts of Mapreduce that are Single Points of Failure with either anternative solutions or with process automation to support an agile release of Hadoop functionality.

    I see it as more of us starting now so we have a running start as we learn about the pain points we discover as we race down to the valley of disalusionment and start up the next hill.
    Reply Vote I'm for Will ramp quickly
  • If it's not simple to understand, simple to implement, and simple to use,

    then, it will be an exercise in futility.

    If the tools aren't there, and companies and developers aren't adopting big data in large and small companies, then, it makes it even worse. SQL works are many different levels, and thus, we'll have SQL serving our basic and big needs for a long time to come, without having to adopt big data. SQL itself is capable of doing "big data", and, until somebody can make big data something that companies, big and small, want to adopt, and until a massive number of developers everywhere want to involved, then, big data will remain a curiosity that, those that absolutely need it, will adopt and use.

    Perhaps the marketing is all wrong, and "big data" sounds like something useful to only those that generate humongous amounts of data. Most companies don't generate "big" data, and many of those that do, can manage with what's already available, namely, SQL and flat data management.
    Reply Vote I'm Undecided
  • Slow uptake is an understatement

    Cool isn't always cost effective! the mid-sized businesses i have worked for recently are just starting to build their Data Warehouses using less complex solutions and in-house know how coupled with standard database technology. Big-data only makes sense for gigantic companies with nearly unmanageable amounts of data. It doesn't make sense for small-mid-sized business(the meat and potatoes of IT shops) to adopt big-data technologies/strategies.
    Reply Vote I'm for Not even close
  • Not sure I really see the benefits . . .

    Not sure I really see the benefits - data mine huge amounts of data in hopes that you might find something special?

    If you find something special, great. But if you don't - haven't you just thrown out millions of dollars?

    Frankly, it sounds to me more like large scale gambling than a real business plan.
    Reply Vote I'm for Not even close
  • Big data needs a strategic approach

    Big data, and big data management, have existed for some time in the form of massive numbers of transactions and huge amounts of data. For example, a typical digital media retailer now manages hundreds of suppliers and customers across tens, if not hundreds, of countries.

    Many firms over recent years have gained strategic advantage through customer data and CRM, processing millions of transactional data points with near real-time associated details linked to spending habits and behaviours. Big data of course will continue to become more complex with multi-channel, social media and mobile being added to the mix; but this does not need to be an overwhelming challenge if the board recognises this and implements a robust and transparent data management strategy.
    T Atkins - Microgen
    Reply Vote I'm for Will ramp quickly
  • just getting started

    While the technology stack is new, Big Data (and in particular machine learning algorithms) will automate some BI work out of existence.

    For the DBAs above that don't yet get it, your relational SQL database can't scale to terra/petabytes without a lot of pain and $$$. Hadoop, BigQuery, Drill, etc can and do. You can run a very capeable 8 node Hadoop cluster at AWS for < $1K a month (and easily process terrabytes per day).

    You might ask who has that data much data? I would counter anyone who has weblogs, text, or large tables across systems. The easy winds are in data we are already collecting but throwing away. We (as technologists) need to get away from the mindset that we don't have the space or processing capacity to interrogate what we are now throwing away. We do.

    Even better is the ability to deal with complexity, being able to throw hardware at complex joins that bring DWs to their knees is life changing. Right now I am doing million/million row full table scans in minutes and for spare change (EMR/S3).

    And it is not just SQL. Mahout and R give you the ability to create ML algorighms for predicting and driving real time system decisions. And not stuck in some DW somewhere!

    When I told our DW/BI team I had every action of our 50K daily users in our application, and I had the last few years of actions mapped to outcomes they just didn't get it. I have that data raw (not stripped down summary data), and can move it in and out of my cluster as needed. I can create models where I look at things happening in real time, and then based on what I have seen in the past try to steer things in the right direction.

    That, is where we are headed.
    Reply Vote I'm for Will ramp quickly
  • Uhm what?

    It kind of sounds like these guys are almost saying the same thing in all these questions. Is it just me?
    Chris Schrader
    Reply Vote I'm for Not even close
  • I didn't learn anything from Lawrence's arguments

    NoSQL + Hadoop + Datawarehouse = ? That's just the storage and the framework. With this your data will just sit there. You still need data processing and analytics tools.

    Andrew's ones make sense and are consistent with the questions asked. Thanks for the dabate.
    Reply Vote I'm for Not even close
  • Larry is Always Right

    Voting on a concept that is best understood by the top analyst at ZDnet is like asking children to verify their father's calculus.

    You don't have to understand it for it to be true.
    Reply Vote I'm for Will ramp quickly
  • Big data might be ready, but enterprises are not

    The technology is there, but that must first be backed by strategy and what you want to achieve. mobile_manny gives an excellent example of using big data for systems analysis, but big corporates would ultimitely want to use big data to better understand their customers.

    They will have many different data silos, different formats and standards. A strategy will have to be put in place to effectively use this data first, and that will take time, LOTS of time.

    So although the technology is there and ready, big corporations are not ready yet.
    Reply Vote I'm for Not even close