Big Data: How close to enterprise-ready?

Moderated by Steve Ranger | October 1, 2012 -- 07:00 GMT (00:00 PDT)

Summary: Big Data technology can absolutely take Enterprise analytics to the next level....but when?

Lawrence Dignan

Lawrence Dignan

Will ramp quickly

or

Not even close

Andrew Brust

Andrew Brust

Best Argument: Not even close

44%
56%

Audience Favored: Not even close (56%)

The Rebuttal

  • Great Debate Moderator

    Mic check

    Are my debaters standing by? We'll be starting promptly at 11am ET / 8am PT

    Posted by Steve Ranger

    Ready to go

    Lawrence Dignan

    I am for Will ramp quickly

    Bring it on!

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    The big fix

    Companies have always struggled to turn data into information. Do you think Big Data can really fix this now?

    Posted by Steve Ranger

    I think Big Data can help but it's not a cure all.

    Enterprises will always be plagued by the following questions. What do I want to know? What do I do with the data? What are the returns and ultimate business impact? Big Data is another tool that enterprises will take for a spin sooner rather than later. Clueless companies and management will exist no matter what the tools are.

    Lawrence Dignan

    I am for Will ramp quickly

    Turning data into information has indeed been a struggle for some time...

    ... in many cases because of the politics and logistics of acquiring, sharing and cleansing data in the enterprise. Big Data won’t fix this any more than Business Intelligence did. Making data “bigger” only increases the surface area of the data to be governed and arguably makes its analysis more complex.

    One thing Big Data has on its side is a more flexible and agile approach to schema, allowing it to be defined at query/analysis time, thus removing some of the complexity and bureaucracy in curating the data. But the tooling for managing unstructured data is relatively immature, and data specialists in the enterprise are not conceptually accustomed to it.
    The long term potential for Big Data here is good, as it should shorten innovation cycles. But in the near term things just aren’t that actionable yet.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Big Data for all

    Is Big Data only for big companies? Or will every firm need a Big Data strategy?

    Posted by Steve Ranger

    I think every company---and employee---will have to become data centric.

    Use cases will be all over the place. A SMB that can monitor sentiment and tie to to growth will ultimately become a large enterprise. In some respects, smaller companies may be at an advantage. They will be more nimble and willing to experiment. Large companies just don't move that fast.

    Lawrence Dignan

    I am for Will ramp quickly

    Data can help everyone...

    If I’m a quick-serve entrepreneur owning five McDonald’s franchises in a mid-sized city, it’s not obvious how I could take advantage of Hadoop and MapReduce to get more customers and more visits. If I’m a large Web company, a major financial services firm, a manufacturing concern or a major retailer, with big, continuous streams of clickstream, market or sensor data, then the appeal of Big Data is much more straightforward. I do think that smaller businesses should begin developing their big data strategies now though. Even they will have substantial clickstream data if they’re online (and most are) and even brick-and-mortar operations can start to accumulate volumes of in-store video recordings (which can reveal shopping habits, store layout effectiveness and product affinities). Data can help everyone, and when you stop throwing it away it becomes Big Data. But mining it has to get easier or small and medium businesses won’t be able to make their move.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Is Big Data worth the effort?

    Where’s the ROI? How do you build a business case for Big Data?

    Posted by Steve Ranger

    The business case revolves around revenue, risk and costs.

    At this juncture, the Big Data pilots should probably start small with some underserved department who has a leader willing to make a few strategic bets. The business case today won't be the one tomorrow. I think Big Data as a service isn't too far away. When that happens the business case will be low risk.

    Lawrence Dignan

    I am for Will ramp quickly

    In the Internet world, Big Data can pay off in increased eyeballs, session lengths and corresponding monetization

    In the manufacturing world it can pay off in reduced or eliminated downtime of assembly lines (through predictive analytics on equipment breakdowns). In the Financial Services world, Big Data can lead to better, more effective, and therefore more lucrative, trading strategies. Media companies can sell more ad impressions. E-Commerce concerns can sell more product. But each of these companies has something in common that the average Enterprise business unit may not have: the ROI for them is demonstrable enough and big enough to get them over the barriers to entry. Will the line-of-business teams in the Enterprise have the smarts, the budget, or the appeal to bring in the Hadoop specialists, statisticians and data scientists necessary to attain compelling ROI? Probably not. Big Data value through off-the-shelf products and professional services has to get good enough, cheap enough and mature enough for these customers to buy in.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Putting Big Data to work

    What are the essential Big Data technologies?

    Posted by Steve Ranger

    Essential technologies

    Essential technologies to me include NoSQL, Hadoop and your friendly neighborhood data warehouse today. Unstructured databases are likely to complement your relational ones. It will be a fabric of things melding Big Data, data warehousing and analytics.

    Lawrence Dignan

    I am for Will ramp quickly

    Top of the list

    Hadoop, and some of its companion components, like Hive and Pig, are top of the list. For those that can afford them, Massively Parallel Processing (MPP) data warehouse appliances, which use in-memory and column-store technology to deal with Big Data workloads, are all but required as well. General purposes analytics tools like “R” (and its commercial competitors from SAS and IBM/SPSS) should be on the shopping list too. True predictive analytics technologies, including the Mahout machine learning tool that works on top of Hadoop, or more commercial Data Mining products, are excellent, provided you have the talent available to use them.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Developing the best strategy

    How does Big Data tie into an organization’s broader IT strategy? Does it stand alone or will it force companies to rethink their architecture?

    Posted by Steve Ranger

    More of a business strategy

    The model I'm seeing is that big data is more of a business strategy that happens to have IT. Shared services for analytics/big data are common at large enterprises. There will be a rethinking of architecture, but that's a few years away.

    Lawrence Dignan

    I am for Will ramp quickly

    Big Data definitely has the potential to transform organizations’ overall IT strategy.

    That’s because Big Data is about more than Big Data per se. For example, Hadoop’s use of direct-attached storage and commodity hardware is very disruptive to the common enterprise deployment of storage networks and expensive servers and appliances. Hadoop may also cause enterprises to emphasize Java skills more and SQL skills less, among other shifts in skillset priorities. The clustering approach used by Hadoop may accelerate adoption of hybrid on-premise and cloud strategies too: it’s easier to push data to on-premise servers, but the elastic nature of the cloud may be more effective in addressing intermittent demand for extremely large clusters.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Training camp

    What new skills will enterprises need to make Big Data work?

    Posted by Steve Ranger

    Data literacy

    The biggest skill will be data scientists (ask the questions, know the math, know how to use the answers). Database admins will be big data admins. The biggest skill will be data literacy on everyone's part.

    Lawrence Dignan

    I am for Will ramp quickly

    May be very hard to recruit people

    Math, statistics and data modeling skills are needed, and there’s a shortage of these. Universities are only now addressing this problem with degree programs in analytics and data science. Java programming skills, as I mentioned above, will be highly useful, even for jobs that are data-oriented and not developer positions. What may be most important though, and most difficult to find, are individuals who have these tech skills in combination with strong industrial domain expertise. That’s the winning formula, and it may be very hard to recruit people who fit with it.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Who's going to be first?

    Which industries are mostly likely to adopt Big Data? And are there industries where Big Data is going to be a waste of time and effort?

    Posted by Steve Ranger

    Today, financial services and healthcare are leading the big data charge.

    Retail will have to get some serious mojo out of big data. Can a retailer combine sentiment and social buzz with transactional data ultimately? Sure. I think all industries will be impacted because all use data. What will change is whether the data is used for supply chain, costs, revenue or customer service. That may vary by mature vs growing industry.

    Lawrence Dignan

    I am for Will ramp quickly

    Every organization has Big Data...

    Again, Internet-focused organizations, as well as media, financial services, online retail and manufacturing are the industries who may have the most to gain. Supply chain companies, be they purveyors of parts and components, or distributors, can certainly be added to the list. So too can healthcare, be it in the areas of research, hospital management or payer/insurance operations. Marketing organizations across industries can get great benefit from Big Data. I think every organization has Big Data…it’s just that some don’t monitor it, and many do not retain it. Those that can and do are in the best position to derive advantage from Big Data. Those that don’t have to evaluate the costs and benefits of changing their operations model in order to become data-driven.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Who's ready to take charge?

    Which vendors are most likely to become synonymous with Big Data?

    Posted by Steve Ranger

    I think the usual suspects---IBM, Oracle, SAP---are already talking the big data game.

    However, there are dozens of smaller vendors who have pushed the big data agenda. Cloudera and Splunk are notable, but there are companies like Opera Solutions. It's too early to tell really, but the field is open.

    Lawrence Dignan

    I am for Will ramp quickly

    Cloudera and IBM come to mind right away...

    ... as they have the best public profiles in the Big Data arena. Eventually I think most other big software providers (like Oracle, Microsoft and SAP), storage and networking companies (like EMC, NetApp and Cisco) and certainly BI companies (like MicroStrategy and SAS) will be identified as credible in the space.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Happy together

    What about the cloud and Big Data – do they play nicely together?

    Posted by Steve Ranger

    No choice

    Cloud and big data have to play together because not every enterprise is going to have the horsepower to crunch the information. Luckily, there are cloud options for that like AWS.

    Lawrence Dignan

    I am for Will ramp quickly

    This cuts both ways.

    The commodity hardware and add-more-as-needed clustering approach of Hadoop has huge affinity to the cloud computing model. In general, elasticity is a feature of both. On the other hand, upstream bandwidth is still a limiting factor for Big Data in the cloud – it’s much easier to stream new data and maintain cloud databases (including Hadoop Distributed File System files) than it is to migrate that data en masse and establish the databases in the first place. This is yet another area where things will change, and barriers will melt away…eventually.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Troubleshooting

    What are the potential pitfalls of Big Data projects?

    Posted by Steve Ranger

    Asking the wrong questions

    The biggest pitfall to me will be asking the wrong questions and making the wrong moves based on the data. I think companies are also going to think they are know-it-alls because they have all the data. Just because you have the data doesn't mean you know what to do with it.

    Lawrence Dignan

    I am for Will ramp quickly

    Data quality is probably the biggest.

    So too is the broader question of data governance. In both cases, the prevalence of unstructured data can make data integration difficult and curb success rates. As well, the still modest level of maturity in many Big Data technologies is a potential pitfall as well. Lots of companies are in the R&D phase with Big Data because of this. The technology will have to become much more accessible, and defensive project management skills more widespread, before Big Data can truly go mainstream.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    The need to know

    Does the CEO understand or want big data? Does it matter?

    Posted by Steve Ranger

    The CEO gets it because it's ultimately his or her arse on the line.

    CEOs are responsible for the bottom line, revenue and risk. Big data will look like a cure all in many respects. All CEOs will be talking the data game.

    Lawrence Dignan

    I am for Will ramp quickly

    Has to get easier

    I think lots of CEOs understand Big Data at a high level and therefore, yes, they do want it. But their management teams have to understand Big Data more granularly, and execute on Big Data initiatives. At the risk of being repetitive, I don’t think we’re there yet. Big Data has to get easier for non-specialists to partake in, and for managers to understand fully, before it becomes pervasive.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    The best person for the job

    Who is going to run Big Data projects – the CIO or CFO? Which is more likely to succeed?

    Posted by Steve Ranger

    Neither.

    The big data projects will be overseen by the business line executive. To really work, all C-levels will have to be on board. The big data is really a way of thinking, experimenting and letting the data complement human judgement.

    Lawrence Dignan

    I am for Will ramp quickly

    It’s an interesting question...

    ...because in many companies the Business Intelligence buying decision lies in the CFO suite. And if Big Data is the successor to BI, then it might stand to reason that CFOs will maintain this control. But Big Data project leaders will more likely come from IT and the line-of-business corners of the Enterprise. Being hands-on with the technology and being acquainted first-hand with the data are likely prerequisites for project success. And data for financials is relatively discrete too – maybe there are petabyte-scale general ledgers out there, but I haven’t come across them yet.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Fiive-year plan

    How does Big Data develop over the next five years? What will stop it being just another fad?

    Posted by Steve Ranger

    Big data is likely to develop like Linux.

    Right now the big data talk is everywhere and hype is at its peak. That'll die down and we'll move on to some other buzzword. However, in five to 10 years big data tools will be in every enterprise. We just won't call it out. Think about Linux. It made a big splash, then largely lost its buzz. The funny part is Linux is now in every data center.

    Lawrence Dignan

    I am for Will ramp quickly

    Definitely not a fad

    Big Data may be at the top of its hype cycle right now (or it might not be), but it’s definitely not a fad. In my experience, very little around data is. Whether it’s line-of-business app development and corresponding transactional database needs, or dimensional analysis, or the kind of predictive analytics and other insights to be gleaned from Big Data, we’re talking about useful, important technology.

    Typically, a new data technology starts out as innovative and ground-breaking, then becomes mainstream and mission-critical, and eventually commoditizes, but it doesn’t often just fizzle out and go away. There’s no question in my mind that Big Data is going to be big and mainstream in the Enterprise, in the future. That future may or may not be within the five-year time horizon, depending on whether Big Data can get past its fragmented, cottage industry phase in that time interval.

    Technology has to get very mature – even a bit boring – before it enjoys truly widespread Enterprise adoption and deployment. Big Data will get there, but it’s got to overcome several hurdles first.

    Andrew Brust

    I am for Not even close

  • Great Debate Moderator

    Great debate guys!

    Look for closing arguments from Andrew and Larry which will be posted on Wednesday. Voting closes at 2pm ET Thursday and soon after you'll see my final verdict. Thanks for joining us!

    Posted by Steve Ranger

Talkback

16 comments
Log in or register to join the discussion
  • The low hanging fruit is driving fast adoption

    There are some fast wins (projects that are easy to impliment, low cost, small in scope, and will deliver business value greater than TCO) that will allow companies to say "We are also doing BigData" which seems to drive a lot of executive motivation (similar to cloud efforts in 2009). In the trenches I plan to see how we can virtualize the Hadoop products and minimize or replace parts of Mapreduce that are Single Points of Failure with either anternative solutions or with process automation to support an agile release of Hadoop functionality.

    I see it as more of us starting now so we have a running start as we learn about the pain points we discover as we race down to the valley of disalusionment and start up the next hill.
    Datalas
    Reply Vote I'm for Will ramp quickly
  • If it's not simple to understand, simple to implement, and simple to use,

    then, it will be an exercise in futility.

    If the tools aren't there, and companies and developers aren't adopting big data in large and small companies, then, it makes it even worse. SQL works are many different levels, and thus, we'll have SQL serving our basic and big needs for a long time to come, without having to adopt big data. SQL itself is capable of doing "big data", and, until somebody can make big data something that companies, big and small, want to adopt, and until a massive number of developers everywhere want to involved, then, big data will remain a curiosity that, those that absolutely need it, will adopt and use.

    Perhaps the marketing is all wrong, and "big data" sounds like something useful to only those that generate humongous amounts of data. Most companies don't generate "big" data, and many of those that do, can manage with what's already available, namely, SQL and flat data management.
    adornoe
    Reply Vote I'm Undecided
  • Slow uptake is an understatement

    Cool isn't always cost effective! the mid-sized businesses i have worked for recently are just starting to build their Data Warehouses using less complex solutions and in-house know how coupled with standard database technology. Big-data only makes sense for gigantic companies with nearly unmanageable amounts of data. It doesn't make sense for small-mid-sized business(the meat and potatoes of IT shops) to adopt big-data technologies/strategies.
    ammohunt
    Reply Vote I'm for Not even close
  • Not sure I really see the benefits . . .

    Not sure I really see the benefits - data mine huge amounts of data in hopes that you might find something special?

    If you find something special, great. But if you don't - haven't you just thrown out millions of dollars?

    Frankly, it sounds to me more like large scale gambling than a real business plan.
    CobraA1
    Reply Vote I'm for Not even close
  • Big data needs a strategic approach

    Big data, and big data management, have existed for some time in the form of massive numbers of transactions and huge amounts of data. For example, a typical digital media retailer now manages hundreds of suppliers and customers across tens, if not hundreds, of countries.

    Many firms over recent years have gained strategic advantage through customer data and CRM, processing millions of transactional data points with near real-time associated details linked to spending habits and behaviours. Big data of course will continue to become more complex with multi-channel, social media and mobile being added to the mix; but this does not need to be an overwhelming challenge if the board recognises this and implements a robust and transparent data management strategy.
    T Atkins - Microgen
    Reply Vote I'm for Will ramp quickly
  • just getting started

    While the technology stack is new, Big Data (and in particular machine learning algorithms) will automate some BI work out of existence.

    For the DBAs above that don't yet get it, your relational SQL database can't scale to terra/petabytes without a lot of pain and $$$. Hadoop, BigQuery, Drill, etc can and do. You can run a very capeable 8 node Hadoop cluster at AWS for < $1K a month (and easily process terrabytes per day).

    You might ask who has that data much data? I would counter anyone who has weblogs, text, or large tables across systems. The easy winds are in data we are already collecting but throwing away. We (as technologists) need to get away from the mindset that we don't have the space or processing capacity to interrogate what we are now throwing away. We do.

    Even better is the ability to deal with complexity, being able to throw hardware at complex joins that bring DWs to their knees is life changing. Right now I am doing million/million row full table scans in minutes and for spare change (EMR/S3).

    And it is not just SQL. Mahout and R give you the ability to create ML algorighms for predicting and driving real time system decisions. And not stuck in some DW somewhere!

    When I told our DW/BI team I had every action of our 50K daily users in our application, and I had the last few years of actions mapped to outcomes they just didn't get it. I have that data raw (not stripped down summary data), and can move it in and out of my cluster as needed. I can create models where I look at things happening in real time, and then based on what I have seen in the past try to steer things in the right direction.

    That, is where we are headed.
    mobile_manny
    Reply Vote I'm for Will ramp quickly
  • Uhm what?

    It kind of sounds like these guys are almost saying the same thing in all these questions. Is it just me?
    Chris Schrader
    Reply Vote I'm for Not even close
  • I didn't learn anything from Lawrence's arguments

    NoSQL + Hadoop + Datawarehouse = ? That's just the storage and the framework. With this your data will just sit there. You still need data processing and analytics tools.

    Andrew's ones make sense and are consistent with the questions asked. Thanks for the dabate.
    RelaxWalk
    Reply Vote I'm for Not even close
  • Larry is Always Right

    Voting on a concept that is best understood by the top analyst at ZDnet is like asking children to verify their father's calculus.

    You don't have to understand it for it to be true.
    tomogden
    Reply Vote I'm for Will ramp quickly
  • Big data might be ready, but enterprises are not

    The technology is there, but that must first be backed by strategy and what you want to achieve. mobile_manny gives an excellent example of using big data for systems analysis, but big corporates would ultimitely want to use big data to better understand their customers.

    They will have many different data silos, different formats and standards. A strategy will have to be put in place to effectively use this data first, and that will take time, LOTS of time.

    So although the technology is there and ready, big corporations are not ready yet.
    nicopretorius
    Reply Vote I'm for Not even close