The human element is critical in applying big data

The human element is critical in applying big data

Summary: Often the focus of the industry is on the tools used in big data applications. Revolution Analytics believes that we need to remember the human element too.

TOPICS: Big Data

Revolution Analytics' Chief Community Officer, David Smith, came by to talk about big data, data science, and the important role the human element plays in successful applications of big data. I thought that was a given, but learned that some organizations are so wrapped up in the tools they're using they don't bring in the appropriate experts.

Many, Smith pointed out, focus on the tools (such as Hadoop), using the R language, MongoDB, MariaDB, Splunk, and a few others, rather than really understanding what data is available, what data is needed, and what role an expertise in computer science and applications, modeling, statistics, analytics, and math play. Organizations can easily find themselves having done quite a bit of work to analyze machine and operational data and still not have a clear understanding of what's going on.  Even worse, Smith said, they can believe that they understand what the data says and really have only proved out the old maxim "Garbage in/Garbage Out."

Smith says organizations need to realize that data science is built upon, but is different than, business analytics. Business analysts would take the results provided by a business system, such as a customer relationship management system, and make predictions about how a product will sell or what's important to customers. Data Scientists would collect data from other systems, such as retail point-of-sale data, weather data, or other information and then ask many different questions.

Smith pointed out that staff that focus on a single source of data often come to incorrect conclusions. Subject matter experts in the areas of information processing, mathematics, business, manufacturing, or other relevant areas of expertise are needed in the organization's line of business to determine what questions to ask, where to find data that shines a light on that question, and only then conduct analysis to learn about the story the numbers are telling.

Is this a painful study of the obvious? If your company is deploying big data solutions, does your team include a data scientist? Please comment and add your thoughts to the discussion.

Topic: Big Data


Daniel Kusnetzky, a reformed software engineer and product manager, founded Kusnetzky Group LLC in 2006. He's literally written the book on virtualization and often comments on cloud computing, mobility and systems software. In his spare time, he's also the managing partner of Lux Sonus LLC, an investment firm.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Data Analysis

    Data collection and analysis is not about blindly collecting data but about asking the proper questions. Most "big-data" hype gloss over this point, it is not the amount of data per se but does the data contain the information needed to answer relevant questions. If not, does the data exist somewhere or how can be collected as well as how is needed must be addressed.
    • Which is why...

      ...the demand for real live statisticians and analysts isn't likely to go down for the foreseeable future. Any fool can build a model, but it takes some work to produce and properly interpret a good one, even with modern tools.
      John L. Ries