The good news is that 10-year-old Hadoop is maturing quickly. The bad news is that many companies are still struggling to get beyond pilot projects and support many applications on this new data-management platform. That's why I wrote my latest report, "Democratize Big Data: How to Bring Order and Accessibility to Data Lakes."
First, let's consider the industry leaders that are doing great things with data lakes, drawing on examples from the recent Hadoop Summit:
In financial services CapitalOne has been making waves this year, appearing at multiple events to talk about its Hadoop and Spark-based fraud-detection and its big data analytics, streaming, and security work.
In retail Macy's embraced Hadoop more than five years ago to power insights for Macys.com. Today it's doing more sophisticated cross-channel analysis, driving personalized promotions encouraging online customers to shop in stores and in-store customers to obtain out-of-stock and online-only items at Macys.com.
In manufacturing Ford relies on Hadoop for connected car capabilities. Ford does filtering and decision-making at the sensor and car level while uploading crucial data points for centralized insight and analysis. For example, FordPass app users can remotely check their car's fuel level, location, and diagnostic error codes, but detailed data used by service technicians remains in the car's black box.
In insurance Progressive has been a pioneer of usage-based pricing with Progressive SnapShot. The company has more than 15 billion miles' worth of driving data in a Hadoop-based data lake, but it can drill down and offer discounts to individual policy holders based on factors such as their total miles driven, nighttime driving, and speed and braking habits.
These examples are inspiring, but behind every breakthrough there's been a lot of hard work. And many fast followers are still struggling. Here's a recent sampling of criticisms I've heard from Hadoop users:
These and other comments led me to publish my latest research: "Democratize Big Data: How to Bring Order and Accessibility to Data Lakes." The report explores three areas where commercial vendors are filling gaps in the Hadoop stack: data management and governance, data cataloging and metadata management, and data discovery and self-service data prep.
These three gaps are being filled by two camps of vendors that are complementing what's available in Hadoop. Incumbent data integration vendors focusing on the data lake include IBM, Informatica, Oracle, Pentaho/Hitachi, SnapLogic, Syncsort, and Talend. Next-generation vendors that have emerged in the big data era include Alation, Collibra, Datameer, Podium Data, Paxata, Trifacta, Tamr, Waterline, and Zaloni.
Both camps are bringing automation and repeatability to data lake management and governance. They're also making the contents of the data lake more accessible and many are abstracting users from the complexities of manual coding in Pig, Hive, Spark, and other open source components.
As the report explains, a data lake is not a replacement for a conventional enterprise data warehouse, but many data-processing and data-analysis workloads are shifting to this new platform. The choice among incumbents and next-generation vendors depends on the specifics of your deployment. The report offers vendor-, category- and capability-specific descriptions and selection criteria as well as big-picture advice on setting your analytic direction. Here's a peek inside the table of contents:
Click here to download an excerpt of the report.
How is your organization filling the gaps in the Hadoop stack? Get in touch with me on the Constellation Research websiteor on Twitter @DHenschen.