Data integration issues still impede digital progress, survey shows

The foundation of a digitally savvy enterprise is data, and lots of it, coming from all corners, and having the right data at the right time being digested and turned into insights that guide both humans and machines to make the right things happen. But organizations are still fumbling in their efforts to bring it all together. Cloud services help -- but only go so far, and may even complicate things even more. 

That's the word from a recent survey of 1,400 executives released by Progress, which finds data integration to be the number-one challenge to enterprises seeking to expand their digital repertoire. Close to half of respondents pinpoint ever-increasing disparate data sources as a major pain point. Another 44 percent of respondents are worried about integrating cloud data with on-premises data. 

While Progress, a provider of data integration tools, has an obvious stake in reporting these results, the findings point to a pain point that doesn't get addressed enough amidst all the talk of the glorious digital and AI revolution. Sort of akin to building the first automobiles without figuring out ways to refine gasoline from oil sludge. 

Data is streaming in from an ever-growing number of "data sources, hybrid environments, constantly evolving APIs and new, disruptive data types," the report's authors state. Each data source "has a unique set of APIs, and 47 percent of survey respondents pointed to integrating all these sources as their most challenging task. Another 44 percent agree that the biggest challenge is incorporating all relevant data across an ever-increasing number of cloud, database with on-premises database. The rapid advancement of social media and IoT contribute greatly to soaring volume of data circulating in the networks, especially with the rise number of connected devices. And 35 percent are worried about the volume of data they're trying to handle." 

Greatest Data Integration Challenges 

• Data spread across an increasing number of data sources   47% 
• Integrating cloud data with on-premises data   44% 
• Data veracity - data inconsistency, data uncertainty, ambiguous data, incomplete data etc.   36% 
• Data volume - big data, IoT, social media, enterprise data    35% 
• Data velocity - batch, near real-time, real-time, streaming etc.   32% 
• Data variety - structured, unstructured, semi-structured    31% 
  

So, what to do? Standardization offers the best hope, and there has been an increasing embrace of some key ones-- especially REST. Organizations are increasingly adopting standard SQL and REST data access standards for easier and faster connectivity to disparate data from BI, reporting and ETL tools, while ensuring interoperability and compatibility with existing systems, the survey's authors report. "While ODBC remains at the top of the SQL heap and is expected to gain further share, the move from SOAP to REST is significant. REST-specifically OData-is expected to see the most growth in adoption over the next two years." The survey report authors are especially keen on the potential of the OData (Open Data Protocol) standard, which defines a set of best practices for building and consuming RESTful APIs.  

Data Access Interfaces Used Now, and Planned for Adoption in the Next 2 Years

• ODBC   58%  (1%) 
• REST   54%   5%)
• JDBC    48%   (1%)
• SOAP   40%   (2%)
• ADO.NET  31%   (1%)
• Native Database Access   28%   (1%)
• XQuery   10%   (2%)
• XPath    10%   (1%)
• OData    8%   (3%)

The survey also tracked cloud adoption, finding the leader, Amazon Web Services, employed by 44 percent  (up 12 percent over the previous year's survey); Microsoft Azure, present at 39 percent of sites  (up seven percent) ; and VMware at 22 percent. Another growing cloud platform is Google Cloud, used by 18 percent and up three percent over the previous year.