Software as a Service and cloud are great for single-purpose projects, but can become very problematic when it comes to data integration.
My colleague Dave Linthicum, the go-to expert on both cloud and integration issues -- once put it very nicely: "The beauty -- and the downside -- of SaaS is that the businesspeople don't need IT to establish accounts and to get up and running. IT has less work to do in the short term. But without integration, SaaS silos spring up, resulting in duplicate data, inaccurate reports, and ultimately, damaging data discrepancies." He wrote this a few years back, and it still holds true today.
While IT gets bypassed, it ultimately, gets dragged back in to clean up the mess.
A new survey of 200 active Salesforce users, professionals and admins finds data integration pains haunt many of these deployments. The survey, released by Progress and conducted in conjunction with Dimensional Research, finds close to nine in 10 plan new Salesforce implementations over the next two years, but data integration is their greatest challenge. Sixty-three percent (63%) stated integration difficulties result in slower Salesforce performance.
A majority, 54 percent, say that application and data source integration is their most pressing challenge. These integration challenges stem from the need to share data between Salesforce and on-premises applications (48%), legacy applications (47%), connectivity (40%) and shared data sources (37%).
The survey report's authors indicate that Salesforce integrations are now "spidering" (that's a great way to put it) into a wide array of enterprise applications -- including support systems (46%), ERP (42%), and accounting (41%). Legacy applications account for 38 percent of all integration requirements, the survey also shows. At least 12 percent of these organizations now have more than 10 applications and data sources connected to Salesforce.
The survey report didn't offer solutions to this challenge. But it's a given that Progress DataDirect, which sponsored the survey, along with a number of other vendors, offer solutions that help meld cloud-based data with on-prem enterprise data. As part of my work with Database Trends & Applications, I recently helped put together a "Best Practices" report on what it takes to achieve this integration from a management perspective, as summarized below:
There must be 50 ways to integrate your data. Numerous technologies and approaches are available, from master data management to data warehousing to data virtualization to Hadoop.
Not everything needs to be -- or should be -- integrated at once. An emerging approach, "data lakes," will position data in its raw form in one place, for later integration and processing as needed. Here, I quote from David Mariani, CEO of AtScale, in a panel I led at DBTA's recent Data Summit 2015 conference: "Data is like water. It's very expensive and difficult to move once it lands someplace." Today's data volumes have"grown beyond our ability to pre-process it or to pre-structure it, to build structures to answer questions today."
New standards are emerging to help simplify cloud-based integration. The industry has been rallying around the Open Data Protocol (OData),which promises to replace the web services standards REST and SOAP to enable greater interoperability between enterprises and across the cloud. As OASIS,the standards body supporting OData,describes it, "OData metadata, a machine-readable description of the data model of the APIs, enables the creation of powerful generic client proxies and tools." The emerging data format "helps you focus on your business logic while building RESTful APIs without having to worry about the approaches to define request and response headers, status codes, HTTP methods,URL conventions, media types, payload formats and query options."
The cloud itself has become the ultimate data integration platform. A recent survey that I helped conduct for Unisphere Research in conjunction with the Independent Oracle Users Group (underwritten by Oracle) finds that a majority of enterprises, 57 percent, either are already employing cloud to manage their big data challenges or plan to within the next few years. Clouds and Platform as a Service (and Integration Platform as a Service) offerings are gaining traction as an online means to manage and process data. The cloud offers a major venue for big data solutions, since they are faster to deploy and easier to operate and scale than with typical on-premises systems.
(Disclosure: I have performed paid project work for DBTA and Oracle, mentioned in this post, in the last 12 months.)