The art of the possible: Pervasive integration of enterprise systems and data arrives
Usually the most underappreciated, hard-to-deliver, and flat-out-difficult of strategic activities in any major IT project is the integration of multiple systems or sources of data together to help support a new unified business solution. Long a one-off activity that required talented people who could hand-develop robust point-to-point integrations -- overcoming thorny issues with data ownership, control, and licensing along the way -- the recently improving state of commercial integration stacks has steadily raised the bar on what's possible.
That art of the possible continues to improve more quickly than it has in quite a while. Today in the industry we're witnessing what is probably the final set of breakthroughs that will make rapid integration between the hundreds of systems that operate the business a straightforward and everyday task in most organizations.
Surprisingly, integration is actually a job too often delegated to business users themselves, who typically have to use many different systems to complete their tasks, often carefully feeding downstream apps exports of other systems, copying and pasting between different solutions, and keeping interim data stuffed away in unstructured repositories like Word documents, or worse, spreadsheets and online databases. The reality even today is that most organizations are silos of hundreds of systems and sources of data.
In my research, I find that average worker currently uses over two dozens applications a day to get their jobs done, and they remain the integration point for most of that work. The dream has always been to contextually connect the systems that they use for key processes and tasks, so that the current data in the right system of record is being referenced and pulled in automatically. The result are experiences or "microapps" that can quickly connect relevant systems together to support, for example, high value sales processes, project management activities, operational tasks, etc.. These integrated solutions are simpler to use, easier to carry out, less failure prone, and faster on which to train workers, embodying usage best practices and the right IT systems all in integrated concert together.
Better connected IT systems doesn't just help workers either, though that is where a lot of the efficiencies, cost reduction, and productivity gains of better integration can come from. The new generation of omnichannel user experiences, either customer or employee, requires much deeper integration across all systems. The long-vaunted -- and much sought-after -- 360 degree view of the customer, using every scrap of data the enterprise knows about them, is another laudable goal that's been held back by the slow, expensive, and difficult to sustain level of integration previously required across many systems, applications, and databases.
Enter the Mature Enterprise Stack
Featured
I've lived this reality for over two decades as I've worked and researched in solving integration issues. Integration of systems and data stubbornly remains the longest, most costly, and most hard-to-get-right activity when it comes to implementing technology-based business solutions. This despite enormous efforts to address it in the industry, from XML integration, Web services, and heavyweight service-oriented architectures (SOA) of over a decade ago, to much more lightweight, low-code, Web-oriented approaches today.
The good news is that we are now witnessing the rise of integration platforms that have the maturity level that can tackle enterprise-class real-world scenarios. One of the shortcomings of previous enterprise-scale integration platforms was that they often lacked enough understanding of an organization's existing systems and data. They had too few out-of-the-box integrations, they didn't know how to access multiple clouds, they had no common semantic vocabulary to make mapping between systems straightforward. And they were hard to use.
So while enterprise service buses (ESBs), enterprise mashups, and low code/no code integration solutions have most recently attempted address the market, they were relatively simple solutions themselves that often lacked an understanding of the great many existing enterprise systems that they had to integrate with.
Now it's common to see the leading integration platforms have pre-existing integration counts in the hundreds, which is critical when the average 100,000 person enterprise typically has over 2,000 applications that run the organization. Though there's certainly room for improvement, the critical mass in the number of ready-to-go integrations these platforms have today greatly speeds up the delivery process, as more and more otherwise expensive and time-consulting integration work can be pulled in off-the-shelf. This allows the core customer experience or employee experience to be much more rapidly connected together and crafted into new solutions across many disparate systems and user experience touchpoints.
In fact, I've argued recently that the lack of such a linchpin integration stack designed for high velocity has led organizations to continue fraught one-off integrations on an infrequent basis, rather than mechanizing integration into a more assembly line process to meet relatively high internal and external demand, especially as business apps continue to proliferate. (This in contrast to APIs and developers networks on the Internet, which have long been the common, standardized way to address high levels of integration.)
Thus, pretty much everyone I talk to in organizations today would like higher levels of integration in the growing number of systems they use, but it hasn't been readily achievable. This is now finally changing.
A Leading Example of Rapid, Smart Integration
This backstory was then the stage for my most recent exploration of one poster child of a contemporary enterprise integration platform, in this case the SAP Cloud Platform Integration Suite. I was able to get a much better understanding of this more state-of-the-art example of an enterprise integration platform (what some would call a integration-platform-as-a-service, or iPaaS) recently at SAP TechEd Barcelona. I caught up with Gunther Rothermel, SVP and Head of SAP Cloud Platform and SAP Leonardo for a discussion on the the possibilities that such an enterprise integration platform can realize, from faster business agility, more widely used integration services, and better business performance.
What was fascinating to me is that Rothermel and team are attempting to create a much more decentralized, and user-shaped model for created an integrated employee experience, which is one of the more significant trends in digital workplace right now.
The most germane part of our conversation is below, highlighting the wider level of usability, more personalization, and high level of integration of an employee experience that can be delivered with a modern enterprise integration platform:
Hinchcliffe: "So if I was looking at modernizing my digital workplace, how does this really differ from let's say, the traditional intranet, which allows you to bring all these little pieces together? It sounds like you're doing something a little bit more profound."
Rothermel: "What we have is a fully cloud-based, microservices-based approach. It's more modular than the classical internet approaches of the past. So there's a lot of flexibility in this concept. That's an important quality so that these digital workplaces can also grow in an incremental way. This bridging between line of business needs and IT from a tooling perspective matters a lot. So you have an authoring environment, sometimes I call this a mashup as well, where where you can really drive the adaptation and content in your own workplace as an employee [my emphasis]. So I feel there are some of these qualities make it different to the traditional internet approaches that we've seen in the past."
Hinchcliffe: "It sounds like the sweet spot is that [SAP Cloud Platform Integration Suite] can be used to to integrate systems and processes and data from all the way back-end to the front of the business. So marketing and sales and customer experience, through the whole value stream to the back end of the business. And to do so securely and manageable way."
Rothermel: "The flow from the front end to the back end, which for us at SAP does translate into a strong connection, of course, into our applications and systems. It also includes, in most cases, non-SAP systems as well. We've invested a lot in openness together with partners. We have this open connect initiative where we offer a access to a wide range of API's. So to be able to connect from front end to back end across the landscape of customers there's a lot of connectivity we have to provide. And this is what we what we have now been announcing."
Hinchcliffe: "So can IT administrators bring App Center in the into the play and can users can pick the applications from there and shape their digital workplace?"
Rothermel: "The way we envision this is that there is an IT-centric, developer-centric lifecycle where developers can produce widgets and cards as well as these artifacts which are part of the workplace. And this is posted to a gallery, and then people can pick it from the gallery also partners can add widgets into this gallery as well. So these two cycles of on the one side, with IT on the other side. The end users, then they use this content gallery. This is a new great way of collaborating between the business users and the IT teams."
Hinchcliffe: "So it sounds like with things like microservices [in the SAP Cloud Platform Integration Suite], that you can go a little bit deeper on the integrations. Not just superficial sitting next to each other on the screen [as in the traditional Intranet.] You can actually connect business processes and content together?
Rothermel: "That's really true. You need to integrate such a solution on many, many levels. Of course, there is a pure UI, rendering level kind of integration as well, where you where we also use upcoming technologies like Web components and upcoming standards. But the second is, of course, on the, I would say on the actual content of the workplace itself."
Below is the full 3 part podcast of my conversation this topic with Gunther Rothermel:
Enterprise Integration for Democratized Solution Assembly
Ultimately, the Holy Grail of integration remains being able to take systems and data and assemble them -- and not reinvent the wheel -- quickly, safely, and securely, with full lifecycle manageability of the resulting IT solutions to meet emerging and opportunistic stakeholder demand. Evidence is mounting that we are reaching a new level of industry maturity where we can actually do this consistently and well, with most integration scenarios requiring minimal new integration work. That means most new solutions can be build out of existing in-inventory integrations.
It helps that the newer integration solutions have recommendation engines to sort through the tough issues of field and schema mapping, choosing between different integrations and sources of data, and providing other assistance that reduces the complexity of what formerly used to be one of the most complicated and difficult, albiet highest value, activities in enterprise IT. I'll be following the developments in the space with great interest and look forward to talking with enterprises that are starting to use these next generation of integration platforms.
Note: SAP covered my travel to and from TechEd Barcelona for the purposes of this interview.