Transformation, agility: are expectations too high for SOA?

There's been a lot of discussion lately about SOA "failures," and already, statistics are showing up showing that this percentage or that percentage of companies consider their SOA projects to be failures. Expect to see plenty more of such surveys over the coming months and years.

What, exactly, "failed" with these SOA projects? Were the expectations for SOA set too high among these companies? In many cases, failure is regarded as the inability to achieve the business transformation, buy-in, and all that agility that SOA proponents and vendors had promised. 

Is it really fair to set the bar for SOA so high at such an early stage? After all, IT can only do so much; it's up to the rest of the business to help realize the loftier promises of SOA. And, as I've said before in this blogsite, many organizations are too hidebound to accept or support the transformations in corporate culture that are part of SOA -- even though these are the organizations that need SOA the most.

Perhaps a more realistic assessment for SOA is that it delivers benefits in a step-by-step fashion, starting with the smaller, nitty-gritty integration projects. First, there's relatively straightforward internal integration projects, such as establishing a service that can call out data from multiple back-end mainframes. Then, there are efforts to integrate a business partner process with your own. Finally, after some time with this kind of work, the business can start to look at SOA as a catalyst for greater transformation. 

Last week, I had the opportunity to host an ebizQ Webinar in which Forrester's Mike Gilpin and Skyway Software's Jared Rodriguez talked about what constitutes success for SOA, and what common elements they've seen in successful SOA projects. (A writeup is also here at my SOA in Action blogsite.)

Gilpin said that new data confirms what many SOA proponents in the trenches have probably suspected for some time: SOA, at least at this stage, is still mostly about integrating two or more applications. Gilpin cited Forrester Research survey data that shows about 62% of organizations of all sizes are looking into SOA strategies, up from 53% a year ago, adding that most are not actual implementations yet, but "mostly in the 'will pursue' and the 'enterprise commitment' categories."

Forrester also found that "integration is the dominant goal of SOA," Gilpin related, adding that at least "83% to 85% of organizations are focused on internal integration to drive their usage of SOA." 

Most SOA initiatives initially begin as integration projects,he related. "One of the key ways that SOA tends to come into organizations is as a way to deal with integration problems," he explained. Such segways evolve out of "trying to work with a business partner through an interface today that would tend to be Web services-based; integration with back end systems, ERP. CRM, or other systems; or more focused toward front-end activities, such as content management or Web self service; or multi-channel integration of user experiences across Web, call center, email, devices, interactive voice response."

Nevertheless, there is also some movement afoot to focus on strategic business transformation as a downstream goal of SOA, he added. But this comes later, with experience. "There’s a tendency in the SOA lifecycle adoption to start with internal integration, then move to external integration with business partners after the security and standards have matured. Then, ultimately to move to strategic business transformation as the more strategic enterprise-wide approach to SOA."

Typically, Gilpin added, "each of those individual technology areas tended to be addressed from an integration perspective in a different unique stovepiped way. You might have one Web-to-host approach that would be used to integrate a customer front-end application to a front-end, and a different approach used to integrate with partners."

SOA brings all these efforts together, he said. "With SOA, you start by building services in the middle, which are a representation of your business in digital form."

We're still a long way off from licking the integration challenge, however, so it's likely SOA will remained anchored to many internal integration projects for some time to come. Another survey conducted by Forrester and released by Progress Software found, for example, that 80% of the 250 companies surveyed still manually change schemas as required. According to the findings, while these percentages will slightly decrease over the next two years, manual processes will continue to trump automation. Despite the popularity of these brute force approaches, an SOA-based approach is gaining ground quickly; while 44% use SOA today, 59% of respondents reported that they plan to use SOA for integration efforts over the next two years.