Data services provide catalyst for SOA, set stage for cloud-based data models

Read a full transcript of the discussion. Listen on iTunes/iPod. Sponsor: WSO2.

In the past, data was structured, secure and tightly controlled. The bad news is that the data was limited by the firewall of personnel, technologies, and process rigidity. Today, however, the demand is for just-in-time and inclusive data, moving away from a monolithic data system mentality to multiple sources of data that provide real-time inferences on consumers, activities, events, and transactions.

The move is in the ownership of data value to the very people who really need it, who help define its analysis, and who can best use it for business and consumption advantage. Analysis and productivity values rule the future of data as services.

But how to jibe the best of the old with the needs of the new? How to use data services as onramps to SOA? How to bring data together for federated analysis? And how to use the power of open source licenses and community to encourage the further federation of data as standardized consumable services?

To answer these questions and to learn more about the quickly evolving data services landscape, I recently spoke with Paul Fremantle, the chief technology officer at WSO2; Brad Svee, the IT manager of development at travel management leader Concur Technologies; and James Governor, a principal analyst and founder at RedMonk.

Here are some excerpts from the resulting podcast:

The live, connected world needs to be managed properly and it's very, very difficult to build that as a single monolithic system. ... The [new] model is of keeping the data where it belongs and yet making it available to the rest of the world.

Our data is trapped in these silos, where each department owns the data and there is a manual paper process to request a report. Requesting a customer report takes a long time, and what we have been able to do is try to expose that data through Web services using mashup type UI technology and data services to keep the data in the place that it belongs, without having a flat file flying between FTP servers, as you talked about, and start to show people data that they haven't seen before in an instant, consumable way.

It seems clear that the status quo is not sustainable. There is inherent risk in the current system, and simply retrofitting existing data and turning it on as a service is not sufficient.

We have been trying to free up our data, as well as rethink the way all our current systems are integrated. We are growing fairly rapidly and as we expand globally it is becoming more and more difficult to expose that data to the teams across the globe. So we have to jump in and rethink the complete architecture of our internal systems.

The browser becomes the ubiquitous consumption point for this data, and we are able to mash up the data, providing a view into several different systems. Before, that was not possible, and the additional piece of moving the file between financial system, for example, we are able to not have to pull files, but actually use Web services to send only the data that has changed, as opposed to a complete dump of the data, which really decreases our network bandwidth usage.

[Yet] most of the data that we are dealing with is fairly sensitive, and therefore almost all of it has a need for at least per-user access basis, as well as, when we are transporting data, we will have to make sure that it's encrypted or at least digitally signed.

What we have built is what we call WSO2 Data Services, which is a component of our application server. The WSO2 Data Services component allows you to take any data source that is accessible through JDBC, MySQL databases, Oracle databases, or DB2, but, in addition, we also have support for Excel, CSV files, and various other formats and very simply expose it as XML.

Now this isn't just exposed to, for example, Web Services. In fact, it can also be exposed by REST interfaces. It can be exposed through XML over HTTP, can even be exposed as JSON. JavaScript Object Notation (JSON) makes it very easy to build Ajax interfaces. It can also support it over JMS, and messaging system.

So the fundamental idea here is that the database can be exposed through a simple mapping file into multiple formats and multiple different protocols, without having to write new code or without having to build new systems to do that.

What we're really replacing is ... where you might take your database and build an object relational map and then you use multiple different programming toolkits -- one Web services toolkit, one REST toolkit, one JMS toolkit -- to then expose those objects. We take all that pain away, and say, "All you have to do is a single definition of what your data looks like in a very simple way, and then we can expose that to the rest of the world through multiple formats."

I think that open source is absolutely vital to making this work, because fundamentally what we're talking about is breaking down the barriers between different systems. As you say, every time you're pushing the proprietary software solution that isn't based on open standards, doesn't have open APIs, and doesn't have the ability to improve it and contribute back, you're putting in another barrier.

Everyone has woken up to this idea of collaboration through Web 2.0 websites, whether through Flickr or FaceParty or whatever. What the rest of the world is waking up to is what open-source developers have been discovering over the last five to 10 years. Open source is Web 2.0 for developers.

Open source drives this market of components, and it's exactly the same thing that happened in the PC market. As soon as there was an open buy off that wasn't owned by a single company, the world opened up to people being able to make those components, work in peace and harmony, and compete on a level playing field. That's exactly where the open-source market is today.

That's exactly the sweet part in my opinion. I can shake and bake, I can code up a bunch of stuff, I can prototype stuff rapidly, and then my boss can sleep well at night, when he knows that he can also buy some support, in case whatever I cook up doesn't quite come out of the oven. I see there's a kind of new model in open source that I think is going to be successful because of that.

Read a full transcript of the discussion. Listen on iTunes/iPod. Sponsor: WSO2.