Middleware is cool again (just ask eBay)

A few years back, 'middleware' was proclaimed dead. But like an old industrial town that gets reinvented into a trendy tourist destination, middleware is the place to see and be seen.

The challenge is being able to tap into multiple data silos across the enterprise and bring the information into an abstracted layer that is independent of any underlying systems. Many companies are finding the best approach is to build an abstracted information based on the fusion of SOA and enterprise information management.

However, as I noted in an article just published in Database Trends & Applications, there is little consensus as to what this middleware should look like, or what technologies will power it.  Or even what it should be called -- variations run from "information layer," "integration tier," "enterprise data management" to "enterprise data fabric." Some say SOA is the way to go, while others look to data extract, transfer, and load approaches -- enterprise information management (EIM).

In a survey I recently authored for the Independent Oracle Users Group (IOUG), (in conjunction with Unisphere Media) we found that 89 percent of enterprise data managers consider some form of middleware essential to their efforts going forward.

eBay is one company that has found it profitable and manageable to abstract enterprise information from underlying systems, and boy, do they need it. James Barrese, vice president of systems development at eBay, told me that his company stores more than two petabytes of data; or 200 times the size of the Library of Congress. He added that eBay is made up of more than six million lines of code and the company rolls out more than 100,000 lines of new code every two weeks. In addition, there are 30,000 software builds per week.

To address this complexity, Barrese said, "eBay has built a software-based Integration Tier.  This contains both a data access layer (DAL) and a services framework.  The Integration Tier acts as an abstraction layer for software engineers to work with many disparate back-end data sources through a consistent set of abstractions."

Does SOA play a role in this infrastructure? Yes, Barrese said. "SOA is part of the base infrastructure," he explained. "We leverage both component-oriented and service-oriented architecture technologies. eBay has built a service architecture and uses it to enable integration across disparate technology stacks. For example, we have enabled open interoperation between our C++ and Java technologies via services. "

SOA has also played a role in integrating new functionality from PayPal, Barrese added. "Additionally, we see that by accelerating the integration of acquisitions such as PayPal into our platform, service-oriented approaches allow us to realize significant benefits in reduced cost and faster time to market."

SOA and EIM go well together. SOA is opening the door to the first information layer truly detached from underlying systems. Enterprise information management enables organizations to get at the right information at the right time. By being able to sift and sort through data, decision makers can predict new product demand, monitor inventory flows, and spot potential fraud.

This combined application and data architecture "creates the secret sauce for extreme scalability" for eBay, Barrese said.  "The integration tier is the keystone in this solution.  This allows us to efficiently scale and manage our back-end resources." Plus, he added, "Our abstraction layers make it easier to train new engineers, constantly improve the availability of the site and make live data architecture changes with no code changes and no downtime."