Big data and transaction processing

Organizations need to compile and hold larger amounts of data. Many big data tools are useful when the goal is discovery and analysis of that data. What needs to be done if a transactional system needs access to a huge, rapidly changing data store.

After enjoying a lovely conversation with the folks at NuoDB and publishing profile of one of the developers using their technology (see NuoDB developer profile for more information), I started getting messages from other suppliers addressing similar industry issues. ScaleBase reached out and informed me that it is addressing the very same issues using a different, more traditional, set of tools

What is the problem ScaleBase is addressing?

Organizations are increasingly needing to collect large and complex stores of rapidly changing data as a way to better understand their customers, understand what their competitors are doing and to comply with regulations. This puts a great deal of stress on more traditional relational and non-relational database engines. Relational database engines simply weren't designed to address the rapid growth in the size of the data store, the many different types of data that must be analyzed, or the number of people needing access to the data simultaneously.

The traditional approach of meant that the organization got the fun of riding an IT roller coaster, that is over provisioning to address future needs, having enough capacity and then having the application overshoot the current configuration and facing the need to buy more capacity. This, of course, also meant that meeting service level objectives was challenging.

This has lead some organizations to consider how to expand beyond the capabilities of traditional clustered database solutions and consider other approaches.

How does ScaleBase support transactional applications in a big data environment?

While NuoDB is an advocate of adding a relational, ACID compliant front end to a Big Data engine, ScaleBase suggests that intelligent middleware combined with a large number of servers supporting more traditional relational database engines will do the trick as well.

How ScaleBase describes what it is doing

  • Employ complex algorithms that analyzes relations, structure and size of data
  • Utilize parallel (map reduce) execution to reconstruct a single result from multiple databases
  • Seamlessly integrates with customer environment - 100% compatible with standard database protocol
  • Policy based command routing to maintain data distribution and achieve high availability
  • Provides a single point of management for a complex, distributed database environment
  • Delivered as simple, easy-to-use, software that runs on commodity hardware - requiring NO changes to existing infrastructure

Snapshot analysis

It is alwas interesting to watch the process of innovation. As Big Data concepts are applied by more and more organizations, it was inevitable that some would want to use huge and rapidly changing data stores to support transactional systems. Unfortunately, most Big Data, NoSQL, tools are designed to address the needs of discovery and analysis rather than supporting ACID compliant transactional applications.

  NuoDB says the answer is building a new database engine, one that combines the best of Big Data tools and more traditional database engines.

ScaleBase disagrees and suggests if the problem is segmented into many smaller pieces, traditional database engines still have plenty of life left.

Which is right? It all depends upon the application in question. What's exciting is innovative solutions are being offered to address the market's needs.