The emergence of NoSQL and convergence with relational databases
This guest post comes from Matt Allen, Senior Product Marketing Manager, MarkLogic
For over two decades, Oracle, IBM, and Microsoft relational databases were the only consistent leaders in the Gartner Magic Quadrant for Operational Database Management Systems--and there were few others to compete.
But as data structures become more complex and data volume increases, traditional relational databases are no longer the only game in town - and that includes in the Enterprise. Organizations are adopting new operational NoSQL databases for better data integration. As such, the leader's quadrant includes 10 companies, half of which are NoSQL-focused companies.
Today's data has a broader profile, in terms of shape and size, than before: It is big, fast, complex, and changing. At most organizations, the data is spread around a complex web of applications, documents, and ETL scripts. This is a major driver of disruption, and presents a danger of silos, conflicting data and increased complexity.
Data silos can cost organizations money, slow them down, and lead projects to failure. Today, 50 to 80 percent of data scientists' time is just spent wrangling data. And, 40 percent of the cost of information systems is due to data integration problems [M. L. Brodie and J. T. Liu. "The power and limits of relational technology in the age of information ecosystems."]. This only compounds the other related challenges of privacy and security and the need to migrate to the cloud.
Although relational database schemas can be molded, eventually the schema is set and inflexible to change. On the other hand, NoSQL document databases are schema-agnostic. They can handle data variety in the form of JSON or XML documents, and schemas can evolve as data changes over time.
NoSQL vendors have been eating into the relational database market, helping organizations shed silos over time. It is not a fast process for organizations to transition out of their trusted databases. There have to be several requirements checked off the list, including whether the NoSQL solution can match up to enterprise rigor. Most NoSQL databases are good at offering schema-flexibility and scale-out but other enterprise features such as security and transactional capabilities are only on a roadmap.
If you look at many use cases for NoSQL databases, you see non-mission-critical stuff, lots of analytics, rather than transactional systems.
Featured
But there are counterexamples. For instance, MarkLogic worked with a well-established healthcare company supporting a massive data integration undertaking. The project was estimated to take over 40,000 hours of development using traditional relational database technology. However, the company chose to use NoSQL for a data integration project involving over 140 human resources-related data feeds, consisting primarily of complex, structured data such as payroll data, employee evaluations, promotions, benefits data, and more.
With our NoSQL product, the company successfully completed the project and put it into production in under a year. The new system handles the complex data ingestion and now relies on enterprise NoSQL technology at a reduced cost compared to their previous system that involved a myriad data silos.
The line between NoSQL and relational will continue to blur as relational database vendors work to incorporate NoSQL into their mix of products. And NoSQL vendors are trying to make their product more like traditional relational databases.
The convergence of the two, while disruptive, has spurred an evolution in today's market. We are establishing a new level of focus to meet the requirements of big data and a larger move to digital approaches of doing business.
In turn, the operational database of choice in the future will likely be one that provides the best of both worlds--a multi-model, operational database that provides a flexible data model and enterprise reliability.
Many leading organizations recognize the arrival of NoSQL, have already adopted the multi-model approach and are seeing the benefits of more options. As Gartner states, "By 2017, all leading operational DBMSs will offer multiple data models, relational and NoSQL, in a single platform."