Open source company MongoDB wants to make sure that the type of database you chose doesn't get in they way of building the applications you need. ZDNet spoke to the company's co-founder and CTO Eliot Horowitz at the recent Big Data LDN conference in Olympia, London.
ZDNet: What was the original idea that led to MongoDB?
Horowitz: The decade before we started Mongo, Dwight [Merriman] and myself were building database products and we invariably had to work around databases. What actually happened was that we were thinking about some new application that we were going to build and we realised that we were going to have to work around database problems almost immediately. We started designing the database so that it would service the application.
SEE: 60 ways to get the most value from your big data initiatives (free PDF)
We quickly realised that the database was actually more interesting than the application that we were going to build and so we sat down and designed the database that we always wanted. Rather than build these four clustered databases for each application, we thought why don't we just build the database that we wished we had had for our entire careers.
And we went off and built it.
What attributes did the database need that you thought essential?
We thought it has to be based around documents. It has to be document-centric with all the good features of relational databases so it had to have a good query language, indexing and lots of other good things. We laid out a spec.
First, it had to be based around documents and not just tables and rows
Two, it had to be used to build good systems, so things like high availability and charting and so on, had to be features.
And three, it had to be open source and run everywhere. We didn't want to be stuck on a platform that was going nowhere. At the time the cloud hadn't got started but it was pretty obvious that it was going to happen.
The open source choice was absolutely deliberate?
Yes. It's my belief that there will never be another piece of systems software that isn't open source. Open source gives you freedom and it also gives you better software. People look at the software, find bugs and get patches and build a better solution.
So you were getting started and were up and running, what were the key moments in your progress?
The first published version of Mongo was in 2009. Then people started to pick up on it. And then somebody wrote this blog post about how they had ported their entire application to Mongo 0.8 and how great it was. And so people started picking it up pretty quickly from there.
And in 2010 we did our first Mongo event. We just thought, "Let's see if this is going to work" and we put together this one-day conference in San Francisco for 200 people. It sold out in 48 hours. And it was at that moment that I thought, "OK, I think we've got something here".
If you think about it, it takes a while to build a database. The entire first two years were about building enough of a database so that people would understand what we were trying to do. And people wanted a document database. People understand what documents do. We had got what we wanted but we thought, are we the crazy ones or are we just cutting edge?
You're up and running, what was the next step or next issue?
The next issue, and it was going to be an issue for a long time, was just scaling -- scaling the product and scaling the capabilities.
The thing that was interesting and, in fact so challenging, was that Mongo was very hot and very fast. It was such a novel concept and the difficult part was keeping up with the numbers. Adoption took off faster than we could handle, just because there was this reaction, "Oh my god, this is just what I wanted".
It seemed that everyone thought the same way, so scaling capabilities and the ability of the team to keep up became very challenging.
That's interesting - is such an issue today?
From a team standpoint we're still growing incredibly rapidly -- the number of products is increasing, the features are increasing. For us, a lot of what we focus on is just execution.
We get a lot of feedback from our customers, our users, and the focus has to be execution -- having the right people building the right stuff.
Who are your peak users in that area?
I can't mention many publicly, but Barclays is one, Amadeus is another. Obviously, there are many more that we can't mention by name.
As the systems that were being used got bigger and bigger, was is it a case of you driving it or are the users driving it for you?
It's always the user driving it. In some ways, our main goal is to get out of the way. Our role is that people who are building applications do not have to worry about the database. If they can do that then we have done our job. We're just there for whatever data they need, whatever volume they need. We just want to help them.
You see, that was my whole problem before Mongo. When ever I tried to build something interesting I couldn't because the database got in the way.
Could you go over some of your latest new products?
One of the big ones was transaction support -- ACID transactions. This is an interesting feature because most Mongo applications don't need it very often, but it makes people a lot more comfortable because there are cases where it is useful.
Another was mobile synchronisation, because if you're building a mobile app you want to keep a subset of our data so, for example, if you're writing a story when you're on a train it's in the background synchronising your data so if you lose connectivity on the train it still works, it still keeps going. So, for mobile it turned out to be a pretty big story.
We have also done a ton of stuff with the analytics -- that is one trend we see a lot of with the merging of transactional and analytical workloads. A lot of people don't want to have separate workloads for analytical and transactional. They want to be doing real-time analytical on their transactional systems.
SEE: Special report: The future of Everything as a Service (free PDF)
And that works because we have this feature called Workload Isolation. So, you can actually have one logical area where you dedicate servers to two different services, so you can have one server dedicated to analytics processing and another dedicated to transaction processing.
Then, [if] your business analyst goes across to your transaction systems and puts in some query that really hammers your database? It has no impact on your system.
But, at the same time, it's one logical cluster, so we handle all the replication so you've got this real safety net.
What are your ambitions going forward?
I think Mongo has changed the way developers look at databases because it's not just about storing data. There's a lot more analytics now, so there's lot more analytics in Mongo.
It's about going from just being a database to being more like a platform. And, whatever you need to do with it, it just works. So, in transactional use cases and analytics use cases, it's really a sort of one-stop-shop for most of your data needs.
It's back to that core concept that data should be easy to work with so that you can develop those applications, be they web application, a mobile application or an analytics application it should really, just work.
It all goes back to that core concept that data should be easy to work with so that you can just focus on building the application.
How about you?
I'm just a developer. I like meeting developers, I like focusing on applications. One of the great things for me is that I get to meet people from every industry. I go from a health care company to a social start-up and aero-industry company and so on. That's great fun.
PREVIOUS AND RELATED COVERAGE
AI-based facial recognition and fingerprinting are allowing new levels of authentication, but privacy advocates should be concerned.
At Oracle world the company introducing an option for autonomous database that allows customers to get an isolated, single-tenant Exadata rack in the Oracle Public Cloud.
Understanding more about diabetes is key to treating and preventing the disease. Germany's DZD is using graph database technology from Neo4j to learn more about the illness.
When a national fingerprinting company joins forces with a startup that authenticates identity using AI-based facial recognition and behavioral prediction in natural settings, the future of human identification tech starts to look an awful lot like sci fi.
MongoDB wants to make companies building cloud services from open source projects open up the infrastructure for those projects. Could it work and should we want it to?
InfoSight, which gathers operational intelligence from infrastructure to offer predictions and suggestions, was previously only available on storage.
Analytics startup, EDO's founders believe they can show the causal relationship of what a consumer watches on TV and what they then do in search, Wikipedia lookups, and other forms of brand-related activity.
This is personal.