Evolution of search in big data as told by LucidWorks

LucidWorks presents that the use of search in big data applications is evolving. It is no longer about sifting through mounds of data to discover what was previously hidden. It is now an information source of note and it is all about making data available searchable as soon as possible after its creation. Do you agree?

Paul Doscher, president and CEO, and Grant Ingersoll, CTO, both from LucidWorks, stopped by to speak about how their tools, which are based upon Apache Lucene/Solr and are now "cloud ready".

As often happens in such discussions, the conversation galloped through quite a bit of territory. Topics such as the Apache Software Foundation's role in making open-source software available and enterprise ready, the difference between offering raw and enterprise-ready technology, and the evolution of big data in the enterprise were all considered.

The Apache Software Foundation's role in open source

Doscher and Ingersoll made a point of praising the efforts of the Apache Software Foundation during the initial moments of our rambling conversation. They pointed out that the technology, not vendor, focus of the organization makes it possible for open source project groups to gather and rapidly build technology focused on new tasks, and make it available broadly.

They went on to point out the work being done on Lucene, a text search engine written entirely in Java, and Solr, an enterprise search server based upon the Lucene library.

Raw versus enterprise-ready technology

The conversation turned to discuss how LucidWorks and its Apache Software Foundation colleagues are doing their best to offer packaged, enterprise-ready software, rather than just creating another open-source project. This means offering usable examples, good documentation, and both representational state transfer (REST) and web application programming interfaces.

Doscher and Ingersoll discussed how LucidWorks' customers have been able to take LucidWorks Search and LucidWorks Big Data, and put them immediately to work because of the company's efforts in making the raw technology into enterprise-ready tools. In their view, these customers have seen the value of the open-source technology, and are pressing it into service.

I'm hoping to speak with one or more of their customers to present a customer profile here.

The evolution of big data and search

Doscher and Ingersoll then went on to discuss how enterprise search and big data have evolved from looking carefully into the past to tease out what insights and learning are available to becoming another important information source for companies.

The key challenge today is moving from rigid systems that gathered, organized, and analyzed data based upon previously known questions to dynamic systems that can immediately look at streams of rapidly changing data coming from many sources. Their hope is that today's systems can help analysts discover the right questions to ask, rather than just providing the answers to previously known questions.

Snapshot analysis

LucidWorks is one of a growing number of technology companies that are building products based upon open-source software that was created in products hosted by the Apache Software Foundation. It is fascinating how they are cooperating to build the basic technology and then focusing on different competitive market niches. Each time I mentioned what I thought was a competitor, the folks from LucidWorks pointed out that those companies are partners that are trying to use their individual strengths together to serve the market.

This is an area that is worth watching.