A search engine for system admins

While roaming the exhibit hall at LinuxWorld, I ran into Michael Baum of Splunk. A serial entrepreneur and former e-commerce exec at Yahoo and Infoseek, Baum has spent nearly three years developing a search engine for log files and machine-generated data. Splunk classifies and indexes log data from whatever devices and applications, and then analyzes the collection of events with its secret sauce--proprietary, real-time pattern analysis that dynamically reverse engineers event relationships--which makes troubleshooting systems easier than  with traditional systems management applications.

"We figure out relationships," Baum told me. "We studied how expert administrators made correlations in their heads. It turns out that we could automate a bunch of things in software, as well as do things that humans couldn't do." 

Baum wants to spread the word with a Splunk personal server, which is available for free in beta. A for-pay Splunk server capable of handling terabytes of data per day, clustering and with more advanced security will be available later this year, Baum said. Like every good citizen participating in the open source ecosystem , Splunk also has an open source community, SplunkForge, and a few projects in the works.