Government agencies, which are typically skittish of new technologies, are embracing the Hadoop open-source data analysis tool popular among the web cognoscenti.
Hadoop, which combines a flexible filesystem with a variety of data analysis tools in an open-source software package, is growing its user base beyond Twitter, Facebook, Yahoo and the other web darlings, Hortonworks' chief executive Eric Baldeschwieler told ZDNet UK on Tuesday.
"The US government and several financial houses are getting tremendous value out of it today," Baldeschwieler said. "The US government has a lot of projects that they don't talk about using Hadoop... as do a number of other governments."
Hortonworks employs many of the core team who originally developed Hadoop at Yahoo.
When asked what uses Hadoop might have for government intelligence agencies — the National Security Agency, Britain's GCHQ, Israel's Mossad and so on — Baldeschwieler said Hadoop was suited to solving their kind of data problems.
If you're an intelligence agency "it's really all the same kinds of problems [as Hadoop is typically used with] — you've got data from all over the world in all kinds of formats — textual data, voice data, whatever, and they need to process that and find patterns," he said. "Putting government aside, whatever your use case, Hadoop is great for this simple bulk data-processing work"
He said Hadoop is increasingly finding use in "sophisticated" tasks, such as analysing networks to see "who's the recommender or the bad guy or the influencer."
No government agencies or financial companies have listed themselves as Hadoop users on the Apache project's page.