Uncle Sam gives Hadoop a whirl

Summary:Government agencies, which are typically skittish of new technologies, are embracing the Hadoop open-source data analysis tool popular among the web cognoscenti.Hadoop, which combines a flexible filesystem with a variety of data analysis tools in an open-source software package, is growing its user base beyond Twitter, Facebook, Yahoo and the other web darlings, Hortonworks' chief executive Eric Baldeschwieler told ZDNet UK on Tuesday.

Government agencies, which are typically skittish of new technologies, are embracing the Hadoop open-source data analysis tool popular among the web cognoscenti.

Hadoop, which combines a flexible filesystem with a variety of data analysis tools in an open-source software package, is growing its user base beyond Twitter, Facebook, Yahoo and the other web darlings, Hortonworks' chief executive Eric Baldeschwieler told ZDNet UK on Tuesday.

"The US government and several financial houses are getting tremendous value out of it today," Baldeschwieler said. "The US government has a lot of projects that they don't talk about using Hadoop... as do a number of other governments."

Hortonworks employs many of the core team who originally developed Hadoop at Yahoo.

When asked what uses Hadoop might have for government intelligence agencies — the National Security Agency, Britain's GCHQ, Israel's Mossad and so on — Baldeschwieler said Hadoop was suited to solving their kind of data problems.

If you're an intelligence agency "it's really all the same kinds of problems [as Hadoop is typically used with] — you've got data from all over the world in all kinds of formats — textual data, voice data, whatever, and they need to process that and find patterns," he said. "Putting government aside, whatever your use case, Hadoop is great for this simple bulk data-processing work"

He said Hadoop is increasingly finding use in "sophisticated" tasks, such as analysing networks to see "who's the recommender or the bad guy or the influencer."

No government agencies or financial companies have listed themselves as Hadoop users on the Apache project's page.

Topics: Storage

About

Jack Clark has spent the past three years writing about the technical and economic principles that are driving the shift to cloud computing. He's visited data centers on two continents, quizzed senior engineers from Google, Intel and Facebook on the technologies they work on and read more technical papers than you care to name on topics f... Full Bio

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.