British intelligence agency GCHQ's Gaffer goes open source

GCHQ has released a graph storage framework to the open-source community.

British spy agency GCHQ is often associated with secretive surveillance tools and practices, but in this instance has decided to release a useful graphical storage framework to the open-source community.

As reported by Security Affairs, the UK agency's Gaffer framework is a storage system designed to support large-scale graphs containing nodes and edges with statistics including counts, histograms and sketches.

Stored on code repository Github, the software "summarise the properties of the nodes and edges over time windows, and they can be dynamically updated over time."

"Gaffer is a graph database, rather than a graph processing system. It is optimised for retrieving data on nodes of interest," GCHQ says.

While not necessarily suitable software for your average user, the graphical system is likely to be of interest to researchers and analysts. It is possible that this software may have been used by the intelligence agency to discern patterns and plot the actions of groups of interest -- such as criminal gangs or terrorists -- statistically.

Gaffer uses Accumulo for storing data, although other options are available. Accumulo , built on Apache, is an open-source framework built upon Google's BigTable design.

Gaffer's list of features include:

  • Allow the creation of graphs with summarised properties within Accumulo with minimal coding.
  • Allow flexibility of statistics that describe the entities and edges.
  • Allow easy addition of new types of nodes and edges.
  • Allow quick retrieval of data on nodes of interest.
  • Deal with data of different security levels -- all data has a visibility, and this is used to restrict who can see data based on their authorizations.
  • Support automatic age-off of data.

Why GCHQ released the software to the open-source community is of anyone's guess. Perhaps by releasing the software, the agency may be able to track down new talent capable of improving such software, as well as encourage others to improve the functionality of the software without additional investment.

Gaffer is distributed under the Apache 2.0 license, which permits anyone to modify or distribute the code. Another version of Gaffer, Gaffer2, is also on the horizon, which is designed for use with both large and small-scale graphs.

Read on: Top picks