Big data, bigger brother: The rise of James Bond, data scientist
The ability to analyse data is becoming just as important as more traditional trade craft.
Previously confidential documents have shed light on just how much data the UK's intelligence services have been sifting through for over a decade.
MORE ON ZERO DAY
The types of data gathered by the intelligence services are wide ranging - Privacy International which revealed the documents said the agencies had the potential to requisition medical records and confidential information shared with a doctor, travel records, financial records, population data, commercial data, and feeds from internet and phone companies including billing data or subscriber details, content of communications and records from government departments. The agencies use this data to spot individuals of interest who might otherwise remain invisible.
The agencies argue that these legally-acquired databases of 'bulk personal data' are necessary to fight crime and terrorism: privacy campaigners insist that the databases represent the "staggering extent" to which the intelligence agencies hoover up and pour over data about millions of entirely innocent people in a bid to snare the few.
"The papers released today act as proof of, and show the sheer scale of, British intelligence agency surveillance of our personal data. It goes far beyond monitoring our text messages, email messages, and social media posts. The intelligence agencies have secretly given themselves access to potentially any and all recorded information about us," Privacy International said.
But the use of these sorts of databases by the intelligence services is only going to increase. That's because there is much more data to trawl through these days: some of this data (like medical and financial records) until recently would have been kept in paper files and impossible to access; now it's digital.
But it's also because so much of it is new, and constantly updated, like what you are doing on your phone, right now.
Another reason that spies are turning to these massive datasets is that other ways of conducting surveillance have recently stopped working for them.
Following the Edward Snowden revelations, many tech companies improved protection for customer data flowing over the internet, making it hard for spies to tap in. And consumers are increasingly using end-to-end encrypted communications, which not even the companies providing the service can read, shutting out the spies again. As a result hacking and big data analytics will be big areas of interest for intelligence agencies. Data scientists and hackers will be as important to intelligence agencies from now on as old-fashioned spies.
For some, this use of data is unsurprising: the spies are only doing exactly what big data has promised for years: taking diverse sets of data and crunching them together to gain insights -- or in the case of GCHQ uncover potential criminals and terrorists. Others will be concerned that agencies have been able to access and analyse this data -- which will also include the records of millions of innocent people -- in secret and for so long.
But it should also make us think a little harder about big data in general -- how much data are we willing to turn over, to whom, and what do we get in return?
More on cyberwarfare and surveillance
- UK boosts spending on cyber army to launch hack attacks on enemies
- The UK's international snooping plan is probably going to end in failure, again
- Defending the last missing pixels: Phil Zimmermann speaks out on encryption, privacy, and avoiding a surveillance state
- Inside the secret digital arms race: Facing the threat of a global cyberwar
- Surveillance laws need rethink, but bulk collection of web data will continue
- The undercover war on your internet secrets: How online surveillance cracked our trust in the web
- The impossible task of counting up the world's cyber armies
- Encryption: More and more companies use it, despite nasty tech headaches