Police and intelligence agencies have long worried about the risk of communications 'going dark' with encryption making it impossible for law enforcement to eavesdrop on criminal or terrorist plots.
The argument was that police needed a way to break through the encryption – a backdoor – in order to read these messages and potentially save lives.
Critics of this approach warned that, because most digital services use encryption for one reason or another, deliberately creating backdoors would cause more damage to society long-term (by increasing the risk of letting in crooks and hackers) than the inability of the police to read a few messages.
SEE: Cybersecurity in an IoT and mobile world (ZDNet special report) | Download the report as a PDF (TechRepublic)
This was the basis of the so-called crypto wars a couple of decades ago, which ended with the general acceptance that widespread access to strong cryptography is a good thing.
Every so often since then governments have attempted to re-run the debate, with limited success. But as the use of encryption has grown from widespread to ubiquitous, governments are having another go. For example both Australia and the UK have introduced laws that could require tech companies to strip encryption from communications, although how effective this legislation will be remains to be seen. If the encryption is easy to remove there's little point having it in the first place, warn tech companies.
But now UK surveillance agency GCHQ has come up with one potential alternative answer to the apparently intractable problem of what to do about encryption. They've described their approach as a modern version of the crocodile-clips on the telephone wire of a few decades ago. Their answer is not to break the encryption that surrounds a conversation, but to insert the police or intelligence agencies into the conversation itself before the encryption is applied. Think of it as a silent addition to your chat with your friend, a hidden eavesdropper on your group chats.
"We're not talking about weakening encryption or defeating the end-to-end nature of the service. In a solution like this, we're normally talking about suppressing a notification on a target's device, and only on the device of the target and possibly those they communicate with. That's a very different proposition to discuss and you don't even have to touch the encryption," said GCHQ in the paper that outlined the concept late last year.
The benefit of this idea is that rather than weakening the encryption that we rely on across the internet, it involves modifying the communication system – a small but important difference which leaves the encryption, which we all rely on, intact.
But the first outraged response arrived quickly: NSA contractor-turned-whistleblower Edward Snowden described the idea at the time as "Absolute madness." The British want companies to poison their customers' private conversations by secretly adding the government, he said.
This week companies, including Microsoft, Apple, Google and WhatsApp, plus civil liberties organisations and security experts, made a somewhat more delayed, longer but equally negative response. In an open letter to GCHQ they warned that the idea would break systems and damage trust.
"The GCHQ's ghost proposal creates serious threats to digital security: if implemented, it will undermine the authentication process that enables users to verify that they are communicating with the right people, introduce potential unintentional vulnerabilities, and increase risks that communications systems could be abused or misused," they wrote.
GCHQ's ghost protocol or crocodile clips concept does have the benefit of being an elegant solution to the deadlock over encryption. It effectively means that we can use strong encryption to protect our messages from everyone – except the government, which gets to see everything. Many would see that as a compromise they are willing to make to be a little safer, but not everyone will agree.
And fundamental real-world problems remain to the idea of the ghost protocol. The most obvious of these is that many tech companies will simply refuse to do it.
For some of the biggest tech firms, ensuring the privacy of their customers data has become a defining characteristic of their business – and a distinct competitive advantage. For them to cheerfully agree to add a spy into every chat would seem deeply unlikely.
Also, these tech companies have customers living around the world, which complicates the issue further. Even in the unlikely event that tech companies might be willing to allow the NSA or GCHQ a ghostly ring-side seat on every conversation they want to snoop, many firms would still want to refuse that capacity to other regimes around the world.
But while tech companies might not want to offer Russia or China the same capabilities, once that ability is built into their code it is much harder to resist that pressure. Once created, these tools are inevitably misused. For tech companies, it's much easier to say no to everyone.
The big tech companies will resist then, and there will also be other messaging apps which will simply be able to ignore the pleas of GCHQ because they aren't based in the UK. Beyond this, strong encryption is effectively a commodity now, which is easy enough to implement, so it's very hard for any government to clamp down on all encrypted apps. And this ghost protocol is only of limited use; the idea won't work for encrypted hard drives or encrypted email, which would require different tools and approaches.
In reality, of course, the debate has never really been just about encryption; the bigger issue is how much privacy we should be allowed in the digital age, which explains the reaction to the GCHQ idea.
Until recently there was, for most people in most countries, an effective right to privacy.
SEE: 10 tips for new cybersecurity pros (free PDF)
The state just couldn't spy on every conversation or listen in to what was happening in every home in real time, and nor did they (in most cases) especially want to.
Now the technology exists to do exactly that. And in many cases we cheerfully carry that technology around in our pockets and install it in our homes. We are creating vast amounts of data every day, and law enforcement inevitably and understandably wants access to that data when necessary to fight crime. Many people, just as understandably, want to keep that data private.
The rise of encrypted services came in part as a response to this tension, to try and recreate that private world of the past. Encryption is one of the best ways that we can keep that information private, which is why the crypto debate has mattered so much. Attempts by government to weaken encryption were a really visible sign of their attempts to reduce that effective right to privacy. Privacy campaigners could, rightly, warn that putting backdoors in encryption put us all at risk.
But now the debate has shifted away from whether governments should be allowed to put backdoors in encryption, to who decides how encryption is applied. It's a subtle shift, but important.
Police and intelligence agencies will always want greater access to data, in order to reduce the risk of crime or terrorism. But those are not the only risks we face. It's already clear that allowing governments too much information quickly leads to the creation of a surveillance state from which it is impossible to escape. Few in the West would want to travel very far down that road.
GCHQ's suggestions, even though many would see them as reasonably modest, are unlikely to get far with the tech community unless they are backed up with legislation – and may well struggle even then. But the battle over digital privacy is far from over.
The Monday Morning Opener is our opening salvo for the week in tech. Since we run a global site, this editorial publishes on Monday at 8:00am AEST in Sydney, Australia, which is 6:00pm Eastern Time on Sunday in the US. It is written by a member of ZDNet's global editorial board, which is comprised of our lead editors across Asia, Australia, Europe, and North America.