For the moment, monitoring threats using big-data technology complements traditional perimeter defences. But ultimately it will replace measures such as antivirus and firewalls, according the head of machine-learning security company Darktrace.
Stephen Huxter, managing director at the Cambridge UK firm, which has a former MI5 head on its advisory board, argues that what he calls the "old Tuscan walled town" IT security method of locks and gates to keep people out is increasingly ineffective.
"The benefit of that approach is on the decline. In the future there's a question mark over whether we'll have very much of that at all," Huxter said.
He believes using machine learning to model normal network behaviour in real time to detect anomalous activities complements conventional techniques — but only for the moment.
"It's part of a completely new era that we're setting out on now in cyber defence, which is more about assuming that you can't keep people out and even if you wanted to, you wouldn't be able to and you couldn't afford it," he said.
In September, Mike Lynch, founder of UK software firm Autonomy bought by Hewlett-Packard two years ago for £7.1bn ($11bn), invested between $10m and $20m in Darktrace.
By applying the techniques of recursive Bayesian mathematics developed at Cambridge University, Darktrace says its software, which it calls Behavioral Cyber Defence (BCD), can model the normal state of an organisation's systems by examining human and network activity.
"It's a machine-learning system that figures out what's normal and therefore what's not normal at all," Huxter said.
"It's part of a completely new era that we're setting out on now in cyber defence, which is more about assuming that you can't keep people out and even if you wanted to, you wouldn't be able to and you couldn't afford it."
— Stephen Huxter, MD of Darktrace
"Once you've modelled that and with no prior knowledge — so no knowledge of yesterday's attacks and signatures — you can then work out what's abnormal and take some action."
The sophisticated mathematics developed by university academics over many years has been applied to the complex environment of a large organisation, with lots of people, connections and data, according to Huxter.
The software conducts passive collection at the network layer to create a picture of all the packets flowing around the system.
"We get a view across the whole organisation and then in real time the core mathematical algorithms go to work on that and compare today versus the normal model that it has derived for the network over time," he said.
"Because we're focused on how these attackers operate in real time, we can see [their activities] across a number of our customers and then update and tune our software to keep it up to date."
Also in September, Darktrace revealed that Sir Jonathan Evans, who had stepped down as director-general of UK domestic counter-intelligence agency MI5 five months earlier, is now on the company's advisory board.
Huxter says Evans has brought expertise in the threats facing organisations and government, developing the concept set out in the UK's national cybersecurity strategy of a new type of cooperation between state departments and the private sector.
"What the government areas can really add is they take a view on risk across the whole piece and they really have a good insight into how some of these attackers are operating. Those kinds of relationships and that insight are going to be valuable," Huxter said.
With attacks coming in from nation states and lone criminals as well as from well-resourced gangs and hacktivists, not only have the threats become more diverse but their approaches have become more sophisticated.
"They're willing to be much more patient, they're much cleverer, and lots of them are willing to stay under the radar as well," Huxter said.
In that context, conventional perimeter defences are inadequate and even rules-based technologies such as logs are "all about closing the stable door once the horse has bolted".
For example, a suspicious behaviour by system administrators might be repeatedly rebooting a particular machine to allow them to load unauthorised technology.
"So let's put a rule in to flag things when they do it more than 50 times. But if you're inside, you can see that and then you'll just say, 'Well, tomorrow I'll do it 40 times'. It's that basic assumption that needs to change," Huxter said.
The Darktrace product is located on a server at a customer's site, positioned for the best view across the whole organisation. Depending on the network topography, in some situations that could mean plugging it into, say, a spam port in a central datacentre. In others scenarios it might sited at two or three locations.
"It sits off to the side and there's absolutely no interference with their current network. It's a one-way feed out to our box," Huxter said.
Darktrace can provide senior management with a threat visualiser as well as an operations room real-time dashboard of existing threats.
Because of the assumption that intruders are already in the system, Darktrace also adopts a more active approach in addition to the passive information gathering.
"We can create false bits of information — honeypots, if you like — which help us with our estimation of whether there's an attack going on," Huxter said.
"If you've created a completely new part of the network with some very sensitive but false material there, if you see that on the move, then it gives you a much stronger indication that probably bad things are happening."
He says the technology world will only grow more complicated with more employee devices used in the workplace and the growth of the internet of things.
"In one sense that's really good for organisations. It will enable them to work differently and probably have some cost advantages. But again, if you're an attacker, that gives you more opportunities to get in and make the boundary of the organisation even more porous," Huxter said.