The big data technology behind online threat detection at Symantec

Symantec acknowledged that in order for the company to help customers defend against attackers online, it needed to not only be able to provide preventative solutions, but also be able to help them detect and respond to threats.

Speaking to media recently in Singapore, Peter Sparkes, Symantec APAC cybersecurity services senior director, said for a long time the company was focused on delivering prevention-only technologies, but as there has been an increase in new threats, the company's future will also be focused on helping customers detect and, if required, respond to threats.

He said a key part of the company's detection and response strategy will be to rely on data analytics, something in which, he said the company has been investing in.

"To do detection and response, you need very good advanced security analytics. In particular, the question about time comes into play: 'How do you reduce the time factor between detection of the threat and the response team going in?' he said.

Sparkes explained that two years ago the company saw a spike in the amount of customer data logs the company was receiving through its security operations centre (SOC), and as a result realised it needed to invest in more processing power to allow Symantec to deal with the influx of data.

"Six or seven years ago I was getting about 60 million logs from our customers for instance. Nowadays I'm getting billions of logs from my customers per month. We started to see scalability issues. If I wanted to introduce more analytics it delayed the logs getting through the system, which meant I couldn't meet my SLA (service-level agreement) in 10 minutes," he said.

According to Sparkes, Symantec had to look at technologies suitable for big data, and that meant looking to both open source and commercial solutions for a new platform. In the end, Symantec created a new data analytics platform based on Hadoop for batch analytics, and Kafka and Storm for stream analytics.

As a result, what previously took Symantec 14 racks of equipment to do, now only takes one server, Sparkes said. "Because now I have processing power I can now do more advanced analytics, such as looking at traffic flows from different IP addresses and looking for anomalies within those traffic flows."

In turn, Symantec is now able to generate linguistic analytics, carry out machine learning, cloud monitoring, and conditional event monitoring as part of its global intelligence network.

At the same time, the company's SOCs are able to use more analytic modules to detect certain types of threats. In fact, implementing a new module now only takes the company less than three months, in comparison with the 18 months it used to take.

"Every log line is two modules, Hot IP and URL. We match every log line against all the known bad IP addresses from our intelligence and also all the known bad URLs from our intelligence. We're getting new feeds every 10 minutes; it's pure data matching," Sparkes said.

"What is difficult is the volume of data that I need to use. I'm taking 2 terabytes of data per hour from my customers and matching that up with terabytes of intelligence and doing that in near real-time."

Most recently, Symantec introduced the smoke detector module to carry out retrospective analytics. This process uses 90 days' worth of a customer's data to examine low confidence events or events that wouldn't normally be viewed as high risk.

"We're looking for patterns within those events and patterns not just from one customer but across multiple customers, and trying to make connections between these patterns.

"For beta testing, we're picking three more critical events per week that we weren't able to detect through any other technology, and the customers we were using have every technology you could think of, and that's the power of advanced analytics," he said.

To complement this strategy, earlier in the year the company acquired Boeing's online security unit, Narus, in a bid to grow its analytical capabilities.

Disclosure: Aimee Chanthadavong travelled to Singapore with Symantec.