Inside Symantec's nuclear bunker

In one of the rolling hills above Winchester rests a decommissioned nuclear bunker, which is now owned by IT security company Symantec. The facility,  built at enormous cost to the taxpayer at the end of the Cold War in the early 1990s,  now houses the company's UK Security Operation Centre (SOC).

The popular image of a bunker is a dank, rat-infested hole in the ground, but luckily for Symantec's team the interior looks surprisingly like any other office. The facility houses Symantec's UK Managed Security Service (MSS) team whose main task is to filter and monitor data fed back from customers' intrusion prevention systems (IPSs), firewalls and intrusion detection systems (IDSs).

The Winchester team alone analyses some 1.5 billion lines of code per day, according to Jeff Ogden, Symantec's director of managed security services for EMEA. "We spend our lives gathering and analysing information and intelligence," he says. "This is an enormous amount of information, and we're trying to pull it into a coherent state."

The MSS team is located in a room glassed-off from the main bunker, with 15 workstations ranged in three rows of five. Four large, flat-screen monitors mounted on the wall face the workstations. Sky News plays constantly in the background to help the team "monitor the geopolitical situations that may affect the info-threat landscape".

Tight security
No one outside the SOC bunker has access — even other Symantec personnel cannot enter the building without prior clearance. Any visits must be announced at least 24 hours in advance. Symantec customers must sign non-disclosure agreements before visiting.

Once inside, all employees must log in at a separate work station and must log out when leaving. The three separate external cameras have a 360 degree view of the building. The digital recorder has 30-days' backup. The bunker runs 24/7, with a minimum of four analysts and a maximum of fifteen.

Even the atmosphere inside is highly managed. It is pressurised to one and half ppsi greater than outside air pressure, so air is constantly being forced out — handy if someone decides to drop an atomic bomb in the vicinity. In the event of a nuclear attack the air can be filtered through charcoal and there are still safeguards in place against a gas attack.

The bunker has features like a security alarm — two strips of black plastic with glowing red insides — that's activated if any unauthorised visitor steps inside the glassed-off internal perimeter of the SOC, where the analysts beaver away. Get too close to them and they bleep and register an unwanted intruder.

If anyone gets past that lot they have one last line of defence to deal with. "That's when I appear with a baseball bat," says Symantec's ex-forces facilities manager Gordon May.

Globally, there are 120 million desktops and servers using Symantec's products, which all feed back samples...

For more, click here...

...of malicious code. Symantec uses basic agent technology to collect the information, or customers can choose to send in the information manually. "We deploy a small agent onto the customer collection point — the firewall, or the syslog server. The agent is a small piece of software that collects, compresses, signs and encrypts the data before forwarding it to us," says Ogden.

The data process
Once the data has been collected, it is sent to Symantec where it is analysed and, if there is any danger of attack, a report is speedily sent to the client. "If the situation is critical or an emergency, we pick the phone up to the customer and say 'You could be under attack'," says Ogden.

All customer information is stored centrally and run through two filters — a "progressive threat model" that decides whether the code is a threat, and an "expert query engine". The expert query engine decides what the threat is targeting, where it's coming from and what the threat is. This code is then analysed by a Symantec engineer and the incident classified according to its threat level:

Informational — the client has been scanned by hackers but no more action is required

Warning — the client has been scanned and a vulnerability has been detected by hackers

Critical — the client has been scanned and vulnerable machines are being targeted

Emergency — there is a possibility of code being deposited on vulnerable machines

During ZDNet UK's visit to the facility, details of an attempted distributed denial of service attack, that had been launched using a botnet in Romania, were detected. "We profile the threat by finding out where it's being launched from, who it's being aimed at and what it's trying to achieve," says Ogden.

On a wider network
The Security Operations Centres (SOCs) Winchester facility is part of Symantec's global network of information monitoring stations. Customer data is monitored in five SOCs located in Sydney, Munich, the UK and two in the US — in Alexandria and San Antonio.

The SOCs work closely with Symantec's seven security response centres (SRCs). Where the primary role of the SOC is to identify attacks against customers, the SRCs work on a higher level and collate information from a wider variety of sources.

The seven SRC's are located around the globe, in locations including the US, Canada, Ireland, Japan and Australia.

As well as monitoring viruses directly detected by customers, Symantec also scans 25 percent of global email traffic for malicious code — Symantec has a number of "honeypot email boxes", which are accounts provided by ISPs. They are not used, so anything that ends up there is usually spam, trojans, viruses or other forms of malware. An attack quarantine system linked to the honeypot network captures malicious code such as worms and trojans. "It is a virtual network that simulates servers, and so looks like a real network," says Art Wong, vice-president of security response and managed security services for Symantec.

Symantec maintains a list of all the vulnerabilities found across its network called Bugtraq. According to Wong, it's both a clearing house and a database of vulnerabilities. This list is shared...

For more, click here...

...with other anti-malware vendors to speed up the process of issuing patches. "We encourage responsible sharing of information. I helped create the NIAC , [a US body] which encourages responsible disclosure of vulnerabilities, so people can come up with a solution or patch," adds Wong.

The threat of botnets
Given its role as one of the leading IT security vendors, Symantec is well positioned to identify future threats. Some of the biggest offenders on the radar at the moment are botnets. These are extensive networks of compromised computers controlled by hackers. The botnets are usually used to launch distributed denial of service attacks — effectively flooding Web servers or mail boxes with traffic until they fall over.

The growth of botnets is a major problem, according to Symantec. There has been a 100 percent increase in botnets year on year in the UK since 2004. Moreover, Symantec believes that the UK currently contains the highest number of botnets in the world. "Just over a third of the botnets we've seen are in the UK," says Wong, quoting figures from Symantec's Internet Security Report VIII, published in September 2005. This is higher than the US, which has traditionally had more botnets.

The high incidence of botnets in the UK is probably to do with the recent explosion in broadband usage and the fact that most UK home users wouldn't know if their computer was compromised. "Maybe there's a slightly lower awareness level in Britain of botnets," he says. "The IP addresses could come from legitimate machines that have been compromised by hackers. Maybe the machines don't have patches, or are not running up-to-date anti-malware products. Plus, if you have 10,000 machines in a botnet it's difficult to track back to each IP address," says Wong.

Taking control
On average, it takes eight minutes for a new machine to be compromised when hooked up to the Web for the first time, according to Symantec tests on a Windows box not running XP Service Pack 2 or antivirus software.

There is a particular danger for businesses using the same network as a compromised machine, as once one box has been infected behind the firewall, hackers can use the machine to infect others. "If attackers manage to infect a machine within an organisation, they can profile additional machines within that subnet. Executable code can be injected onto other machines to profile the users," says Symantec's Ogden.

Symantec does not tell those people with compromised IP addresses that their computers are being controlled by hackers, due to the sheer scale of the problem. "A botnet can consist of thousands of machines, and we just don't have the time to contact everyone. Our first priority is our customers," says Ogden.

However, when it comes to serious incidents, Symantec does support the police. But the company is keen to point out that it doesn't supply any direct customer information. "The information we supply to our customers belongs to them, and it's up to them to provide information to law enforcement agencies regarding any suspect activity. When companies are targeted, it's the customer who initiates giving information about the offending individuals," says Ogden.

It also supports the police in its efforts to counter botnets. "In the UK the National Hi-Tech Crime Unit has been proactive in trying to close down botnet activity. We welcome any initiative which closes down botnets," says Ogden. "We have had some contact with the authorities in the past and it works quite successfully."

If a company is the subject of an attack, Symantec recommends it goes to the police, if it is aware of the attack. "The focus of managed security services is to protect the customer. In the extortion cases last year, for example, data was fed back to the authorities by those organisations. In such a circumstance we can recommend that the companies contact the relevant authorities," says Ogden.

But Symantec will only go so far with chasing potential criminals. If an attack has been unsuccessful, they are unlikely to be hunted down, says Ogden. "If we have controlled and closed down a particular threat to a customer, there's not a great deal of benefit in tracking down the individuals who mounted the attack."