Is spyware clogging your firewall?

Organizations frequently ask me for assistance in diagnosing and resolving Internet problems. After a bit of detective work, I usually find that the problems are not really an Internet security issue. There's so much complexity in the corporate network these days, and so many places where a problem can occur, that simply identifying the true source of a networking problem is increasingly complex.

Earlier this month, a hospital that I periodically do some consulting for contacted me and asked for some assistance. Because I've worked there on other projects, I was already quite familiar with its network configuration and equipment.

This organization uses Check Point's FireWall-1, a modular firewall platform. Depending on your network, this can either be just what you need or overkill.

The company also uses Websense Enterprise, an HTTP content-filtering system that monitors and restricts Web sites. Websense interacts with the HTTP proxy on Firewall-1 (the HTTP Security Server) using the URL Filtering Protocol (UFP).

After weeks of trouble, the organization called me in to help solve one of the more frustrating computer problems: intermittent failure. During normal business hours--but not always--Web surfing didn't always work. The problem sometimes occurred even with accessing internal Web sites not proxied by Firewall-1.

At first, the description of the error sounded like a DNS failure, but this wasn't the case. Further details suggested a failure of the Firewall-1 HTTP proxy.

After reviewing the log files, we discovered that one particular Web site was repeatedly turning up in the logs, and Websense was consistently denying access to this Web site. But for some reason, it was also randomly dropping legitimate URLs as well--sometimes not even showing up in the log files.

We finally discovered that the URL that Websense was blocking was evidence of a spyware program transmitting information. It began at 7:30 A.M. and continued throughout the day, and other workstations were also showing up in the logs.

After further investigation, we determined that a program called Wild Tangent Updater was responsible for all of the log entries. The Wild Tangent Updater was attempting to transmit usage information, but it was failing because outbound HTTP requests required authentication by Firewall-1.

Firewall-1 and Websense were doing exactly what they should. So why were they also blocking legitimate Web sites?

All network-connected devices using TCP have limits to their ability to communicate. TCP is a connection-oriented protocol, and it uses a socket for communication.

Checkpoint Firewall-1 employs many individual proxy servers using TCP to handle communication from the internal network to and from the public Internet. Firewall-1 also uses TCP to communicate with Websense to determine whether to allow a URL.

I suspected that Wild Tangent Updater was causing either Firewall-1 or Websense to run out of TCP sockets. TCP sockets have timeouts, so they don't just disappear when you're finished with communication.

My theory seemed to explain the problems quite well. After a quick Google search and a visit to phoneboy.com, I felt that I was on the right track. So we increased the socket limits for Firewall-1 and Websense from their default values, and the problem went away.

Whether the Wild Tangent Updater caused the problem or merely precipitated it, there are certainly a lot of other firewall systems out there that could also experience this type of problem. If you're having similar difficulties, check your firewall: Spyware may be clogging it.

Jonathan Yarden is the senior UNIX system administrator, network security manager, and senior software architect for a regional ISP.