Patch management: Filling security holes

Nobody knows when the first patch was issued, but it was almost certainly shortly after the first release of the first software package. No matter how much testing is done in-house, the real world and real users always exercise applications in ways the writers never foresaw or tested: even the largest software company can't afford the time or resources to exhaustively check complex products -- which is not to say they can't do better. Patches will be with us always.

At its simplest, where a software company sends out a small piece of code that must be transplanted into the body of the ailing software, patching takes a few moments. This doesn't scale well. Ask anyone responsible of late for collections of Windows machines, as soon as there are multiple machines with multiple configurations, even deploying and checking a single patch can take a long time. When there are multiple patches, patch management threatens to become a full-time job. And multiple patches are becoming the norm: the CERT security advisory team reported over 4,000 vulnerabilities in 2002, fuelling a veritable industry of patching that is variously estimated to cost businesses and governments up to $1.5bn worldwide.

The nature of patches
The problem is complicated by the nature of patches. They are often produced under great time pressure, especially when the vulnerability they seek to fix has the potential for quick exploitation over the Internet. There is rarely the time for a full quality assurance check by the patch writers, who see themselves as in competition with exploitative hackers who neither know nor care about QA.

Patch management must be owned by a person or a group with responsibility for receiving advisories, checking them in the context of the company and triaging them into must-do, might-do and reject. The patching process has to be integrated with standard system administration techniques if it itself is to be properly managed. It is not in itself risk-free, therefore it should be assessed as part of risk management and treated as any other network-wide change. Not every patch is critical, and not every patch is appropriate for all systems -- if the fix is for a service that isn't deployed, then the benefits from applying it will be outweighed by the time to apply it and the risk that the patch itself may have knock-on effects.

However, this approach isn't useful unless you have a full inventory of your system, including each computer's operating system and applications complement, the versions of each package and the services that are actually running. You must also know what patches have already been applied -- or not -- and who's responsible for each computer. In many companies, this discipline is imperfectly observed with an attitude that as long as every computer's got regularly updated antivirus (AV) protection and there's a decent corporate firewall then nothing too bad can happen. Such approaches are dangerously outdated: threats such as the Blaster worm will bypass AV protection by attacking vulnerabilities directly in the operating system, and just one infected computer can compromise everything behind the firewall. If you don't know where it is, who to call to turn it off or what's missing from its system, it can take far too long to root out the culprit. Taking a full inventory must be the first priority.

Once you know the who, what and where of your systems, you'll be in a position to start assessing patches as they become available. Does the patch affect any of your computers? A buffer overrun in SNMP may be very serious, but only if SNMP is in use: it may be quicker, easier and more effective to make sure that you have disabled the service. Microsoft is finally moving to a model where services default to off when installed, but most of us are still running systems with unused yet active components. Locking down is cheaper and more effective than fixing up.

Controlled testing
When you have assessed that a patch is relevant and necessary, it should be deployed first in a controlled test. A common approach is to first release the patch to internal developers, who have the appropriate skills to check for problems that the patch may bring and are usually highly motivated to ensure the integrity of their systems. If the patch passes that test, then a limited deployment on a subset of live systems is a good next step -- if possible, with a check of any rollback procedure so that further problems with the patch can be reversed. Only then can the patch be considered as safe for universal application.

This process would seem ideal for patch management software, and there are many packages that promise to help. However, to date the situation is overly complex: Microsoft itself has around eight different patch management tools, not all of which agree with each other. The company is working towards integrating these tools into two or three main applications, and to further integrate these into its main system management software. But for now, patch managers have to consider Windows Update -- where the clients themselves detect and install patches -- alongside the Baseline Security Analyzer, Office Update, the Office Update Inventory Tool, and the Software Update Service (SUS) -- which lets managers check and control patch deployment by server. Lately, Microsoft has integrated SUS with its System Management Server 2003, which can automate patch roll-out, make patches mandatory after a specified period, and report on which users have been patched and which remain untouched.

Outside Microsoft, the patch management industry is serviced by a small group of companies. The tools they provide cover much more than just the Windows environment -- every large application and operating system needs similar care -- but none as yet is capable of coping with everything. Companies to investigate include BigFix, Ecora, Kintana, Novadigm, Shavlik Technologies, Patchlink and On Technology, and always consult your application and OS vendors for recommendations.

A serious business
Patch management is finally becoming recognised as a serious and important part of overall system management. Like so many aspects of IT, it has grown up in a haphazard fashion from the days of stand-alone PCs and tightly controlled centralised mainframes, but its true nature has been sharply illuminated by the evolution of the Internet. In the end, responsibility for making it work well is divided between vendors, users, managers and systems designers: now the tools are becoming available, it is up to those at the sharp end to evolve the processes and discipline required to use them sensibly and effectively.