A day in the life of a Microsoft security patch

On the heels of the Sobig.F and MSBlaster security breaches, I now find myself checking Microsoft's Windows Update site at least once a day. Perhaps I'm overreacting, but I want to make sure that I have that latest critical patch that keeps my system from riding the information highway with its hatchback wide open.

A barrage of security patches have emerged from Microsoft over the past few months. For the most high profile of these --- called 03-026 within the company, but known to the rest of us as the patch for MSBlaster --- it took Microsoft only 17 days to turn the discovery of a vulnerability into a patch for it that was available through a variety of online channels.

Momentarily setting aside any criticisms or prognostications about why the vulnerabilities are there in the first place, I started to wonder how the company manages to turn around these patches on such short order. Considering the delicate balance that must exist in the millions of lines of code making up Microsoft's operating systems and applications, 17 days hardly seems like enough time to engineer and test the integrity of a patch. Most corporations wouldn't dream of putting their own 17 day-old code into production.

So, I asked Microsoft for a backstage pass to view the inner workings of its emergency response team; officially known as the Microsoft Security Response Center (MSRC).

Considering all the attention that's been given to Microsoft's Trustworthy Computing Initiative, I expected the company would have a platoon of security specialists sequestered in a NORAD-like bunker under Washington State's Mt. Rainier, all staring at giant plasma displays showing DEFCON ratings and graphical virus and worm progress reports. I imagined various personnel with strange symbolic shoulder patches on their uniforms running off to secret elevators with bodyguards and bulletproof briefcases handcuffed to their wrists.

In reality, the MSRC process and facilities aren't quite so glamorous. However, the resources that Microsoft applies to each vulnerability do put the company's money where its mouth is when it comes to Trustworthy Computing.

According to MSRC security program manager Stephen Toulouse, the first step in the security response process is the point at which Microsoft is made aware of a vulnerability. "We receive vulnerability reports through a variety of channels," said Toulouse. In some cases, the MSRC is notified by security researchers and others through a widely publicized e-mail address --- secure@microsoft.com. Some researchers have a direct line to a specific member of the MRSC team. Researchers are not compensated for their efforts, according to Toulouse. In addition to the researchers in the security community, Microsoft also has teams internally that find and report vulnerabilities to the MSRC.

The majority of reports, however, come through the e-mail address, which is not tied to a distribution list; instead, members of the MSRC take turns monitoring it. Given how patching a vulnerability can be a race against the clock, I was surprised to learn that the inbox corresponding to secure@microsoft.com is not monitored 24/7. You'd think Microsoft would be mobilizing personnel the minute a legitimate report of a vulnerability came through any channel--if only to make the Trustworthy Computing Initiative look good. "We don't monitor the inbox 24/7," said Toulouse, "but it never goes beyond a certain amount of time without being checked. We respond to everything within 24 hours."

Once an MSRC member confirms receipt of a legitimate vulnerability report, Toulouse said, "The first thing we do is immediately acknowledge the report. We want a great working relationship with the security community. Then, we check to make sure that it contains enough specifics to reproduce the vulnerability. If we have questions, we engage the sender in an open channel of communication."

Next, more MSRC members and other Microsoft personnel are brought into the loop. "It could be one person, or it could be a team," said Toulouse. "If a report comes in and says 'I found a buffer overflow in Windows,' then I, as a security program manager, would make the determination that the report doesn't have enough information to get others involved yet. Only after I'm able to get more specifics from the 'finder' will I widen the number of people involved and the initial steps of investigation, which typically involve reproducing the problem.. Then, the Secure Windows Initiative Team [will] work with us to evaluate the overall risk of threats and help us determine the impact on Microsoft's products."

According to Toulouse, the 03-026 vulnerability associated with MSBlaster was a textbook application of the process. "With MS03-026, a security research outfit in Poland known as The Last Stage of Delirium sent us an e-mail on June 27 that went very much like, 'We would like to report a security vulnerability in Windows and we're asking for a relationship with [Microsoft].' Within a couple of hours, we replied to them and said 'We'd like to know more and we'd like to work with you directly on this issue.' "

Toulouse was the program manager on duty when The Last Stage of Delirium reached out. "They provided the information we needed to widen the scope of the investigation and to pull in the necessary teams. From what they sent us, I was able to furnish the Windows product team with the steps they needed to reproduce the problem as well as an explanation of how The Last Stage of Delirium discovered it."

Now, Microsoft is under the gun to release fixes for all supported platforms in all supported languages. Depending on the patch, this could mean more than 100 separate patches. "With Internet Explorer, for example," Toulouse said, "we're concurrently supporting five operating systems and 25 languages. That equates to 125 different patches that must be written and tested." And Microsoft has to release all those patches at the same time. Why? Theoretically, if Microsoft released the patches for different versions in stages, there would be a Window of opportunity following the first stage during which a malicious intruder could take advantage of unprotected users. "If we do a patch in English-only first and let it out before the other languages," said Toulouse, "we raise the awareness of the vulnerability to the bad guys, thus leaving the other language versions vulnerable."

At this point, the workflow branches off in different directions, involving more teams of people. "At the heart of the process is the Secure Initiative Team --- with established procedures for getting the updates engineered and tested as soon as possible," said Toulouse. "This is where hundreds of people start to get involved. And while all this is happening, we're still getting more mail at secure@microsoft.com."

Indeed, unbeknownst to me, while I was conducting interviews for this story, work was secretly underway for 03-029-- the vulnerability that was addressed with a patch issued earlier this week. According to Microsoft spokesperson Casey McGee, "There are multiple patches under development at any given time, but Microsoft can't discuss them until the comprehensive fix can be made available. [This week's] bulletin was prompted by a report that Microsoft received on July 25th, which led to an investigation that uncovered additional issues which were also fixed."

As the response engine gathers steam, the MSRC makes decisions about escalation. For example, does the vulnerability and the subsequent work require the attention of Mike Nash, vice president for Microsoft's Security Business Unit? Toulouse admitted that there are e-mail distribution lists that include members of Microsoft's executive team, too; but he declined to offer say who is on these lists or what sort of event triggers the use of those distribution lists when escalating news of a vulnerability.

While writing the patch is obviously a critical step, the entire process is heavily biased towards the testing of that patch. For example, in the case of 03-026, Microsoft estimates that it devoted more than 1,150 person hours across 17 days to release the patch. Of those hours, more than 1,000 were spent on testing. "We do all sorts of testing," said Toulouse. "We obviously do OS compatibility testing and we also do a certain level of application compatibility testing that encompasses more than just Microsoft applications. The list of what we test varies depending on the vulnerability, and the tests themselves look at a number of different things. For example, we test the impact on the operating systems' performance and stability. We also test a variety of configurations. Some of the testing is automated. Other testing is manual. Throughout the company, we have specific teams that test for certain things. In addition, depending on the vulnerability, we might engage third parties as well to provide an outside perspective."

Patches are not the only solutions Microsoft develops in response to a vulnerability. While patch development and testing is taking place, Microsoft must also develop other workarounds that don't involve patching an operating system or an application. This parallel track was developed in response to corporate customers who wanted protection from vulnerabilities as they arise, but were loath to deploy a patch to anything without first conducting their own tests. Since Microsoft doesn't predisclose vulnerability information to anyone --- even its biggest customers --- it's imperative that the company release alternative workarounds at the same time the patch is released. "In the case of 03-026," said Toulouse, "we identified specific ports that customers could block on their firewalls, and provided other steps that disabled the internal interface in Windows to block the pathways of the vulnerability. While these steps just blocked pathways and didn't fix the problem, they did offer the sort of temporary remediation that some customers prefer."

Two other groups that get pulled into the process almost immediately are Microsoft's Security Engineering Strategy (SES) and its Product Support Services (PSS) teams.

While other teams are focused on rolling out a solution to Microsoft's customers, the Security Engineering Strategy team studies the vulnerability in depth in an effort to prevent the error from occurring again. Meanwhile, the PSS prepares the customer outreach, making sure that each of Microsoft's distribution channels for information and software are primed and ready to blanket the world. Toulouse said, "If it's a Windows technology that ships as a part of Windows, then we have Windows Update. XP users in particular can be updated automatically. For Windows 2000, there's a download that gives them the automatic update functionality. For Microsoft Office, we have Office Update. Finally, another way to get the updates is through Microsoft's download system. We know this isn't optimal, so, moving forward, you can expect to see us consolidate [these channels]."

Sometimes, developing these solutions forces Microsoft to make some tradeoffs. Depending on who needs to get involved, it's not unlikely that resources might be shifted from other projects. Beyond a kernel of people dedicated to the MSRC's activities, many other Microsoft workers get involved in the MSRC on an as-needed basis. This shifting of resources is usually not enough to delay a product under development, Toulouse said. But, indicative of Bill Gates' prioritization of security, Toulouse said the company would have no qualms about delaying a product if it meant securing something else first.

There's more to releasing a patch than writing it and testing it. While various engineering teams are finishing up work on the patch itself, Microsoft starts revving up the communications process. The company must publicize detailed information on the vulnerability and provide patching instructions.

With few exceptions, the MSRC releases its updates on Wednesdays. According to Toulouse, this "schedule" came in response to customers who wanted to know when to expect an update rather than wait for news of a problem to surface randomly. "That's not to say we won't have a release on another day," Toulouse told me. "MS 03-007, which affected a distributed authoring component of windows called WEBDAV, was released on a Monday. In general, we strive for Wednesday releases."

The patch for 03-026, following standard operating procedure, was released on Wednesday, July 16-or 17 days after The Last Stage of Delirium reported it to Microsoft.

One of the communications teams involved in the process is Microsoft's public relations firm Waggener-Edstrom. Typically, journalists like me hear the news of an MSRC activity through an e-mail from Waggener-Edstrom. This week's news regarding 03-039 showed up in my inbox on Wednesday.

Before a bulletin can be released, Microsoft must assign a rating to it. Microsoft has four classifications for its bulletins: low, moderate, important, and critical. The key distinction between the latter two has to do with whether or not the vulnerability can be exploited by a worm-- the type of exploit that self-replicates from one machine to another without human involvement. Nimda and Blaster are worms.

Once the threat is rated, and Microsoft's PSS is ready to release the bulletin, the company must make sure that it tips its hat to the researcher or outfit that discovered the vulnerability. This gesture also gives the "finder" the opportunity to release its own information about the vulnerability at the same time that Microsoft releases its bulletin.

Then, on whichever Wednesday comes next, if everything is working right, the entire world is notified of the vulnerability, and can begin applying Microsoft's various suggested countermeasures.

Toulouse stressed that the day in the life of a patch doesn't end here. "We continue the investigation internally to learn from the vulnerability, and to determine how we can better improve our engineering practices, and we monitor PSS very closely to keep tabs on any deployment problems with the patch."

In what is perhaps a testimony to how well the whole process is working, the last security patch to be so problematic that Microsoft had to issue a "patch to a patch" happened over a year ago with update MS02-029 (a patch that was issued in June 2002). Since then, Microsoft has not re-issued a security patch. That's not to say there haven't been deployment problems. For example, after reports circulated that the patch for 03-026 wasn't "taking" in all situations, I reviewed the Windows Update service and concluded that the service itself could use an update--a criticism with which Microsoft generally agreed. In its current state, the Windows Update service is doing an injustice to what has otherwise become a time-tested and generally excellent response procedure.

One improvement that Toulouse said is coming will be better roll-ups of all the patches. Installing the security patches can sometimes be very cumbersome. If there are several, you sometimes have to apply one and reboot before applying another--and rebooting again. Toulouse acknowledged the awkwardness of this process. Then again, some people prefer single function updates. In the case of this week's patch to the 03-039 vulnerability, Microsoft's Windows Update Web site sports a message that the new patch will cover users for both the 03-039 and 03-026 (Blaster) vulnerabilities.

Another improvement needed: more granular controls over Windows Update. After learning that Microsoft normally issues updates on Wednesdays, I thought the prudent thing to do would be to go into the Automatic Update settings tab in Windows Control Panel's System Applet, and set it to check for updates every Wednesday evening. But, by checking that, one must disable the ability to review the updates before they happen. I don't want Microsoft doing anything with my system automatically.

OK, so Microsoft's MSRC is not a NORAD-like operation buried underneath Mt.Rainier. But for the most part, it appears to be working. The question for you is whether you can get as organized and proactive as Microsoft is when it comes to security transgressions. For starters, do what I do: Run Windows Update from Internet Explorer's Tools menu at least every Wednesday (as I said earlier, I run it every day). That ought to keep your system more secure than most.

A lot of people don't trust Microsoft as far as you could throw Bill Gates. After "touring" Microsoft's Security Response Center , do you feel even a little bit better? I do. Share your thoughts, fears, nightmares, and conspiracy theories with your fellow readers using TalkBack . Or write to me at david.berlind@cnet.com. If you're looking for my commentaries on other IT topics, check the archives.