Dealing with a bad patch in Asia

update Microsoft's flawed patch serves as a timely reminder that it is important to test a patch before deployment, even as the software vendor admits it has more to learn about how it deals with patches.

update Microsoft's flawed patch last week serves as a timely reminder that it is just as important to test a patch before deployment, as it is to apply one quickly. The software vendor also acknowledges that improving the reliability of patches is still "a continuous learning process" for the company.

Security experts in Asia have expressed little surprise over the software vendor's patch blunder, which caused users to get locked out of their PC, and prevented Microsoft's own Windows Firewall from launching.

"The problem arising from (the flawed patch) showed that there are more that we should, and can do, in (the way Microsoft handles patches)."
--Kang Meng Chow
Asia-Pacific chief security advisor, Microsoft

Said Neal Gemassmer, PatchLink's vice president for Asia: "With the overall complexity involved in trying to provide security patches on an ongoing basis, there are bound to be mistakes and faults to be found."

Ken Low, a senior manager for security at networking equipment maker 3Com, recalled that Microsoft had postponed its regular monthly patch update last month due to quality problems. "So I wasn't surprised that there were problems (with this month's update)," he said.

Low reckoned a significant number of Windows business users in the Asia-Pacific region would have been affected, and added that the urgency with which Microsoft pushed the bulletin would have also escalated the problem.

"What's really serious about this is that, when Microsoft released the bulletin last Tuesday, they told everyone to install the patches as soon as possible or risk having the vulnerability being exploited by worm writers," he said.

"So a lot of (their) customers took the advice, rushed to install the patches…and would have ended up with this problem."

And instead of pushing the blame to users for fiddling with the system's default settings, Microsoft should have offered more information on how the patch could affect machines that are configured differently, Low said.

More importantly, he stressed, patches should work regardless of how organizations tweak their machines. "They could have provided more information on what configuration to avoid (before getting users to download the patch)," he said. "Even then, Microsoft shouldn't be dictating how customers configure their systems but to provide a patch that works for all systems. It shouldn't be an issue."

Continuous learning for Microsoft
Kang Meng Chow, Microsoft's Asia-Pacific chief security advisor, acknowledged that the problems arising from the MS05-051 patch, showed that "there are more that we should, and can do" in the way the company handles security patches.

He added that improving the reliability of patches and a user's experience with them is "a continuous learning process".

Kang stressed this incident is not a step back for Microsoft, but rather, a useful lesson on how it can make "further progress in this area".

He noted that, based on feedback from customers, the number of people impacted by the flawed patch is "very low" and the problem can be resolved by following the directions provided on Microsoft's Web site.

"It remains critical for customers to continue to apply this patch to keep their systems updated, while taking precautionary steps to prevent occurrence of the 'side effects' by ensuring appropriate security permission setting for the COM+ Catalog director and files," Kang said.

PatchLink's Gemassmer is also optimistic that the problem is contained in this region, simply because Asian enterprises have yet to understand the importance of deploying patches quickly.

"At this point, what you'll find is that 85 to 90 percent of companies have yet to apply this patch, even with Microsoft's strong urging," he said. "It generally takes 60 to 90 days, or more, for them to apply a security patch across the whole organization."

He added that companies in the Asia-Pacific region presume they are well-protected from malicious attacks if they have already implemented a security infrastructure with the fundamental components such as firewall, antivirus tools and intrusion prevention system (IPS).

"They don't see a need to patch, and assume they've already fixed the problem when they've applied all these tools at the network edge," he said. "But, they don't realize that deploying of security patches is their last line of defense and it needs to be done properly for their networks to be completely secure."

Low noted that Microsoft's flawed patch has "created a dent" in customer's confidence of security patches, though he acknowledged that the software vendor has done a better job in security over the last few months, than they had over the last few years.

"They've certainly shown their commitment to improving security, and I would say they have made some progress," he said.

Test, then patch
The flawed patch incident also serves as a timely reminder for enterprises to test all patches before deployment. "Businesses will probably now pay more attention to patch testing, instead of blindly downloading patches as they come," Low said.

Gemassmer noted: "It raises questions on how they can better evaluate a vendor's security patch and the steps they need to take to apply patches more effectively in their environment. It doesn't mean that a vendor's patch is going to be fool-proof."

He added that IT managers, who only test patches for mission-critical applications and systems, should also rethink their strategy.

"If the large organizations think (they don't have to test patches meant for desktops), they're going to get thousands of blue screens," he said. "Think about the downtime, the impact of that on productivity, and the IT resources you'll need to deploy to get everything up and running."

Between 20 percent and 30 percent of patches that organizations roll out, are deployed inaccurately or are overwritten by subsequent patches, he revealed.

He recommends that patches are thoroughly tested before deployment, and applied across the organization in stages.

Software vendors may do a good job testing their patches, but that does not take away the responsibility of an IT administrator to ensure the patch is not going to break his company's system, he added.

"It's everyone's responsibility," Gemassmer said. "Technology is never going to be perfect, and it's going to behave differently in every corporate environment. There's really not one process that can be universally deployed in all companies, and not one toolset that will fix security on a whole."

"Security needs to be layered...you need your antivirus, your firewall, your IPS. You also need to invest in having better processes for vulnerability assessment and remediation tools. It's a combination of people, tools and processes," he said.

James Yeo, a security consultant with Symantec Singapore, noted that a good patch management strategy should emphasize the importance of user awareness and education.

He added that organizations, particularly the larger ones, are becoming more proactive in dealing with security patches.

"Generally, the ones I've visited see patch as a key focus area," he said. "They have some form of testing procedure before applying a patch, and are very proactive in asking their vendors about patches and the impact of applying these fixes."

According to Kang, Microsoft remains focused on enhancing the security and reliability of its patches. The company in Jan. 2005 introduced an initiative to recruit consultants and selected enterprise customers to test Microsoft's software patches before they are released to the public.

"The program has helped to improve and assure the quality and reliability of (our) patches, and we are continuing with the initiative," Kang said.