Several other operating systems that use open-source components are vulnerable too varying degrees as well.
The software bug--known as a double-free vulnerability--causes key memory-management functions in the zlib compression library to fail, a condition that could allow a smart attacker to compromise computers over the Internet, said Dave Wreski, director for open-source security company Guardian Digital.
"It is just a matter of time before an exploit is developed," Wreski said.
The flaw, discovered by Linux user Matthias Clasen and Owen Taylor, an engineer at Linux-software company Red Hat, affects any Linux program that uses the zlib library for decompression, including the core software of the operating system, the kernel.
Because the problem is in a library--a set of code that can be shared by any application that links to it--multiple programs could be affected by the flaw. In fact, many non-Linux operating systems use the library, making them vulnerable as well, said Mark Cox, senior director of engineering at Red Hat.
"Zlib is used on all sorts of operating systems: the BSDs and even Solaris," Cox said. "While any operating system that uses the library is affected, the ability to exploit the vulnerability depends on the operating system."
The graphical basis for the Linux desktop, X11, uses the library, as does the common software foundation for the Linux-based Netscape and Galeon browsers. Many image-editing programs, which use the library for compression, also will be affected by the flaw.
The library's functions are "used in network compression, so connecting to untrusted services could allow a hostile site to allocate space in a way that triggers a buffer overflow," Wreski said.
"Because the vulnerability is in a library, that means that the attacker has to identify programs that use the library," said Dave Ahmad, threat analysis manager for security information company SecurityFocus. "There are also a bunch of applications that borrow code from the library."
Weaving the code directly into another application--known as statically linking--means that fixing the programs is much more difficult. Where simply installing a new version of the zlib software on systems will repair the flaw in applications that merely access the library, any program that has borrowed the code itself will have to be patched on its own.
Known as a "double-free vulnerability," the software bug causes programs that use the zlib compression library to behave unpredictably when a malicious program tries to free memory more than once. Most legitimate programs wouldn't try to repeatedly free memory except by accident, but attackers could use such a technique to attempt to force the operating system to run code designed to take over the computer.
Originally, Clasen, a Linux user, found the problem when an image he had created in the open-source Portable Network Graphics, or PNG, format crashed a popular image program. When notified of the problem, Red Hat's Taylor discovered that the issue wasn't with the program but the library used for decompression.
"Owen found that it was a bigger problem than was first thought," said Red Hat's Cox. "At that stage, we realized that there was a significant security hole."
Red Hat worked with the Computer Emergency Response Team (CERT) Coordination Center at Carnegie Mellon University to disseminate information about the flaw to software companies.
CERT/CC is expected to release more information Monday afternoon, but would not comment on the vulnerability.