'Code Red': What went wrong?

At midnight Thursday, July 19 GMT, more than 350,000 servers infected with the so-called Code Red worm stopped hammering the Internet with scans searching for vulnerable computers. Instead, the servers targeted an Internet address used as the hub for the White House's public Web site with a denial-of-service attack of such proportions that some feared parts of the Internet would shut down, unable to cope with the unprecedented flood of data.

"If this goes along what it's looking like, parts of the Net will go down," predicted Marc Maiffret, chief hacking officer at network-protection company eEye Digital Security. A month earlier, the Aliso Viejo, Calif., company discovered the flaw exploited by the worm in Microsoft's Web servers and was the first to decode the malicious program.

In the end, a design flaw in the worm's programming stymied the attack, but the potential threat of hundreds of thousands of servers flooding the wires with garbage data has resurrected concerns about security among those who consider themselves the guardians of the Internet.

The Internet was lucky this time, as this particular Code Red program squandered its advantage and left itself vulnerable to security measures. That will not always be the case, said Vern Paxson, staff computer scientist at the Lawrence Berkeley National Laboratory, who analyzed Code Red's quick spread.

"This could have been so much worse," he said.

Worms have become the tool of choice among malicious vandals on the Internet, but the Code Red strain has proven particularly fast and effective in commandeering a significant portion of the Internet. Unlike other worms that hide in e-mail attachments, such as LoveLetter and SirCam, Code Red does not require fooling an unwitting recipient into opening a document.

Paxson said a better author could have clogged the entire Net with garbage data or hit critical parts of the global network with a more effective denial-of-service attack--things that the inevitable variants of this version could still do.

"We are in for bumpy times," he said. "I don't see any way out of that."

While you were sleeping...
Like many new worms, Code Red took full advantage of its element of surprise.

On Thursday, July 12, things were going smoothly at the Black Hat Security Briefings in Las Vegas, where several hundred consultants in the computer-security industry hobnobbed with each other. The day before, one researcher had predicted that worms would continue to threaten the Internet. Most considered it an obvious conclusion.

Unknown to the attendees, however, that day a program had started infecting computers running Microsoft's Internet Information Server. The servers had a security hole that had been discovered the month before, leaving them open to attack if not repaired with Microsoft's specific software patch.

The security hole, known officially as the Index Server ISAPI vulnerability, allowed an attacker--whether a network intruder or a worm--to take control of a server by specially formatting a Web page request.

Each hole among the hundreds of thousands of vulnerable IIS servers represented a chink in the armor of the Internet that allowed the worm to spread.

That Thursday, the intrusion-detection system at publishing company Chemical Abstract Services recorded three illegal Web access attempts from a single Internet address. The original attacker's address apparently belongs to a server at the University of Foshan in China, though Ken Eichman, senior security engineer for CAS, stressed that an online vandal could have infected the server from practically anywhere.

Eichman didn't notice the scans until the next day, July 13, when 611 attacks from 27 sources appeared in the company's logs. "It wasn't really intense, and it really didn't bother me," he said.

By the end of the day, however, the scans started getting worse. At one point, Eichman thought that hostile hackers were targeting his company's network. On Saturday, when the number of servers attacking his system jumped from 27 the day before to more than 1,000, he knew it was no minor mischief.

"By Saturday night, it was getting more intense," Eichman said. "By Sunday morning, I got up and hoped it would be gone, but it wasn't."

That Sunday, Eichman sent his findings to a security mailing list hosted by intrusion-detection project DShield.org. He described the attacks affecting servers that used the most common service on the Internet: the Web. He expected help; what he got in return was derision and sarcasm.

"You never heard about Web browsers?" wrote one person on the list. "Please get real. (That's) a Web browser, not an attack," another offered.

But Eichman was a frequent contributor to DShield, which used his logs to correlate disparate incidents on the Net in an effort to identify some sort of patterns. Because he seemed knowledgeable, he was taken seriously by Johannes Ullrich, editor of DShield.org and the chief technology officer of the Internet Storm Center for the System Administration Networking and Security Institute.

"The first suspicion was that there was something wrong with his firewall," Ullrich said. "But he was a long-time submitter, so we kept notifying the people" who were attacking CAS' network, he added.

On Monday, July 16, researchers got the first confirmation that Eichman was right. The immediate conclusion: It was a worm.

Fauna of the Internet
A worm is a program, most often malicious, that can spread from computer to computer without needing to infect files first.

One of the most infamous examples caused a password-collection program to become the Cornell Internet Worm, which spread to 3,000 to 4,000 servers, or about 5 percent of the Internet, in November 1988. Created by then-graduate-student Robert T. Morris, the worm exploited flaws in two well-known Internet services and attempted to masquerade as a legitimate user by trying passwords stolen from other systems.

Lured by the efficiency of self-propagating worms' ability to spread code widely, online vandals have begun using such worms to deface and hack servers. Starting with the Linux Ramen worm in January, a steady stream of such programs have leveraged widespread flaws in computer systems to spread across the Internet.

When Microsoft announced June 18 that a flaw had been found in the company's IIS Web server software--the software basis of nearly 6 million Web sites--it seemed only a matter of time before virus writers and vandals created a worm to attack it.

So for eEye's Maiffret, it came as no surprise when Internet hosting service Left Coast Systems reported the discovery of just such a worm a month later.

The British Columbia-based company discovered that one of their servers had been infected on Friday, July 13, by a new worm exploiting the vulnerability. They decided to directly contact eEye, the company that had found the flaw.

Maiffret immediately asked for a copy of the program to analyze, but his investigation was delayed by the weekend. The worm kept working overtime, though, infecting almost 3,600 hosts by Sunday night. Part II--Dissecting the worm
On Monday, several programmers at eEye began analyzing the code, working through the night on adrenalin fed by large amounts of "Code Red"-branded Mountain Dew, a highly caffeinated soft drink that has become a staple among the code warriors of Silicon Valley. The group dubbed the worm Code Red in honor of the drink and in wry political reference to the worm's habit of defacing Web sites with pages that read "Hacked by Chinese!"

By Tuesday morning, the bleary eEye crew had discovered how the worm worked.

A worm that already had infected a server would scan the Internet using 100 "threads," or sub-programs. When one of the threads locates a vulnerable computer, the worm infects it and begins the process all over again.

The company also discovered two important properties of the worm: Code Red defaces Web pages, and the part of the program used to generate a list of random addresses to attack had an error. Each instance of the worm, once it had infected a server, would not randomly attack the Internet but instead follow the same path as all its brethren.

Any computer attacked by the first Code Red worm would, in the end, be attacked by each of its offspring.

The error had an interesting side effect. The owner of any computer attacked by the worm could make a definitive list of compromised machines, because every infected server would eventually attack the computer. This allowed eEye and others to track the growth of the worm, though it could also allow a person with malicious intent to build a list of known vulnerable systems.

Throughout the day, eEye continued to decode the worm. By Tuesday evening, worm infections had topped 10,000.

Fix it, NOW
For eEye's Maiffret, the virulent spread of the worm drove home a point that the security community had been making for at least two decades: System software must be patched regularly. And when flaws are found in software as widely used as Microsoft's in Web servers, fixing the problem is even more critical.

"We were telling people how bad it was, and Microsoft was telling people how bad it was, but they still didn't install the patch," Maiffret said on July 18.

Scott Culp, program manager for Microsoft's security response center, also put out a dire warning to customers: Patch now, or else.

"We are going back out to customers and telling them that if they didn't put the patch on before, this is all the reason they need to put the patch on now," he said.

However, many security researchers are questioning that common wisdom. If the spread of the Internet worm shows anything, it's that publicizing vulnerabilities and trying to persuade system administrators to plug the holes doesn't work, said LBNL's Paxson.

"I would not at all be surprised if 30 percent or 50 percent (of system administrators) have no clue," he said.

Even the most diligent administrators have trouble keeping abreast of security holes and patches. "Just watching a single site like LBNL--where part of the mission is cybersecurity--they take it seriously," Paxson said. "It's really so hard."

Yet, with new attacks that quickly spread, system administrators have taken on the mantle of responsibility--however reluctantly--not only for their systems, but also what their systems do to the Internet.

The Code Red worm proved that individual, insecure systems can quickly become a global problem.

Mission possible
On Wednesday, July 18, after completely dissecting the worm, eEye's team discovered it had a new mission: The next day, at midnight GMT, every worm would stop attempting to infect other computers on the Net and instead level a denial-of-service attack at an IP address used by the White House Web site.

Still worse, each copy of the worm--which totaled almost 14,000 by Wednesday evening--would send 400MB of garbage data every 4.5 hours.

Many thought the massive influx of data could slow parts of the Internet to a crawl. Others thought the Web could handle the load.

Then, on Thursday morning, the worm soared from slow growth to an epidemic. To experts, it was obvious what had happened: Someone had created a variant of Code Red and fixed the random-number generator, enabling the worm to spread much faster.

Within three hours, the worm had topped 100,000 infections, and by the midnight GMT deadline--5 p.m. PDT--the worm had hit more than 359,000 computers, according to an analysis by David Moore, staff researcher at the Cooperative Association for Internet Data Analysis.

"Had the worm not been programmed to stop spreading at midnight, additional hosts would have been compromised," Moore said in the analysis.

Of those machines, almost 44 percent were in the United States, 11 percent in South Korea, 5 percent in China and the rest scattered around the globe. At its peak, around 9 a.m. PDT, the worm had infected more than 2,000 servers every minute.

The worm's growth slowed as midnight GMT approached, indicating it had saturated the Net, LBNL's Paxson said. Otherwise, every unpatched server would eventually have been infected.

"If you were vulnerable, you were nailed," he said.

Better software a necessity
While there are almost 6 million Web sites hosted on Microsoft's IIS software, according to Internet survey firm Netcraft.com, it's uncertain how many servers that equates to, because a single server can host several sites.

While system administrators should take responsibility for the security of their systems, software makers need to start taking more responsibility for their software as a whole, according to the Computer Emergency Response Team (CERT) Coordination Center, the group responsible for passing information between corporate security managers.

System administrators should not have to deal with the unending task of patching the holes in such software, CERT Coordination Center manager Jeffrey Carpenter said in a statement.

"As we've seen with the 'Code Red' worm and other distributed attacks, even sites that do everything correctly can be severely impacted when new vulnerabilities are discovered," he said.

Microsoft and the IIS flaw were not mentioned by name, but the criticism was clearly aimed at the software giant and the 40 bugs the company has acknowledged in the first seven months of this year.

"The kinds of problems caused by Code Red will continue until vendors substantially reduce the number of vulnerabilities in their products in the first place," Carpenter said.

"As long as software is built by human hands, there will always be software bugs and some of those bugs will result in security vulnerabilities," said Microsoft's Culp, who agreed with CERT's assessment that security is a software problem that needs work.

Microsoft was not even immune to its own software's flaws. Several of the giant's own sites--including some servers related to the company's update and support Web site--fell prey to the worm. Part III--Deflecting the attack
Whether the White House Web site ran on Microsoft's IIS Web server didn't matter, however.

On Thursday at 5 p.m. PDT, servers infected with Code Red were scheduled to overwhelm the Whitehouse.gov site, and potentially parts of the Internet, with a flood of data, according to the analysis by eEye.

As reports came in that the worm's phenomenal growth had started affecting various companies' network performance, White House system administrators worked to defend against the attack.

In the end, a simple flaw in the makeup of the worm saved the White House from an attack that could have taken it down for days.

By design, the worm would try to connect to the original address and unleash its deluge of data only if the server responded. Since the worm targeted the specific IP address for the White House's Web site--198.137.240.91--administrators for the site dodged the onslaught by apparently moving the site to a neighboring IP address, 198.137.240.92.

By playing a shell game with the site's IP address, and junking any data sent to the original address, the White House's system administrators dodged the attack. White House spokesman Jimmy Orr acknowledged that the site's technicians took precautions but would not discuss the address switch.

The attack continues to go on, however. While unsuccessful, the worm's programming will keep attempting to access the Whitehouse.gov site until Friday at 5 p.m. PDT, when the worm will go into hibernation until the end of the month, according to eEye.

While the White House sidestepped the deluge of data, an old debate resurfaced, and eEye found itself under attack by critics of its "tell-all" policy regarding security holes.

Information overload
While the company says it didn't reveal the recipe of how to turn the security hole into a worm, details in its original June 18 advisory were indirectly responsible for causing the rewrite of the Code Red worm, said Russ Cooper, self-proclaimed "Surgeon General for the Internet" and the editor of NTBugtraq mailing list for security service provider TruSecure.

"Their original analysis contained everything required to place code in an executable position within IIS, as well as necessary information bout how to make that code properly execute," Cooper said in a post to the NTBugtraq mailing list.

eEye may not have given a blueprint to worm writers, but they certainly provided pointers on how to exploit the code. In a section of the June 18 advisory titled "The Exploit, as taught by Ryan 'Overflow Ninja' Permeh," the company outlined several issues that hamper programs that may seek to exploit the hole.

But Maiffret says such details are necessary to outline the danger the vulnerability could cause.

"You're damned if you do, and damned if you don't," eEye's chief hacking officer said. "If you have a program that tells people there is a hole and a tool that leaves a file on their hard drive, it's the file that will convince them to patch their server."

CAS security guru Eichman agreed that responsible disclosure of information is a hard balance to maintain. "It's a fine line," he said. "It's tough to stay on that line without pissing someone off in one direction or another."

While Microsoft questioned the necessity of the details of eEye's advisory, the software giant did praise the company for alerting them first and giving its developers a month to create a fix before going public.

July 31 revival
Yet, like the problems for Internet security, the worm won't go away.

On Monday, July 31 at 5 p.m. PDT, the worm will awake and again attempt to infect servers. Malicious programmers will likely be modifying the worm's code with an even more devastating payload.

Have system administrators, software makers and security professionals taken to heart the lesson of the Code Red attack? LBNL's Paxson fears that the lesson may not have been driven home.

"If it had attacked Whitehouse.gov successfully, that might have been more effective in the long run," he said, pointing out that the failure of the worm to shut down the site may actually hurt security because the resurgence could be worse.

"There is some sort of tension between an ugly public security event that teaches and one that hurts people," he said. "This one probably wasn't visible enough to really change our mind-set, so really, we haven't learned anything."