Apache.org hit by targeted XSS attack, passwords compromised

Apache.org hit by targeted XSS attack, passwords compromised

Summary: The hackers hit the server hosting the software that Apache.org uses to it to track issues and requests and stole passwords from all users.

SHARE:
111

Combining a cross-site scripting (XSS) vulnerability with a TinyURL redirect, hackers successfully broke into the infrastructure for the open-source Apache Foundation in what is being described as a "direct, targeted attack."

The hackers hit the server hosting the software that Apache.org uses to it to track issues and requests and stole passwords from all users.  The software was hosted on brutus.apache.org, a machine running Ubuntu Linux 8.04 LTS, the group said.

The passwords were encrypted on the compromised servers (SHA-512 hash) but Apache said the risk to simple passwords based on dictionary words "is quite high" and urged users to immediately rotate their passwords.  "In addition, if you logged into the Apache JIRA instance between April 6th and April 9th, you should consider the password as compromised, because the attackers changed the login form to log them," Apache said.

Here's what happened, in Apache.org's own words:follow Ryan Naraine on twitter

On April 5th, the attackers via a compromised Slicehost server opened a new issue, INFRA-2591. This issue contained the following text:

ive got this error while browsing some projects in jira http://tinyurl.com/XXXXXXXXX [obscured]

Tinyurl is a URL redirection and shortening tool. This specific URL redirected back to the Apache instance of JIRA, at a special URL containing a cross site scripting (XSS) attack. The attack was crafted to steal the session cookie from the user logged-in to JIRA. When this issue was opened against the Infrastructure team, several of our administators clicked on the link. This compromised their sessions, including their JIRA administrator rights.

At the same time as the XSS attack, the attackers started a brute force attack against the JIRA login.jsp, attempting hundreds of thousands of password combinations.

On April 6th, one of these methods was successful. Having gained administrator privileges on a JIRA account, the attackers used this account to disable notifications for a project, and to change the path used to upload attachments. The path they chose was configured to run JSP files, and was writable by the JIRA user. They then created several new issues and uploaded attachments to them. One of these attachments was a JSP file that was used to browse and copy the filesystem. The attackers used this access to create copies of many users' home directories and various files. They also uploaded other JSP files that gave them backdoor access to the system using the account that JIRA runs under.

By the morning of April 9th, the attackers had installed a JAR file that would collect all passwords on login and save them. They then sent password reset mails from JIRA to members of the Apache Infrastructure team. These team members, thinking that JIRA had encountered an innocent bug, logged in using the temporary password sent in the mail, then changed the passwords on their accounts back to their usual passwords.

[ ALSO SEE: Apache.org hit by SSH key compromise ]

Then the attack spread to Bugzilla:

The group said that one of the hijacked passwords was the same as the password to a local user account on brutus.apache.org that had full sudo access. The attackers were thereby able to login to brutus.apache.org, and gain full root access to the machine. This machine hosted the Apache installs of JIRA, Confluence, and Bugzilla.

Once they had root on brutus.apache.org, the attackers found that several users had cached Subversion authentication credentials, and used these passwords to log in to minotaur.apache.org (aka people.apache.org), our main shell server. On minotaur, they were unable to escalate privileges with the compromised accounts.

About 6 hours after they started resetting passwords, we noticed the attackers and began shutting down services. We notified Atlassian of the previously unreported XSS attack in JIRA and contacted SliceHost. Atlassian was responsive. Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly.

Apache said the use of one-time passwords was a "lifesaver" because it limited the damage and stopped the attack from spreading to other services/hosts. "The attackers could have caused widespread damage to the ASF's infrastructure. Fortunately, in this case, the damage was limited to rooting a single host," it said.

[ SEE: Exploit code published for Apache Tomcat flaw ]

However, there were some worrying security weaknesses that caused problems for Apache.  For example, the same password should not have been used for a JIRA account as was used for sudo access on the host machine.  The group also lamented the inconsistent application of one-time passwords, which were required for other machines, but not on the brutus server.

"SSH passwords should not have been enabled for login over the internet," Apache acknowledged.

This is the second major Apache compromise in less than a year.  Last August, the main site of the Apache Foundation was hacked through an attack that used a compromised SSH key.

Topics: Security, Open Source

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

111 comments
Log in or register to join the discussion
  • Embarrassing!

    apache.org themselves? Oh, the irony....

    And they are running Linux, which according to
    some advocates is the most secure operating
    system on the planet!

    When even apache can't get it right, who can
    you trust?

    (ok, couldn't help myself. sorry)
    honeymonster
    • I'm sure he will stop by to tell us

      that it was all some sort of misunderstanding. ;)
      John Zern
      • But wait a minute .. isn't Linux un-hackable? Yea, right. nt

        nt
        babyboomer57
        • Do you know what XSS is?

          Oh... you don't?!!

          I'm done with you then.
          Great Kahuna
          • Yes. It is a vulnerability. And it has been exploited.

            [i]Tinyurl is a URL redirection and shortening tool. This specific URL redirected back to the Apache instance of JIRA, at a special URL containing a cross site scripting (XSS) attack. The attack was crafted to steal the session cookie from the user logged-in to JIRA. When this issue was opened against the Infrastructure team, several of our administators clicked on the link. This compromised their sessions, including their JIRA administrator rights.[/i]

            Why did Apache "administrator" use a Windows machine to open the URL? (The Linux browsers are invincible, remember?)

            Then why did they open the URLs while having administrative access to the JIRA?

            [i]At the same time as the XSS attack, the attackers started a brute force attack against the JIRA login.jsp, attempting hundreds of thousands of password combinations.

            On April 6th, one of these methods was successful. Having gained administrator privileges on a JIRA account, ...[/i]

            Why an account that is accessible from the Internet and is susceptible to brute force attacks can ever have administrative access to the system?

            Why wasn't there a password strength check in place, especially for the admin accounts?

            Why wasn't there a brute force attack detection system that would alert the administrators about such attempts and, perhaps, automatically lock down the access?

            The security is only as strong as the weakest link. The kernel may not be vulnerable, the AppArmor may theoretically be invincible, but as soon as a little whole is found, even <a href="http://blogs.zdnet.com/hardware/?p=7728">local attacks</a> against the Kernel may lead to a disaster.
            Earthling2
          • A browser vulnerability to be exact

            XSS and weak passwords at play, that's all.

            Apache and Linux have got nothing to do with it.
            Great Kahuna
          • Which proves once again that Linux is totally worthless....

            if you try to actually do anything with it (i.e. run software). I mean that is the gist of what you are saying. Ah, no problem with linux or apache, just XSS and weak passwords, that's all. Nothing at all really, in fact this never happened, that is how darned silly this all is. LOL.

            Gotta love how the kernel is defended instead of the entire system from kernel to end user, whenever a company using Linux based systems is successfully attacked.

            I assume you know about the Linux kernel vulnerabilities. Had to wonder when Linus decided he had to keep up with the features of other systems and in doing so started cranking out more lines of code per day than ever and cutting back too much on the QA testing that was a hallmark of the kernel.

            How about when Ubuntu servers were hacked to attack each other? Was that an "app" problem? LOL

            http://blogs.zdnet.com/security/?p=453&tag=col1;post-770

            Just another high profile Ubuntu attack easily done. And the childish war between Cannonical and the community admins taking care of the servers. It was obviously partially user blame, but why would Cannonical allow obvious hacks to run their systems?
            It brings up another point since they could not upgrade beyond Breezy due to hardware incompatibiliy with newer kernel releases.

            And today we get an article lauding Ubuntu/Linux based systems in general for their frequent updating. Well when you flag an OS at EOL when users are still stuck using it, is that a good thing?
            If MS dropped support on an OS in just over a year, like most on the Ubuntu versions on the EOL list, it would be front page news and the entire industry would be up in arms. Esp. the Linux zealots who seem to have more problems with an OS they don't use, than those who use it.

            Oh, by the way, there is no statement from Apache anywhere that claims the brute force password attack (weak passwords), is how accounts were compromised.

            <i>On April 6th, one of these methods was successful. Having gained administrator privileges on a JIRA account......</i>

            You do realize, like many other high profile Linux attacks, some on our nation's defense programs, the hackers were messing around on the Apache systems for at least 4 days. It doesn't say when they were done.
            FOUR DAYS.
            Our network has attempted attacks made on it frequently but I'm happy to say we've never been compromised.

            But when 2 of the worlds premiere Linux sites, which one would think are run by capable people, are compromised quite easily, it makes one start to wonder just what would happen to Linux in the hands of the average users who are not computer scientists for a living.

            So seriously, does this say that the worlds highest profile Linux sites are hiring or have voluteers who are clueless, or does it say that Linux based systems are hard to use and secure? Which one fits?
            xuniL_z
          • I have absolutely no worries.

            But you sure seem to worry a lot about my posts or you wouldn't be spamming them all of the time.
            LOL!

            What are you so worried about, O' petty one? <br>:)
            l0l

            As for me, I am just a good man who sees people saying things that are not true, or half truths and am correcting those for the good people that might visit this site.
            xuniL_z
          • I'll stop saying RTFA when people start reading the article before replying

            [i]Which proves once again that Linux is totally worthless....
            if you try to actually do anything with it (i.e. run software).[/i]


            No, it proves that the vulnerability had nothing to do with Linux.

            [i]I mean that is the gist of what you are saying. Ah, no problem with linux or apache, just XSS and weak passwords, that's all. Nothing at all really, in fact this never happened, that is how darned silly this all is. LOL. [/i]

            It happened, it just wasn't related to Linux or Apache any more than it was related to Windows or IIS.

            [i]Gotta love how the kernel is defended instead of the entire system from kernel to end user, whenever a company using Linux based systems is successfully attacked.[/i]

            Why shouldn't it be? Windows isn't attacked every time somebody running Windows gives out [i]their[/i] password. Why hold an operating system that is free to much higher standards than a commercial one? That isn't very fair.

            [i]I assume you know about the Linux kernel vulnerabilities.[/i]

            At least read the title please. If you don't know what the term means look it up. Hint; it has nothing to do with the Linux kernel.

            [i]Had to wonder when Linus decided he had to keep up with the features of other systems and in doing so started cranking out more lines of code per day than ever and cutting back too much on the QA testing that was a hallmark of the kernel.[/i]

            Other way around. Windows has been playing catch-up with Linux since.. pretty much forever.
            And beta versions of Linux have more stability than release versions of Windows.

            Ooo, UAC? UNIX series had it decades before Windows. Linux had it from the beginning. It's called sudo.

            64-bit? Working great Linux since 2001.. where-as the first serious 64-bit release of Windows was Vista, in 2007. XP got a 64-bit version in 2005 but it was largely dismissed even by the Windows community as being unstable/unusable.

            Multicore? Again, first seen in Linux.

            Internet access? MS copy and pasted it without paying from a UNIX variant known as BSD.

            3D desktop effects and virtual desktops? Again, Linux leads by a huge margin.

            [i]How about when Ubuntu servers were hacked to attack each other? Was that an "app" problem? LOL

            http://blogs.zdnet.com/security/?p=453&tag=col1;post-770[/i]

            Seriously, what is so hard about reading something before commenting on it? The affected systems were running Ubuntu 5.10 (the current release is 9.10), and had security updates completely disabled. People were sending their passwords over clean text meaning anyone could simply steal them and log in with them, regardless of what the OS was.

            It would be like if someone using an unpatched version of Win2k got hacked by sending their login password unencrypted and having somebody find it, was used as an example of why Windows sucks.

            How can you hold a free product to significantly higher standards than a commercial product whose parent company gets 58 billion dollars a year? I can see holding it to the same standards, since MSFT really suck, but holding it to much higher standards is just not cool.

            [i]Just another high profile Ubuntu attack easily done. And the childish war between Cannonical and the community admins taking care of the servers. It was obviously partially user blame, but why would Cannonical allow obvious hacks to run their systems?[/i]

            Good questions, not relevant to the product itself though.

            [i]It brings up another point since they could not upgrade beyond Breezy due to hardware incompatibiliy with newer kernel releases.[/i]

            It's not their fault that the hardware manufacturer refused to provide hardware specs for the Linux community to make new drivers with, and also refused to update their closed drivers themselves. Either letting other people do it, or doing it themselves, would have prevented this problem. Nobody else had any say.

            [i]And today we get an article lauding Ubuntu/Linux based systems in general for their frequent updating. Well when you flag an OS at EOL when users are still stuck using it, is that a good thing?
            If MS dropped support on an OS in just over a year, like most on the Ubuntu versions on the EOL list, it would be front page news and the entire industry would be up in arms. Esp. the Linux zealots who seem to have more problems with an OS they don't use, than those who use it. [/i]

            Okay, for one, what you've said is <a href=https://wiki.ubuntu.com/Releases>factually wrong</a>.
            For two, even if it wasn't, there's still no comparison; to upgrade to a new version of Windows you must pay around $100, and reinstall all of your programs and reconfigure all of your settings completely from scratch, [i]every time[/i].

            [i]Oh, by the way, there is no statement from Apache anywhere that claims the brute force password attack (weak passwords), is how accounts were compromised. [/i]

            When the alternative is reversing a one-way hash function, I think it's a safe assumption to make.

            [i]You do realize, like many other high profile Linux attacks, some on our nation's defense programs, the hackers were messing around on the Apache systems for at least 4 days. It doesn't say when they were done.
            FOUR DAYS.
            Our network has attempted attacks made on it frequently but I'm happy to say we've never been compromised.[/i]

            Four days of misconfiguration unrelated to the underlying. Microsoft can leave in known vulnerabilities being used to hack properly configured systems in their software for weeks, and people like you and Loverock [i]praise[/i] them in response. How can you hold Windows to such abyssal standards while holding Linux to impossible ones, while keeping a straight face?
            AzuMao
          • still not nice....

            I'm glad I can be entertainment for you, apparently you don't require much in that way. I've heard you can be content all day examining the wonder of your opposable thumbs.
            LOL.

            Matter of fact, I have stated the half truths Microsoft has put out there. Especially those from Steve Ballmer. If you knew me, you'd know I can't stand Steve Ballmer and would love to see MS can his azz as much as anyone else.

            He is bad for technology and has the mentality of "milking" products, which he is too asinine to see that while he spent all that time milking XP, and telling university business students that you have learn to "milk" your products for everything they are worth, the world did not stop for him and now he's put the company in a hard position because of the technology being built all around him, while he's totally oblivious to it.

            So yes. But when someone tries to say that MS developers and engineers are somehow half rate, or anything of that regard, I take issue with it.
            Obviously they've had employees move to, and come from the open source world. Happens more often than you'd know. Same with Google. Google got it's best employees from Microsoft when it went on it's spree of actively trying to steal MS employees by setting up shop right next to the Redmond campus.
            xuniL_z
        • LOL. After years of bragging of "more secure than IIS"

          ... and countless embarrassment to disapprove it, how many more blunders does it take for the FOSS nuts to finally fold it?
          LBiege
          • Bozo the clown used to laugh a lot too

            and just like you he had no clue as to why.
            Great Kahuna
          • Speaking of clowns.....and Larry harmon knew why he was laughing....

            Larry Harmon turned the Bozo the Clown franchise into a huge business. Anyone that wanted to portray bozo had to license the rights from Harmon before they could hire actors to play Bozo.
            He portrayed bozo in countless appearances and died a very happy and wealthy man who had brought joy to millions of children.

            Yep, he knew why he was laughing.

            Just like we all know that the high profile Linux based systems hacks are hilarious and laughable because of people like you telling us for years that Linux systems are inpenetrable.
            You never said they were any less secure by simply running software on them...which must have been what you meant then? The linux kernel is secure, as long as you keep it on a disc in your locked safe and never let it see the light of day....yeah, then it's downright the most secure OS for sure.

            Speaking of clowns, you are the house zdnet.com clown.
            xuniL_z
        • They show transparency

          Troll !!!
          jmary@...
        • This was social engineering using XSS on another site

          From the article, the XSS was run on another server when the url link was clicked on. This is social engineering. They baited them to click on the link which took them to the compromised XSS server. This is shown in the article by the phrase "a special URL containing a cross site scripting (XSS) attack." Then they exploited the browser of the one that clicked on the link which is shown by "The attack was crafted to steal the session cookie from the user logged-in to JIRA."
          The Ubuntu Linux server at Apache did not even have to have a vulnerability for this type of attack to work. They did not mention what OS the url server was running but it would not have mattered because it was very likely under the control of the ones that setup the url.

          This sounds like more of a browser issue than a server issue. With NoScript on Firefox, the XSS could not have run.

          Mike
          21_years_IT
          • Social engineering wasn't an excuse when Google was hacked

            As far as I remember, browser vulnerabilities and social engineering didn't work an excuses for 9-year old distribution through which Google was compromised.
            Earthling2
          • Actually, It was

            If you go back to the Google hack, it was a browser vulnerability in a Windows client that led to the compromise...

            Also note that that, too, was a social engineering hack.
            blarman_z
        • Linux wasn't hacked. RTFA before replying next time.

          [b] [/b]
          AzuMao
          • Message has been deleted.

            still not nice
          • RTFA Your New Fave ...

            Got a keyboard shortcut to speed up your turbo-typing nonsense for it yet?
            PMC-CON