Apache.org hit by targeted XSS attack, passwords compromised
Summary: The hackers hit the server hosting the software that Apache.org uses to it to track issues and requests and stole passwords from all users.
Combining a cross-site scripting (XSS) vulnerability with a TinyURL redirect, hackers successfully broke into the infrastructure for the open-source Apache Foundation in what is being described as a "direct, targeted attack."
The hackers hit the server hosting the software that Apache.org uses to it to track issues and requests and stole passwords from all users. The software was hosted on brutus.apache.org, a machine running Ubuntu Linux 8.04 LTS, the group said.
The passwords were encrypted on the compromised servers (SHA-512 hash) but Apache said the risk to simple passwords based on dictionary words "is quite high" and urged users to immediately rotate their passwords. "In addition, if you logged into the Apache JIRA instance between April 6th and April 9th, you should consider the password as compromised, because the attackers changed the login form to log them," Apache said.
Here's what happened, in Apache.org's own words:
On April 5th, the attackers via a compromised Slicehost server opened a new issue, INFRA-2591. This issue contained the following text:
ive got this error while browsing some projects in jira http://tinyurl.com/XXXXXXXXX [obscured]
Tinyurl is a URL redirection and shortening tool. This specific URL redirected back to the Apache instance of JIRA, at a special URL containing a cross site scripting (XSS) attack. The attack was crafted to steal the session cookie from the user logged-in to JIRA. When this issue was opened against the Infrastructure team, several of our administators clicked on the link. This compromised their sessions, including their JIRA administrator rights.
At the same time as the XSS attack, the attackers started a brute force attack against the JIRA login.jsp, attempting hundreds of thousands of password combinations.
On April 6th, one of these methods was successful. Having gained administrator privileges on a JIRA account, the attackers used this account to disable notifications for a project, and to change the path used to upload attachments. The path they chose was configured to run JSP files, and was writable by the JIRA user. They then created several new issues and uploaded attachments to them. One of these attachments was a JSP file that was used to browse and copy the filesystem. The attackers used this access to create copies of many users' home directories and various files. They also uploaded other JSP files that gave them backdoor access to the system using the account that JIRA runs under.
By the morning of April 9th, the attackers had installed a JAR file that would collect all passwords on login and save them. They then sent password reset mails from JIRA to members of the Apache Infrastructure team. These team members, thinking that JIRA had encountered an innocent bug, logged in using the temporary password sent in the mail, then changed the passwords on their accounts back to their usual passwords.
[ ALSO SEE: Apache.org hit by SSH key compromise ]
Then the attack spread to Bugzilla:The group said that one of the hijacked passwords was the same as the password to a local user account on brutus.apache.org that had full sudo access. The attackers were thereby able to login to brutus.apache.org, and gain full root access to the machine. This machine hosted the Apache installs of JIRA, Confluence, and Bugzilla.
Once they had root on brutus.apache.org, the attackers found that several users had cached Subversion authentication credentials, and used these passwords to log in to minotaur.apache.org (aka people.apache.org), our main shell server. On minotaur, they were unable to escalate privileges with the compromised accounts.
About 6 hours after they started resetting passwords, we noticed the attackers and began shutting down services. We notified Atlassian of the previously unreported XSS attack in JIRA and contacted SliceHost. Atlassian was responsive. Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly.
Apache said the use of one-time passwords was a "lifesaver" because it limited the damage and stopped the attack from spreading to other services/hosts. "The attackers could have caused widespread damage to the ASF's infrastructure. Fortunately, in this case, the damage was limited to rooting a single host," it said.
[ SEE: Exploit code published for Apache Tomcat flaw ]
However, there were some worrying security weaknesses that caused problems for Apache. For example, the same password should not have been used for a JIRA account as was used for sudo access on the host machine. The group also lamented the inconsistent application of one-time passwords, which were required for other machines, but not on the brutus server."SSH passwords should not have been enabled for login over the internet," Apache acknowledged.
This is the second major Apache compromise in less than a year. Last August, the main site of the Apache Foundation was hacked through an attack that used a compromised SSH key.
Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.
Talkback
Embarrassing!
And they are running Linux, which according to
some advocates is the most secure operating
system on the planet!
When even apache can't get it right, who can
you trust?
(ok, couldn't help myself. sorry)
I'm sure he will stop by to tell us
But wait a minute .. isn't Linux un-hackable? Yea, right. nt
Do you know what XSS is?
I'm done with you then.
Yes. It is a vulnerability. And it has been exploited.
Why did Apache "administrator" use a Windows machine to open the URL? (The Linux browsers are invincible, remember?)
Then why did they open the URLs while having administrative access to the JIRA?
[i]At the same time as the XSS attack, the attackers started a brute force attack against the JIRA login.jsp, attempting hundreds of thousands of password combinations.
On April 6th, one of these methods was successful. Having gained administrator privileges on a JIRA account, ...[/i]
Why an account that is accessible from the Internet and is susceptible to brute force attacks can ever have administrative access to the system?
Why wasn't there a password strength check in place, especially for the admin accounts?
Why wasn't there a brute force attack detection system that would alert the administrators about such attempts and, perhaps, automatically lock down the access?
The security is only as strong as the weakest link. The kernel may not be vulnerable, the AppArmor may theoretically be invincible, but as soon as a little whole is found, even <a href="http://blogs.zdnet.com/hardware/?p=7728">local attacks</a> against the Kernel may lead to a disaster.
A browser vulnerability to be exact
Apache and Linux have got nothing to do with it.
Which proves once again that Linux is totally worthless....
Gotta love how the kernel is defended instead of the entire system from kernel to end user, whenever a company using Linux based systems is successfully attacked.
I assume you know about the Linux kernel vulnerabilities. Had to wonder when Linus decided he had to keep up with the features of other systems and in doing so started cranking out more lines of code per day than ever and cutting back too much on the QA testing that was a hallmark of the kernel.
How about when Ubuntu servers were hacked to attack each other? Was that an "app" problem? LOL
http://blogs.zdnet.com/security/?p=453&tag=col1;post-770
Just another high profile Ubuntu attack easily done. And the childish war between Cannonical and the community admins taking care of the servers. It was obviously partially user blame, but why would Cannonical allow obvious hacks to run their systems?
It brings up another point since they could not upgrade beyond Breezy due to hardware incompatibiliy with newer kernel releases.
And today we get an article lauding Ubuntu/Linux based systems in general for their frequent updating. Well when you flag an OS at EOL when users are still stuck using it, is that a good thing?
If MS dropped support on an OS in just over a year, like most on the Ubuntu versions on the EOL list, it would be front page news and the entire industry would be up in arms. Esp. the Linux zealots who seem to have more problems with an OS they don't use, than those who use it.
Oh, by the way, there is no statement from Apache anywhere that claims the brute force password attack (weak passwords), is how accounts were compromised.
<i>On April 6th, one of these methods was successful. Having gained administrator privileges on a JIRA account......</i>
You do realize, like many other high profile Linux attacks, some on our nation's defense programs, the hackers were messing around on the Apache systems for at least 4 days. It doesn't say when they were done.
FOUR DAYS.
Our network has attempted attacks made on it frequently but I'm happy to say we've never been compromised.
But when 2 of the worlds premiere Linux sites, which one would think are run by capable people, are compromised quite easily, it makes one start to wonder just what would happen to Linux in the hands of the average users who are not computer scientists for a living.
So seriously, does this say that the worlds highest profile Linux sites are hiring or have voluteers who are clueless, or does it say that Linux based systems are hard to use and secure? Which one fits?
I have absolutely no worries.
LOL!
What are you so worried about, O' petty one? <br>:)
l0l
As for me, I am just a good man who sees people saying things that are not true, or half truths and am correcting those for the good people that might visit this site.
I'll stop saying RTFA when people start reading the article before replying
if you try to actually do anything with it (i.e. run software).[/i]
No, it proves that the vulnerability had nothing to do with Linux.
[i]I mean that is the gist of what you are saying. Ah, no problem with linux or apache, just XSS and weak passwords, that's all. Nothing at all really, in fact this never happened, that is how darned silly this all is. LOL. [/i]
It happened, it just wasn't related to Linux or Apache any more than it was related to Windows or IIS.
[i]Gotta love how the kernel is defended instead of the entire system from kernel to end user, whenever a company using Linux based systems is successfully attacked.[/i]
Why shouldn't it be? Windows isn't attacked every time somebody running Windows gives out [i]their[/i] password. Why hold an operating system that is free to much higher standards than a commercial one? That isn't very fair.
[i]I assume you know about the Linux kernel vulnerabilities.[/i]
At least read the title please. If you don't know what the term means look it up. Hint; it has nothing to do with the Linux kernel.
[i]Had to wonder when Linus decided he had to keep up with the features of other systems and in doing so started cranking out more lines of code per day than ever and cutting back too much on the QA testing that was a hallmark of the kernel.[/i]
Other way around. Windows has been playing catch-up with Linux since.. pretty much forever.
And beta versions of Linux have more stability than release versions of Windows.
Ooo, UAC? UNIX series had it decades before Windows. Linux had it from the beginning. It's called sudo.
64-bit? Working great Linux since 2001.. where-as the first serious 64-bit release of Windows was Vista, in 2007. XP got a 64-bit version in 2005 but it was largely dismissed even by the Windows community as being unstable/unusable.
Multicore? Again, first seen in Linux.
Internet access? MS copy and pasted it without paying from a UNIX variant known as BSD.
3D desktop effects and virtual desktops? Again, Linux leads by a huge margin.
[i]How about when Ubuntu servers were hacked to attack each other? Was that an "app" problem? LOL
http://blogs.zdnet.com/security/?p=453&tag=col1;post-770[/i]
Seriously, what is so hard about reading something before commenting on it? The affected systems were running Ubuntu 5.10 (the current release is 9.10), and had security updates completely disabled. People were sending their passwords over clean text meaning anyone could simply steal them and log in with them, regardless of what the OS was.
It would be like if someone using an unpatched version of Win2k got hacked by sending their login password unencrypted and having somebody find it, was used as an example of why Windows sucks.
How can you hold a free product to significantly higher standards than a commercial product whose parent company gets 58 billion dollars a year? I can see holding it to the same standards, since MSFT really suck, but holding it to much higher standards is just not cool.
[i]Just another high profile Ubuntu attack easily done. And the childish war between Cannonical and the community admins taking care of the servers. It was obviously partially user blame, but why would Cannonical allow obvious hacks to run their systems?[/i]
Good questions, not relevant to the product itself though.
[i]It brings up another point since they could not upgrade beyond Breezy due to hardware incompatibiliy with newer kernel releases.[/i]
It's not their fault that the hardware manufacturer refused to provide hardware specs for the Linux community to make new drivers with, and also refused to update their closed drivers themselves. Either letting other people do it, or doing it themselves, would have prevented this problem. Nobody else had any say.
[i]And today we get an article lauding Ubuntu/Linux based systems in general for their frequent updating. Well when you flag an OS at EOL when users are still stuck using it, is that a good thing?
If MS dropped support on an OS in just over a year, like most on the Ubuntu versions on the EOL list, it would be front page news and the entire industry would be up in arms. Esp. the Linux zealots who seem to have more problems with an OS they don't use, than those who use it. [/i]
Okay, for one, what you've said is <a href=https://wiki.ubuntu.com/Releases>factually wrong</a>.
For two, even if it wasn't, there's still no comparison; to upgrade to a new version of Windows you must pay around $100, and reinstall all of your programs and reconfigure all of your settings completely from scratch, [i]every time[/i].
[i]Oh, by the way, there is no statement from Apache anywhere that claims the brute force password attack (weak passwords), is how accounts were compromised. [/i]
When the alternative is reversing a one-way hash function, I think it's a safe assumption to make.
[i]You do realize, like many other high profile Linux attacks, some on our nation's defense programs, the hackers were messing around on the Apache systems for at least 4 days. It doesn't say when they were done.
FOUR DAYS.
Our network has attempted attacks made on it frequently but I'm happy to say we've never been compromised.[/i]
Four days of misconfiguration unrelated to the underlying. Microsoft can leave in known vulnerabilities being used to hack properly configured systems in their software for weeks, and people like you and Loverock [i]praise[/i] them in response. How can you hold Windows to such abyssal standards while holding Linux to impossible ones, while keeping a straight face?
still not nice....
LOL.
Matter of fact, I have stated the half truths Microsoft has put out there. Especially those from Steve Ballmer. If you knew me, you'd know I can't stand Steve Ballmer and would love to see MS can his azz as much as anyone else.
He is bad for technology and has the mentality of "milking" products, which he is too asinine to see that while he spent all that time milking XP, and telling university business students that you have learn to "milk" your products for everything they are worth, the world did not stop for him and now he's put the company in a hard position because of the technology being built all around him, while he's totally oblivious to it.
So yes. But when someone tries to say that MS developers and engineers are somehow half rate, or anything of that regard, I take issue with it.
Obviously they've had employees move to, and come from the open source world. Happens more often than you'd know. Same with Google. Google got it's best employees from Microsoft when it went on it's spree of actively trying to steal MS employees by setting up shop right next to the Redmond campus.
LOL. After years of bragging of "more secure than IIS"
Bozo the clown used to laugh a lot too
Speaking of clowns.....and Larry harmon knew why he was laughing....
He portrayed bozo in countless appearances and died a very happy and wealthy man who had brought joy to millions of children.
Yep, he knew why he was laughing.
Just like we all know that the high profile Linux based systems hacks are hilarious and laughable because of people like you telling us for years that Linux systems are inpenetrable.
You never said they were any less secure by simply running software on them...which must have been what you meant then? The linux kernel is secure, as long as you keep it on a disc in your locked safe and never let it see the light of day....yeah, then it's downright the most secure OS for sure.
Speaking of clowns, you are the house zdnet.com clown.
They show transparency
This was social engineering using XSS on another site
The Ubuntu Linux server at Apache did not even have to have a vulnerability for this type of attack to work. They did not mention what OS the url server was running but it would not have mattered because it was very likely under the control of the ones that setup the url.
This sounds like more of a browser issue than a server issue. With NoScript on Firefox, the XSS could not have run.
Mike
Social engineering wasn't an excuse when Google was hacked
Actually, It was
Also note that that, too, was a social engineering hack.
Linux wasn't hacked. RTFA before replying next time.
Message has been deleted.
RTFA Your New Fave ...