Some Linux Foundation crack attack details emerge

Some Linux Foundation crack attack details emerge

Summary: The Linux Foundation and its sites are still down after a hack attack.

SHARE:

A well-maintained secure operating system, like Linux, can be safe. But, that doesn't mean that a Website built on top of it is necessarily safe. The Linux Foundation has found out the hard way. The Linux Foundation's main site, and related sites such as Linux.com are still down after a break-in was discovered on September 8th.

This attack came on the heels of the main Linux development site, kernel.org, being compromised in late August. Kernel.org is still down. In the meantime, Linus Torvalds has uploaded the mainline Linux source code to GitHub. This is a site that uses Git, a distributed version control system, for distributed software development. Once kernel.org is back in working order though Torvalds will be returning the code to it.

But while work continues apace on this site and over the Linux Kernel Mailing List (LKML), the Linux Foundation sites remain dark. If you visit these sites you'll find the following message:

Linux Foundation infrastructure including LinuxFoundation.org, Linux.com, and their sub-domains are down for maintenance due to a security breach that was discovered on September 8, 2011. The Linux Foundation made this decision in the interest of extreme caution and security best practices. We believe this breach was connected to the intrusion on kernel.org.

We are in the process of restoring services in a secure manner as quickly as possible. As with any intrusion and as a matter of caution, you should consider the passwords and SSH [secure shell] keys that you have used on these sites compromised. If you have reused these passwords on other sites, please change them immediately. We are currently auditing all systems and will update this statement when we have more information.

We apologize for the inconvenience. We are taking this matter seriously and appreciate your patience. The Linux Foundation infrastructure houses a variety of services and programs including Linux.com, Open Printing, Linux Mark, Linux Foundation events and others, but does not include the Linux kernel or its code repositories.

That said, according to a Linux Foundation representative, "We believe there is a connection [between the kernel.org and Linux Foundation sites attacks] but are working with security experts and authorities to confirm the details." In addition, the spokesperson said, "We are working with authorities and aggressively working to restore services."

When pressed as to who these "authorities" were, I didn't get an answer. I presume though that police and other legal agencies are looking into this as being more than just a random attack. According to the site's FAQ, "We are aggressively investigating the source of the attack. Unfortunately, we can't elaborate on this for the time being."

So if you have a Linux.com account are you in any possible trouble? Maybe. The site's FAQ notes that while the "Linux Foundation does not store passwords in plaintext. However an attacker with access to stored password would have direct access to conduct a brute force attack. An in-depth analysis of direct-access brute forcing, as it relates to password strength, can be read at Choosing Secure Passwords .We encourage you to use extreme caution, as is the case in any security breach, and discontinue the use of that password if you re-use it across other sites."

I think you should assume that, unless you used a passphrase instead of a password, that your password has been compromised. If you only used it only on that site, you're probably fine. But, if, like many people, you use the same password on many sites, change your password on those sites immediately.

How did this happen? We don't know yet. Paul Ducklin, security firm Sophos's Head of Technology, Asia Pacific, speculated that the breech was made by a malware attack. What kind of attack? We don't know that either. From an e-mail sent by John 'Warthog9' Hawley, Chief Kernel.org Administrator, it appears that the first attack came in through a malware compromised PC.

If, as appears likely, a cracker obtained high level passwords, it would have been easy to "break" into the sites. It's like "breaking" into a house if you have the key-there's really nothing to it.

Eventually, we'll find out exactly what happened. What I already know today is that no operating system, not even such security heavy-weights as Chrome OS or OpenBSD, are somehow magically immune to attacks.

Anything can be successfully attacked. It's just that some systems are easier than others. This should serve as a reminder that Linux too can be vulnerable and needs to be guarded with proper security measures. Given how slowly and carefully The Linux Foundation is restoring its systems, it's clear they've learned that lesson.

Related Stories:

Hackers break into Linux Foundation

Ghost in the Wires: The Kevin Mitnick Interview

If you have a mysterious problem with a Linux box, try bashing your system with sys_basher

Fake SSL certificates pirate Web sites

Topics: Linux, Browser, Open Source, Operating Systems, Security, Software

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

51 comments
Log in or register to join the discussion
  • RE: Some Linux Foundation crack attack details emerge

    You have to be pretty brave to hack that site.
    khess
    • must heve been

      @khess M$ moles in the Linux foundation
      The Linux Geek
      • RE: Some Linux Foundation crack attack details emerge

        @The Linux Geek You seem to have lost your mind !!!
        1773
      • RE: Some Linux Foundation crack attack details emerge

        @The Linux Geek Face the facts Linux Geek - The Linux Foundation got hacked. Period, end of story. This was not an issue of having the front door open as DTS originally postulated as a possible cause but an attack that brought down Linux hard. Tough break dude, you just found out that you guys are just as vulnerable as Mac users to malware attacks. Now you can either act on that hard truth by learning how to defend or you can keep on trollin' and stick your head in the sand and pretend this did not happen. Knowing you I'd guess it'll be the latter course of action.
        athynz
      • They do not need moles. The Linux developer's own incompetence

        @The Linux Geek
        is obviously all that was needed.

        :|
        Tim Cook
      • RE: Some Linux Foundation crack attack details emerge

        @The Linux Geek
        @Pete "athynz" Athens

        "first attack came in through a malware compromised PC"

        It wasn't malware on the linux systems, it was malware on a PC that did the attack. Most likely, passwords were compromised giving the attackers access to the systems. If you search for lists of the most commonly used passwords, you would be astonished to find out how many system admins still use these, regardless of the operating system being run. I know, I see it all the time as a consultant. Unix/Linux is still the most secure OS, but nothing is secure if you give away all the keys.

        As for some Linux Geek's posts, he gives other linux users a bad name. If this was a game, I would wonder what side he was really on.
        linux for me
      • RE: Some Linux Foundation crack attack details emerge

        @linux for me [i]It wasn't malware on the linux systems, it was malware on a PC that did the attack.[/i]

        From the link.

        [b]Earlier today discovered a trojan existing on
        HPA's personal colo machine, as well as hera. [/b]

        So... sure, it was a personal computer. Running what OS I'm not sure, but wouldn't you think folks used to running Linux would know what to look for on their personal computers? If that Trojan was on a machine running Linux, that would be telling indeed.
        Badgered
      • RE: Some Linux Foundation crack attack details emerge

        @linux for me

        [b]"first attack came in through a malware compromised PC"[/b]

        At least use the full quote:"[i]How did this happen? We don???t know yet. Paul Ducklin, security firm Sophos???s Head of Technology, Asia Pacific, speculated that the breech was made by a malware attack. What kind of attack? We don???t know that either. From an e-mail sent by John ???Warthog9??? Hawley, Chief Kernel.org Administrator, it appears that the first attack came in through a malware compromised PC.[/i]" Note the words "...it [i]appears[/i] that the first attack came in through a malware compromised PC" (Emphasis mine). So it was a PC - was it a PC running Windows, Mac, or Linux? Or are you going with the PC = Windows based Personal Computer thing?[b]


        It wasn't malware on the linux systems, it was malware on a PC that did the attack. Most likely, passwords were compromised giving the attackers access to the systems. [/b]

        Again not the "...appears to be..." part of that quote. At this point that is still not certain. But was the "malware compromised PC" actually at The Linux Foundation? How was it connected? How was it able to compromise the Linux based servers?[b]

        If you search for lists of the most commonly used passwords, you would be astonished to find out how many system admins still use these, regardless of the operating system being run. I know, I see it all the time as a consultant. Unix/Linux is still the most secure OS, but nothing is secure if you give away all the keys.[/b]

        NOW we come to another excuse - the "it was secured using a common password" theory... which is similar to DTS's "the front door was open" theory. Next.[b]

        As for some Linux Geek's posts, he gives other linux users a bad name. If this was a game, I would wonder what side he was really on.[/b]

        I seriously thing that Linux Geek needs some heavy duty meds - he really does give Linux fans a bad name. As for me I don't hate Linux at all - I do not find it to be very useful for me as a desktop OS as I play MMORPGs and sync my iPhone which is something Linux cannot do and Windows can.

        But all of that aside, how can you explain the following from the link claiming it was a malware compromised PC:

        [i]As you can guess from the subject line, I've not had what many would
        consider a "good" day. Earlier today discovered a trojan existing on
        HPA's personal colo machine, as well as hera. Upon some investigation
        there are a couple of kernel.org boxes, specifically hera and odin1,
        with potential pre-cursors on demeter2, zeus1 and zeus2, that have been
        hit by this.[/i]

        Now if Linux - as I have been told by quite a few Linux fans here on ZDNet - is invulnerable or not susceptible to malware how were those boxes hit? And by what?
        athynz
      • The malware seems to have been on a Linux machine

        @ Badgered

        The answer to your question is in the link provided by Vaughan-Nichols.

        The term 'malware compromised PC' is something that Vaughan-Nichols simply made up (as he tends to do), unless he's posted the wrong link. The link he posted makes no reference to a PC. Rather, it states that a trojan was discovered on 'HPA's personal colo machine' -- a 'personal machine', not a 'PC'.

        More importantly, the source also states that a 'trojan startup file was added to rc3.d'. As anyone familiar with Linux will know, 'rc3.d' is a directory containing start-up scripts for run level 3. The Linux run level scheme was copied from Unix, and as anyone familiar with Windows will know, Windows does not use run levels, nor has it ever.

        In short, what Vaughan-Nichols calls a 'malware compromised PC' was apparently a 'personal co[-]lo[cation] machine' running Linux. It was apparently infected, along with several other Linux machines, by a trojan that targets Linux. It was Linux malware, full stop.

        Anyone who's puzzled by a high-profile infection of Linux systems should consider the following:

        1. Every production operating system contain bugs

        2. Every user/administrator makes mistakes (much more important than 1)

        3. Containing user/administrator mistakes and managing problems caused by bugs requires considerable resources

        4. It's exceedingly unlikely that the Linux Kernel Organization, a non-profit, can match the resources of large commercial firms

        5. Despite the myths spread by the technically inept, Linux isn't inherently more secure than Windows (indeed, as Charlie Miller has pointed out, Linux desktops are probably easier to hack than Windows desktops)

        To those who haven't the first clue about security and think Linux is magically protected by pixies (i.e. most Linux zealots), the fact that hackers were able to compromise kernel.org and apparently remain undetected for some time must come as a shock. To anyone who actually understands the Linux, Unix and Windows security models, however, it isn't the least bit surprising.
        WilErz
      • RE: Some Linux Foundation crack attack details emerge

        @The Linux Geek
        There is no such company as "M$".
        tom@...
    • RE: Some Linux Foundation crack attack details emerge

      @khess
      "You have to be pretty brave to hack that site."

      Why? I can think of a lot of other sites that would be more dangerous to attack. ONE HOP is all a hacker needs to hide himself and I'll bet the number was a lot higher than 1!
      Now, if ISPs were to do their job, and authenticate EVERY piece of mail right at the first knocking on the door, and refuse/accept right there, the perps would have a LOT more trouble hiding. But hell, even I could leave a message that you'd never track down to me rght now, ISP affadavits forced by gvt subpoenas or not! ISPs just-do-not-care to be part of the solution and thus are part of the trackiing problem. Then eventually, ANY ISP not conforming would be a known rogue or personally owned source and lists of perps suddenly begin to appear in public.
      tom@...
  • Interesting

    I hope they describe how they did it. It would definitely be an interesting read.

    Despite their efforts, the Kernel code is safe, and so far, no credentials have been reported as misused (not to be confused with 'stolen'). Not too shabby.
    CommonOddity
    • Why?

      @ CommonOddity

      Why would it be interesting? At least one bit of malware involved was a simple trojan script in rc3.d. Something as obvious as a new start-up script in a privileged directory went unnoticed for weeks, on several Linux machines. Even worse, it sounds as if it was only discovered by chance, because of a bug. To top it off, they still don't even know exactly when the intrusion occurred, which implies there wasn't proper auditing in place.

      I'm sorry, but the administrators of the Linux systems behind kernel.org are clearly inept. The same applies, I'm afraid, to most Linux users/administrators I've known (I used to be a Linux user myself). They seem to think the Linux pixies are magically protecting their systems from harm. If anyone actually bothered to attack these Swiss cheese Linux machines, they could probably break in within seconds, steal all the information and leave without a trace (maybe leaving behind some hidden key loggers for good measure). I doubt the kernel.org admins are as bad as that, but it doesn't sound like they'd last long in any organisation routinely targeted by hackers (e.g. just about any large firm).

      The insecure state of so many of the Linux servers I've encountered does make me wonder how important compromised Linux servers (e.g. web servers serving malware) are to the overall malware ecosystem.
      WilErz
  • So, no update then?

    This is the exact same information that I knew yesterday. Why is it taking so longer to make information about the exploit public? We would not tolerate this from a non open source organzation, we should demand the same from Linux Foundations.
    Your Non Advocate
    • RE: Some Linux Foundation crack attack details emerge

      @facebook@... wrote:<br>"We would not tolerate this from a non open source organzation<br><br>You mean like HBGary, RSA or Lockheed Martin? I think that both kernel.org and The Linux Foundation have been VERY forthcoming so far. And bravo to SJVN for writing this article, though a bit late for kernel.org.
      Rabid Howler Monkey
    • RE: Some Linux Foundation crack attack details emerge

      @facebook@... On the contrary, this is a group who are essentially all unpaid volunteers and they've behaved much better than many corporations so far - disclosing the compromise as soon as it was found out, taking all systems that could have been affected offline, putting some back together from the ground up, etc. That's what we're owed and that's the priority - assessing damage and repairing damage and then closing security holes. The Linux kernel folks aren't under a pressing demand to explain to you in detail how to steal a member's password and get access to their systems. You're not likely to hear anything in regards to how until everything is back online, the attack is fully understood, and measures are in place to prevent it from being successful again. If law enforcement is involved, you're certainly not going to be getting details about the attack at this time.
      jgm@...
  • RE: Some Linux Foundation crack attack details emerge

    Recapping from the link SJVN provided for the kernel.org hack:<br><br><a href="http://pastebin.com/BKcmMd47" target="_blank" rel="nofollow"><a href="http://pastebin.com/BKcmMd47" target="_blank" rel="nofollow">http://pastebin.com/BKcmMd47</a></a><br><br>The attackers gained root level privileges and a trojan startup file was added to rc3d, presumably /etc/rc3.d, on both HPA's personal machine and kernel.org server, Hera. The trojan was initially discovered through Xnest /dev/mem error messages without Xnest installed.<br><br>The break-in is believed to have occurred no later than August 12, 2011, and was discovered on August 29, 2011, a duration of at least 17 days.<br><br>Doesn't sound like they had ossec-hids, or similar, installed on their servers, pushing periodic reports to sysadmins mail boxes. The directory /etc/rc3.d surely would be a prominent monitoring location.<br><br>One wonders what the kernel.org security policy looks like. I'm sure some improvements will be made once all of the details are uncovered and digested.
    Rabid Howler Monkey
  • SJVN, the Hera machine was *rooted*

    In fact, the entire infrastructure was compromised. Either way you try to spin this, a web site vulnerability can *never* explain how a machine gets root'ed.

    I think you will find that either the kernel.org maintainers were incompetent (not patching with their own patches) or that the hackers used a zero-day vulnerability.

    Personally my money would be on a vulnerability which has been fixed in the kernel repository but which hasn'tmade its way through to the distro on which kernel.org runs.

    This is a systemic problem with the Linux open source model which relies on distros to repackage. Vulnerabilities are effectively disclosed when fixes are committed to the kernel.org. But they are not <i>patched</i> until the distros incorporate those fixes into actual patch packages.
    honeymonster
    • Couldn't it have been as simple as a keylogger on a PC?

      @honeymonster - why do you say the above? Couldn't the attack boiled down to something as simple as a keylogger on a PC being used by an admin to access the web servers for admin purposes?
      daboochmeister
    • The Turnaround for Security Patches Is Very Short

      @honeymonster
      The turnaround for security patches from a project to a distribution's repositories is generally very short. It's not much of a window of opportunity.

      According to the information that's been released so far about this attack, the initial vector seems to be a compromised password, although it's unclear from the reports just how the password was compromised.

      What has yet to be released about the attack is how the attackers got from having user level access, which they probably used to ssh into the server, to root level access. Until we know how that was done, it's hard to make any pronouncements about who was incompetent or how.

      It would be possible for elevated privileges to be obtained by installing a keylogger into the account of a user who knew the root password and waiting for him to su into the root account. For the moment we don't know it was done.

      I suspect that carelessness about security on someone's part led to the obtainment of the initial password, but it would be nice to have more details about the progression of events.
      CFWhitman