Mistaken Heartbleed clean-up efforts accidentally leaving thousands of servers vulnerable

Mistaken Heartbleed clean-up efforts accidentally leaving thousands of servers vulnerable

Summary: Security researcher blames media hype for admins adding the Heartbleed flaw to previously unaffected servers.

TOPICS: Security, Servers

The flaw in the OpenSSL cryptographic system that sent the world into conniptions last month appears to have prompted some admins to patch unaffected systems with the buggy update, leaving them with an estimated $12m cleanup bill.

Security researcher Yngve Nysæter Pettersen has found one reason system admins should stay calm and focused when a major security flaw is discovered. In the weeks after Heartbleed's disclosure, system administrators, probably under pressure to "do something", added the flaw to around 2,500 previously unaffected web servers, according to Petterson.

Petterson, who discovered the trend during six internet scans he's run since 11 April, notes two reasons this is bad news. After disclosure, there's a heightened risk that attackers will exploit a flaw, as Canada's Revenue agency found out. The other is that there's now estimated collective cleanup bill of $12m to fix a problem that didn't exist.

"It is difficult to definitely say why this problem developed, but one possibility is that all the media attention led concerned system administrators into believing their system was unsecure," Petterson wrote.

"This, perhaps combined with administrative pressure and a need to 'do something', led them to upgrade an unaffected server to a newer, but still buggy version of the system, perhaps because the system variant had not yet been officially patched."

Admins that did patch unaffected systems with the bug should follow Petterson's advice: patch the servers with the correct update, revoke and update certificates, and change passwords, in that order.

Despite the mis-patched servers, the researcher's scan of 500,000 servers revealed that concern over the flaw did have a positive impact, with around 75 percent of affected servers patched before Petterson's first scan four days after the bug was revealed.

The problem now is that the patching effort has come to a halt with almost no change in the number of affected servers in the past fortnight. According to Petterson, the percentage of vulnerable servers has dropped from 5.36 percent on 11 April to 2.33 percent this week.

However, the percentage was already down to 2.77 percent just two weeks after his initial scan "indicating that patching of vulnerable servers has almost completely stopped".

Security researcher Robert Graham of Errata Security, who has conducted separate scans over the past month, made similar findings, this week estimating there were 318,239 Heartbleed vulnerable systems, down from 600,000 a month ago. 

Another problem that some admins have glossed over is revoking and updating certificates after patching. Petterson estimated that two-thirds of the patched servers are still using the same certificates which should, in his view, be assumed to be compromised.

"Given that any server that was patched after April 7 has to be assumed to have had its certificate private key compromised (because criminals may have used Heartbleed to compromise their server), this indicates a serious problem for the users of those sites," Petterson noted.

A third reported problem, said Petterson, comes from F5 Network's BigIP SSL/TLS accelerator, some of which are running vulnerable versions of OpenSSL. Petterson advised admins to ensure that if they've installed a new BigIP server that they should upgrade the firmware before deploying it.

However, in a 12 May blog update, Petterson revised this advice based on further research: "After closer investigation together with F5, it seems that, due to an issue with the network connection of the prober the test used to detect F5 BigIP server showed higher numbers than it should have, and the numbers of such servers therefore got very inflated for the scans that were run in the past month. This means that the BigIP related information and conclusions are not correct, and I have therefore moved down and struck out the section regarding BigIP servers. My apologies to F5 and their customers for this mistake".

Admins should also remember to patch non-web systems. Calling for calm following the bug's disclosure, security researcher Dan Kamisnky advised admins to hunt down all servers that rely on SSL, including VPN.

"Find anything moving SSL, particularly your SSL VPNs, prioritising on open inbound, any TCP port. Cycle your certs if you have them, you’re going to lose them, you may have already, we don’t know. But patch, even if there’s self signed certs, this is a generic Information Leakage in all sorts of apps. If there is no patch and probably won’t ever be, look at putting a TLS proxy in front of the endpoint. Pretty sure stunnel4 can do this for you."

Read more on Heartbleed


This article was revised on May 14 to reflect a correction from security researcher Yngve Nysæter Pettersen to change his conclusions related to F5 BigIP servers.

Topics: Security, Servers

Liam Tung

About Liam Tung

Liam Tung is an Australian business technology journalist living a few too many Swedish miles north of Stockholm for his liking. He gained a bachelors degree in economics and arts (cultural studies) at Sydney's Macquarie University, but hacked (without Norse or malicious code for that matter) his way into a career as an enterprise tech, security and telecommunications journalist with ZDNet Australia. These days Liam is a full time freelance technology journalist who writes for several publications.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • it doesn't effect me

    because we use windows servers so we don't need to worry about it.
    • Nope...

      You only have to worry about getting audited...
      virus infections...
      And general insecurity...
    • Looks like shellcodes_coder has switched sides (as well as handles)

      Rabid Howler Monkey
    • No platform is secure

      Therefore hubris is not allowable by anybody.
    • lol.. clearly you don't know IIS history.

      You do know that IIS was once a star performer in the Microsoft virus transport system right? It was right up there with IE and OE in serious security flaws. In fact with activeX, IE and IIS the bad guys got up to no end of bad 5h1t. Only reason it didn't affect as many as heartbeat is because IIS has never been a class leader... Apache on the other had has been for an eternity.

      Its good that it happened to someone else for a change right?
      • You made an interesting statememt

        you talked about IIS in the past, while he was talking about OpenSSL in the present.

        I'm not sure how that negated what he said.
  • Fool me once ...

    By itself I would tend to just read this story and move on. But remember that earlier story about 25 brands of DSL wireless routers that were found to have a back door? A patch was issued, but security researchers found that the patch did not close the back door, but only made it harder to detect!

    When Heartbleed first surfaced, it was revealed that US intelligence agencies, most especially the NSA, had known about and been exploiting Heartbleed for at least two years prior to the public becoming aware of the problem. This suggested that Heartbleed was not a coding error as claimed, but yet another intentional back door built into the system by the NSA.
    Michael Rivero
    • Since I got on the meds

      I don't need my tin hat anymore. I'll sell it to you cheap.
    • Re: Fool me once ...

      The keyword in open source is open. Could an NSA coder, or a coder working for any other country's version of NSA, contribute code to OpenSSL? Certainly. But they do so with the knowledge that any coder could discover their code at any time, particularly a coder working for their "competition". Not a very reliable method of installing back doors, to say the least.

      Even if a coder with some spy agency contributed code purposefully installing a back door in OpenSSL, we still would have no way to know that NSA did it. Discovering it does not equate to guilt in causing it, so there is no evidence whatsoever that any intelligence agency had any part in causing the flaw. In truth, that NSA knew about the flaw is only rumor. The only evidence to support such a suggestion is that they do in fact make an effort to look for exploits, as of course does every other intelligence agency.

      I would lean more towards believing that nearly all of the world's intelligence agencies knew about heartbleed, yet none of them actually inserted the code. It was just a programming mistake that they discovered and took advantage of, rather than reporting.

      Naturally, any intelligence agency, or spy agency, is by definition unethical and can be counted on to report an exploit if, and only if, they are fairly certain their counterparts already know about it as well, in which case it is better to report the flaw and get it fixed so as to deny their counterparts its use. So you see, because of the increased risk of discovery, open source is not a good target for the insertion of back doors. Better to look for the inadvertent mistakes made by legitimate coders, which are much harder for their counterparts to discover.
      • Don't rely on phantom eyeballs

        Yes, those phantom eyeballs could pick up malicious code introduced (perhaps as a "fix") by malicious contributors.

        Just as they picked up an obvious defect in SSL that allows input to ask for arbitrary length strings to be returned from memory. Oh wait, Heartbleed is exactly a failure to do that, missing a defect that was there for years?
    • Wow talk about chicken little -

    • I fed my password through a secret decoder ring app

      It came back with an NSA message that read "Be sure to drink your Ovaltine" 8-[
    • DSL Router

      DSL routers/modems are supposed to have 'backdoors', else how else is the ISP supposed to do any troubleshooting when the customer complains that their 'internet is broken'? News outlets who are otherwise having a slow news day are perfectly willing to downplay that aspect in favor of more sensationalist headlines.

      If you want some kind of actual control and are forced to use an awful DSL router then DMZ an actual, proper router (preferably something with DD-WRT/OpenWRT) to it and just use that.
    • Can you support those claims?

      Do you have any authoritative reports or links to back up those claims Michael?

      The NSA explicitly denied knowing about it prior to 2014. http://www.bloomberg.com/video/what-and-when-did-nsa-know-about-heartbleed-bug-CcNm~m5ZSzC9GWx4na6L2Q.html
    • Or those backdoors were for ISP upgrades and troubleshooting?

      A proper domain setup has the users accessing their computers with a user logo, with rights restrictions, while us admins have total access to the computer with our logon.

      Now, going by your theory, that's just because we like to snoop on employee's computers....
  • so blame media?

    Sloppy work after inattentiveness isn't the problem. Nope, because the media publicized the problem to make consumers aware, that's the problem!
    • No, not media, per say.

      I think what everyone is dancing around, but doesn't want to say, is that many people were so giddy that an open-source project actually had a serious flaw, that they all happily jumped on the panic button. Look at Larry Seltzer's posts on Heartbleed here on ZDNet for example. It seemed to me, that he was advocating for giving Microsoft a complete monopoly on all Internet security software, because apparently, he is from a parallel universe where MS has never had a software security issue. Posts like those and others all over Internet, caused naive admins around the world to do many things that were not required, while not doing things they should have.

      We had several customers, that, in the past, have had all kinds of malware on Windows servers and PCs, but never once called to ask about any MS zero-day security issue. With Heartbleed, they were having managers fly in from all over for meetings and dragging IT people onto the carpet to explain. I had one CFO ask about updating his 8 year old flip phone.

      The result, and poiint of this article: Lots of admins with too little knowledge went way overboard.
      • Admins should know better than to "whack all the proverbial buttons"...

        ... and patch everything in sight, "just in case".

        IMO, the documentation was pretty clear with regard to which versions were affected and how to patch them. It does indeed seem to be case of Admins who didn't RTFM and who probably should never have been given root in the first place.
        • part of the deal with open source

          Open to screw-ups. Good thing it was free... or at least some of it was free... but they paid in the end.
      • CFO's flip phone

        Save time with the CFO by just telling him that his 8 year old flip phone is not vulnerable since it runs on DC.