Mistaken Heartbleed clean-up efforts accidentally leaving thousands of servers vulnerable
The flaw in the OpenSSL cryptographic system that sent the world into conniptions last month appears to have prompted some admins to patch unaffected systems with the buggy update, leaving them with an estimated $12m cleanup bill.
Security researcher Yngve Nysæter Pettersen has found one reason system admins should stay calm and focused when a major security flaw is discovered. In the weeks after Heartbleed's disclosure, system administrators, probably under pressure to "do something", added the flaw to around 2,500 previously unaffected web servers, according to Petterson.
Petterson, who discovered the trend during six internet scans he's run since 11 April, notes two reasons this is bad news. After disclosure, there's a heightened risk that attackers will exploit a flaw, as Canada's Revenue agency found out. The other is that there's now estimated collective cleanup bill of $12m to fix a problem that didn't exist.
"It is difficult to definitely say why this problem developed, but one possibility is that all the media attention led concerned system administrators into believing their system was unsecure," Petterson wrote.
"This, perhaps combined with administrative pressure and a need to 'do something', led them to upgrade an unaffected server to a newer, but still buggy version of the system, perhaps because the system variant had not yet been officially patched."
Admins that did patch unaffected systems with the bug should follow Petterson's advice: patch the servers with the correct update, revoke and update certificates, and change passwords, in that order.
Despite the mis-patched servers, the researcher's scan of 500,000 servers revealed that concern over the flaw did have a positive impact, with around 75 percent of affected servers patched before Petterson's first scan four days after the bug was revealed.
The problem now is that the patching effort has come to a halt with almost no change in the number of affected servers in the past fortnight. According to Petterson, the percentage of vulnerable servers has dropped from 5.36 percent on 11 April to 2.33 percent this week.
However, the percentage was already down to 2.77 percent just two weeks after his initial scan "indicating that patching of vulnerable servers has almost completely stopped".
Security researcher Robert Graham of Errata Security, who has conducted separate scans over the past month, made similar findings, this week estimating there were 318,239 Heartbleed vulnerable systems, down from 600,000 a month ago.
Another problem that some admins have glossed over is revoking and updating certificates after patching. Petterson estimated that two-thirds of the patched servers are still using the same certificates which should, in his view, be assumed to be compromised.
"Given that any server that was patched after April 7 has to be assumed to have had its certificate private key compromised (because criminals may have used Heartbleed to compromise their server), this indicates a serious problem for the users of those sites," Petterson noted.
A third reported problem, said Petterson, comes from F5 Network's BigIP SSL/TLS accelerator, some of which are running vulnerable versions of OpenSSL. Petterson advised admins to ensure that if they've installed a new BigIP server that they should upgrade the firmware before deploying it.
However, in a 12 May blog update, Petterson revised this advice based on further research: "After closer investigation together with F5, it seems that, due to an issue with the network connection of the prober the test used to detect F5 BigIP server showed higher numbers than it should have, and the numbers of such servers therefore got very inflated for the scans that were run in the past month. This means that the BigIP related information and conclusions are not correct, and I have therefore moved down and struck out the section regarding BigIP servers. My apologies to F5 and their customers for this mistake".
Admins should also remember to patch non-web systems. Calling for calm following the bug's disclosure, security researcher Dan Kamisnky advised admins to hunt down all servers that rely on SSL, including VPN.
"Find anything moving SSL, particularly your SSL VPNs, prioritising on open inbound, any TCP port. Cycle your certs if you have them, you’re going to lose them, you may have already, we don’t know. But patch, even if there’s self signed certs, this is a generic Information Leakage in all sorts of apps. If there is no patch and probably won’t ever be, look at putting a TLS proxy in front of the endpoint. Pretty sure stunnel4 can do this for you."
Read more on Heartbleed
- Internet slowed by Heartbleed identity crisis
- Heartbleed: Open source's worst hour
- Heartbleed: Serious OpenSSL zero day vulnerability revealed
This article was revised on May 14 to reflect a correction from security researcher Yngve Nysæter Pettersen to change his conclusions related to F5 BigIP servers.