Sophos update bug cripples businesses; customers affected
Summary: Sophos' antivirus update bug may have overloaded systems administrators email inboxes, but it also had a knock-on effect for businesses and end-consumers around the globe.
Sophos' anti-virus software hiccuped this week as a result of a false positive update that quarantined or deleted crucial files of the anti-virus software itself; a 'friendly-fire' situation that saw many corporate and business networks vulnerable.
But the knock-on effect to businesses were wide ranging, including in one case sending a U.K. building society into the Stone Age for a day after their banking systems crumbled as a result of the anti-virus software's failure.
Earlier this week, Sophos antivirus software recognized a malware definition update as malware. The update files were detected as SSH/Updater-B malware by the Sophos software, which forced crucial update and core antimalware files into quarantine, resulting in the software grinding to a halt. Alert emails were sent en masse to administrators which required manual intervention on each and every machine, according to reports.
A subsequent update fixed the issue later on Wednesday, more than half a day since Sophos-running machines around the world first went haywire.
A senior security consultant, with knowledge of the situation, who declined to be named, told ZDNet that some businesses in the U.K. were particularly affected, including major supermarkets and banking groups.
The source said that in some cases, areas at point-of-sale -- where shoppers check out their goods for purchase -- were down for lengthy periods throughout the day, leaving shoppers unable to purchase goods through shopping tills and self-checkout areas.
What happened? Not only did the Sophos software detect its own update as a false positive, putting the anti-malware definitions and critical files in quarantine, in some cases the software also detected custom-built business applications and their files as malware threats, the source added.
This caused widespread disruption beyond the email alerts to administrators and affected customers in at least one case. Many retailers named by the source thought to be affected did not respond outside U.K. business hours.
However, one of those named by the source responded for comment. A Yorkshire Building Society spokesperson told ZDNet in an emailed statement:
...I can confirm that yes, this disruption did result in us reverting to manual transactions in a number of instances (within branch) to avoid any issues and to ensure that we continued to provide the best possible service to our customers.
The issue has now been resolved. This did not result in any security issues for our customers.
The whole experience is reminiscent of the Natwest fiasco earlier this year.
The Royal Bank of Scotland-owned Natwest banking group stumbled as a result of a hardware failure stemming from a company upgrade. The banking system was down for days, leaving many without pay checks and in some cases unable to withdraw cash from hole-in-the-wall machines.
Sophos spokesperson Dan Ring told ZDNet in an emailed statement that the firm apologized for the disruption, and has not only increased to maximum capacity our phone support centers around the world, but has also released online tools to help with those still affected by the bug.
"We will be conducting a thorough investigation into what went wrong, and we'll be communicating more about the background to the incident in the coming days. But right now our priority is on fixing the systems of those customers who have been impacted. As you can imagine, we are very sorry about what has occurred."
We're still waiting to hear back from a few Sophos-named clients in the U.S. and the U.K., and will update the piece if and when we hear back.
In the meantime, if you a business that has seen customers directly affected by the Sophos glitch, please leave a comment, or you can also get in touch via the email form.
Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.
Talkback
Affected Mostly 24/7 Systems?
Perhaps this affected systems in large companies more than those in SMEs?
SMEs
Got news
Journalistic ineptitude
ZDNet needs to apply some degree of editorial control over its weaker contributors.
Took down Quickbooks and Solidworks
Inevitable
I am a small business customer, so receiving a fix in less than 4 hours after the problem occurred outside of office hours is a pretty phenomenal response.
The contrast of speed and openness of Sophos compared to Natwest who took 15 months ad me reporting them to the banking ombudsman before they would they would listen to me about the security loophole I found (and could lock everyone out of their accounts) is telling.
With something as complex as modern IT, one should expect that problems will happen from time to time. We would rather they didn't. However, it is the speed of reponse and honesty from the outset that matters most, and I personally cannot fault Sophos for the way they have handled things.
What Sophos has not shared yet!
All folders with the name *update* has been put in quarantine. Since we had the delete option set at the management console we're serverely hit.
Ie: Sophos update, Java Update, iTunes update, HP service manager update, HP printer update etc. We hope that no users has lost data, next week we're going through SMC to find out which computers needs to be reinstalled!
Since almon.exe is in quarantine or deleted, you can not uninstall Sophos without trouble.
The remedy script delivered by Sophos is bloated and of no use, since the check of files is binary and not only a file check.
We made our own script copying the needed files to version 8,9,10, uninstalling and installing Sophos AV.
/ahave
Not down long....
Ian H. - Sys Admin for a SMB with 100+ machines