Sophos update bug cripples businesses; customers affected

Sophos update bug cripples businesses; customers affected

Summary: Sophos' antivirus update bug may have overloaded systems administrators email inboxes, but it also had a knock-on effect for businesses and end-consumers around the globe.

SHARE:

Sophos' anti-virus software hiccuped this week as a result of a false positive update that quarantined or deleted crucial files of the anti-virus software itself; a 'friendly-fire' situation that saw many corporate and business networks vulnerable.

But the knock-on effect to businesses were wide ranging, including in one case sending a U.K. building society into the Stone Age for a day after their banking systems crumbled as a result of the anti-virus software's failure. 

Read more

Cloaking tech

Image: the alert many network administrators saw when the Sophos software detected an update as false positive malware.

Sophos antivirus detects own update as false positive malware

Earlier this week, Sophos antivirus software recognized a malware definition update as malware. The update files were detected as SSH/Updater-B malware by the Sophos software, which forced crucial update and core antimalware files into quarantine, resulting in the software grinding to a halt. Alert emails were sent en masse to administrators which required manual intervention on each and every machine, according to reports. 

A subsequent update fixed the issue later on Wednesday, more than half a day since Sophos-running machines around the world first went haywire. 

A senior security consultant, with knowledge of the situation, who declined to be named, told ZDNet that some businesses in the U.K. were particularly affected, including major supermarkets and banking groups. 

The source said that in some cases, areas at point-of-sale -- where shoppers check out their goods for purchase -- were down for lengthy periods throughout the day, leaving shoppers unable to purchase goods through shopping tills and self-checkout areas. 

What happened? Not only did the Sophos software detect its own update as a false positive, putting the anti-malware definitions and critical files in quarantine, in some cases the software also detected custom-built business applications and their files as malware threats, the source added. 

This caused widespread disruption beyond the email alerts to administrators and affected customers in at least one case. Many retailers named by the source thought to be affected did not respond outside U.K. business hours.

However, one of those named by the source responded for comment. A Yorkshire Building Society spokesperson told ZDNet in an emailed statement:

...I can confirm that yes, this disruption did result in us reverting to manual transactions in a number of instances (within branch) to avoid any issues and to ensure that we continued to provide the best possible service to our customers.

The issue has now been resolved. This did not result in any security issues for our customers.

The whole experience is reminiscent of the Natwest fiasco earlier this year. 

The Royal Bank of Scotland-owned Natwest banking group stumbled as a result of a hardware failure stemming from a company upgrade. The banking system was down for days, leaving many without pay checks and in some cases unable to withdraw cash from hole-in-the-wall machines. 

Sophos spokesperson Dan Ring told ZDNet in an emailed statement that the firm apologized for the disruption, and has not only increased to maximum capacity our phone support centers around the world, but has also released online tools to help with those still affected by the bug.

"We will be conducting a thorough investigation into what went wrong, and we'll be communicating more about the background to the incident in the coming days. But right now our priority is on fixing the systems of those customers who have been impacted. As you can imagine, we are very sorry about what has occurred."

We're still waiting to hear back from a few Sophos-named clients in the U.S. and the U.K., and will update the piece if and when we hear back. 

In the meantime, if you a business that has seen customers directly affected by the Sophos glitch, please leave a comment, or you can also get in touch via the email form

Topics: Security, E-Commerce, Government UK, EU

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

8 comments
Log in or register to join the discussion
  • Affected Mostly 24/7 Systems?

    Perhaps I've been lucky, but not yet noticed any problems with my Sophos protected PCs.

    Perhaps this affected systems in large companies more than those in SMEs?
    richard.s
    • SMEs

      As an IT support provider to SMEs, we actually saw almost all of our clients running Sophos affected by this problem. And when you have companies with very little in house IT experience, even following the knowledge base is a frightening task they they wouldn't want to undertake themselves. We have found the last week an incredibly busy time trying to remedy for all our customers ranging in user size and market. Also, for those paying for Ad Hoc support, this has in some circumstances added a large extra expense in the technical resource to fix this problem.
      Victoria.
  • Got news

    A friend said his University was affected and had to use Group Policy to prevent the update
    greenoil
    • Journalistic ineptitude

      There may be an interesting story behind this shockingly inept piece of journalism, but strip away Zack's hyperbole and distortion and you are left with very thin content.

      ZDNet needs to apply some degree of editorial control over its weaker contributors.
      donaldvc
  • Took down Quickbooks and Solidworks

    We had a few machines lose quickbooks, but the biggest hiccup was sophos deleted solidworks installation files from our workstations and the install files on the server. Only way to fix it was to disable on access scanning and restore the solidworks installation files from backups. This was not a fun two days at all and now all of our workstations are stuck in sophos limbo. I cannot uninstall or reinstall sophos. We'll see how it goes over the weekend.
    Anthony Pumilia
  • Inevitable

    If you stop and think about it, this sort of thing will happen from time to time. I happened to spot it about an hour after it occurred, contacted Sophos around 21:30 and received am email with instructions in to fix the problem at 23:35.

    I am a small business customer, so receiving a fix in less than 4 hours after the problem occurred outside of office hours is a pretty phenomenal response.

    The contrast of speed and openness of Sophos compared to Natwest who took 15 months ad me reporting them to the banking ombudsman before they would they would listen to me about the security loophole I found (and could lock everyone out of their accounts) is telling.

    With something as complex as modern IT, one should expect that problems will happen from time to time. We would rather they didn't. However, it is the speed of reponse and honesty from the outset that matters most, and I personally cannot fault Sophos for the way they have handled things.
    tony@...
  • What Sophos has not shared yet!

    We are are large enterprise customer (+5000) and make use of SMC v9 and v10.
    All folders with the name *update* has been put in quarantine. Since we had the delete option set at the management console we're serverely hit.

    Ie: Sophos update, Java Update, iTunes update, HP service manager update, HP printer update etc. We hope that no users has lost data, next week we're going through SMC to find out which computers needs to be reinstalled!

    Since almon.exe is in quarantine or deleted, you can not uninstall Sophos without trouble.

    The remedy script delivered by Sophos is bloated and of no use, since the check of files is binary and not only a file check.

    We made our own script copying the needed files to version 8,9,10, uninstalling and installing Sophos AV.

    /ahave
    ahave2009
  • Not down long....

    Yes, we were hit but what saved us was the files were qurantined and not moved or deleted. I had problems with the download from the update manager (which turned out to be a bad password) but all I did was turn off the live protection (this shutdown the emails and warnings) and I was able to manually download and install the IDE files - which then were sent out and populated my machines with the updated files. All is well....I did see the issue target some google apps some of my users had installed but nothing fatal - I think the admins who had their systems remove and/or delete files on detection faired the worse of this problem.

    Ian H. - Sys Admin for a SMB with 100+ machines
    ian_hardie@...