Sophos' anti-virus software hiccuped this week as a result of a false positive update that quarantined or deleted crucial files of the anti-virus software itself; a 'friendly-fire' situation that saw many corporate and business networks vulnerable.
But the knock-on effect to businesses were wide ranging, including in one case sending a U.K. building society into the Stone Age for a day after their banking systems crumbled as a result of the anti-virus software's failure.
Earlier this week, Sophos antivirus software recognized a malware definition update as malware. The update files were detected as SSH/Updater-B malware by the Sophos software, which forced crucial update and core antimalware files into quarantine, resulting in the software grinding to a halt. Alert emails were sent en masse to administrators which required manual intervention on each and every machine, according to reports.
A subsequent update fixed the issue later on Wednesday, more than half a day since Sophos-running machines around the world first went haywire.
A senior security consultant, with knowledge of the situation, who declined to be named, told ZDNet that some businesses in the U.K. were particularly affected, including major supermarkets and banking groups.
The source said that in some cases, areas at point-of-sale -- where shoppers check out their goods for purchase -- were down for lengthy periods throughout the day, leaving shoppers unable to purchase goods through shopping tills and self-checkout areas.
What happened? Not only did the Sophos software detect its own update as a false positive, putting the anti-malware definitions and critical files in quarantine, in some cases the software also detected custom-built business applications and their files as malware threats, the source added.
This caused widespread disruption beyond the email alerts to administrators and affected customers in at least one case. Many retailers named by the source thought to be affected did not respond outside U.K. business hours.
However, one of those named by the source responded for comment. A Yorkshire Building Society spokesperson told ZDNet in an emailed statement:
...I can confirm that yes, this disruption did result in us reverting to manual transactions in a number of instances (within branch) to avoid any issues and to ensure that we continued to provide the best possible service to our customers.
The issue has now been resolved. This did not result in any security issues for our customers.
The whole experience is reminiscent of the Natwest fiasco earlier this year.
The Royal Bank of Scotland-owned Natwest banking group stumbledstemming from a company upgrade. The banking system was down for days, leaving many without pay checks and in some cases unable to withdraw cash from hole-in-the-wall machines.
Sophos spokesperson Dan Ring told ZDNet in an emailed statement that the firm apologized for the disruption, and has not only increased to maximum capacity our phone support centers around the world, but has also released online tools to help with those still affected by the bug.
"We will be conducting a thorough investigation into what went wrong, and we'll be communicating more about the background to the incident in the coming days. But right now our priority is on fixing the systems of those customers who have been impacted. As you can imagine, we are very sorry about what has occurred."
We're still waiting to hear back from a few Sophos-named clients in the U.S. and the U.K., and will update the piece if and when we hear back.
In the meantime, if you a business that has seen customers directly affected by the Sophos glitch, please leave a comment, or you can also get in touch via the email form.