How an NHS email testing mistake caused a messaging meltdown for 840,000 workers
NHS staff replying to the 'test' email only made the problem much, much worse...
An IT contractor working for a branch of the NHS in Croydon accidentally sent a test email to hundreds of thousands of staff across the entire National Health Service, causing the internal NHSmail system to seize up as workers began to respond.
Speaking to ZDNet - over the phone, of course - an NHS Digital spokesperson confirmed that the incident had taken place, with 840,000 staff affected.
The incident has been blamed on "a bug in the supplier's system" which "inadvertently included everyone on the NHS Mail list".
As NHS IT staff tried to control the situation, it was made worse by staff replying to the accidental test email asking to be removed from the list - which in turn resulted in hundreds of millions more emails being sent, further crashing the system. NHS IT staff eventually removed the ability to send to all users as they try to fix the problem.
A service status for the NHSmail 2 Outlook web application diplomatically describes the incident as "an issue with a distribution list" which has meant "several test emails have been widely received by users".
"This has been exacerbated by recipients replying in response and increasing the volume of emails associated with the list," the statement continues, adding "The impact of this issue has meant that some users are unable to access OWA due to the volume of emails being circulated"
As a result, users are suffering from email delivery delays while "the activity to clear down the emails associated with the distribution list remains ongoing".
An NHS spokesperson told ZDNet that they "think" the system is working now to the extent some messages can be sent and received - although the failure of an promised email statement from NHS Digital to ZDNet about the situation to arrive suggests that there may still some issues with the system.
"Some users have experienced short delays in the NHS Mail system this morning. Action has been taken to resolve this issue," said a statement published on the NHS Digital website.