[Update 18.23BST: 123reg statement]
A case of fat fingers or dodgy code? We don't know -- but it's likely someone in 123reg's VPS hosting department will be looking for a new job soon.
Over the weekend, 123reg, known for providing cheap domain name registration and hosting, messed up. In what is being considered a mass deletion spree, an unspecified number of virtual private server (VPS) web hosting clients have realized their websites have faded from existence as if they never were to begin with.
For large corporations and SMBs alike, the eradication of a business' website without warning can be incredibly damaging and cause chaos for months to come -- a problem 123reg is going to have to face and explain to a horde of angry clients.
At the least, hundreds are in trouble. However, it could be thousands.
The company said in a status update on 16 April that an unspecified problem beyond "connectivity issues" and the "performance of our VPS product" was being investigated by engineers.
In a flurry of updates, 123reg said "all hands were on deck" to fix the problem, and the issue has been damaging enough to bring in external help.
123reg's Twitter account has been incredibly vague, which is likely to incense companies which are now hobbled -- or put on ice -- without a digital profile. The firm says that engineers are working on the problem, and VPS packages are being restored "using data recovery tools," which means that customer data has been wiped, somehow.
The advisory says:
"We are attempting to recover your packages bit by bit, while checking data integrity at the same time.
The process is quite time-consuming and the undergoing work is being done in parallel on multiple servers, in several locations.
Due to the nature of this process, we are unable to prioritise the recovery of specific hosts, but we are focused on restoring bulks of data at the moment. Progress is visible, and we have already managed restoring a certain number of hosts. However, we are unable to estimate when will the recovery be finished on all of the affected packages."
To make matters worse, 123reg is asking customers with backups to fix their own problem (although 123reg has been nice enough to provide a basic how-to guide).
One customer, software firm InnMaster, told The Register that on Saturday morning, someone from the web host "ran a script that had a catastrophic error in it," resulting in the deletion of servers and customer websites.
"Our hosting provider still hasn't restored anyone's data and most of their customers' websites are still offline," the company said in a statement. "They have told all customers that they would be better off starting from scratch with a blank server and rebuilding it, which is what we are now doing. Building a new web server from a fresh install is a full days' work."
As you can imagine, there are hundreds of business owners and angry customers flocking to social media to vent their frustration. Just a snapshot of the situation is below, and unfortunately, there is no word on how long it will take for service to be fully restored -- or even if this is possible.
123reg told ZDNet:
"We suffered a technical fault, which has affected a minority of 123 Reg customers. The fault was limited to 67 servers out of 115,000 (across Europe) and it is important to note that only a selection of customers on the 67 servers affected has been impacted.
We are investigating the restoration of each VPS on a case-by-case basis and are working individually with customers to keep them informed of the website recovery process. We are working as quickly as we can to restore service to normal."
Read on: Top picks