Dropbox not hacked, just stupid

Summary:Far from suffering a hacking incident, the file hosting service fell victim to its update scripts and MySQL infrastructure.

Despite reports over the weekend claiming that Dropbox was hacked, the company has today released a post-mortem of its downtime, with the blame falling onto its update and database infrastructure.

The company said that the weekend's outage was due to a bug within an update script reinstalling a number of machines containing production traffic for photo sharing, camera uploads, and some APIs.

"On Friday at 5.30pm PT, we had a planned maintenance scheduled to upgrade the OS on some of our machines. During this process, the upgrade script checks to make sure there is no active data on the machine before installing the new OS," wrote Dropbox head of infrastructure Akhil Gupta.

"A subtle bug in the script caused the command to reinstall a small number of active machines. Unfortunately, some master-slave pairs were impacted, which resulted in the site going down.

"Your files were never at risk during the outage."

Gupta said that Dropbox was able to recover from backups, and able to restore "most functionality" within three hours.

However, the company was not able to fully restore complete functionality until Sunday afternoon, Pacific Time, due to the large size of the MySQL databases that the company uses.

Seemingly shocked by the time taken using the standard tooling to restore from MySQL backups, Dropbox said it has developed a tool that will allow for faster restorations by parallelising the replay of binary logs. The company plans to open source this tool in future.

In order to prevent the updating script from reinstalling active machines in Dropbox's master and dual-slave database infrastructure, Gupta said that active machines would now be able to ignore such commands.

"Over the past few years, our infrastructure has grown rapidly to support hundreds of millions of users. We routinely upgrade and repurpose our machines. When doing so, we run scripts that remotely verify the production state of each machine," Gupta said.

"We've since added an additional layer of checks that require machines to locally verify their state before executing incoming commands. This enables machines that self-identify as running critical processes to refuse potentially destructive operations."

1775 Sec, the group claiming to be behind the supposed hacking of Dropbox, later said that it was all a hoax.

Topics: Cloud, Data Centers, Data Management, Security

About

Chris started his journalistic adventure in 2006 as the Editor of Builder AU after originally joining CBS as a programmer. After a Canadian sojourn, he returned in 2011 as the Editor of TechRepublic Australia, and is now the Australian Editor of ZDNet.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.