GitHub shocks top developer: Access to 5 years' work inexplicably blocked

Three incidents in the past week illustrate the sometimes unavoidable risks involved in relying on cloud providers.

Developer conferences: Have they jumped the shark and forgotten about developers? ZDNet's Mary Jo Foley and Larry Dignan lament how developer conferences such as Microsoft Build, Google I/O, Facebook F8 and Apple WWDC are more about corporate vision statements and less about the actual tools developers actually use. Read more: https://zd.net/2V0mt0A

Microsoft's code-sharing site GitHub has caused a scare for developer Jason Rohrer after the company, without explanation or warning, blocked him from all his code repositories.    

GitHub has a pretty good rapport with most developers but its reputation could take a ding after Rohrer's experience with the site, which has 36 million users

Rohrer is the developer of One Hour One Life and over a dozen other games, whose open-source code is all hosted on GitHub. 

SEE: How to build a successful developer career (free PDF)

The developer yesterday posted a warning on Twitter about the potential risk to developers of using GitHub "for your life's work" after he was abruptly locked out, apparently following a single complaint from another user. 

"If you're thinking about using @github for your life's work, FYI, they may remove it without any warning or notice, based on some user 'report' made out of spite. That happened today for the 5+ years of One Hour One Life work that I'm hosting there. They didn't even email me," Rohrer wrote

Rohrer added he was "astounded by the completely unprofessional behavior" of the services he's using to run One Hour One Life. 

"If you want to position yourself as a cornerstone, you've got to behave like a cornerstone," he noted.

GitHub CEO Nat Friedman offered Rohrer an apology on Twitter over the block and confirmed Rohrer's account has been restored, also promising an investigation into why the block was implemented in the first place. 

Despite the apology from Friedman and after having received an email from GitHub support, Rohrer is still in the dark about why his account was blocked, preventing him from accessing the 5,000 commits and the 23 repositories that he's created over the past five years. 

Besides that lack of explanation, he wasn't informed by GitHub when the block was initiated. 

"The biggest problem here was that I wasn't even emailed when my account was blocked. GitHub emails me notifications all the time. For such an active account with such a deep history, taking it down in a silent POOF with no notification? I was greeted with a 404," he wrote. 

Rohrer told ZDNet that although he has backups of all his data, he was using GitHub as hub for his operation.

"My servers auto-update by pulling from GitHub, for example, and there are several third-party tools, like a tech tree browser for the game, that also pull from GitHub," he said.

Rohrer said the incident seemed to start with an issue he posted a few days earlier that had triggered GitHub's spam filter.

"It was actually the first time I ever needed to include a URL in an issue – a reference to a post in my forums where the issue was originally described. I guess posting a link in an issue is suspicious," he said.

"But blocking the entire account, and all 23 repositories, seems like an extreme response, even if what I posted was actually spam. Why not just block that one spammy message?"

GitHub told Rohrer that his account has now been pre-vetted, so there should be no repetition of the incident.

"They're also updating their spam algorithms to take account age and activity level into consideration. My account was five years old with 5,000 commits – pretty unlikely that I'd suddenly start spamming, right? 

GitHub confirmed that Rohrer's account was mistakenly flagged by its spam algorithm.

"We restored it promptly after learning of the mistake. We work hard to make GitHub a safe and inclusive place to host developer content and are constantly working on improvements to our spam filtering," a GitHub spokesperson said.

But Rohrer's experience serves as yet another reminder of the risks of depending on service providers whose technology and employees can combine to exacerbate problems for customers. 

Cloud-hosting outfit DigitalOcean caused an uproar last week after blocking the account of a small AI firm called Raisup. DigitalOcean locked the account after detecting what it thought was malicious code from Raisup's account. 

As Raisup CTO Nicolas Beauvais explained on Twitter, that supposedly malicious code was a actually a legit Python script it periodically uses to manage its databases. 

The service was effectively shut down until DigitalOcean restored the account 12 hours later, and after being restored was shut down for a further 29 hours, both times due to an error on DigitalOcean's part.   

DigitalOcean this week in a lengthy blogpost explained that the account was locked and its resources powered down "due to a false positive generated by our anti-fraud and abuse automation system". 

The system is used to monitor for cryptocurrency mining activity by looking at 'Droplet' virtual-machine CPU loads and the creation of new Droplets. 

SEE: Six in-demand programming languages: Getting started (free PDF)

After the initial 12-hour block, a mistake made by one the provider's support agents meant that when the automated system flagged the same code as malicious again, a different agent failed to recognize the alert was false positive and so "fully denied access" to the account. This decision led to the lockout of 29 hours. 

"We lost everything, our servers, and more importantly one year of database backups. We now have to explain to our clients, Fortune 500 companies, why we can't restore their account," said Beauvais. 

DigitalOcean concluded the incident was the result of "failures across people, process, and technology". The company has promised to implement several changes to avoid a repeat, including peer review for account terminations and killing off automated email responses in instances where accounts are terminated. 

Finally, Google this week had some explaining to do regarding Sunday's extensive four-hour outage, which impacted Google services as well as firms that rely on Google Cloud

A configuration change destined for a small group of servers in one region was wrongly applied by a machine or human to a larger number of servers across several neighboring regions. It resulted in regions having less than half their network capacity. 

More on Microsoft GitHub and cloud issues