How GitHub handled getting hacked

Here is the story of how GitHub was hacked, and how the company reacted to the event. The first response was very poor, but thankfully the second one was a significant improvement.
Written by Emil Protalinski, Contributor

GitHub was hacked today in a way that exposed every repository. Russian hacker Egor Homakov discovered a public key form update vulnerability that allowed him (or anyone else, for that matter) to access any GitHub repository with full administrator privileges. As a result, anyone could, for example, commit to master, reopen and close issues in Issue Tracker, or even wipe the entire history of any GitHub project.

This was possible because of how Rails handles mass assignment of attributes (see Enlight Solutions). In short, if developers don't protect against mass assignment, it means that a malicious user can set any value in your models. GitHub has since fixed this vulnerability, but the way the company handled the event is, unfortunately, the real story.

Let's take a look at what happened this week, as well as today's culmination. It all started when Homakov opened an issue in the rails repository on GitHub titled "Mass assignment vulnerability - how to force dev. define attr_accesible?" The majority of Rails applications are likely vulnerable, but Homakov's issue was closed multiple times and he reopened it again and again to try to get his point across.

The discussion got pretty heated and it appeared as if Homakov wasn't getting anywhere because Rails maintainers simply said this was not a Rails problem. Their argument was that it's up to the developer to secure his or her code, not Rails.

You should know, if you don't already, that GitHub is written Ruby on Rails and Erlang. Homakov was also talking to GitHub privately on how to exploit this vulnerability. He responsibly disclosed it on Friday and GitHub fixed it. On Sunday though, Homakov exploited the public key form update vulnerability, which is based on the former, without responsible disclosure (meaning he did so publicly instead of contacting GitHub). GitHub then suspended his account.

Based on the issue thread, it appeared as if GitHub was ignoring Homakov and had banned him instead of paying attention to what he had to say. GitHub's first explanation of the issue was very poor. GitHub's first blog post made it look like Homakov was being malicious, as if the company had detected his "attack" and blocked it, instead of admitting that Homakov did not do any actual damage and simply tried to get his point across.

The overall handling of the situation caused many GitHub users to become angry with the company. Thankfully, GitHub reinstated his account and followed up with a better explanation. They also rolled out quick fixes, which deserves some praise.

Here's a timeline of the events, with the actual postings by Homakov and GitHub. First up, we have the commit history on the Rails GitHub project:

wow how come I commit in master? O_o +another showcase of rails apps vunlerability. +Github pwned. again :( +will you pay me for security audit?

Homakov then opened another issue on the Rails project, by overriding the timestamps in the forms to post it 1001 years into the future. It was titled "I'm Bender from Future. (the text doesn't do it justice, see the screenshot above):

Hey. Where is a suicide booth? from 3012 with love

You should check it ... #5228

Homakov followed up with the first details of his hacking in a blog post titled "Egor, stop hacking GH":

I'm not done yet. Why I do this? Since guys in rails issues ingored me and my issue I got spare time to test it on the first website i had in mind. github. That was pretty funny. Firstly, I could write post from 1234 year or 4321. Then, I could make a post pretending i am DHH. That was funny too.

Then I could wipe any post in any project. That wasn't that funny but pretty dangereous. It got more curious. Today I can pull/commit/push in any repository on github. Jack pot.

I will write big post regards this topic - examples(not only github is vulnerable this way - I found a lots of rails apps that are waiting for my hack! Yeah, it is only start). stay tuned. P.S. GH sorry, I was bored.

GitHub responded with a first blog post, the one with the poor explanation, titled "Public Key Security Vulnerability and Mitigation:

At 8:49am Pacific Time this morning a GitHub user exploited a security vulnerability in the public key update form in order to add his public key to the rails organization. He was then able to push a new file to the project as a demonstration of this vulnerability.

As soon as we detected the attack we expunged the unauthorized key and suspended the user.

At 9:53am Pacific Time this morning we rolled out a fix to the vulnerability and started an investigation into the impact of the attack. Database and log analysis have shown that the user compromised three accounts (rails and two others that appear to have been proofs of concept). All affected parties have been or will be contacted once we are certain of the findings.

The root cause of the vulnerability was a failure to properly check incoming form parameters, a problem known as the mass-assignment vulnerability. In parallel to the attack investigation we initiated a full audit of the GitHub codebase to ensure that no other instances of this vulnerability were present. This audit is still ongoing, and I am going to personally ensure that we have a strategy going forward to prevent this type of vulnerability from happening again.

I sincerely apologize for allowing this to happen. Security is our priority and I will be arranging additional external security audits above and beyond our normal schedule to further test our security measures and give you peace of mind.

When Homakov's account was suspended he posted a blog post titled "i'm disappoint, github":

Yes I behaved like a jerk. But why you suspended my account? Oh yea, Terms. But, let's get it real. It is not the way you were supposed to fix things.

I, dammit, LOVE YOU

P.S. All of you who makes fun of my posts - I'm not an english speaker at all, so please nevermind wrong using of your language. peace

An hour later, he followed up with a more detailed explanation of what he did in a blog post titled "How-to":

current page views count: 43559. is it really interesting? If so, let's walk through what I did(since GH guys told me they fixed it) 1. we have relations. Let me imagine what gh got inside of app: class PublicKey cat ~/.ssh/*pub and paste it and then submit. Then press edit on fresh-created public key and, e.g. open webinspector to add new field, like below: input type=hidden value=USER_ID name=public_key[user_id] so for my stupid prank I used USER_ID which i got at https://api.github.com/users/rails id = 4223 then press update. So, what goes on on back end? I can guess: @pk = PublicKey.find(params[:id]) @pk.update_attributes(params[:public_key]) #Oh no! We passed public_key[user_id] of our victim!

after that procedure your victim got your public key. Enjoy your pushing

Finally, GitHub published a blog post titled "Responsible Disclosure Policy," which I think is what the company should have written the first time around:

There has been some confusion over today's security vulnerability and our policy on responsible disclosure and account suspension that I'd like to clear up.

Three days ago, user @homakov opened an issue on rails/rails about the prevalence of the mass-assignment vulnerability. Two days ago he responsibly disclosed a security vulnerability to us and we worked with him to fix it in a timely fashion. Today, he found and exploited the public key form update vulnerability without responsible disclosure. For this reason we temporarily suspended his account for violation of section A8 of the GitHub Terms of Service pending a full investigation into what happened. Now that we've had a chance to review his activity, and have determined that no malicious intent was present, @homakov's account has been reinstated.

We haven't been as clear as we should have been on how to responsibly disclose security problems, and for that I'm sorry. To prevent future confusion about security-related account suspension, and to make explicit our stance on responsible disclosure, we have added a section entitled Responsible Disclosure of Security Vulnerabilities to our Security policy.

By working together we can make the development community safe and productive for everyone. Thank you for your support, and to all those that have helped us keep GitHub safe and secure.

Everything is back to normal now, right? Well, it would appear so, for GitHub, unless Homakov has more up his sleeve. That being said, GitHub isn't the only project that depends on Rails. This vulnerability will likely crop up in many other Rails projects, but that's a story for another day.

See also:

Editorial standards