An unusual traffic spike is what alerted Facebook engineers that something might be wrong, and it was an investigation into this heightened activity that led engineers to discover a massive security breach this week.
Facebook confirmed the hack earlier today, in a press release. It said hackers stole access tokens for roughly 50 million users.
Access tokens are alpha-numeric codes that are generated when a user logs in, and are saved in the user's browser and Facebook's servers at the same time. They are used to let users access the Facebook site without prompting the user to log in on every visit --with Facebook servers checking the browser's access token in a page's background.
Facebook said earlier today hackers obtained access tokens for 50 million users by abusing a vulnerability in "View As," a feature present on each Facebook's user's profile that lets them see how their account looks through the eyes of another user.
The social network said a change in Facebook's code in July 2017 introduced this vulnerability, which the company spotted being exploited for the first time on September 16.
This is the day Facebook believes hackers started massively abusing this flaw to access the View As feature and obtain access tokens for other accounts.
The access token harvesting operation triggered the massive traffic spike on Facebook servers. Sifting through the traffic, Facebook engineers realized what was happening on September 26, and rushed to put together a patch for the vulnerability last night, on September 27, before going public with their findings this morning.
While Facebook held an initial phone conference with reporters in the morning and answered general questions, the company scheduled a second press call in the afternoon, during which Nathaniel Gleicher, Head of Cybersecurity Policy, and Guy Rosen, VP of Product Management, answered more technical inquiries.
During this second press call, Rosen said that the vulnerability in the View As feature was actually a combination of three bugs.
"The vulnerability that we fixed was the result of three distinct bugs, and was introduced in July 2017 when we created a certain new video uploader," Rosen said.
"The first bug was that when using the View As product, the video uploader actually shouldn't have showed up at all, but in a very specific case, around posts that encouraged people to wish Happy Birthday, it did show up.
"Now the second bug was that this video uploader incorrectly used SSO to generate an access token that had the permissions of the Facebook mobile app. That's not how SSO was intended to be used on our platform.
"The third bug was that when the video uploader showed up past as View As, which is something that it wouldn't do except in the case of that first bug we had, and then it generated an access, which is again something that it wouldn't do except in the case of that second bug, it generated the access token not for you the viewer, but for the user you were looking up [to preview via the View As feature].
"So it's the combination of these three bugs that created a vulnerability," Rosen said. "This vulnerability was discovered by hackers, and the way they exploited it is not just finding this vulnerability and using it to get an access token, but then every time they have an access token, pivoting from that to other accounts, to other friends to that user to get further access token."
Gleicher cited the early stage of the investigation and wasn't able to comment on the geographical reach of the breach, or if it targeted only particular types of accounts.
"We haven't yet been able to determine a pattern. The early indication makes it seem like it is very broad and there is no specific country or area targeted, but it is in its early days, and as we learn more, we will update with what we learn."
But even before going public, Facebook already countered the hack. The company invalidated 90 million access tokens in total in the past days. It invalidated tokens for the 50 million accounts that hackers had abused using the vulnerability, and the access tokens for another 40 million users who used the View As feature.
Gleicher said the company invalidated tokens for the latter out of an abundance of caution, even if they didn't see the hackers trying to interact with these accounts.
All Facebook users can access the Login and Security Facebook options page and review a list of all the devices that logged into their account, along with their respective geographical locations.
The Facebook exec said that there are cases where an attacker could use developer tools, such as libraries and command-line tools, to provide the access token and interact with a Facebook account, an interaction that would not appear on the above login list.
But because Facebook already invalidated all supposedly stolen access tokens, this means that even if hackers have these codes, they can't use them to log into an account.
But if users want to feel safe, they can click the link at the bottom of the login list that says "Log Out Of All Sessions," which would invalidate all access tokens, regardless.
Gleicher also confirmed there is no connection between today's security breach and the threats made by a security researcher yesterday, threatening to delete Mark Zuckerberg's Facebook profile during a live feed on Sunday using a new vulnerability he discovered.
Article updated at 20:35 ET, September 28, to correct attribution of some quotes.
RELATED AND PREVIOUS COVERAGE:
6 ways to delete yourself from the internet
Amid the ongoing trust crisis, Facebook users get an easier way to download their data and new mobile privacy settings.
New CJEU ruling in Facebook case could have "far-reaching effects" for GDPR contracts.
Facebook's new report attempts to convey how effective its AI is at flagging bad content and fake accounts.
Browser maker faces backlash for failing to inform users about Chrome Sync behavioral change.