Almost a week ago, Google Project Zero security researcher Tavis Ormandy discovered an issue with Cloudflare's edge servers, finding corrupted web pages being returned by some HTTP requests run through Cloudflare.
Cloudflare's edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data, with some of that data cached by search engines.
"I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything," Ormandy wrote.
The researcher's theory is that Cloudfare had code in its "ScrapeShield" feature that performed something similar to this:
int Length = ObfuscateEmailAddressesInHtml(&OutputBuffer, CachedPage);
write(fd, OutputBuffer, Length);
"But they weren't checking if the obfuscation parsers returned a negative value because of malformed HTML," Ormandy added.
"My working theory was that this was related to their "ScrapeShield" feature which parses and obfuscates HTML -- but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers."
In response to Ormandy's findings, Cloudflare identified the cause of the problem, and turned off three minor Cloudflare features that were using the same HTML parser chain that was causing the leakage: Email obfuscation, Server-side Excludes, and Automatic HTTPS Rewrites.
As a result, it was no longer possible for memory to be returned in an HTTP response.
After fetching a few live samples and observing encryption keys, cookies, passwords, chunks of POST data, and HTTPS requests for other major Cloudflare-hosted sites from other users, Ormandy said he immediately paused his investigation and contacted Cloudflare.
"We keep finding more sensitive data that we need to cleanup. I didn't realise how much of the internet was sitting behind a Cloudflare CDN until this incident," Ormandy wrote.
After back and forth with Cloudflare regarding its time to disclosure throughout the week, the DNS provider told Ormandy it would be notifying customers by Wednesday of the data leak. However, with notification still not sent by Thursday, and Project Zero's "7-day" policy for actively exploited attacks being increasingly pushed, Ormandy made public his findings on Thursday.
"Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug," the company wrote in a blog post following Ormandy's public disclosure on Thursday.
"We are disclosing this problem now as we are satisfied that search engine caches have now been cleared of sensitive information. We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence."
Cloudflare confirmed that customer SSL private keys were not leaked.
Cloudflare also confirmed that the greatest period of impact was during the period of February 13 and February 18, with around one in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage.