Mozilla to track infrastructure time-bombs in wake of recent Firefox armagadd-on

Browser maker lays out initial details of how an intermediate certificate expiration played hell with its add-on ecosystem.

mozilla-firefox-add-on-new-cert.png

(Image: Mozilla)

In the wake of the mass disablement of Mozilla Firefox's add-on ecosystem last weekend, Mozilla has committed to improving its asset tracking and developing a mechanism that can quickly push updates to users when needed.

Due to an intermediate certificate expiring on May 4 at 1AM UTC, users found their browser add-ons were switched off and could not be re-enabled. Thanks to timezones and the rotation of the planet, users on the western side of the Pacific were the first hit.

Writing in a blog post, Firefox CTO Eric Rescorla detailed some initial thoughts and announced a formal post-mortem would be published next week.

"First, we should have a much better way of tracking the status of everything in Firefox that is a potential time bomb and making sure that we don't find ourselves in a situation where one goes off unexpectedly. We're still working out the details here, but at minimum we need to inventory everything of this nature," Rescorla wrote.

"Second, we need a mechanism to be able to quickly push updates to our users even when -- especially when -- everything else is down.

"Finally, we'll be looking more generally at our add-on security architecture to make sure that it's enforcing the right security properties at the least risk of breakage."

Rescorla said the browser maker considered using a Firefox point release to change the date used to validate the expired certificate. It also considered generating a replacement certificate that was valid as it worked out how to get it to existing Firefox users. The latter was the eventual way armagadd-on was mitigated before Firefox 66.0.4 was released.

The Firefox Normandy Studies mechanism was chosen to get a new system add-on containing the new certificate out to users -- the mechanism has previously run foul of users for pushing unwanted code.

Responding to criticism on how long it took Mozilla to push a fix, Rescorla said the response was "quite good" from a standing start.

"First, it took a while to issue the new intermediate certificate ... the root certificate is in a hardware security module which is stored offline. This is good security practice, as you use the root very rarely and so you want it to be secure, but it's obviously somewhat inconvenient if you want to issue a new certificate on an emergency basis," he wrote.

"Second, developing the system add-on takes some time. It's conceptually very simple, but even simple programs require taking some care, and we really wanted to make sure we didn't make things worse."

Thanks to Firefox only checking Normandy updates every 6 hours, it took a while for the update to propagate to users, Rescorla said, while for those not opted into Normandy, the issue would be fixed through a point release that will be made later on.

As users who had opted out of Normandy needed to switch on the mechanism to restore add-on functionality, Mozilla said it would delete a week's worth of telemetry collected via Studies for its entire user base.

The Firefox CTO also conceded that users lost data in some cases, and it is something that will need to be looked at. He also added that a new method will need to be created for users to swiftly get updates if they had opted out of Normandy.

"The update channel should be more responsive than what we have today. Even on Monday, we still had some users who hadn't picked up either the hotfix or the dot release, which clearly isn't ideal," he wrote.

Overall, Rescorla said the Firefox team did "amazing work" in shipping a fix in less than 12 hours from it first being reported.

"As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted."

Related Coverage

Firefox add-ons disabled en masse after Mozilla certificate issue

Firefox users report having add-ons disabled, being unable to re-activate or (re)-install extensions.

Microsoft security chief: IE is not a browser, so stop using it as your default

Internet Explorer is a 'compatibility solution' and should only be used selectively, warns Microsoft exec.

Former Mozilla exec: Google has sabotaged Firefox for years

Former and current Mozilla engineers are reaching their boiling points.

Firefox to add Tor Browser anti-fingerprinting technique called letterboxing

Firefox gets another new feature from the Tor Uplift project started in 2016.

How Mozilla uses AI to manage Firefox bug reports (TechRepublic)

The company created a homegrown artificial intelligence tool dubbed BugBug to classify and categorize each bug report.