Meltdown and Spectre response hampered by 'exclusive club' secrecy

The secrecy that still surrounds many details of the Meltdown and Spectre vulnerabilities caused problems, and continues to cause problems, according to speakers at the linux.conf.au open-source software conference in Sydney on Thursday.

Information about newly-discovered software security vulnerabilities is normally kept under embargo until a fix is ready for distribution. It's a mature and well-understood process. But in the case of these hardware vulnerabilities, things didn't run as smoothly.

"Normally when an embargo ends we get timelines, and we get pretty much a full disclosure of what happened," said Jonathan Corbet, who maintains the documentation for the Linux kernel, and is a member of the Linux Foundation's Technical Advisory Board.

"In this case there's still a lot of secrecy around what went on with Meltdown and Spectre, and how they were handled, and exactly what was happening in the three months or so between the initial disclosure [and] when the Linux kernel community began to hear about it," he said.

"We do actually have mechanisms for disclosure within the community that have solved a number of these problems, at least for most of the issues that we've had. It's not perfect, but it works fairly well. Those mechanisms were not used this time around. This disclosure process was handled very differently. Why? I don't actually have an answer to that."

Jess Frazelle, who works on open-source software, containers, and Linux at Microsoft, was even blunter -- though like all of the panellists she was speaking in a personal capacity.

"I think maybe a way to fix it in the future would obviously be not having an absolute sh*t show of an embargo," Frazelle said.

Some of the secrecy has been surreal, even Kafkaesque.

"There are people who've said publicly at this conference they were not even allowed to say the names of these vulnerabilities," said Corbet, referring to Intel's own Casey Schaufler.

Schaufler was presenting a session on the future of security in the Linux kernel, but was banned from even mentioning the most important problem in his company's products since the Pentium FDIV bug a generation ago.

"I would like to see an end to that," Corbet said. "I'd like the industry to end at least that piece of it, so that we can get the whole story out there, and figure out how to do better the next time around."

Katie McLaughlin is the site reliability engineer for Divio, a Zürich-based Django and Python-based cloud hosting service. Even though they're a second-tier cloud provider, she only heard about the vulnerabilities when information started to appear on Twitter.

"It seemed like there was kind of a decision that some cloud providers know and some don't," McLaughlin said. Smaller cloud providers knew nothing, even though they actually pay for hardware.

"I'm unsure exactly what happened there, but it seems to be like an exclusive club as to whether you know or don't know, and it's not really clear the lines of who should be informed."

Corbet agreed. "I'd hate to see a world where only the very biggest cloud providers have access to information about something like this because, you know, that's a competitive [issue]."

Benno Rice, a core team member of the FreeBSD operating system's development community, said that their developers had little warning.

"From the FreeBSD perspective, our primary gripe is that despite having relationships with a lot of the vendors involved, we didn't find out until very, very late in the piece," Rice said.

"We had, I think, 11 days between when we were told, from when the embargo stopped, to develop ... kernel page table isolation or something like that."

Rice said he didn't put the late notice down to any malice, and said to laughter that he hoped it was a once in a lifetime event.

Now read: Cybersecurity in 2018: A roundup of predictions

"It's the largest customers of Intel products that get the first drops from Intel, and as community projects that don't have a specific vendor relationship with Intel, that puts us kind of on a 'dunno' list."

Even Google's kernel developers had trouble getting information, at least initially.

"Within Google, even though it was pretty well contained, not a lot of people knew about it," said Kees Cook, a Linux kernel security engineer who works on Android and Chrome OS.

"When I did find out about it, it was pretty constrained ... Trying to make sure that everyone who needed to know about it got through some approval process to learn about it proved to be somewhat difficult," he said, although things improved after the initial problems.

"A lot of people talk about how the embargo was a complete disaster. From my perspective, it seemed like notification internally during the embargo was where most of the problem was. The embargo itself was relatively successful, and only broke six days early from something that started in June the year prior ... I thought that was relatively successful, and the things that could be developed in the open were developed in the open, and that seemed to go pretty well."

Is open hardware the answer?

Could vulnerabilities like Meltdown and Spectre be spotted more quickly if processors moved to more open architectures, designs that could be more directly reviewed and influenced by software communities?

Singapore-based hardware hacker Andrew "bunnie" Huang thinks not.

"Unfortunately, I think in the case of this particular bug, all the ingredients that were necessary to make it happen were actually publicly disclosed information. We all know that speculative execution occurs. We all know there's timing side channels [that can be used in attacks]," Huang told the conference.

"One of my favourite quotes from [mainframe computing pioneer] Seymour Cray was that memory is like an orgasm. It's better if we don't have to fake it. And every single time that you try to fake some bit of performance, you're going to have a timing side channel," he said.

Huang thinks that open hardware might help with finding other kinds of bugs, however.

"There's a whole class of things that exploits hardware that haven't even been disclosed yet, that they're just waiting to come out ... All these things you don't even know about that are inside the processors, you would be able find about, and review, and be like, 'Holy cow that's really scary'."

According to Cook, Meltdown and Spectre highlight the need to pay attention to the more paranoid individuals.

See also: Spectre puts the brakes on CPU need for speed

"We've understood timing side channels for a long time, but there wasn't what you could consider a practical attack using them," Cook said. "Yeah, but maybe I'm not smart enough to find the practical attack, but it might still be there. Someone else might have found it."

The problem, of course, is that end users will continue to demand better performance.

As Huang put it, "It's an arms race to get there very quickly, and the thing that was exploited was, you know, a thing that was involved in getting you good performance on those devices."

Speculative execution was a concept built into the design, and that concept proved to be flawed. Huang said it'll be interesting to see how this plays out for Intel, because the Pentium FDIV bug cost them $475 million in 1994 money.

"You can scale that up to what it might look like for Intel now," Huang said.

Huang thinks the fear of such massive payouts might dissuade vendors from moving to more open designs.

"The response will be in terms of, well, this is absolutely why we should keep everything closed, because it's really expensive if you guys find out about our bugs that we happen to ship in our hardware that's been there for years and years and years," he said.

"It will be interesting to see how that all plays out, and how that interacts with the chip designers and their sort of paranoid mentality about sharing documentation."

Huang also wonders whether tight embargoes really help.

"Who are you actually trying to protect against by embargoing? Are you trying to make sure that random script kiddies don't use this? Or are you looking to keep state-level actors away from trying to exploit every computer in the world? ... If you're actually looking to protect, for example, against state-level actors, those guys may already be listening to your comms, and they would have known the exploit the same time you guys would have known it," Huang said.

"Actually just opening it up to the whole community to solve the problem, and having us all band together against state-level actors, would have been a much more powerful response," he said.

"As a hardware engineer, I find it insane that you guys think that you can run secrets with non-secrets on the same piece of hardware."

Related Coverage

• Linux and Intel slowly hack their way to a Spectre patch
• Industrial equipment manufacturers reporting difficulties with Meltdown and Spectre patches
• Spectre and Meltdown: Linux creator Linus Torvalds criticises Intel's 'garbage' patches
• Why Intel x86 must die: Our cloud-centric future depends on open-source chips
• Spectre and Meltdown: Cheat sheet (TechRepublic)
• Intel: Don't install our Spectre fix, risk of unwanted reboots is too great (TechRepublic)