'

Major Linux distros have Meltdown patches, but that's only part of the fix

All the major Linux distributions have now released their Intel chip meltdown patches. But, someone must retune all those servers to get their performance up to speed and replace network devices and servers running up-to-date Linux distros.

Video: Why microprocessor systems' architecture needs to go open-source

The Intel Meltdown security problem is the pain that just keeps hurting. Still, there is some good news. Ubuntu and Debian Linux have patched their distributions. The bad news? It's becoming clearer than ever that fixing Meltdown causes significant performance problems. Worst still, many older servers and appliances are running insecure, unpatchable Linux distributions.

But first, let's look at what happens to performance with the patches. Red Hat's Meltdown/Spectre performance benchmarks found with the Linux Meltdown patches have the following performance problems:

  • Measurable: 8 percent to 19 percent -- Highly cached random memory with buffered I/O, OLTP database workloads, and benchmarks with high kernel-to-user space transitions are impacted between 8 percent to 19 percent. Examples include OLTP Workloads (tpc), sysbench, pgbench, netperf (< 256 byte), and fio (random I/O to NvME).
  • Modest: 3 percent to 7 percent -- Database analytics, Decision Support System (DSS), and Java VMs are impacted less than the "Measurable" category. These applications may have significant sequential disk or network traffic, but kernel/device drivers are able to aggregate requests to moderate level of kernel-to-user transitions. Examples include SPECjbb2005, Queries/Hour, and overall analytic timing (sec).
  • Small: 2 percent to 5 percent -- HPC (High Performance Computing) CPU-intensive workloads are affected the least, with only 2 percent to 5 percent performance impact, because jobs run mostly in user space and are scheduled using cpu-pinning or numa-control. Examples include Linpack NxN on x86 and SPECcpu2006.
  • Minimal: Linux accelerator technologies that generally bypass the kernel in favor of user direct access are the least affected, with less than 2 percent overhead measured. Examples tested include DPDK (VsPERF at 64 byte) and OpenOnload (STAC-N). Userspace accesses to VDSO like get-time-of-day are not impacted. We expect similar minimal impact for other offloads.

You can expect to see similar problems with Windows servers. There is no way to patch this performance problem -- even with chip microcode -- with any of today's processors.

True, Intel has released microcode definitions for all its processors, but we don't know yet how much this will mitigate the overall performance problems.

Google claims its "Retpoline," a binary modification technique that protects against "branch target injection" attacks, doesn't damage cloud performance. Early benchmarks don't back up those hopeful claims. Just as with Linux's Kernel Page Table Isolation (KPTI) patches, "most of the Retpoline performance impact comes down to I/O workloads and those with high kernel interactivity."

The bottom line: On the Linux desktop, just as on Windows, you won't see that much of a slowdown from the patches. It's a different story with your servers -- whether you're running Linux on a standalone server or on a cloud's virtual machines (VM)s and containers. If you're a sysadmin, you'll be doing a ton of application performance testing and rebalancing.

Performance woes and all, at least you can protect yourself from attacks after the patches. On far too many systems, patching isn't an option.

Some of this you probably already know. Many consumer electronic devices use Linux, but they can't be patched.

As security expert Bruce Schneier wrote, Meltdown and Spectre "affect embedded computers in consumer devices. Unlike our computer and phones, these systems are designed and produced at a lower profit margin with less engineering expertise. There aren't security teams on call to write patches, and there often aren't mechanisms to push patches onto the devices. We're already seeing this with home routers, digital video recorders, and webcams. The vulnerability that allowed them to be taken over by the Mirai botnet last August simply can't be fixed."

It's not just Linux-powered consumer devices. Linux and open-source software powers many firewall, Domain Name System (DNS), load balancing, internet gateways, VPN hardware, and authentication and key encryption appliances. CentOS, the Red Hat Enterprise Linux (RHEL) clone, is the most frequently used distribution.

But, as Richard Morrell, CTO and security lead of Falanx, a cyber defense company, points out: "Many (a lot) of these devices are still running platforms that started out in the development lab at the vendor as CentOS 4/5/6/7 development trees. For the later versions that's fine and dandy, kernel and microcode patches are available due to CentOS benefitting from the hard work Red Hat did to get the patches out for a multitude of architectures." But, many of the older "devices are running versions 4 and 5 and have long since departed from being 'standard builds.'"

These won't be patched. Morrell continued [sic], "A huge chunk of our security estate is built out on non-supported, non-patchable variants of CentOS and other Linux variants. ... Many of these devices are end of life and still in use in many organisations who haven't removed them ... because they still work and are the glue they require, and if it ain't broke why fix it."

Well, now they are broken. Since patching them in a non-starter, you'll need, at a minimum, to monitor your systems more closely than ever for attackers.

A wiser decision for companies will be to replace this newly vulnerable gear. As for security vendors, it is long, long past time that, when they refresh their appliances, they do so to fully supported and patchable Linux distributions. If they don't, they'll be in a world of legal pain after the first Meltdown attacks materialize.

Related stories