Mission-critical system alert: 40-year-old OpenVMS hit by exploitable bug

The OpenVMS bug affects systems running on VAX and Alpha processors, and could impact Intel Itanium systems in mixed-architecture clusters.

Demand for new security tools grows as businesses adopt cloud computing

Video: Demand for new security tools grows as businesses adopt cloud computing

A patch is available for a privilege-escalation flaw affecting the 40-year-old OpenVMS operating system on hardware powered by ancient VAX and Alpha processors from Digital Equipment Corporation.

The OS, which has been supported by HP, is known for its reliability and has historically been used for core business systems that require high availability, including nuclear power plants and process-control systems.

The Register reports that a patch for the privilege-escalation flaw, CVE-2017-17842, has been made available ahead of a detailed description of the issue due in March. The delay is to give admins time to patch affected systems.

VMS Software Inc (VSI), the company to which HP licensed OpenVMS in 2014, said a "malformed DCL command table may result in a buffer overflow allowing a local privilege escalation in non-privileged accounts". DCL is the VMS shell.

The vulnerability affects all versions of VMS and OpenVMS dating back to version 4.0, when it was just called VMS.

While this vulnerability is exploitable on VAX and Alpha hardware, it only causes a crash on Intel Itanium-based hardware and isn't directly exploitable.

However, according to Simon Clubley, the researcher who found the flaw, a different version of the same vulnerability could make Itanium systems exploitable.

"The only reason Itanium is not compromisable with this specific version of the exploit is because the return address is handled very differently on Itanium," he wrote.


The decades-old OpenVMS operating system is known for its reliability and has historically been used for core business systems.

Image: Vaxomatic/Flickr

"It is not beyond the bounds of possibility that someone could find a different variant that could be used to compromise an Itanium system. For example, if you can overwrite a pointer to a data structure, then you can force code within DCL to process memory that you control."

Additionally, Itanium systems can be indirectly compromised using the exploit he has if they're part of a cluster with affected VAX or Alpha processors.

See also: How to manage the lifecycle of your company's computers

"If your Itanium systems are part of a mixed-architecture cluster, then you can use the vulnerability to compromise a vulnerable cluster member and then use that cluster member to compromise your Itanium systems," he said.

Clubley told The Register that anyone with shell access can compromise any version of OpenVMS released for VAX or Alpha architecture in the past 30 years.

There are different courses of action to remedy the issue for different customers, according to VMS Software's VP of software engineering, Eddie Orcutt.

Alpha customers running VSI OpenVMS V8.4-2L1 or VSI OpenVMS V8.4-2L2 for Alpha need to contact VSI support.

Customers with Itanium running VSI OpenVMS V8.4-1H1, VSI OpenVMS V8.4-2, or VSI OpenVMS V8.4-2L1 can contact HPE if they have a HPE support contract for their version. Otherwise customers need to contact VMS Software VSI support.

Customers running HPE OpenVMS versions prior to and including V8.4 must contact HPE customer support.

Previous and related coverage

Pwning the mainframe: How to hack the "most secure" platform on Earth

A researcher found a security flaw that granted him access to a mainframe's vital, sensitive data.

Making the mainframe relevant in the world of agile development and DevOps

Q&A: Compuware CEO Chris O'Malley on how the company is breathing new life into the mainframe with agile and DevOps.

It's not the SAN: ATO says mainframe needed a reboot

The Australian Taxation Office said it identified intermittent system issues affecting its mainframe and impacting services five days into tax time.

8 best practices for managing software patches (TechRepublic)

Patching software is not a glamorous job, but it can prevent disasters like the recent Equifax breach. Following these eight tips can keep your company's data safe.