CISA swiftly ordered all federal agencies to patch Log4J to the best of their abilities a week after the remote code execution flaw was disclosed on December 9, 2021. It was considered a serious threat because the free and open-source logging component, maintained by the Apache Software Foundation (ASF), was implemented in products from VMware, IBM, Oracle, Cisco, SolarWinds, and others.
According to the CSRB, Log4Shell is now "endemic" and is expected to affect systems until at least 2032.
"Most importantly, however, the Log4j event is not over. The board assesses that Log4j is an 'endemic vulnerability' and that vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer. Significant risk remains," the board writes.
CSRB details the havoc it caused researchers, admins, software engineers and open-source maintainers who were meant to be heading for a Christmas break after nearly two years of beefing up WFH tech during the pandemic.
"Defenders faced a particularly challenging situation; the vulnerability impacted virtually every networked organization and the severity of the threat required fast action. The fact that there is no comprehensive 'customer list' for Log4j, or even a list of where it is integrated as a sub-system, hindered defender progress," writes the CSRB's authors Robert Silvers and Heather Adkins.
The report said enterprises and vendors scrambled to discover where they used Log4j and the "pace, pressure, and publicity" compounded the defensive challenges as security researchers quickly found additional vulnerabilities in Log4j, contributing to confusion and "patching fatigue". It said defenders struggled to distinguish vulnerability scanning by bona fide researchers from threat actors; and responders found it difficult to find authoritative sources of information on how to address the issues.
"This culminated in one of the most intensive cybersecurity community responses in history," it said.
"Responders, spanning the public and private sectors, the open source community, and researchers globally, collaborated and communicated in a dedicated fashion, working through weekends and the December holidays."
Explaining the enormity of the task at hand, CSRB notes that one federal agency spent 33,000 hours patching Log4j exposures on the network. This created a backlog of work on other critical vulnerabilities, which by that stage were being flagged by CISA as mandatory for federal agencies to patch via its Known Exploited Vulnerabilities Catalog.
Yet, the board acknowledged "somewhat surprisingly" that "to date, generally speaking, exploitation of Log4j occurred at lower levels than many experts predicted, given the severity of the vulnerability".
"It has been difficult to arrive at this conclusion," the authors continue. "While cybersecurity vendors were able to provide some anecdotal evidence of exploitation, no authoritative source exists to understand exploitation trends across geographies, industries, or ecosystems. Many organizations do not even collect information on specific Log4j exploitation, and reporting is still largely voluntary."
Organizations should prepare to address Log4J in the long term while state and federal regulators should push CISA's mitigation recommendations, according to the board's review. Organizations should also have documented vulnerability response and disclosure programs.
Vendors of open-source software to federal agencies should also have a baseline for transparency. And the US should invest in incentive structures required to build secure software.
The focus on Log4Shell might be a good way to jolt US agencies and businesses into action over the security of software development, but Dan Lorenc, CEO of Chainguard and a leader of the Sigstore open-source code-signing effort, said that is going to be hard when flaws like Log4J are a vulnerability because of the way technology is implemented by customers. That is, when admins don't apply the principle of least privileges devices.
"With Log4j, preventing the entire class of bugs that cause it is going to be hard with today's technology, but stuff like fuzzing and safe-by-default language/library design can help a lot. But the full exploit still required the library to be running in an environment with significant permissions, many of which were unlikely needed," he said.