Log4j flaw: Why it will still be causing problems a decade from now
Despite a well-coordinated effort to rally organizations to patch to the major open-source software flaw, cybersecurity officials don't see an end to the Log4Shell problems for at least a decade.
That's the conclusion of The Cyber Safety Review Board (CSRB), a group established in February on the back of US President Joe Biden's cybersecurity executive order, which demanded a national response to major cybersecurity incidents. The group's first task was to review the Log4Shell flaws in the Log4j Java app-logging library. The board's report and recommendations are out now via the Department of Homeland Security's Cybersecurity and Infrastructure Security Agency (CISA).
Privacy
CISA swiftly ordered all federal agencies to patch Log4J to the best of their abilities a week after the remote code execution flaw was disclosed on December 9, 2021. It was considered a serious threat because the free and open-source logging component, maintained by the Apache Software Foundation (ASF), was implemented in products from VMware, IBM, Oracle, Cisco, SolarWinds, and others.
SEE: These are the cybersecurity threats of tomorrow that you should be thinking about today
Log4Shell was present in potentially hundreds of millions of enterprise devices exposed to the internet. Yet, despite or because of the massive US-led patching effort, a month after the flaw was disclosed, CISA hadn't observed any significant intrusions.
At the time, CISA warned it could be months before attackers pounced and the Federal Trade Commission signaled it might punish US organizations for inaction.
According to the CSRB, Log4Shell is now "endemic" and is expected to affect systems until at least 2032.
"Most importantly, however, the Log4j event is not over. The board assesses that Log4j is an 'endemic vulnerability' and that vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer. Significant risk remains," the board writes.
That view aligns with Microsoft's standing warning that the threat of Log4Shell to its Azure customers remains high. Microsoft in December had only observed it being used to deploy crypto-mining malware but warned state-sponsored hackers were probing systems for it.
CSRB details the havoc it caused researchers, admins, software engineers and open-source maintainers who were meant to be heading for a Christmas break after nearly two years of beefing up WFH tech during the pandemic.
"Defenders faced a particularly challenging situation; the vulnerability impacted virtually every networked organization and the severity of the threat required fast action. The fact that there is no comprehensive 'customer list' for Log4j, or even a list of where it is integrated as a sub-system, hindered defender progress," writes the CSRB's authors Robert Silvers and Heather Adkins.
The report said enterprises and vendors scrambled to discover where they used Log4j and the "pace, pressure, and publicity" compounded the defensive challenges as security researchers quickly found additional vulnerabilities in Log4j, contributing to confusion and "patching fatigue". It said defenders struggled to distinguish vulnerability scanning by bona fide researchers from threat actors; and responders found it difficult to find authoritative sources of information on how to address the issues.
"This culminated in one of the most intensive cybersecurity community responses in history," it said.
"Responders, spanning the public and private sectors, the open source community, and researchers globally, collaborated and communicated in a dedicated fashion, working through weekends and the December holidays."
Explaining the enormity of the task at hand, CSRB notes that one federal agency spent 33,000 hours patching Log4j exposures on the network. This created a backlog of work on other critical vulnerabilities, which by that stage were being flagged by CISA as mandatory for federal agencies to patch via its Known Exploited Vulnerabilities Catalog.
Yet, the board acknowledged "somewhat surprisingly" that "to date, generally speaking, exploitation of Log4j occurred at lower levels than many experts predicted, given the severity of the vulnerability".
"It has been difficult to arrive at this conclusion," the authors continue. "While cybersecurity vendors were able to provide some anecdotal evidence of exploitation, no authoritative source exists to understand exploitation trends across geographies, industries, or ecosystems. Many organizations do not even collect information on specific Log4j exploitation, and reporting is still largely voluntary."
The CSRB's list of recommendations include many activities the US government and major tech firms are already doing to improve the security of open-source software. Google's efforts include improving the security of packages for programming languages, supporting critical open-source projects with more funds, curating open-source packages, pushing the Supply chain Levels for Software Artifacts framework, or SLSA, and giving Python developers two-factor authentication security keys. These investments come at a time when 41% of organizations doubt the security of their open source software.
SEE: Google: Half of zero-day exploits linked to poor software fixes
Organizations should prepare to address Log4J in the long term while state and federal regulators should push CISA's mitigation recommendations, according to the board's review. Organizations should also have documented vulnerability response and disclosure programs.
Vendors of open-source software to federal agencies should also have a baseline for transparency. And the US should invest in incentive structures required to build secure software.
The focus on Log4Shell might be a good way to jolt US agencies and businesses into action over the security of software development, but Dan Lorenc, CEO of Chainguard and a leader of the Sigstore open-source code-signing effort, said that is going to be hard when flaws like Log4J are a vulnerability because of the way technology is implemented by customers. That is, when admins don't apply the principle of least privileges devices.
"With Log4j, preventing the entire class of bugs that cause it is going to be hard with today's technology, but stuff like fuzzing and safe-by-default language/library design can help a lot. But the full exploit still required the library to be running in an environment with significant permissions, many of which were unlikely needed," he said.