We Are in So Much Trouble – InApps Technology 2022

The open source Java logging library Apache Log4j is used a lot. And, when I say a lot, I mean it’s used in hundreds of applications. And, these aren’t just any old programs, they include <deep breath> Apache projects, such as Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark, and Struts; Apple iCloud; numerous Cisco programs; Elastic LogStash; GrayLog2; Minecraft client and server; Neo4J, Steam, Twitter, and such VMware programs as Horizon, vCenter, vRealize and Tanzu. It just goes on and on.

I’m sure there are more popular libraries out there. I just can’t think of any off the top of my head.

Check Your Apps

To see which–not if–of your applications are vulnerable check out SwitHak’s, Royce Williams’, and the Netherland’s National Cyber Security Centrum (NCSC) lists.

The zero-day vulnerability, CVE-2021-44228 – also known as Log4Shell — is both easy to exploit and can be used to grab complete control of vulnerable servers. The family of attacks that can be made via this zero-day attack is called Log4sholdShell.

As Bad as It Gets

So how bad is Log4Shell really? As bad as it gets. According to the National Vulnerability Database (NVD), it’s rated as 10.0 CVSSv3 which, by my count of a 0.1 to 10 scale, is the worst possible. If successfully exploited, attackers can hit you with a Remote Code Execution (RCE) attack, which can be used to compromise your servers. Given how easy it is to exploit, even as you’re reading this, odds are decent you’re being attacked.

As security company Nextron Systems’ head of research, Florian Roth tweeted, “The #Log4Shell vulnerability isn’t just a RCE 0day. It’s a vulnerability that causes hundreds and thousands of 0days in all kinds of software products. It’s a 0day cluster bomb.”

Use the Patch!

Yeah, this is going to be so much fun.

Any systems or services that use Apache Log4j between versions 2.0 and 2.14.1 are open to attack. There is a fixed version out now: Apache Log4j 2.15.0. You should update this as soon as possible (Editor’s note: This update turned out to be incomplete, check this post for updated details).

Well, what are you waiting for? Seriously, patch it now!

Back? Patched? Good.

What’s that? You can’t take your program down because it’s mission-critical? It’s going to get a whole lot more critical if it takes your server down or an attacker uses it to start playing Doom instead of running your service. Fortunately, Amazon Web Services (AWS) developers have created a patching program, Log4jHotPatch, that works on Amazon Corretto 8, its OpenJDK 8 implementation. Armed with the code, it shouldn’t be too much work to port it to other platforms.

Finding the Problem

But you’re not done yet. You see Log4j is used and called in so many programs even if you patch it in your systems, it’s all too likely to be hiding in other applications. And all it takes is one hole, one vulnerable program and down goes your server.

As Josh Bressers, Anchore vice president of security explained, “One of the challenges the log4j vulnerability poses is actually finding it. Java applications and dependencies are usually in some sort of packaging format that makes the distribution and running really easy, but it can make figuring out what’s inside of those software packages difficult.”

You see, Bressers continued, “When working within Java, dependencies are cataloged as Java archive files, typically called JAR files. These are packages that can then be used as a Java library but the issue here is that JAR files can start to look like Russian nesting dolls. A JAR file can contain JAR files, and those also contain JAR files. It’s basically JAR files all the way down.”

“All hope is not lost though!” ended Bressers. You can use two Anchore open-source projects Syft and Grype, to detect and identify log4j, even if it’s buried in the third circle of JAR files.

How It Works

That said, here’s how Log4Shell works.

Apache log4j library makes it very easy for developers to log various data within their applications. Users can also submit data, which will be logged. What log4j doesn’t do is sanitize that data.

Should that input contain special characters that are then logged, the Java method lookup will be called. This supports several kinds of lookup, such as the Java Naming and Directory Interface (JNDI) Lookup, JVM Input Arguments Lookup (JMX), and Web Lookup. The JNDI lookup allows variables to be retrieved and it supports remote lookup, such as Lightweight Directory Access Protocol (LDAP) and Java Remote Method Invocation (RMI). Once called, these execute the user-defined remote Java class in the local log4j server and then Bad Things Happen. Such as an RCE on the victim server.

Good News: A Fix Is in

The good news is that this zero-day, once it was spotted, was quickly fixed. Yea Apache! The bad news is that it appears to have been being attacked even before most people knew of the security vulnerability.

Cloudflare CEO Matthew Prince tweeted, the “Earliest evidence we’ve found so far of #Log4J exploit is 2021-12-01 04:36:50 UTC. That suggests it was in the wild at least 9 days before being publicly disclosed. However, don’t see evidence of mass exploitation until after public disclosure.”

Cisco Talos added that its security researchers had spotted “the vulnerability known as CVE-2021-44228 from December 2.” Therefore Talos suggests, you should look for “indicators of compromise should extend their searches to at least that far back.”

What You Should Do

Good idea. Here’s what you should do now.

First, let me repeat myself, patch it already.

The US Cybersecurity and Infrastructure Security Agency (CISA) director Jen Easterly also strongly suggests you:

1. Enumerate any external-facing devices that have log4j installed.
2. Make sure that your security operations center is actioning every single alert on the devices that fall into the category above.
3. Install a web application firewall (WAF) with rules that automatically update so that your security operating center is able to concentrate on fewer alerts.

Good luck with that last one. Everyone in IT is going to be busy as beavers in a flood for the next few weeks. The Log4j vulnerability is already being used in Muhstik and Mirai botnets to attack vulnerable servers.

Take Drastic Measures

If you want to take more drastic approaches you can take Palo Alto Networks‘ Unit 42 anti-Log4Shell defensive maneuvers:

• Disable suspicious outbound traffic, such as LDAP and RMI on the server with a firewall.
• Disable JNDI lookup.
  • Set up log4j2.formatMsgNoLookups=true
  • Remove the JndiLookup file in the log4j-core and restart the service.
• Disable JNDI
  • Set up spring.jndi.ignore=true

Of course, the problem with these approaches is you may be knocking down your applications in the process. Sometimes the cure can be worse than the disease.

If for some reason, you’re tied to a specific version of Log4j, Red Hat has some security suggestions. These are:

– In releases >=2.10, set the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true.

– For releases >=2.7 and <=2.14.1, all PatternLayout patterns can be modified to specify the message converter as %m{nolookups} instead of just %m.

– For releases >=2.0-beta9 and <=2.10.0, remove the JndiLookup class from the classpath. For example:

You can make your life easier by talking to your vendors and getting their patched programs up and running as soon as possible. You should also look into security company offerings that can address Log4j problems. These currently include offerings from TideLift, Sonatype, and NCCGroup.

You can also look for potential attacks by scanning your own logs. Microsoft has found that attacks can contain strings such as the following:

Here the attacker is performing an HTTP request against a target system, which generates a log using Log4j 2 that leverages JNDI to perform a request to the attacker-controlled site. The vulnerability then causes the exploited process to reach out to the site and execute the payload. In many observed attacks, the attacker-owned parameter is a DNS logging system, intended to log a request to the site to fingerprint the vulnerable systems.”

These strings can be identified through several components. They often contain “jndi”, which refers to the Java Naming and Directory Interface. Next, it comes with a protocol name, such as “ldap”, “ldaps”, “rmi”, “dns”, “iiop”, or “http”, preceding the attacker domain. You should set your WAF to block these strings.

Worst Security Problem in a Generation

Steve Povolny, McAfee Enterprise and FireEye‘s head of advanced threat research has said Log4Shell “now firmly belongs in the same conversation as Shellshock, Heartbleed, and EternalBlue.”

I think he’s wrong. I think this is much worse. Log4Shell may be the worst security problem in a generation. As Bressers, mentioned in a blog post, “There’s going to be a lot of false-negative reports where a product claims it doesn’t use log4j, but it does, only the log4j is hidden somewhere deep in the dependency mines. … there will be products that don’t see updates for weeks or months.”

Hang in there, folks. This is going to be a long, hard ride.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Unit.

Feature Image par Gerd Altmann de Pixabay.