XZ: A Case Study in Open-Source Supply Chain Attacks

CVE-2024-3094 is a CVSS 10.0 backdoor hidden in the release tarballs of XZ Utils 5.6.0 and 5.6.1. It alters liblzma, which some distributions load into OpenSSH’s sshd through libsystemd. Downgrade to 5.4.6 or upgrade to 5.6.2, then use a Kyverno policy to block images whose SBOMs list the affected versions.

CVE-2024-3094 at a glance

CVE	Component	Severity (CVSS 3.1)	Affected	Fixed	Mitigation in this post
CVE-2024-3094	XZ Utils (liblzma)	10.0 Critical	5.6.0 and 5.6.1 release tarballs	5.6.2; CISA advises downgrading to 5.4.6 Stable	Kyverno policy that checks CycloneDX SBOM attestations for liblzma or xz-utils 5.6.0/5.6.1; CI check for Dockerfiles that install xz-utils or liblzma

Sources: the NVD entry, the original oss-security disclosure by Andres Freund, and the CISA alert. For other advisories we track, see Kyverno security advisories.

What happened in the XZ supply chain attack?

The recent compromise of the XZ compression library underscores the dangers present within modern software supply chains. This incident emphasizes why vigilance and strong security measures are necessities, particularly when using open-source projects. The incident is a first of its kind and will have ripple effects throughout the community. We will need to look at what changes to the current maintainer accreditation, and acceptance practices will prevent the issue in the future. Despite the initial breakdown, this incident highlights the power of open source. The rapid detection of this attack underscores the transparency and collaborative nature of open-source models. Such threats often go undetected or unaddressed for far longer within closed ecosystems.

Detection came from one engineer noticing odd symptoms. Andres Freund saw ssh logins on Debian sid using a lot of CPU, plus valgrind errors, and traced them to liblzma.

• Social engineering success: Attackers successfully pressured the project maintainer through a series of well-crafted emails, ultimately gaining commit access to the XZ code repository. This highlights the effectiveness of social manipulation even in security-conscious environments.
• Malicious backdoor: The attackers subtly inserted a backdoor within the library’s test code. A brief code analysis reveals its intent to provide remote command execution capabilities.

How does CVE-2024-3094 affect Kubernetes clusters?

The discovery of the SSH vulnerability (CVE-2024-3094) further illustrates the risks of compromised upstream dependencies. Strictly, the flaw is in liblzma, not in OpenSSH. OpenSSH doesn’t use liblzma directly, but Debian and several other distributions patch it for systemd notification, and libsystemd depends on liblzma. That is how the backdoor reached sshd. Kubernetes environments pose specific concerns:

• Node compromise: Nodes utilizing vulnerable OS versions with the compromised XZ library could be directly targeted.
• Cascading risk to containers: Container images built upon these OS versions inherit the vulnerability, perpetuating the impact.

The two need different fixes. Nodes get patched through the OS package manager. Images need rebuilding on a clean base, and you need a way to stop old images from running in the meantime. The next section covers that second part. For the wider work of driving image CVEs down, see Towards CVE-Free Images.

How do you block XZ-affected images with Kyverno?

Suppose you are a Nirmata customer running Nirmata Control Hub or Kyverno in production. In that case, the following policy can give you a better view of this new vulnerability’s impact.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image-vulns-cve-2024-3094
  annotations:
    policies.kyverno.io/title: Verify Image Check CVE-2024-3094
    policies.kyverno.io/category: Software Supply Chain Security
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      CVE-CVE-2024-3094 is a critical vulnerability in the XZ library. Detecting
      the affected package may be done in an SBOM by identifying the "liblzma & xz-utils" package
      with one of the affected versions. This policy checks attested SBOMs in an image in CycloneDX format.
      specified under `imageReferences` and denies it if it contains versions 5.6-5.6.1 of the
      packages. Using this for your purposes will require customizing the `imageReferences`,
      `subject` and `issuer` fields are based on your image signatures and attestations.      
spec:
  validationFailureAction: audit
  webhookTimeoutSeconds: 10
  rules:
    - name: cve-2024-3094
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "*"
        attestations:
        - type: https://cyclonedx.org/schema
          attestors:
          - entries:
            - keyless:
                subject: "mysubject"
                issuer: "myissuer"
                rekor:
                  url: https://rekor.sigstore.dev
          conditions:
          - all:
            - key: "{{ components[?name=='liblzma'].version || 'none' }}"
              operator: AllNotIn
              value: ["5.6.0","5.6.1"]
            - key: "{{ components[?name=='xz-utils'].version || 'none' }}"
              operator: AllNotIn
              value: ["5.6.0","5.6.1"]

This policy utilizes the Software Bill of Materials (SBOM), attached to the image as signed metadata, i.e., an attestation, and checks the contents of the image. This will allow us to ensure that you do not permit the execution of Pods using vulnerable library versions.

This approach uses SBOMs as signed attestations for software supply chain security, a recommended best practice. However, attestations are typically unavailable for images directly used from public registries. Implementing this requires scanning, attesting all images, and checking that only trusted images are allowed. Contact us to learn how to implement a secure software pipeline.

How do you catch xz in CI before deployment?

Further, Nirmata advocates using a new principle called Shift-Down to ensure we keep our worker nodes from this attack before deployment. We can utilize the following policy inside our CI/CD pipeline to check for vulnerable images using NCTL.

apiVersion: json.kyverno.io/v1alpha1
kind: ValidatingPolicy
metadata:
  name: check-dockerfile-cve-2022-3094
spec:
  rules:
    - name: check-cve-2022-3094
      assert:
        any:
        - message: "xz-utils are being installed"
          check:
            ~.(Stages[].Commands[].CmdLine[]):
              (contains(@, 'xz-utils')): false
        - message: "liblzma is being installed"
          check:
            ~.(Stages[].Commands[].CmdLine[]):
              (contains(@, 'liblzma')): false

Why sign and verify container images?

Container image signing and verification is central to ensuring the authenticity and integrity of the code within your cloud native environments. It works like a digital fingerprint, allowing you to verify that the image you’re deploying hasn’t been tampered with or modified. This explicitly blocks potential vulnerabilities by ensuring you never execute unknown code.

Signing gives you four things:

• Increased security: Signing helps prevent attackers from injecting malicious code into your images, potentially compromising your entire system.
• Enhanced trust: Signing verifies the image’s origin and publisher, ensuring you’re deploying what you intended.
• Improved compliance: Some industries and regulations mandate signed images for security and audit purposes.
• Reduced risk: Identifying and mitigating vulnerabilities becomes easier when you know precisely what code you’re running.

How does image signing work?

The mechanism has three steps:

• Signing key pair: You obtain a signing key pair with a private key (kept secret) and a public key (distributed).
• Signing the image: The private key is used to sign the container image cryptographically, creating a signature.
• Verifying the signature: The public key and signature confirm that the image hasn’t been altered on deployment.

Popular tools for this:

• Cosign: An open-source tool based on the Sigstore standard, supporting keyless signing and various signing authorities.
• Notary: An integrated solution from Docker for signing and verifying images within their ecosystem.
• AWS Signer: A managed service from AWS for signing images stored in their Elastic Container Registry (ECR).
• Venafi CodeSign Protect: A managed X.509 and HSM service with enterprise-grade signing and verification.

Kyverno’s verifyImages rule checks Cosign and Notary signatures when a Pod is admitted. For a worked example, see Kubernetes supply chain policy management with Cosign and Kyverno.

What should you take away from the XZ backdoor?

For long-term security, defense in depth matters most. Each security measure needs constant revision. Advanced, persistent attackers will find ways to circumvent controls. Multi-layered security – from developer awareness to automated scanning tools – remains essential. Tools ensuring reliable tracking of software origins would make detecting malicious build elements easier, and that protects supply chain integrity. Use Nirmata Control Hub and Kyverno together for image security integrations, providing you with the ability to:

1. Implement image signing and verification: This ensures the authenticity and integrity of your container images, preventing vulnerabilities. Consider combining Nirmata Control Hub with tools like Cosign, Notary, AWS Signer, or Venafi CodeSign Protect.
2. Prioritize image trust and attestations: Only pull images from trusted sources and registries. Avoid untrusted images to minimize the attack surface. Attaching signed metadata, i.e., attestations like SBOMs and scan reports, helps quickly identify and address issues.
3. Maintain comprehensive patching: Patch not just Kubernetes but also low-level components like runc. A chain is only as strong as its weakest link.
4. Use Nirmata Control Hub: Use these tools to enforce security policies and automate vulnerability detection and mitigation.

The patch-the-whole-stack point came up two months earlier with the runc flaws covered in Securing containers in the wake of Leaky Vessels. For a broader view, see software supply chain security.

Visit the Nirmata website to learn more about Nirmata Control Hub, Nirmata Enterprise for Kyverno, and Kyverno OSS.

Frequently asked questions

Was the XZ backdoor in the Git repository or only in the release tarballs?

Both, in parts. Andres Freund's disclosure showed that the line that starts the attack, in build-to-host.m4, was only in the release tarballs for 5.6.0 and 5.6.1, not in the Git source. The obfuscated payload itself sat in test .xz files that were committed to the repository. Building from a GitHub-generated source archive did not run the injection step.

Is XZ Utils 5.6.2 safe to install?

The upstream 5.6.2 release, published on 29 May 2024 by the original maintainer, lists 'Remove the backdoor (CVE-2024-3094)' as its first change. Before that release existed, CISA's alert told users to downgrade to an uncompromised version such as XZ Utils 5.4.6 Stable. Either is a reasonable target today. Check what your distribution actually ships, since vendors backport fixes under their own version strings.

What does the CVSS 10.0 score for CVE-2024-3094 mean?

The CNA score uses the vector AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H. In plain terms: the attack works over the network, needs no privileges and no user interaction, and its impact reaches past the vulnerable component, with high loss of confidentiality, integrity and availability. That is the maximum score CVSS 3.1 can give, which is why distributions treated it as an emergency.

Which systems could the XZ backdoor actually run on?

The disclosure lists narrow conditions. The injection step only ran on x86-64 Linux builds made with gcc and the GNU linker, as part of a Debian or RPM package build, and the backdoor likely only works on glibc systems. Andres Freund also noted that 5.6.0 and 5.6.1 had not yet been widely integrated by Linux distributions, and mostly appeared in pre-release versions.