Towards CVE-Free Images

To get CVE-free container images, scan each image with Trivy, trace every flagged library to a direct or transitive dependency with Gradle’s dependencyInsight, then upgrade it, force a fixed version, swap a relocated artifact or upgrade the library that shades it. Test every fix and fail CI builds on new CVEs. This post covers library CVEs in Java services.

This blog post was co-authored by Suhas Gumma and Harshit Raj.

How we fixed each type of library CVE at a glance

Dependency type	Example from our scans	Fix in build.gradle
Direct dependency	CVE-2022-25647 in gson 2.3.1	Upgrade gson to 2.8.9
Transitive, library we own	CVE-2023-6378 in logback-classic 1.2.0, via nirmata-notif-producer	Update our library, release it, and use the new version in the app
Transitive, third-party library	CVE-2024-26308 and CVE-2024-25710 in commons-compress 1.24, via avro 1.11.3	Force commons-compress 1.26.0 with resolutionStrategy
Third-party, no fixed version	CVE-2020-13939 in org.apache.velocity	Add velocity-engine-core 2.3 and exclude the old velocity module
Third-party shaded jar	CVE-2023-1370 in json-smart 2.4.8, inside nimbus-jose-jwt 9.22	Upgrade the parent, oauth2-oidc-sdk, to 11.9.1

Why fix CVEs?

The acronym CVE stands for “Common Vulnerabilities and Exposures,” a publicly known information security vulnerabilities and exposures database. Each entry in the CVE dictionary identifies a unique vulnerability or exposure and includes critical information such as a description, potential impact, and any known fixes or workarounds. It is essential to stay updated on the CVE database to ensure you are aware of any possible security threats or vulnerabilities that may affect your systems. Fixing CVEs is crucial for several reasons:

• Security: Vulnerabilities identified in CVEs can be exploited by attackers to compromise systems, steal data, or disrupt services. Fixing these vulnerabilities helps to protect systems and data from unauthorized access or manipulation.
• Risk Mitigation: Unfixed vulnerabilities pose a risk to the confidentiality, integrity, and availability of systems and data. By addressing CVEs promptly, organizations can reduce their exposure to these risks and safeguard their assets.
• Compliance: Many regulatory frameworks and industry standards require organizations to address known system vulnerabilities. Failure to fix CVEs can result in non-compliance, which may lead to legal consequences, financial penalties, or damage to the organization’s reputation.
• Trust and Reputation: Customers, partners, and stakeholders trust organizations to maintain secure and reliable systems. Demonstrating a commitment to addressing CVEs helps to build and maintain trust, enhancing the organization’s reputation and credibility.

Fixing CVEs is essential for maintaining systems’ security, stability, and trustworthiness and protecting against potential threats and vulnerabilities.

This two-part blog post details how we have removed the CVEs from our services.

How do you scan images for CVEs?

At Nirmata, our services are predominantly Java-based. We use Gradle as the building tool for building the services. The open-source tool Trivy is used to scan our images to detect CVEs. Trivy detects and reports CVEs based on whether they come from the base image or the libraries used in the app.

The following sections give details of the changes required in the Gradle build file for the various types of CVEs reported by Trivy that are coming from the libraries included in the app. A subsequent blog will describe the changes required to fix the CVEs arising from the base image.

Reading the Trivy output

Here is a sample of Trivy on one of our images.

The first row of the above output shows a HIGH CVE (CVE-2022-25647) coming from library com.google.code.gson:gson is used in building the app activity.war. The CVE arises due to using the library version 2.3.1, and it is fixed in the library version 2.8.9. The output also gives a brief description of the CVE.

Where does a vulnerable library come from?

Based on the Trivy output, it is crucial to determine the source of the CVE and whether it originates as a direct or transitive dependency. The Gradle tool dependencyInsight provides information about the library’s source.

Direct dependency

The library is directly included in the app, usually as the following dependency in build.gradle

implementation group: 'lib-package', name: 'lib-name', version: 'lib-version'

Here is an example

./gradlew :service-activity:dependencyInsight --dependency com.google.code.gson:gson 
> Task :service-activity:dependencyInsight
com.google.code.gson:gson:2.3.1
  Variant compile:
    | Attribute Name                 | Provided | Requested    |
    |--------------------------------|----------|--------------|
    | org.gradle.status              | release  |              |
    | org.gradle.category            | library  | library      |
    | org.gradle.libraryelements     | jar      | classes      |
    | org.gradle.usage               | java-api | java-api     |
    | org.gradle.dependency.bundling |          | external     |
    | org.gradle.jvm.environment     |          | standard-jvm |
    | org.gradle.jvm.version         |          | 21           |

com.google.code.gson:gson:2.3.1
\--- compileClasspath

The above output shows that the version 2.3.1 of library com.google.code.gson:gson is a direct dependency in the app and comes from the following dependency in build.gradle.

implementation group: 'com.google.code.gson', name: 'gson', version: '2.3.1'

Transitive dependency

The library is included in another library used in the app. An example is:

./gradlew :service-activity:dependencyInsight --dependency ch.qos.logback:logback-classic
ch.qos.logback:logback-classic:1.2.0
\--- com.nirmata.notification:nirmata-notif-producer:2.10.0
     \--- com.nirmata.status:nirmata-status:4.0.7
          \--- compileClasspath

The above output shows that the library ch.qos.logback:logback-classic:1.2.0 is coming from the Nirmata library com.nirmata.notification:nirmata-notif-producer:2.10.0 which in turn is coming from com.nirmata.status:nirmata-status:4.0.7

The sections below describe how we can fix the CVEs by updating the libraries in the build.gradle file.

How do you fix each type of library CVE?

MVN repository is a great place to find the CVEs associated with a particular library.

The screenshot above shows that the gson library from version 2.8.9 onwards has no CVEs.

Direct dependency

A direct dependency is the easiest to fix as the library version needs to be upgraded directly in the build.gradle for the app

In the example for CVE-2022-25647 coming from gson:2.3.1, we changed

implementation group: 'com.google.code.gson', name: 'gson', version: '2.3.1'

to

implementation group: 'com.google.code.gson', name: 'gson', version: '2.8.9'

Transitive dependency: a library you own

When the source of the transitive dependency is a library that you own, the recommended approach is to upgrade the version in the dependent library and use the updated version of the dependent library in your app.

For the case of CVE-2023-6378 originating from logback-classic:1.2.0, we updated nirmata-notif-producer and incorporated the updated version into nirmata-status to generate a new version. This updated version was then utilized in the app activity.

Transitive dependency: a third-party library

When the source of the transitive dependency is a library that you do not own i.e. a third-party/open-source library, where updates to the library may not be possible, then the recommended workaround is to do a force dependency in the build.gradle file

./gradlew :service-activity:dependencyInsight --dependency org.apache.commons:commons-compress
org.apache.commons:commons-compress:1.24
\--- org.apache.avro:avro:1.11.3
     +--- compileClasspath

For the specific case of CVEs CVE-2024-26308/CVE-2024-25710 originating from org.apache.commons:commons-compress:1.24, the source is traced back to the third-party library org.apache.avro:avro version 1.11.3. Unfortunately, org.apache.avro:avro does not offer a newer version addressing the CVEs. To resolve the CVEs, we enforced the dependency on v1.26.0 by adding the following directive to the build.gradle file.

configurations.all {
    resolutionStrategy {
          force 'org.apache.commons:commons-compress:1.26.0'
    }  
}

Transitive dependency: a third-party library with no fix version

There are some CVEs where Trivy does not show a fixed version e.g. CVE-2020-13939

In such scenarios, it is best to consult the documentation for the library or the Maven repository. In this case, the Maven repository indicated that the artifact had been moved from org.apache.velocity to velocity-engine-core.

To fix the CVE, we forced the dependency on the new library and excluded the old library in build.gradle by adding the following directives.

implementation 'org.apache.velocity:velocity-engine-core:2.3'
configurations.configureEach {
     exclude group:'org.apache.velocity', module: 'velocity'
}

In other cases where no fixed version is available, one may need to contact the maintainer or consider forking the library and resolving the CVE independently.

Transitive dependency: a third-party shaded jar

A shaded JAR is a Java ARchive file that includes all dependencies within it, with package names modified to avoid conflicts. For fixing CVEs arising from such bundled libraries, a forced dependency does not work as the library version is bundled with the shaded jar.

We faced this issue with the CVE CVE-2023-1370

To detect which jar is bundling this library, Trivy scan has to be run with extra arguments.

trivy image <image-name> -f json --list-all-pkgs

The output for which showed that the json-smart v2.4.8 is included in the shaded jar nimbus-jose-jwt-9.22.jar

        {
          "Name": "net.minidev:json-smart",
          "Identifier": {
            "PURL": "pkg:maven/net.minidev/[email protected]"
          },
          "Version": "2.4.8",
          "Layer": {
            "Digest": "sha256:abe2a302ae239ef559a986fe8535abf5406ed654cb9ac1470f4c74e5eaa7c937",
            "DiffID": "sha256:a408e99ba6dfa914f5f89b857004d1c727bb3e532545cd75fa5399eaef6a5829"
          },
          "FilePath": "usr/local/tomcat/webapps/cluster.war/WEB-INF/lib/nimbus-jose-jwt-9.22.jar"
        },

We further used dependencyInsight to get the source of nimbus-jose-jwt, which came out to be the library com.nimbusds:oauth2-oidc-sdk

./gradlew :service-activity:dependencyInsight --dependency nimbus-jose-jwt
com.nimbusds:nimbus-jose-jwt:9.22
\--- com.nimbusds:oauth2-oidc-sdk:6.21.1
     \--- compileClasspath

We fixed the CVE by forcing the dependency of this library to v11.9.1 by adding the following to build.gradle

implementation 'com.nimbusds:oauth2-oidc-sdk:11.9.1'

How do you keep images CVE-free after the fix?

Test every fix

It is crucial to thoroughly test CVE fixes to ensure they don’t break the app’s functionality and introduce regressions. At Nirmata, we used both manual and automated testing. This approach helped us identify instances where updating the affected library version fixed the CVE but broke functionality due to an overlooked update in a dependent library.

Scan on every build, and build daily

We have updated our GitHub Actions to perform a Trivy scan on the built image and fail the build if any CVEs are found. The build failure is then notified to the appropriate stakeholders, who can address the CVEs.

      - name: Perform Trivy Scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{env.REGISTRY}}/${{env.IMAGE_NAME}}:${{env.IMAGE_TAG}}
          format: 'json'
          output: 'scan.json'
          exit-code: '1'

Although we have achieved a clean record of 0 CVEs across all our repositories to date, we cannot guarantee that today’s built image will remain free of future CVEs. To detect such issues proactively, we have implemented daily builds from the main branch of all repositories. This ensures that new CVEs are promptly flagged and addressed, even in repositories with no active development.

What did this get us?

Cybersecurity vulnerabilities, known as CVEs, must be fixed to enhance security measures, mitigate potential threats, and protect sensitive data and systems from exploitation. While addressing CVEs in new projects requires minimal effort, dealing with them in legacy code can be challenging. However, the approaches above took us from hundreds of CVEs across all services to zero. This accomplishment was made possible by combining automated and manual testing, ensuring that integrating new libraries for CVE fixes did not compromise functionality.

While the number of CVEs detected in the libraries amounted to a few dozen for each of our services, the count of CVEs originating from base images used in our services reached several hundred. Our next blog post covers how we addressed CVEs from the base image. For policy-based controls on base images, see securing container base images using Kyverno policies; for why dependencies matter beyond CVEs, see XZ: a case study in open-source supply chain attacks. Related pages: Nirmata scanners and software supply chain security.

References

CVE (Common Vulnerabilities and Exposures): https://cve.mitre.org/

Frequently asked questions

Does fixing library dependencies remove CVEs from the base image too?

No. The steps in this post cover CVEs that come from libraries packaged in the app, which Trivy reports separately from operating system packages in the base image. In our services the base image was the bigger source, with several hundred CVEs against a few dozen from libraries per service, so it needs its own work.

Can forcing a newer transitive dependency break my app?

Yes. A forced version overrides what the parent library was built and tested against, so an API change in the newer version can break the app at runtime. We hit cases where the CVE was fixed but a dependent library had not been updated to match, which is why each change went through manual and automated tests.

What should I read first in a Trivy finding?

Start with four columns: the library name, the installed version, the fixed version and the file or app it was found in. The installed and fixed versions tell you the minimum upgrade. The file path tells you which build to change. If the fixed version is empty, check the Maven repository or the library docs before forcing anything.