This blog post was co-authored by Suhas Gumma and Harshit Raj. 

Introduction

The acronym CVE stands for “Common Vulnerabilities and Exposures,” a publicly known information security vulnerabilities and exposures database. Each entry in the CVE dictionary identifies a unique vulnerability or exposure and includes critical information such as a description, potential impact, and any known fixes or workarounds. It is essential to stay updated on the CVE database to ensure you are aware of any possible security threats or vulnerabilities that may affect your systems. Fixing CVEs is crucial for several reasons:

• Security: Vulnerabilities identified in CVEs can be exploited by attackers to compromise systems, steal data, or disrupt services. Fixing these vulnerabilities helps to protect systems and data from unauthorized access or manipulation.
• Risk Mitigation: Unfixed vulnerabilities pose a risk to the confidentiality, integrity, and availability of systems and data. By addressing CVEs promptly, organizations can reduce their exposure to these risks and safeguard their assets.
• Compliance: Many regulatory frameworks and industry standards require organizations to address known system vulnerabilities. Failure to fix CVEs can result in non-compliance, which may lead to legal consequences, financial penalties, or damage to the organization’s reputation.
• Trust and Reputation: Customers, partners, and stakeholders trust organizations to maintain secure and reliable systems. Demonstrating a commitment to addressing CVEs helps to build and maintain trust, enhancing the organization’s reputation and credibility.

Fixing CVEs is essential for maintaining systems’ security, stability, and trustworthiness and protecting against potential threats and vulnerabilities.

This two-part blog post details how we have removed the CVEs from our services.

Background

At Nirmata, our services are predominantly Java-based. We use Gradle as the building tool for building the services. The open-source tool Trivy is used to scan our images to detect CVEs. Trivy detects and reports CVEs based on whether they come from the base image or the libraries used in the app.

The following sections give details of the changes required in the Gradle build file for the various types of CVEs reported by Trivy that are coming from the libraries included in the app. A subsequent blog will describe the changes required to fix the CVEs arising from the base image.

Trivy output

Here is a sample of Trivy on one of our images.

The first row of the above output shows a HIGH CVE (CVE-2022-25647) coming from library com.google.code.gson:gson is used in building the app activity.war. The CVE arises due to using the library version 2.3.1, and it is fixed in the library version 2.8.9. The output also gives a brief description of the CVE.

Determining the library’s source using dependencyInsight

Based on the Trivy output, it is crucial to determine the source of the CVE and whether it originates as a direct or transitive dependency. The Gradle tool dependencyInsight provides information about the library’s source.

Direct dependency

The library is directly included in the app, usually as the following dependency in build.gradle

implementation group: 'lib-package', name: 'lib-name', version: 'lib-version'

Here is an example

./gradlew :service-activity:dependencyInsight --dependency com.google.code.gson:gson 
> Task :service-activity:dependencyInsight
com.google.code.gson:gson:2.3.1
  Variant compile:
    | Attribute Name                 | Provided | Requested    |
    |--------------------------------|----------|--------------|
    | org.gradle.status              | release  |              |
    | org.gradle.category            | library  | library      |
    | org.gradle.libraryelements     | jar      | classes      |
    | org.gradle.usage               | java-api | java-api     |
    | org.gradle.dependency.bundling |          | external     |
    | org.gradle.jvm.environment     |          | standard-jvm |
    | org.gradle.jvm.version         |          | 21           |

com.google.code.gson:gson:2.3.1
\--- compileClasspath

The above output shows that the version 2.3.1 of library com.google.code.gson:gson is a direct dependency in the app and comes from the following dependency in build.gradle.

implementation group: 'com.google.code.gson', name: 'gson', version: '2.3.1'

Transitive Dependency

The library is included in another library used in the app. An example is:

./gradlew :service-activity:dependencyInsight --dependency ch.qos.logback:logback-classic
ch.qos.logback:logback-classic:1.2.0
\--- com.nirmata.notification:nirmata-notif-producer:2.10.0
     \--- com.nirmata.status:nirmata-status:4.0.7
          \--- compileClasspath

The above output shows that the library ch.qos.logback:logback-classic:1.2.0 is coming from the Nirmata library com.nirmata.notification:nirmata-notif-producer:2.10.0 which in turn is coming from com.nirmata.status:nirmata-status:4.0.7

The sections below describe how we can fix the CVEs by updating the libraries in the build.gradle file.

Fixing CVE

MVN repository is a great place to find the CVEs associated with a particular library.

The screenshot above shows that the gson library from version 2.8.9 onwards has no CVEs.

Direct Dependency

A direct dependency is the easiest to fix as the library version needs to be upgraded directly in the build.gradle for the app

In the example for CVE-2022-25647 coming from gson:2.3.1, we changed

implementation group: 'com.google.code.gson', name: 'gson', version: '2.3.1'

to

implementation group: 'com.google.code.gson', name: 'gson', version: '2.8.9'

Transitive Dependency – Own library

When the source of the transitive dependency is a library that you own, the recommended approach is to upgrade the version in the dependent library and use the updated version of the dependent library in your app.

For the case of CVE-2023-6378 originating from logback-classic:1.2.0, we updated nirmata-notif-producer and incorporated the updated version into nirmata-status to generate a new version. This updated version was then utilized in the app activity.

Transitive Dependency – Third-party library

When the source of the transitive dependency is a library that you do not own i.e. a third-party/open-source library, where updates to the library may not be possible, then the recommended workaround is to do a force dependency in the build.gradle file

./gradlew :service-activity:dependencyInsight --dependency org.apache.commons:commons-compress
org.apache.commons:commons-compress:1.24
\--- org.apache.avro:avro:1.11.3
     +--- compileClasspath

For the specific case of CVEs CVE-2024-26308/CVE-2024-25710 originating from org.apache.commons:commons-compress:1.24, the source is traced back to the third-party library org.apache.avro:avro version 1.11.3. Unfortunately, org.apache.avro:avro does not offer a newer version addressing the CVEs. To resolve the CVEs, we enforced the dependency on v1.26.0 by adding the following directive to the build.gradle file.

configurations.all {
    resolutionStrategy {
          force 'org.apache.commons:commons-compress:1.26.0'
    }  
}

Transitive Dependency – Third-party library with no fix version

There are some CVEs where Trivy does not show a fixed version e.g. CVE-2020-13939

In such scenarios, it is best to consult the documentation for the library or the Maven repository. In this case, the Maven repository indicated that the artifact had been moved from org.apache.velocity to velocity-engine-core.

To fix the CVE, we forced the dependency on the new library and excluded the old library in build.gradle by adding the following directives.

implementation 'org.apache.velocity:velocity-engine-core:2.3'
configurations.configureEach {
     exclude group:'org.apache.velocity', module: 'velocity'
}

In other cases where no fixed version is available, one may need to contact the maintainer or consider forking the library and resolving the CVE independently.

Transitive Dependency – Third-party shaded jar

A shaded JAR is a Java ARchive file that includes all dependencies within it, with package names modified to avoid conflicts. For fixing CVEs arising from such bundled libraries, a forced dependency does not work as the library version is bundled with the shaded jar.

We faced this issue with the CVE CVE-2023-1370

To detect which jar is bundling this library, Trivy scan has to be run with extra arguments.

trivy image <image-name> -f json --list-all-pkgs

The output for which showed that the json-smart v2.4.8 is included in the shaded jar nimbus-jose-jwt-9.22.jar

        {
          "Name": "net.minidev:json-smart",
          "Identifier": {
            "PURL": "pkg:maven/net.minidev/json-smart@2.4.8"
          },
          "Version": "2.4.8",
          "Layer": {
            "Digest": "sha256:abe2a302ae239ef559a986fe8535abf5406ed654cb9ac1470f4c74e5eaa7c937",
            "DiffID": "sha256:a408e99ba6dfa914f5f89b857004d1c727bb3e532545cd75fa5399eaef6a5829"
          },
          "FilePath": "usr/local/tomcat/webapps/cluster.war/WEB-INF/lib/nimbus-jose-jwt-9.22.jar"
        },

We further used dependencyInsight to get the source of nimbus-jose-jwt, which came out to be the library com.nimbusds:oauth2-oidc-sdk

./gradlew :service-activity:dependencyInsight --dependency nimbus-jose-jwt
com.nimbusds:nimbus-jose-jwt:9.22
\--- com.nimbusds:oauth2-oidc-sdk:6.21.1
     \--- compileClasspath

We fixed the CVE by forcing the dependency of this library to v11.9.1 by adding the following to build.gradle

implementation 'com.nimbusds:oauth2-oidc-sdk:11.9.1'

Ensuring Functionality Integrity through Comprehensive Testing

It is crucial to thoroughly test CVE fixes to ensure they don’t break the app’s functionality and introduce regressions. At Nirmata, we utilized both manual and automated testing. This approach helped us identify instances where updating the affected library version fixed the CVE but broke functionality due to an overlooked update in a dependent library.

Enhancing Security with Automated CVE Scanning and Daily Builds

We have updated our GitHub Actions to perform a Trivy scan on the built image and fail the build if any CVEs are found. The build failure is then notified to the appropriate stakeholders, who can address the CVEs.

      - name: Perform Trivy Scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{env.REGISTRY}}/${{env.IMAGE_NAME}}:${{env.IMAGE_TAG}}
          format: 'json'
          output: 'scan.json'
          exit-code: '1'

Although we have achieved a clean record of 0 CVEs across all our repositories to date, we cannot guarantee that today’s built image will remain free of future CVEs. To detect such issues proactively, we have implemented daily builds from the main branch of all repositories. This ensures that new CVEs are promptly flagged and addressed, even in repositories with no active development.

Conclusion

Cybersecurity vulnerabilities, known as CVEs, must be fixed to enhance security measures, mitigate potential threats, and protect sensitive data and systems from exploitation. While addressing CVEs in new projects requires minimal effort, dealing with them in legacy code can be challenging. However, implementing the abovementioned strategies successfully eliminated hundreds of CVEs across all services to zero. This accomplishment was made possible by combining automated and manual testing, ensuring that integrating new libraries for CVE fixes did not compromise functionality.

While the number of CVEs detected in the libraries amounted to a few dozen for each of our services, the count of CVEs originating from base images used in our services reached several hundred. Our upcoming blog post will explore our approach to addressing CVEs stemming from the base image. Stay tuned for further details.

References

CVE (Common Vulnerabilities and Exposures): https://cve.mitre.org/