Introduction

In our previous blog post, we delved into the importance of addressing CVEs within application libraries and outlined strategies for remediation. Continuing on this journey towards enhanced cybersecurity, we now turn our attention to the critical task of eliminating CVEs originating from base images. These foundational components play a pivotal role in the security posture of containerized applications, making their integrity paramount.

Addressing the Base Image Challenge

To ensure optimal performance and security for our Java services within containers, the foundational image requires the following components:

1. Operating System (OS) such as Ubuntu or Alpine, equipped with essential OS libraries
2. Java runtime environment
3. Necessary security packages

Initially, we utilized a prebuilt base image from a reputable vendor, encompassing the aforementioned elements. However, upon closer inspection, we discovered that this base image contained over 200 packages, totaling a hefty 400 MB in size. Trivy, our vulnerability scanner, flagged over 100 Common Vulnerabilities and Exposures (CVEs). With a multitude of Java microservices in our ecosystem, this presented several challenges:

• A sprawling attack surface due to the abundance of packages
• Prolonged build and load times stemming from the image’s substantial size
• Compromised security posture owing to the high number of CVEs

To contextualize the severity, the cumulative CVE count from our stack’s libraries barely exceeded 100, while the base image harbored over 1000 vulnerabilities. This prompted us to explore tools capable of crafting leaner, CVE-free base images, tailored precisely to our requirements.

The ensuing sections delineate our journey towards achieving a CVE-free status for our base images.

Utilizing Apko for Base Image Optimization

To streamline the creation of minimal base images tailored to our microservices’ requirements, we turned to Apko [1]. This powerful tool enables a declarative approach to bundle only the necessary packages, facilitating faster builds and reducing image footprints. By leveraging Apko, we achieved significant reductions in package count and image size while ensuring a secure foundation for our applications.

Transitioning to Alpine-based Base Images

Recognizing Alpine Linux’s reputation for simplicity, security, and efficiency, we adopted Alpine packages for our base images. Alpine’s lightweight nature and security-focused design align perfectly with our objectives of minimizing attack surfaces and enhancing overall security posture. The Apko configuration file provided a clear blueprint for assembling the essential components, including the Java runtime and security packages, resulting in lean and CVE-free base images.

The listing below shows the apko configuration for building the base image using alpine packages required for running our services.

contents:

  repositories:
  - https://dl-cdn.alpinelinux.org/alpine/v3.19/main
  - https://dl-cdn.alpinelinux.org/alpine/v3.19/community

  packages:
  - ca-certificates-bundle
  - java-cacerts
  - libcrypto3
  - libssl3
  - openjdk21-jre-headless

environment:
  JAVA_HOME: /usr/lib/jvm/java-21-openjdk
  PATH: /usr/lib/jvm/java-21-openjdk/bin:/usr/sbin:/sbin:/usr/bin:/bin

os-release:
  version-id: '3.19'

archs:
  - amd64
  - arm64

The command to build the base image is

apko publish alpine-base.yaml ghcr.io/nirmata/alpine-base:<tag>

Addressing CVEs Beyond Trivy: The Role of Grype and Wolfi

The base image generated through the aforementioned configuration yielded a pristine slate with zero CVEs upon examination with Trivy [2]. However, when subjected to analysis with Grype [3], additional vulnerabilities surfaced, notably within components such as BusyBox, libcrypto, and ssl_client. Trivy’s omission of these CVEs stems from its reliance on the Alpine security database, which does not flag them [4].

To address these gaps in vulnerability detection, we explored Wolfi [5] packages, a pioneering initiative in community-driven Linux (un)distribution tailored for bolstering container security. Integrating Wolfi packages into our base images fortified our defensive perimeter against potential threats, culminating in a more resilient and robust infrastructure.

The listing below shows the apko configuration for building the base image using wolfi packages. It mirrors the configuration for Alpine, with the sole distinction being the specification of the repository.

contents:

  keyring:
    - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

  repositories:
    - https://packages.wolfi.dev/os

  packages:
    - wolfi-base
    - ca-certificates-bundle
    - java-cacerts
    - libcrypto3
    - libssl3
    - openjdk-21-jre

entrypoint:
  command: /bin/sh -l

environment:
  JAVA_HOME: /usr/lib/jvm/java-21-openjdk
  PATH: /usr/lib/jvm/java-21-openjdk/bin:/usr/sbin:/sbin:/usr/bin:/bin

archs:
  - amd64
  - arm64

The command to build the base image is:

apko publish wolfi-base.yaml ghcr.io/nirmata/wolfi-base:<tag>

Comparing Base Images: Alpine vs. Wolfi Packages

In comparing the base images created with Alpine and Wolfi packages:

• Package Count: Alpine’s image had 15 packages, while Wolfi’s had 32
• Image Size: Alpine’s image was approximately 208 MB, slightly smaller than Wolfi’s 240 MB
• CVE Vulnerabilities: While Alpine’s image revealed non-zero CVEs when scanned by Grype, Wolfi’s image maintained a pristine record with zero CVEs detected

In summary, while Alpine excelled in minimalism and compactness, the broader package selection offered by Wolfi was a key factor in our decision to use it. Additionally, the fact that Wolfi’s image maintained a CVE-free status when scanned by Grype underscored its robust security posture, further solidifying our choice.

Ensuring Long-term Stability through Package Pinning

To maintain consistency and mitigate future risks, we adopted a strategy of pinning specific package versions. By pinning packages to predefined versions, we ensure that subsequent builds of our base images remain consistent and predictable. This proactive approach enables us to swiftly address any future CVEs by updating package versions and rebuilding the base images, thereby safeguarding our applications against emerging threats.

    - wolfi-base=1-r4
    - ca-certificates-bundle=20240226-r0
    - java-cacerts=20230106-r3
    - libcrypto3=3.2.1-r0
    - libssl3=3.2.1-r0
    - openjdk-21-jre=21.0.2-r2

Base image for Go based services

We utilized ko [6] to create container images for our Go-based services, resulting in remarkably low-footprint images. However, we encountered CVEs within the image originating from stdlib, posing a significant challenge to rectify.

To address this issue, we delved into melange [7], a tool for generating apk images through declarative pipelines. Integrating this with Apko, we established a pipeline for generating base images utilizing the apk packages generated by melange. This strategic approach enabled us to mitigate the CVEs stemming from stdlib, ensuring the integrity of our Go-based service images.

Comprehensive Testing: Safeguarding Functionality and Performance

Throughout the transition to optimized base images, ensuring the integrity of our applications’ functionality and performance remained paramount. Rigorous automated and manual testing processes were employed to validate the stability and reliability of the updated base images. By conducting thorough testing, we mitigated the risk of regressions and ensured that our applications continued to meet stringent quality standards.

Conclusion

In our quest to fortify our product’s security and compliance, the journey towards eliminating CVEs from base images represents a significant milestone. Through the strategic utilization of Apko, melange, Wolfi packages, and comprehensive testing practices, we successfully transitioned from vulnerability-laden base images to lean, CVE-free alternatives. This achievement underscores our commitment to prioritizing cybersecurity and lays a robust foundation for future advancements in our containerized environment. As we continue to evolve our security practices, we remain steadfast in our dedication to delivering secure and resilient solutions to our customers.

References

[1] apko

[2] Trivy 

[3] Grype 

[4] Trivy GitHub Discussion about security database 

[5] Wolfi

[6] ko

[7] melange