This is a joint blog post from Dan Lorenc (@lorenc_dan), a software engineer at Google and a maintainer of Cosign, and Jim Bugwadia (@JimBugwadia), co-founder of Nirmata and a maintainer of Kyverno.

Introduction

Modern software applications are built on decades of prior art and can have hundreds of dependencies across open source, internal, and other 3rd party components. Hence, securing software supply chains is complex. Rapid adoption of DevOps and cloud-native technologies like containers and Kubernetes also add complexity, but have the potential of bringing new best practices and automation tools. In this post we will demonstrate how Cosign, an image signing and verification tool, can be used with Kyverno, a Kubernetes policy engine. 

Image Signing with Cosign

Supply chains can be modeled as a series of links where source code, metadata. and artifacts change hands from producer to consumer, making their way from keyboard to production deployment. Unfortunately each of these links is a potential place for something to go wrong, either by accident or on purpose from an attacker!

One method for protecting these links against attacks (and mistakes!) is to use digital signatures, which can help detect if an artifact or metadata has been tampered with. Artifact publishers can generate a cryptographic keypair, and use the private key to sign the artifact. They then distribute the public key widely, and consumers can use that to verify the signature and artifact. When these pieces all get put together, they can be used to protect the links in a software supply chain.

Unfortunately these pieces are often harder to put together than they need to be, which is what the Sigstore project aims to help fix! Sigstore is a set of open source projects and shared community-driven infrastructure to help make key distribution, certificate verification and signature discovery easy and transparent. While Sigstore is aimed at all types of software artifacts, the cosign sub-project is specifically targeted at signing and verifying container images. You can get up and running quickly with cosign by following the getting started guide here, where you can generate a keypair, sign images and verify them on the command line.

Or, you can follow the rest of this blog post to see how to automatically verify images before they’re deployed using Kyverno!

Cosign Image Signature Verification with Kyverno

Kyverno is a policy management solution built for Kubernetes. Kyverno enables managing policies and policy results as Kubernetes resources. Kyverno runs as an admission controller and can allow or deny API requests based on configured policies. Kyverno policies can also be used to mutate resources and be used to trigger the automatic generation of new resources.

To add support for Cosign, a new type of policy rule was introduced. The new “verifyImages” rule performs the following actions:

1. It validate signatures for matching images using Cosign
2. It mutates image references with the digest returned by Cosign

Using an image digest guarantees immutability of images and hence improves security.

The rule is executed in the mutating admission controller, but runs after resources are mutated to allow policies to mutate image registries and other configurations, before the image signature is verified.

Here is a policy that verifies all images from the repository “ghcr.io/kyverno/” that start with the name “test-verify-image” are signed with the provided public key:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image
spec:
  validationFailureAction: enforce
  background: false
  rules:
    - name: check-image
      match:
        resources:
          kinds:
            - Pod
      verifyImages:
      - image: "ghcr.io/kyverno/test-verify-image:*"
        key: |-
          -----BEGIN PUBLIC KEY-----
          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8nXRh950IZbRj8Ra/N9sbqOPZrfM
          5/KAQN0/KjHcorm/J5yctVd7iEcnessRQjU917hmKO6JWVGHpDguIyakZA==
          -----END PUBLIC KEY-----  

The Kyverno policy rule definition allows for multiple entries, and the all matching rules are executed as a logical AND operation i.e. they must all be successful for the policy check to pass. This allows for use cases where an image is signed by a build automation tool, as well as a system administrator. 

While it’s best to write policies that operate on a pod specification at the pod level, pods are almost always managed using pod controllers such as a Deployment or StatefulSet. Kyverno automatically generates policy rules for pod controllers via the auto-gen feature to provide early enforcement of policies and a better user experience.

The Kyverno auto-gen feature also works for the “verifyImages” rule. This means that a Deployment, or any other controller, with a pod template that references a matching image will be validated and the pod template will be updated with the image digest! 

Let’s take a look at this in action with a step-by-step demonstration:

1. Install the latest version of Kyverno (the Kyverno docs have detailed installation instructions from command line and the Helm chart):

kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/main/definitions/release/install.yaml

2. Patch the Kyverno webhook, to allow time for calling the OCI registry (future releases of Kyverno will automatically adjust the defaults):

kubectl patch mutatingwebhookconfigurations kyverno-resource-mutating-webhook-cfg \
--type json \
-p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"},{"op": "replace", "path": "/webhooks/0/timeoutSeconds", "value": 15}]'

3. Install a sample image validation policy:

kubectl apply -f https://raw.githubusercontent.com/kyverno/policies/main/other/verify_image.yaml

4. Try running a signed test image as a Deployment from the Kyverno repository:

kubectl create deployment signed \
--image=ghcr.io/kyverno/test-verify-image:signed

5. Check the image reference in the deployment.

kubectl get deploy signed -o yaml | grep image

Note that the image digest was automatically added:

  - image: ghcr.io/kyverno/test-verify-image:signed@sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105
    imagePullPolicy: IfNotPresent
    name: test-verify-image

6. Try running an unsigned image that matches the configured rule:

kubectl create deployment unsigned \
--image=ghcr.io/kyverno/test-verify-image:unsigned

This will be blocked:

error: failed to create deployment: admission webhook "mutate.kyverno.svc" denied the request:

resource Deployment/default/unsigned was blocked due to the following policies

verify-image:
  autogen-verify-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:unsigned:
    signature not found'

7. Try running an image signed using a different private key that matches the configured rule:

kubectl create deployment signed-other \
--image=ghcr.io/kyverno/test-verify-image:signed-by-someone-else

This will be blocked as well:

error: failed to create deployment: admission webhook "mutate.kyverno.svc" denied the request:

resource Deployment/default/signed-other was blocked due to the following policies

verify-image:
  autogen-verify-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:signed-by-someone-else:
    invalid signature'

8. Finally try running a bare pod, with an unsigned image:

kubectl run unsigned \
--image=ghcr.io/kyverno/test-verify-image:unsigned

This verifies that Kyverno policies are applied at the pod controller and pod levels:

Error from server: admission webhook "mutate.kyverno.svc" denied the request:

resource Pod/default/unsigned was blocked due to the following policies

verify-image:
  verify-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:unsigned:
    signature not found'

Kyverno also supports Cosign integration with private OCI registries. Refer to the Kyverno documentation for details, and to test this feature with your own images.

What this integration provides

While Kubernetes supply chain security is a large topic, the ability to easily sign and verify container images provides a foundational strategy you can implement today. Establishing a baseline level of trust and awareness of everything you’re running in production is a great first step in securing your overall software supply chain. Signing your images, either manually or as part of a build pipeline, and then verifying them before deployment, is one way to get started on the path to more robust Kubernetes supply chain security.

As standards and tools evolve, a more comprehensive strategy would be to use richer attestations that contain data about how an image was built and what source code was used inside of it. These attestations should be generated automatically by a CI/CD pipeline and can be used to write and enforce powerful policies using engines like Kyverno.  The In-Toto project contains a large set of metadata formats that are useful here. More on this topic at: Policy and Attestations. Best Practices for Supply Chain… | by Dan Lorenc | Jul, 2021 | Medium.  

What’s next for Cosign

This is a fast moving space, and these tools are changing rapidly! Up next in Sigstore, we’re working on firming up our APIs and specifications so more tooling can be built on top of our infrastructure. Our signature transparency log and code signing CA are currently Experimental, but we’re aiming to get these up to production quality early this fall. We’re also working to integrate Cosign signatures directly into build systems, so they can be created in CI systems and verified in policy engines from end-to-end, automatically!

What’s next for Kyverno

The Kyverno 1.4.2 verify images policy rule provides a simple yet powerful way to verify container images verified by Cosign during admission controls. The feature in Kyverno is currently in “alpha” status and the current focus is to make this production ready. This will include adding Kyverno CLI support for image verification via policies outside a cluster (e.g. in a CI/CD pipeline) and metrics support. A full list of open issues is available on the Kyverno GitHub. Learn more about Kyverno as a policy manager for Kubernetes while you’re here.

Beyond verifying image signatures, Kyverno will continue to expand support for other supply chain security use cases, such as verify in-toto attestations as these standards mature.

Get Involved

Cosign and Kyverno are both fast moving projects with growing communities. Come say hello in the slack channels (Cosign slack, Kyverno slack) and follow our GitHub repositories (Cosign GitHub, Kyverno GitHub) to stay updated. We’d love to hear from you!

Meanwhile, Nirmata will continue to update readers (and developers) here on important DevOps topics like Kubernetes supply chain security, including Cosign security for Kubernetes. Feel free to contact us here with any specific questions you may have, or sign-up for our free monthly newsletter at the top right of this page to stay informed on all things Kubernetes and Kyverno!

Photo by boris misevic on Unsplash