Kyverno, the Kubernetes native policy engine, has integrated support to verify image signatures prior to the image getting deployed in the Kubernetes cluster. This capability helps enhance any software supply chain security by providing an additional check to ensure that any unauthorized or unverified images do not get deployed in your clusters. 

While there have been several blog posts describing how image verification can be used, there are some specific challenges when enabling image verification on Amazon EKS with images from Amazon ECR. This blog post describes how to enable this capability on Amazon EKS with Amazon ECR.

Amazon EKS

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed container service to run and scale Kubernetes applications in the cloud or on-premises.

Amazon ECR

Amazon ECR is a fully managed container registry offering high-performance hosting, so you can reliably deploy application images and artifacts anywhere. When using private ECR registries, Amazon IAM is used which requires additional configuration to fetch temporary token to authenticate with the registry. As a result, Kyverno image verification capabilities do not work out of the box with images stored in Amazon ECR. 

Kyverno Policy Engine

Kyverno is an open-source Kubernetes-native policy engine that runs as an admission controller and can validate, mutate, and generate any configuration data based on customizable policies. Kyverno can also verify images by validating signatures using Cosign as well as by validating attestations. 

Although other general purpose policy solutions were retrofitted to Kubernetes, Kyverno was designed ground-up for Kubernetes. Like Kubernetes, Kyverno adopts a declarative management paradigm. Kyverno policies are simply Kubernetes resources, and don’t require learning a new language. Kyverno helps secure the Kubernetes configuration by preventing misconfigurations and enhancing security.

Cosign

Cosign is a sigstore project that makes signing and verification of OCI images easy. Cosign aims to make signatures invisible infrastructure.

Putting it All Together

Next, lets see how all the components can be set up to enable image verification for your clusters.

Create Amazon EKS cluster

Create an EKS cluster via AWS CLI or the AWS management console. If you already have an EKS cluster you can use it as well.

Create Amazon ECR registry

You will need a private ECR registry. You can also use an existing ECR registry. In your registry, you will need an image that can be used for this demonstration. We will be using the nginx image.  Note down the image registry URL which is in the format – <aws-account-id>.dkr.ecr.us-west-1.amazonaws.com

Install Kyverno

Next, you can install kyverno. First, create a namespace ‘kyverno’ and create a secret ‘aws-registry’

kubectl create namespace kyverno
kubectl create secret docker-registry aws-registry \
--docker-server=<aws-account-id>.dkr.ecr.us-west-1.amazonaws.com \
--docker-username=AWS \
--docker-password=demo 

Where docker-server has the URL of your ECR registry

Note: The password here is temporary and will be replaced later with the token for the registry.

Now, in Kyverno you need to add the imagePullSecrets argument  to the kyverno deployment (kyverno container)

You can use helm chart to deploy kyverno or use the yam directlyl. Follow the instructions here.

For helm install, add the ‘set’ option:

--set 'extraArgs={--imagePullSecrets=aws-registry}'

 

If you are using the YAML file, you need to add the argument in the kyverno deployment, kyverno container:  

--imagePullSecrets=aws-registry

 

This argument ensures that the image registry credentials will be fetched from the aws-registry secret.

Configure IAM Role for Service Account

Once Kyverno is installed, you need to create an IAM Role for Service Account for the cron job that will fetch the registry credentials and store them in a secret.

Create the IAM role for the kyverno-service-account service account. You can follow the steps here to create the IAM role. 

In the last step, when updating the Trust Relationships, use the following ARN :

system:serviceaccount:kyverno:kyverno-service-account 

 

Here kyverno is the namespace and kyverno-service-account is the service account to bind to the IAM role.

Note the IAM role name. Now, edit the kyverno-service-account and add the following annotations:

 eks.amazonaws.com/role-arn: arn:aws:iam::<aws-account-id>:role/<iam-role-name>
 eks.amazonaws.com/sts-regional-endpoints: "true"

Install CronJob to fetch registry token

Next, we will deploy the cronJob to fetch the registry credentials. Here is the cronJob manifest (aws-registry-credential-cron.yaml). You will need to replace the <aws-account-id> with your AWS account ID.

#filename: aws-registry-credential-cron.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: aws-registry-credential-cron
spec:
  schedule: "* */8 * * *"
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 2
  jobTemplate:
    spec:
      backoffLimit: 4
      template:
        spec:
          serviceAccountName: kyverno-service-account 
          terminationGracePeriodSeconds: 0
          restartPolicy: Never
          containers:
          - name: kubectl
            imagePullPolicy: IfNotPresent
            image: xynova/aws-kubectl:latest
            command:
            - "/bin/sh"
            - "-c"
            - |
              DOCKER_REGISTRY_SERVER=https://<aws-account-id>.dkr.ecr.${AWS_REGION}.amazonaws.com
              DOCKER_USER=AWS
              DOCKER_PASSWORD=$(aws ecr get-login --region ${AWS_REGION} --registry-ids <aws-account-id> | cut -d' ' -f6)
              kubectl delete secret aws-registry || true
              kubectl create secret docker-registry aws-registry \
              --docker-server=$DOCKER_REGISTRY_SERVER \
              --docker-username=$DOCKER_USER \
              --docker-password=$DOCKER_PASSWORD \
              --docker-email=no@email.local

 

Run the cronJob in the kyverno namespace:

kubectl create -f aws-registry-credential-cron.yaml -n kyverno

 

The cronJob will run periodically. Meanwhile, you also need to run a job on-demand to fetch the secret for the very first time.

kubectl create job \
--from=cronjob/aws-registry-credential-cron -n kyverno aws-registry-credential-cron-manual-001

 

Check the logs for any errors:

kubectl logs job/aws-registry-credential-cron-manual-001 -n kyverno
secret "aws-registry" deleted
secret "aws-registry" created

 

Deploy image verification policy

Next, generate the certificates to sign the image using cosign.

​​cosign generate-key-pair

Cosign will generate a public and private key. Add the public key to the policy for image verification. Make sure that the image path points to your ECR registry.

#filename check-images.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image
spec:
  validationFailureAction: enforce
  background: false
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: check-image
      match:
        resources:
          kinds:
            - Pod
      verifyImages:
      - image: "<aws-account-id>.dkr.ecr.<aws-region>.amazonaws.com/*"
        key: |- 
          -----BEGIN PUBLIC KEY-----
          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEdvBaT5usBe/wWKQ2UxocUySQ7X5J
          wEOM0mHrl6m8o5Gll5DdbUhLFPpiCxEyfzkx6cqcGhUAuyOuxa2ygP+Y4g==
          -----END PUBLIC KEY-----

 

Now apply this policy to your EKS cluster

kubectl apply -f check-images.yaml

 

Check if it is possible to deploy the nginx image to your cluster

kubectl  run nginx --image=<aws-account-id>.dkr.ecr.<aws-region>.amazonaws.com/nirmata/nginx:latest
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:

resource Pod/default/nginx was blocked due to the following policies

check-image:
  check-image: 'image verification failed for 12345678.dkr.ecr.us-west-1.amazonaws.com/nirmata/nginx:latest:
    signature mismatch'

 

Sign the image

Now sign the image using the key generated earlier. Note that prior to using cosign, you will need to log in to the ECR registry using docker. You can do that using AWS CLI:

aws ecr get-login-password --region<aws-region> | docker login --username AWS --password-stdin <aws-account-id>.dkr.ecr.<aws-region>.amazonaws.com

cosign sign --key cosign.key <aws-account-id>.dkr.ecr.<aws-region>.amazonaws.com/nirmata/nginx:latest

You should be able to check if the signature has been pushed to your ECR registry.

Run Application

Now, if you run your nginx application, it should successfully be deployed. Try it out!

 

kubectl --kubeconfig=../eks-demo_145506_01312022.yaml run nginx --image=12345678.dkr.ecr.us-west-1.amazonaws.com/nirmata/nginx:latest
pod/nginx created
kubectl --kubeconfig=../eks-demo_145506_01312022.yaml get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          12s

So, Kyverno was able to successfully verify the image signature and as a result, the pod was deployed successfully!

Conclusion 

As supply chain attacks are becoming more common, it has become absolutely necessary to secure various phases in your CI/CD pipeline. Kyverno can not only validate and mutate your Kubernetes manifests to meet your software supply chain security requirements, it can also prevent unsigned and unattested images from being deployed to your Amazon EKS clusters. 

I hope this blog post helps you get started with Kyverno on Amazon EKS. Please reach out to us if you have any questions or need any help getting started with Kyverno on AWS.