Software Supply Chain Security on Amazon EKS clusters using Amazon ECR, Kyverno, and Cosign

Software Supply Chain Security on Amazon EKS clusters using Amazon ECR, Kyverno, and Cosign

Kyverno, the Kubernetes native policy engine, has integrated support to verify image signatures prior to the image getting deployed in the Kubernetes cluster. This capability helps enhance the security of any software supply chain by providing an additional check to ensure that any unauthorized or unverified images do not get deployed in your clusters. 

While there have been several blog posts describing how image verification can be used, there are some specific challenges when enabling image verification on Amazon EKS with images from Amazon ECR. This blog post describes how to enable this capability on Amazon EKS with Amazon ECR.

Amazon EKS

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed container service to run and scale Kubernetes applications in the cloud or on-premises.

Amazon ECR

Amazon ECR is a fully managed container registry offering high-performance hosting, so you can reliably deploy application images and artifacts anywhere. When using private ECR registries, Amazon IAM is used which requires additional configuration to fetch temporary token to authenticate with the registry. As a result, Kyverno image verification capabilities do not work out of the box with images stored in Amazon ECR. 

Kyverno Policy Engine

Kyverno is an open-source Kubernetes-native policy engine that runs as an admission controller and can validate, mutate, and generate any configuration data based on customizable policies. Kyverno can also verify images by validating signatures using Cosign as well as by validating attestations. 

Although other general purpose policy solutions were retrofitted to Kubernetes, Kyverno was designed ground-up for Kubernetes. Like Kubernetes, Kyverno adopts a declarative management paradigm. Kyverno policies are simply Kubernetes resources, and don’t require learning a new language. Kyverno helps secure the Kubernetes configuration by preventing misconfigurations and enhancing security.


Cosign is a sigstore project that makes signing and verification of OCI images easy. Cosign aims to make signatures invisible infrastructure.

Putting it All Together

Next, lets see how all the components can be set up to enable image verification for your clusters.

Create Amazon EKS cluster

Create an EKS cluster via AWS CLI or the AWS management console. If you already have an EKS cluster you can use it as well.

Create Amazon ECR registry

You will need a private ECR registry. You can also use an existing ECR registry. In your registry, you will need an image that can be used for this demonstration. We will be using the nginx image.  Note down the image registry URL which is in the format – <aws-account-id>

Install Kyverno

Next, you can install kyverno. First, create a namespace ‘kyverno’ and create a secret ‘aws-registry’

kubectl create namespace kyverno
kubectl create secret docker-registry aws-registry \
--docker-server=<aws-account-id> \
--docker-username=AWS \

Where docker-server has the URL of your ECR registry

Note: The password here is temporary and will be replaced later with the token for the registry.

Now, in Kyverno you need to add the imagePullSecrets argument  to the kyverno deployment (kyverno container)

You can use helm chart to deploy kyverno or use the yam directlyl. Follow the instructions here.

For helm install, add the ‘set’ option:

--set 'extraArgs={--imagePullSecrets=aws-registry}'


If you are using the YAML file, you need to add the argument in the kyverno deployment, kyverno container:  



This argument ensures that the image registry credentials will be fetched from the aws-registry secret.

Configure IAM Role for Service Account

Once Kyverno is installed, you need to create an IAM Role for Service Account for the cron job that will fetch the registry credentials and store them in a secret.

Create the IAM role for the kyverno-service-account service account. You can follow the steps here to create the IAM role. 

In the last step, when updating the Trust Relationships, use the following ARN :



Here kyverno is the namespace and kyverno-service-account is the service account to bind to the IAM role.

Note the IAM role name. Now, edit the kyverno-service-account and add the following annotations: arn:aws:iam::<aws-account-id>:role/<iam-role-name> "true"

Install CronJob to fetch registry token

Next, we will deploy the cronJob to fetch the registry credentials. Here is the cronJob manifest (aws-registry-credential-cron.yaml). You will need to replace the <aws-account-id> with your AWS account ID.

#filename: aws-registry-credential-cron.yaml
apiVersion: batch/v1beta1
kind: CronJob
  name: aws-registry-credential-cron
  schedule: "* */8 * * *"
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 2
      backoffLimit: 4
          serviceAccountName: kyverno-service-account 
          terminationGracePeriodSeconds: 0
          restartPolicy: Never
          - name: kubectl
            imagePullPolicy: IfNotPresent
            image: xynova/aws-kubectl:latest
            - "/bin/sh"
            - "-c"
            - |
              DOCKER_PASSWORD=$(aws ecr get-login --region ${AWS_REGION} --registry-ids <aws-account-id> | cut -d' ' -f6)
              kubectl delete secret aws-registry || true
              kubectl create secret docker-registry aws-registry \
              --docker-server=$DOCKER_REGISTRY_SERVER \
              --docker-username=$DOCKER_USER \
              --docker-password=$DOCKER_PASSWORD \


Run the cronJob in the kyverno namespace:

kubectl create -f aws-registry-credential-cron.yaml -n kyverno


The cronJob will run periodically. Meanwhile, you also need to run a job on-demand to fetch the secret for the very first time.

kubectl create job \
--from=cronjob/aws-registry-credential-cron -n kyverno aws-registry-credential-cron-manual-001


Check the logs for any errors:

kubectl logs job/aws-registry-credential-cron-manual-001 -n kyverno
secret "aws-registry" deleted
secret "aws-registry" created


Deploy image verification policy

Next, generate the certificates to sign the image using cosign.

​​cosign generate-key-pair

Cosign will generate a public and private key. Add the public key to the policy for image verification. Make sure that the image path points to your ECR registry.

#filename check-images.yaml
kind: ClusterPolicy
  name: check-image
  validationFailureAction: enforce
  background: false
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
    - name: check-image
            - Pod
      - image: "<aws-account-id>.dkr.ecr.<aws-region>*"
        key: |- 
          -----BEGIN PUBLIC KEY-----
          -----END PUBLIC KEY-----


Now apply this policy to your EKS cluster

kubectl apply -f check-images.yaml


Check if it is possible to deploy the nginx image to your cluster

kubectl  run nginx --image=<aws-account-id>.dkr.ecr.<aws-region>
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:

resource Pod/default/nginx was blocked due to the following policies

  check-image: 'image verification failed for
    signature mismatch'


Sign the image

Now sign the image using the key generated earlier. Note that prior to using cosign, you will need to log in to the ECR registry using docker. You can do that using AWS CLI:

aws ecr get-login-password --region<aws-region> | docker login --username AWS --password-stdin <aws-account-id>.dkr.ecr.<aws-region>

cosign sign --key cosign.key <aws-account-id>.dkr.ecr.<aws-region>

You should be able to check if the signature has been pushed to your ECR registry.

Run Application

Now, if you run your nginx application, it should successfully be deployed. Try it out!


kubectl --kubeconfig=../eks-demo_145506_01312022.yaml run nginx
pod/nginx created
kubectl --kubeconfig=../eks-demo_145506_01312022.yaml get pods
nginx   1/1     Running   0          12s

So, Kyverno was able to successfully verify the image signature and as a result, the pod was deployed successfully!


As supply chain attacks are becoming more common, it has become absolutely necessary to secure various phases in your CI/CD pipeline. Kyverno can not only validate and mutate your Kubernetes manifests to meet your security requirements but it can also prevent unsigned and unattested images from being deployed to your Amazon EKS clusters. 

I hope this blog post helps you get started with Kyverno on AWS EKS. Please reach out to us if you have any questions or need any help getting started with Kyverno on AWS.

Kyverno v1.6.0: More Kubernetes Security and Governance Use Cases Through Policy
Kyverno Highlight: Meet Frank Jogeleit, Kyverno Maintainer!
No Comments

Post a Comment