Image Verification with Kyverno, Notation, and Azure AD Workload Identity on AKS

Image Verification with Kyverno, Notation, and Azure AD Workload Identity on AKS

Kyverno Azure VerifyImages

Container security is a paramount concern for organizations deploying applications on Kubernetes clusters. One crucial aspect of securing container workloads is ensuring that only trusted images are used. Azure Container Registry (ACR) is a private registry service for building, storing, and managing container images and related artifacts. However, managing access to ACR and verifying image authenticity can be challenging.

In this blog post, we will explore how to leverage Azure AD workload identity and Kyverno, a powerful Kubernetes policy engine, to enhance container security by verifying ACR images on AKS clusters. By harnessing the capabilities of Azure AD workload identity, we can simplify the authentication process and establish secure connections between AKS clusters and ACR, while Kyverno allows us to enforce fine-grained policies to verify image authenticity. By the end of this post, you will have a clear understanding of how to leverage Azure AD workload identity and Kyverno to strengthen container security, prevent unauthorized image deployments, and ensure the integrity of your AKS clusters.

Create a resource group

First, create a resource group named `kyverno-test-resource-group` in the `eastus` location with the following command:

$ az group create --name kyverno-test-resource-group --location eastus
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/kyverno-test-resource-group",
  "location": "eastus",
  "managedBy": null,
  "name": "kyverno-test-resource-group",
  "properties": {
    "provisioningState": "Succeeded"
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"

To help simplify the steps to configure the identities required, the steps below define environmental variables for reference on the cluster. Here it assumes the default installation of Kyverno. If you have a custom Kyverno installation, replace SERVICE_ACCOUNT_NAMESPACE and SERVICE_ACCOUNT_NAME for the Kyverno admission controller service account with your custom values.

export RESOURCE_GROUP="kyverno-test-resource-group"
export LOCATION="eastus"
export SERVICE_ACCOUNT_NAME="kyverno-admission-controller"
export SUBSCRIPTION="$(az account show --query id --output tsv)"
export USER_ASSIGNED_IDENTITY_NAME="kyverno-test-azure-identity"
export FEDERATED_IDENTITY_CREDENTIAL_NAME="kyverno-test-azure-fed-identity"
export KEYVAULT_NAME="kyverno-test-kv"
export KEYVAULT_SECRET_NAME="kyverno-test-kv-secret"

Build and sign container images using Notation and Azure Key Vault

You can skip this section if you already have a test image. Otherwise, follow this instruction to build test images in ACR and sign them using Notary and Azure Key Vault. 

Note that the registry needs to be under the created resource group, i.e., kyverno-test-resource-group if you follow this blog post.

Create an AKS cluster

Create an AKS cluster using the Azure CLI with OpenID Connect (OIDC) Issuer.

$ az aks create -g "${RESOURCE_GROUP}" -n kyverno-test --node-count 1 --enable-oidc-issuer --enable-workload-identity --generate-ssh-keys

It could take a few minutes to provision the cluster, once completed, it returns JSON-formatted information about the cluster.

Get the OIDC Issuer URL and save it to an environmental variable using the following command.

$ export AKS_OIDC_ISSUER="$(az aks show -n kyverno-test -g "${RESOURCE_GROUP}" --query "oidcIssuerProfile.issuerUrl" -otsv)"s

Create a managed identity

Set a specific subscription as the current active subscription.

$ az account set --subscription "${SUBSCRIPTION}"

Create a managed identity. Managed identities provide an automatically managed identity in Azure Active Directory (Azure AD) for applications to use when connecting to resources that support Azure AD authentication. We will configure the Kyverno service account to consume this managed identity for accessing the ACR Registry and retrieving data.

$ az identity create --name "${USER_ASSIGNED_IDENTITY_NAME}" --resource-group "${RESOURCE_GROUP}" --location "${LOCATION}" --subscription "${SUBSCRIPTION}"
  "clientId": "f8c2fc74-9506-4821-a19b-67c4c58c46f6",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/kyverno-test-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kyverno-test-azure-identity",
  "location": "eastus",
  "name": "kyverno-test-azure-identity",
  "principalId": "d05f8352-476b-46f7-98e6-13d547b78077",
  "resourceGroup": "kyverno-test-resource-group",
  "systemData": null,
  "tags": {},
  "tenantId": "3d95acd6-b6ee-428e-a7a0-196120fc3c65",
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities"

Save the client ID of the managed identity to an environmental variable for later use.

$ export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' -otsv)"

Install Kyverno and configure it to use the workload identity

Switch to use the AKS cluster created above.

$ az aks get-credentials -n kyverno-test -g "${RESOURCE_GROUP}"

Ensure the webhook configurations are in-place to inject workload identity to Kyverno.

$ kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations

NAME                                                                                                        WEBHOOKS   AGE                         1          18m                  1          18m   1          16m

NAME                                                                                      WEBHOOKS   AGE   1          18m

To use the workload identity for Kyverno, you need to:

  1. annotate the service account of the admission controller with azure.workload.identity/client-id, the value is the client ID of the managed identity which we have saved to $USER_ASSIGNED_CLIENT_ID earlier
  2. label the pod of the admission controller with azure.workload.identity/use: “true”

Create a values.yaml file and use it for Helm installation.

    admissions.enforcer/disabled: "true"
        azure.workload.identity/client-id: f8c2fc74-9506-4821-a19b-67c4c58c46f6
    azure.workload.identity/use: "true"
$ helm upgrade --install kyverno kyverno/kyverno --namespace kyverno --create-namespace --values values.yaml

To customize your Kyverno installation, please follow the guidance here.

Verify Kyverno is installed successfully.

$ kubectl get pod -n kyverno   
NAME                                             READY   STATUS     RESTARTS   AGE
kyverno-admission-controller-bd7798fb8-gsp5z     1/1     Running    0          17s
kyverno-background-controller-5c97796cdd-dm8kt   1/1     Running    0          17s
kyverno-cleanup-controller-78c4978cf4-zc7sc      1/1     Running    0          17s
kyverno-reports-controller-6788666c89-sknc4      1/1     Running    0          17s

Verify the environmental variables for the workload identity are injected correctly, AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE, AZURE_AUTHORITY_HOST must be set for the Kyverno admission controller.

$ kubectl describe pod -n kyverno -l | grep Environment -A12

KYVERNO_SERVICEACCOUNT_NAME:  kyverno-admission-controller
INIT_CONFIG:                  kyverno
METRICS_CONFIG:               kyverno-metrics
KYVERNO_NAMESPACE:            kyverno (v1:metadata.namespace)
KYVERNO_POD_NAME:             kyverno-admission-controller-5cb897477-jk9j7 (
KYVERNO_DEPLOYMENT:           kyverno-admission-controller
KYVERNO_SVC:                  kyverno-svc
AZURE_CLIENT_ID:              f8c2fc74-9506-4821-a19b-67c4c58c46f6
AZURE_TENANT_ID:              3d95acd6-b6ee-428e-a7a0-196120fc3c65
AZURE_FEDERATED_TOKEN_FILE:   /var/run/secrets/azure/tokens/azure-identity-token
INIT_CONFIG:                  kyverno
METRICS_CONFIG:               kyverno-metrics
KYVERNO_NAMESPACE:            kyverno (v1:metadata.namespace)
KYVERNO_POD_NAME:             kyverno-admission-controller-5cb897477-jk9j7 (
KYVERNO_SERVICEACCOUNT_NAME:  kyverno-admission-controller
KYVERNO_SVC:                  kyverno-svc
TUF_ROOT:                     /.sigstore
KYVERNO_DEPLOYMENT:           kyverno-admission-controller
AZURE_CLIENT_ID:              f8c2fc74-9506-4821-a19b-67c4c58c46f6
AZURE_TENANT_ID:              3d95acd6-b6ee-428e-a7a0-196120fc3c65
AZURE_FEDERATED_TOKEN_FILE:   /var/run/secrets/azure/tokens/azure-identity-token

Assign the AcrPull role to the workload identity

In this post, we will use container images from kyvernonotarytest ACR Registry for testing. Create the AcrPull role assignment at container registry level for the created identity to authorize Kyverno to fetch images.

Get and save kyvernonotarytest registry ID to resourceID.

$ export resourceID=$(az acr show --resource-group "${RESOURCE_GROUP}" --name kyvernonotarytest --query id --output tsv)

Fetch and save the principal ID of the managed identity to spID.

$ export spID=$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query principalId --output tsv)

Authorize kyvernonotarytest ACR and assign AcrPull role for the managed identity.

$ az role assignment create --assignee "${spID}" --scope  "${resourceID}" --role acrpull

To check if the AcrPull role assignment is created, use the following command:

$ az role assignment list --assignee "${spID}" --scope "${resourceID}"
    "condition": null,
    "conditionVersion": null,
    "createdBy": "020fe09b-b37b-40b7-9c0f-f31f2f72ef13",
    "createdOn": "2023-08-30T09:30:30.623337+00:00",
    "delegatedManagedIdentityResourceId": null,
    "description": null,
    "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/kyverno-test-resource-group/providers/Microsoft.ContainerRegistry/registries/kyvernonotarytest/providers/Microsoft.Authorization/roleAssignments/0e9e05e0-e5da-405d-b008-4938de28c735",
    "name": "0e9e05e0-e5da-405d-b008-4938de28c735",
    "principalId": "d05f8352-476b-46f7-98e6-13d547b78077",
    "principalName": "f8c2fc74-9506-4821-a19b-67c4c58c46f6",
    "principalType": "ServicePrincipal",
    "resourceGroup": "kyverno-test-resource-group",
    "roleDefinitionId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/7f951dda-4ed3-4680-a7ca-43fe172d538d",
    "roleDefinitionName": "AcrPull",
    "scope": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/kyverno-test-resource-group/providers/Microsoft.ContainerRegistry/registries/kyvernonotarytest",
    "type": "Microsoft.Authorization/roleAssignments",
    "updatedBy": "020fe09b-b37b-40b7-9c0f-f31f2f72ef13",
    "updatedOn": "2023-08-30T09:30:30.623337+00:00"

Establish federated identity credential

Create the federated identity credential between the managed identity, service account issuer, and service account subject.

$ az identity federated-credential create --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} --identity-name ${USER_ASSIGNED_IDENTITY_NAME} --resource-group ${RESOURCE_GROUP} --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}
  "audiences": [
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/kyverno-test-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kyverno-test-azure-identity/federatedIdentityCredentials/kyverno-test-azure-fed-identity",
  "issuer": "",
  "name": "kyverno-test-azure-fed-identity",
  "resourceGroup": "kyverno-test-resource-group",
  "subject": "system:serviceaccount:kyverno:kyverno-admission-controller",
  "systemData": null,
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials"

Verify ACR images using Azure AD workload identity

First, let’s inspect signatures for our test images using the notation ls command. The image with tag v1-signed has two signatures attached while there is no signature attached to the image with tag v1-unsigned.

$ notation ls
Warning: Always list the artifact using digest(@sha256:...) rather than a tag(:v1-signed) because resolved digest may not point to the same signed artifact, as tags are mutable.
└── application/vnd.cncf.notary.signature
    ├── sha256:57ae40c92b77f05c57dd4946633a680dab52579c24ae89adcc0c4249acc70f83
    └── sha256:3931e5cfbb523e262d3d70841f7e35b36ccfbc4a34fa936857369b79a772368c
$ notation ls
Warning: Always list the artifact using digest(@sha256:...) rather than a tag(:v1-unsigned) because resolved digest may not point to the same signed artifact, as tags are mutable. has no associated signature

Then install the following Kyverno policy to your AKS cluster. This policy verifies the image signature for pods in test-shuting namespace, and tune the policy if you use a different image for testing.

Note, if you follow this instruction to sign the image using Notation and Azure Key Vault, the certificate that is used in the following policy is named wabbit-networks-io.pem, and you can find it by “notation cert ls” command.

kind: ClusterPolicy
  annotations: none
  name: test-acr
  background: true
  - match:
        - Pod
        - test-shuting
    name: check-digest
    - attestors:
      - count: 1
        - certificates:
            cert: |-
                -----BEGIN CERTIFICATE-----
                -----END CERTIFICATE-----
      mutateDigest: true
      required: true
      type: Notary
      verifyDigest: true
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30

Create the test namespace.

$ kubectl create ns test-shuting
namespace/test-shuting created

Create the pod using the signed image in server-side dry-run mode. The pod creation passes through as expected.

$ kubectl run pod -n test-shuting --dry-run=server
pod/pod created (server dry run)

Create another pod using the unsigned image, this time the pod is blocked due to no signature being found.

$ kubectl run pod -n test-shuting --dry-run=server
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request: 
resource Pod/test-shuting/pod was blocked due to the following policies 
  check-digest: 'failed to verify image
    .attestors[0].entries[0]: failed to verify
    no signature is associated with "",
    make sure the artifact was signed successfully'


In conclusion, leveraging Azure AD workload identity and Kyverno offers a powerful solution for enhancing container security on AKS clusters. By integrating Azure AD workload identity, authentication becomes simplified, enabling secure connections between clusters and ACR. Together, these tools provide a streamlined approach to the authentication process, fostering a secure environment for your applications in containers.


Mitigating the Latest Kubernetes NGINX Ingress Controller CVEs
The Evolution of Kyverno
  • Andrej

    October 18, 2023 at 8:16 am Reply

    What will happen when signing certificate will expire. Do I need to re-sign all the images and change certificate in the repo?

    • Jim Bugwadia

      November 16, 2023 at 7:05 am Reply

      You can introduce a new certificate and check using both, until the workloads have been updated.

Post a Comment