Container security is a paramount concern for organizations deploying applications on Kubernetes clusters. One crucial aspect of securing container workloads is ensuring that only trusted images are used. Azure Container Registry (ACR) is a private registry service for building, storing, and managing container images and related artifacts. However, managing access to ACR and verifying image authenticity can be challenging.
In this blog post, we will explore how to leverage Azure AD workload identity and Kyverno, a powerful Kubernetes policy engine, to enhance container security by verifying ACR images on AKS clusters. By harnessing the capabilities of Azure AD workload identity, we can simplify the authentication process and establish secure connections between AKS clusters and ACR, while Kyverno allows us to enforce fine-grained policies to verify image authenticity. By the end of this post, you will have a clear understanding of how to leverage Azure AD workload identity and Kyverno to strengthen container security, prevent unauthorized image deployments, and ensure the integrity of your AKS clusters.
Create a resource group
First, create a resource group named `kyverno-test-resource-group` in the `eastus` location with the following command:
$ az group create --name kyverno-test-resource-group --location eastus
{
"id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/kyverno-test-resource-group",
"location": "eastus",
"managedBy": null,
"name": "kyverno-test-resource-group",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": "Microsoft.Resources/resourceGroups"
}To help simplify the steps to configure the identities required, the steps below define environmental variables for reference on the cluster. Here it assumes the default installation of Kyverno. If you have a custom Kyverno installation, replace SERVICE_ACCOUNT_NAMESPACE and SERVICE_ACCOUNT_NAME for the Kyverno admission controller service account with your custom values.
export RESOURCE_GROUP="kyverno-test-resource-group"
export LOCATION="eastus"
export SERVICE_ACCOUNT_NAMESPACE="kyverno"
export SERVICE_ACCOUNT_NAME="kyverno-admission-controller"
export SUBSCRIPTION="$(az account show --query id --output tsv)"
export USER_ASSIGNED_IDENTITY_NAME="kyverno-test-azure-identity"
export FEDERATED_IDENTITY_CREDENTIAL_NAME="kyverno-test-azure-fed-identity"
export KEYVAULT_NAME="kyverno-test-kv"
export KEYVAULT_SECRET_NAME="kyverno-test-kv-secret"Build and sign container images using Notation and Azure Key Vault
You can skip this section if you already have a test image. Otherwise, follow this instruction to build test images in ACR and sign them using Notary and Azure Key Vault.
Note that the registry needs to be under the created resource group, i.e., kyverno-test-resource-group if you follow this blog post.
Create an AKS cluster
Create an AKS cluster using the Azure CLI with OpenID Connect (OIDC) Issuer.
$ az aks create -g "${RESOURCE_GROUP}" -n kyverno-test --node-count 1 --enable-oidc-issuer --enable-workload-identity --generate-ssh-keysIt could take a few minutes to provision the cluster, once completed, it returns JSON-formatted information about the cluster.
Get the OIDC Issuer URL and save it to an environmental variable using the following command.
$ export AKS_OIDC_ISSUER="$(az aks show -n kyverno-test -g "${RESOURCE_GROUP}" --query "oidcIssuerProfile.issuerUrl" -otsv)"sCreate a managed identity
Set a specific subscription as the current active subscription.
$ az account set --subscription "${SUBSCRIPTION}"Create a managed identity. Managed identities provide an automatically managed identity in Azure Active Directory (Azure AD) for applications to use when connecting to resources that support Azure AD authentication. We will configure the Kyverno service account to consume this managed identity for accessing the ACR Registry and retrieving data.
$ az identity create --name "${USER_ASSIGNED_IDENTITY_NAME}" --resource-group "${RESOURCE_GROUP}" --location "${LOCATION}" --subscription "${SUBSCRIPTION}"
{
"clientId": "f8c2fc74-9506-4821-a19b-67c4c58c46f6",
"id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/kyverno-test-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kyverno-test-azure-identity",
"location": "eastus",
"name": "kyverno-test-azure-identity",
"principalId": "d05f8352-476b-46f7-98e6-13d547b78077",
"resourceGroup": "kyverno-test-resource-group",
"systemData": null,
"tags": {},
"tenantId": "3d95acd6-b6ee-428e-a7a0-196120fc3c65",
"type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}Save the client ID of the managed identity to an environmental variable for later use.
$ export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' -otsv)"Install Kyverno and configure it to use the workload identity
Switch to use the AKS cluster created above.
$ az aks get-credentials -n kyverno-test -g "${RESOURCE_GROUP}"Ensure the webhook configurations are in-place to inject workload identity to Kyverno.
$ kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations
NAME WEBHOOKS AGE
mutatingwebhookconfiguration.admissionregistration.k8s.io/aks-node-mutating-webhook 1 18m
mutatingwebhookconfiguration.admissionregistration.k8s.io/aks-webhook-admission-controller 1 18m
mutatingwebhookconfiguration.admissionregistration.k8s.io/azure-wi-webhook-mutating-webhook-configuration 1 16m
NAME WEBHOOKS AGE
validatingwebhookconfiguration.admissionregistration.k8s.io/aks-node-validating-webhook 1 18mTo use the workload identity for Kyverno, you need to:
- annotate the service account of the admission controller with azure.workload.identity/client-id, the value is the client ID of the managed identity which we have saved to $USER_ASSIGNED_CLIENT_ID earlier
- label the pod of the admission controller with azure.workload.identity/use: “true”
Create a values.yaml file and use it for Helm installation.
config:
webhookAnnotations:
admissions.enforcer/disabled: "true"
admissionController:
rbac:
serviceAccount:
annotations:
azure.workload.identity/client-id: f8c2fc74-9506-4821-a19b-67c4c58c46f6
podLabels:
azure.workload.identity/use: "true"$ helm upgrade --install kyverno kyverno/kyverno --namespace kyverno --create-namespace --values values.yamlTo customize your Kyverno installation, please follow the guidance here.
Verify Kyverno is installed successfully.
$ kubectl get pod -n kyverno
NAME READY STATUS RESTARTS AGE
kyverno-admission-controller-bd7798fb8-gsp5z 1/1 Running 0 17s
kyverno-background-controller-5c97796cdd-dm8kt 1/1 Running 0 17s
kyverno-cleanup-controller-78c4978cf4-zc7sc 1/1 Running 0 17s
kyverno-reports-controller-6788666c89-sknc4 1/1 Running 0 17sVerify the environmental variables for the workload identity are injected correctly, AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE, AZURE_AUTHORITY_HOST must be set for the Kyverno admission controller.
$ kubectl describe pod -n kyverno -l app.kubernetes.io/component=admission-controller | grep Environment -A12
Environment:
KYVERNO_SERVICEACCOUNT_NAME: kyverno-admission-controller
INIT_CONFIG: kyverno
METRICS_CONFIG: kyverno-metrics
KYVERNO_NAMESPACE: kyverno (v1:metadata.namespace)
KYVERNO_POD_NAME: kyverno-admission-controller-5cb897477-jk9j7 (v1:metadata.name)
KYVERNO_DEPLOYMENT: kyverno-admission-controller
KYVERNO_SVC: kyverno-svc
AZURE_CLIENT_ID: f8c2fc74-9506-4821-a19b-67c4c58c46f6
AZURE_TENANT_ID: 3d95acd6-b6ee-428e-a7a0-196120fc3c65
AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/
--
Environment:
INIT_CONFIG: kyverno
METRICS_CONFIG: kyverno-metrics
KYVERNO_NAMESPACE: kyverno (v1:metadata.namespace)
KYVERNO_POD_NAME: kyverno-admission-controller-5cb897477-jk9j7 (v1:metadata.name)
KYVERNO_SERVICEACCOUNT_NAME: kyverno-admission-controller
KYVERNO_SVC: kyverno-svc
TUF_ROOT: /.sigstore
KYVERNO_DEPLOYMENT: kyverno-admission-controller
AZURE_CLIENT_ID: f8c2fc74-9506-4821-a19b-67c4c58c46f6
AZURE_TENANT_ID: 3d95acd6-b6ee-428e-a7a0-196120fc3c65
AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/Assign the AcrPull role to the workload identity
In this post, we will use container images from kyvernonotarytest ACR Registry for testing. Create the AcrPull role assignment at container registry level for the created identity to authorize Kyverno to fetch images.
Get and save kyvernonotarytest registry ID to resourceID.
$ export resourceID=$(az acr show --resource-group "${RESOURCE_GROUP}" --name kyvernonotarytest --query id --output tsv)
/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/kyverno-test-resource-group/providers/Microsoft.ContainerRegistry/registries/kyvernonotarytestFetch and save the principal ID of the managed identity to spID.
$ export spID=$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query principalId --output tsv)
d05f8352-476b-46f7-98e6-13d547b78077Authorize kyvernonotarytest ACR and assign AcrPull role for the managed identity.
$ az role assignment create --assignee "${spID}" --scope "${resourceID}" --role acrpullTo check if the AcrPull role assignment is created, use the following command:
$ az role assignment list --assignee "${spID}" --scope "${resourceID}"
[
{
"condition": null,
"conditionVersion": null,
"createdBy": "020fe09b-b37b-40b7-9c0f-f31f2f72ef13",
"createdOn": "2023-08-30T09:30:30.623337+00:00",
"delegatedManagedIdentityResourceId": null,
"description": null,
"id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/kyverno-test-resource-group/providers/Microsoft.ContainerRegistry/registries/kyvernonotarytest/providers/Microsoft.Authorization/roleAssignments/0e9e05e0-e5da-405d-b008-4938de28c735",
"name": "0e9e05e0-e5da-405d-b008-4938de28c735",
"principalId": "d05f8352-476b-46f7-98e6-13d547b78077",
"principalName": "f8c2fc74-9506-4821-a19b-67c4c58c46f6",
"principalType": "ServicePrincipal",
"resourceGroup": "kyverno-test-resource-group",
"roleDefinitionId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/7f951dda-4ed3-4680-a7ca-43fe172d538d",
"roleDefinitionName": "AcrPull",
"scope": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/kyverno-test-resource-group/providers/Microsoft.ContainerRegistry/registries/kyvernonotarytest",
"type": "Microsoft.Authorization/roleAssignments",
"updatedBy": "020fe09b-b37b-40b7-9c0f-f31f2f72ef13",
"updatedOn": "2023-08-30T09:30:30.623337+00:00"
}
]Establish federated identity credential
Create the federated identity credential between the managed identity, service account issuer, and service account subject.
$ az identity federated-credential create --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} --identity-name ${USER_ASSIGNED_IDENTITY_NAME} --resource-group ${RESOURCE_GROUP} --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}
{
"audiences": [
"api://AzureADTokenExchange"
],
"id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/kyverno-test-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kyverno-test-azure-identity/federatedIdentityCredentials/kyverno-test-azure-fed-identity",
"issuer": "https://eastus.oic.prod-aks.azure.com/3d95acd6-b6ee-428e-a7a0-196120fc3c65/ed6dcfcc-5a18-4c5b-98b4-ed667aeff22a/",
"name": "kyverno-test-azure-fed-identity",
"resourceGroup": "kyverno-test-resource-group",
"subject": "system:serviceaccount:kyverno:kyverno-admission-controller",
"systemData": null,
"type": "Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials"
}Verify ACR images using Azure AD workload identity
First, let’s inspect signatures for our test images using the notation ls command. The image with tag v1-signed has two signatures attached while there is no signature attached to the image with tag v1-unsigned.
$ notation ls kyvernonotarytest.azurecr.io/net-monitor:v1-signed
Warning: Always list the artifact using digest(@sha256:...) rather than a tag(:v1-signed) because resolved digest may not point to the same signed artifact, as tags are mutable.
kyvernonotarytest.azurecr.io/net-monitor@sha256:cea196b6fb3e9cefa495d40652f3abbd19c858a72d297ebbfa19ce00549f609f
└── application/vnd.cncf.notary.signature
├── sha256:57ae40c92b77f05c57dd4946633a680dab52579c24ae89adcc0c4249acc70f83
└── sha256:3931e5cfbb523e262d3d70841f7e35b36ccfbc4a34fa936857369b79a772368c$ notation ls kyvernonotarytest.azurecr.io/net-monitor:v1-unsigned
Warning: Always list the artifact using digest(@sha256:...) rather than a tag(:v1-unsigned) because resolved digest may not point to the same signed artifact, as tags are mutable.
kyvernonotarytest.azurecr.io/net-monitor@sha256:4a2f25fd8a2e0696614cbb2e55c07367ed77331d4849c7f275f05ced398fb19a has no associated signatureThen install the following Kyverno policy to your AKS cluster. This policy verifies the image signature for pods in test-shuting namespace, and tune the policy if you use a different image for testing.
Note, if you follow this instruction to sign the image using Notation and Azure Key Vault, the certificate that is used in the following policy is named wabbit-networks-io.pem, and you can find it by “notation cert ls” command.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
annotations:
pod-policies.kyverno.io/autogen-controllers: none
name: test-acr
spec:
background: true
rules:
- match:
resources:
kinds:
- Pod
namespaces:
- test-shuting
name: check-digest
verifyImages:
- attestors:
- count: 1
entries:
- certificates:
cert: |-
-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----
imageReferences:
- kyvernonotarytest.azurecr.io/net-monitor*
mutateDigest: true
required: true
type: Notary
verifyDigest: true
validationFailureAction: Enforce
webhookTimeoutSeconds: 30Create the test namespace.
$ kubectl create ns test-shuting
namespace/test-shuting createdCreate the pod using the signed image in server-side dry-run mode. The pod creation passes through as expected.
$ kubectl run pod --image=kyvernonotarytest.azurecr.io/net-monitor:v1-signed -n test-shuting --dry-run=server
pod/pod created (server dry run)Create another pod using the unsigned image, this time the pod is blocked due to no signature being found.
$ kubectl run pod --image=kyvernonotarytest.azurecr.io/net-monitor:v1-unsigned -n test-shuting --dry-run=server
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
resource Pod/test-shuting/pod was blocked due to the following policies
test-acr:
check-digest: 'failed to verify image kyvernonotarytest.azurecr.io/net-monitor:v1-unsigned:
.attestors[0].entries[0]: failed to verify kyvernonotarytest.azurecr.io/net-monitor@sha256:4a2f25fd8a2e0696614cbb2e55c07367ed77331d4849c7f275f05ced398fb19a:
no signature is associated with "kyvernonotarytest.azurecr.io/net-monitor@sha256:4a2f25fd8a2e0696614cbb2e55c07367ed77331d4849c7f275f05ced398fb19a",
make sure the artifact was signed successfully'Conclusion
In conclusion, leveraging Azure AD workload identity and Kyverno offers a powerful solution for enhancing container security on AKS clusters. By integrating Azure AD workload identity, authentication becomes simplified, enabling secure connections between clusters and ACR. Together, these tools provide a streamlined approach to the authentication process, fostering a secure environment for your applications in containers.
Andrej
October 18, 2023 at 8:16 amWhat will happen when signing certificate will expire. Do I need to re-sign all the images and change certificate in the repo?