We are excited to announce the launch of kyverno-notation-aws, a Nirmata extension service for Kyverno that executes the AWS Signer plugin for Notation to verify image signatures and attestations.
AWS Signer, announced by Amazon AWS a few months ago is a new capability that gives us native AWS support for signing and verifying container images stored in container registries like Amazon Elastic Container Registry (Amazon ECR). It supports features like cross-account signing, signature validity duration, and profile lifecycle management with cancellation and revocation operations.
AWS Signer supports signing and verifying container images and is integrated with Notation, an open-source Notary project within the Cloud Native Computing Foundation (CNCF). With contributions from AWS, Microsoft, Docker, and several others, Notary is an open standard and client implementation that allows for vendor-specific plugins for key management and other integrations. AWS Signer manages signing keys, key rotation, and PKI management for you and is integrated with Notation through a binary plugin that provides a simple client-based workflow.
The Nirmata extension service is invoked from a Kyverno policy which then calls the notation service configured with AWS Signer to provide image signature and attestation verification as well as digest mutation.
Key Features
Image Signature Verification
The Nirmata extension service provides image signature verification capabilities. Verifying images establishes authenticity and provenance, giving you the ability to determine if content comes from a particular party. This allows you to only allow images from trusted parties to be used in your container image builds and deployments
The Nirmata extension service is invoked from a Kyverno policy which passes it a list of images to be verified. The service then verifies notation format signatures for container images using the AWS Signer notation plugin and returns responses back to Kyverno.
The service manages Notation trust policies and trust stores as Kubernetes resources.
Image Digest Mutation
Image tags are mutable in nature and can be spoofed. However, we can use image digests that are immutable to ensure better security, and image integrity and prevent tampering. The service allows users to replace image tags with digests.
The Kyverno policy passes the images variable to the services’ /checkimages endpoint. The result returns a list of images with their JSON path and digests so Kyverno can mutate each image in the admission payload.
Here is an example:
Response object structure
{
"verified": true,
"message": "...",
"results": [
{
"op": "replace",
"path": "/spec/containers/0/image",
"value": "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108"
}
]
}
- verified: true means that all the images provided to the extensions were verified. We can use this variable when we only want to validate image verification.
- message: “” returns the error message when verification fails.
- results is an array of objects containing the name of the container image, the path of the container image, and the mutated image.
Kyverno policy fragment
mutate:
foreach:
- list: "response.results"
patchesJson6902: |-
- path: '{{ element.path }}'
op: '{{ element.op }}'
value: '{{ element.value }}'
This policy fragment describes a mutation where we are looping through the array of results we have received from the extension and do a replace operation, where we replace the current image with the same image containing digests.
Attestation Verification
In addition to verifying signatures, the extension service can verify signed metadata i.e. attestations.
To verify attestations, the Kyverno policy can optionally pass a variable called attestations in the request:
- key: attestations
value:
- imageReference: "*"
- name: sbom/cyclone-dx
conditions:
all:
- key: \{{creationInfo.licenseListVersion}}
operator: Equals
value: "3.17"message: invalid license version
- name: application/sarif+json
conditions:
all:
- key: \{{ element.components[].licenses[].expression }}
operator: AllNotIn
value: ["GPL-2.0", "GPL-3.0"]
- imageReference: "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo*"type:
- name: application/vnd.cyclonedx
conditions:
all:
- key: \{{ element.components[].licenses[].expression }}
operator: AllNotIn
value: ["GPL-2.0", "GPL-3.0"]The attestations variable is a JSON array of where each entry has:
- An imageReference to match images;
- A type that specifies the name of the attestation; and
- A list of conditions we want to verify the attestation data
In the example above we are verifying the following:
- The attestations sbom/cyclone-dx and application/sarif+json exist for all images.
- The creationInfo.licenseListVersion is equal to 3.17 in the SBOM and GPL licenses are not present.
- The attestation application/vnd.cyclonedx is available for all versions of the 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo image and does not contain GPL licenses.
NOTE: The conditions key in the attestations must be escaped with \ so Kyverno does not substitute them before executing the extension service.
Caching
To prevent repeated lookups for verified images, the Nirmata extension has a built-in cache.
Caching is enabled by default and can be managed using the –cacheEnabled flag. The cache is a TTL-based cache, i.e., entries expire automatically after some time and the value of TTL can be customized using –cacheTTLDurationSeconds (default is 3600) and the max number of entries in the cache can be configured using –cacheMaxSize (default is 1000).
The cache stores the verification outcomes of images for the trust policy and verification outcomes of attestations with the trust policy and conditions. The cache is an in-memory cache that gets cleared when the pod is recreated. The cache will also be cleared when there is any change in trust policies and trust stores.
Multi-Tenancy
In a shared cluster, each team may have different signatures and trust policies. To support such use cases, the extension allows configuring multiple trust policies and trust stores as Kubernetes custom resources.
The extension service allows specifying what trust policy they want to use for verification thus enabling multi-tenancy. Multiple teams can share one cluster and have different trust policies separate from each other. To specify the trust policy, we can pass the trustPolicy variable in the request:
- key: trustPolicy
value: "tp-{{request.namespace}}"
or we can set the DEFAULT_TRUST_POLICY environmental variable. In the above example, we are dynamically using the trust policy for the namespace of the request.
High Availability
Kyverno-notation-aws can be installed in a highly-available manner where additional replicas can be deployed for the plugin. The plugin does not use leader election for inbound API requests which means verification requests can be distributed and processed by all available replicas. Leader election is required for certificate management so therefore only one replica will handle these tasks at a given time.
Multiple replicas configured for the plugin can be used for both availability and scale. Vertical scaling of the individual replicas’ resources may also be performed to increase combined throughput.
Demo
To install kyverno-notation-aws plugin, follow the instructions in kyverno-notation-aws repository’s readme.
After installation, kyverno-notation-aws namespace should look as follows:
$ kubectl get all,secret,sa,lease,rolebindings -n kyverno-notation-aws
NAME READY STATUS RESTARTS AGE
pod/kyverno-notation-aws-5b49875b94-ftntr 1/1 Running 0 16m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/svc ClusterIP 172.20.43.47 443/TCP 33d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kyverno-notation-aws 1/1 1 1 33d
NAME DESIRED CURRENT READY AGE
replicaset.apps/kyverno-notation-aws-5b49875b94 1 1 1 16m
NAME TYPE DATA AGE
secret/root-secret kubernetes.io/tls 3 33d
secret/svc.kyverno-notation-aws.svc.tls-ca kubernetes.io/tls 2 8d
secret/svc.kyverno-notation-aws.svc.tls-pair kubernetes.io/tls 2 8d
NAME SECRETS AGE
serviceaccount/default 0 33d
serviceaccount/kyverno-notation-aws 0 33d
NAME HOLDER AGE
lease.coordination.k8s.io/kyverno-notation-aws kyverno-notation-aws-5b49875b94-ftntr 8d
NAME ROLE AGE
rolebinding.rbac.authorization.k8s.io/kyverno-notation-aws-rolebinding Role/kyverno-notation-aws-role 33d
Next, we have to create a test namespace to test our plugin:
$ kubectl create ns test-notation
namespace/test-notation created
Now, we have to apply a Kyverno policy so Kyverno can interact with the plugin to verify an image:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: check-images
spec:
validationFailureAction: Enforce
failurePolicy: Fail
webhookTimeoutSeconds: 30
schemaValidation: false
rules:
- name: call-aws-signer-extension
match:
any:
- resources:
namespaces:
- test-notation
kinds:
- Pod
operations:
- CREATE
- UPDATE
context:
- name: tlscerts
apiCall:
urlPath: "/api/v1/namespaces/kyverno-notation-aws/secrets/svc.kyverno-notation-aws.svc.tls-pair"
jmesPath: "base64_decode( data.\"tls.crt\" )"
- name: response
apiCall:
method: POST
data:
- key: images
value: "{{images}}"
- key: trustPolicy
value: "tp-{{request.namespace}}"
- key: attestations
value:
- imageReference: "*"
type:
- name: sbom/example
conditions:
all:
- key: \{{creationInfo.licenseListVersion}}
operator: Equals
value: "3.17"
message: invalid license version
service:
url: <https://svc.kyverno-notation-aws/checkimages>
caBundle: '{{ tlscerts }}'
mutate:
foreach:
- list: "response.results"
patchesJson6902: |-
- path: '{{ element.path }}'
op: '{{ element.op }}'
value: '{{ element.value }}'This policy does the following:
- Checks for Pod creation and update requests in test-notation namespace
- Sends an API call to https://svc.kyverno-notation-aws/checkimages with TLS certs extracted from the secret using an API call. The API call request is a POS request that contains the following
- Kyverno’s images variable which sends the image info of all the images used in the given resource.
- Trust policy name, to specify what trust policy to use for this request. Optional if DEFAULT_TRUST_POLICY environmental variable is set in the plugin.
- An attestations list as described in attestation verification section.
Here, we have an unsigned image in AWS ECR 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned:
$ notation inspect 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned
Warning: Always inspect the artifact using digest(@sha256:...) rather than a tag(:v1-unsigned) because resolved digest may not point to the same signed artifact, as tags are mutable.
844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5 has no associated signature
When we run an unsigned image, it gives an error saying the image was not signed:
$ kubectl -n test-notation run test1 --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned --dry-run=server
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request: mutation policy check-images error: failed to apply policy check-images rules [call-aws-signer-extension: failed to load context: failed to fetch data for APICall: HTTP 406 Not Acceptable: failed to verify container kyverno-demo: failed to verify image {{844333597536.dkr.ecr.us-west-2.amazonaws.com kyverno-demo kyverno-demo v1-unsigned } /spec/containers/0/image}: no signature is associated with "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5", make sure the artifact was signed successfully]
When using an image that has an SBOM attached to it and is signed using notary 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1:
$ oras discover -o tree 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
├── application/vnd.cncf.notary.signature
│ └── sha256:04eed4edc7a549399648e9fdd6d34fb46ff79cd9f8b155827d6c569c7c3c732f
└── sbom/example
└── sha256:352c1a77a9c635b9b2e9bc6b26ccedd9321088e103668ab935a0b6aa3a622aa4
└── application/vnd.cncf.notary.signature
└── sha256:032720a8b067f7cb25b5eb5cecd2b663fa05bc352c8206003a334856ce7b8a4b
The output from notation inspect shows that the image was signed using AWS signer and has signing profile information attached:
$ notation inspect 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
Warning: Always inspect the artifact using digest(@sha256:...) rather than a tag(:v1) because resolved digest may not point to the same signed artifact, as tags are mutable.
Inspecting all signatures for signed artifact
844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
└── application/vnd.cncf.notary.signature
└── sha256:04eed4edc7a549399648e9fdd6d34fb46ff79cd9f8b155827d6c569c7c3c732f
...
│ ├── com.amazonaws.signer.signingJob: arn:aws:signer:us-west-2:844333597536:/signing-jobs/006bb1ca-1380-4b9d-8dba-19a605df8fbb
│ ├── com.amazonaws.signer.signingProfileVersion: arn:aws:signer:us-west-2:844333597536:/signing-profiles/kyvernodemo/Kq390zhFlj
│ └── io.cncf.notary.verificationPlugin: com.amazonaws.signer.notation.plugin
...
├── certificates
│ ├── SHA256 fingerprint: bb0727fbd9a9688f5257ab229fc370b382a034d47552fec272bdd7f912fc1751
│ │ ├── issued to: CN=AWS Signer,OU=AWS Cryptography,O=AWS,L=Seattle,ST=WA,C=US
│ │ ├── issued by: CN=AWS Signer us-west-2 Code Signing CA G1,OU=Cryptography,O=AWS,ST=WA,C=US
│ │ └── expiry: Mon Aug 7 04:31:29 2023
...
└── signed artifact
├── media type: application/vnd.docker.distribution.manifest.v2+json
├── digest: sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
└── size: 526
apiVersion: notation.nirmata.io/v1alpha1
kind: TrustPolicy
spec:
...
trustStores:
- signingAuthority:aws-signer-ts
trustedIdentities:
- "arn:aws:signer:us-west-2:844333597536:/signing-profiles/kyvernodemo"
The attached SBOM sbom/example has the license version set to 3.17:
{
"SPDXID": "SPDXRef-DOCUMENT",
"name": "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo-v1",
"spdxVersion": "SPDX-2.2",
"creationInfo": {
"created": "2023-08-01T08:33:48.450737Z",
"creators": [
"Organization: Anchore, Inc",
"Tool: syft-v0.46.3"
],
"licenseListVersion": "3.17"
},
"dataLicense": "CC0-1.0",
"documentNamespace": "<https://anchore.com/syft/image/844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo-v1-2fe0c3fb-75ce-4c3c-9dee-9c87e8b5e688>",
"packages": []
}
When we run this image, verification succeeds:
$ kubectl -n test-notation run test1 --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1 --dry-run=server
pod/test1 created (server dry run)
If we update the ClusterPolicy to check for “licenseListVersion”: “3.18”:
type:
- name: sbom/example
conditions:
all:
- key: \{{creationInfo.licenseListVersion}}
operator: Equals
value: "3.18"
message: invalid license version
Image verification fails with an invalid license version error:
$ kubectl -n test-notation run test1 --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1 --dry-run=server
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request: mutation policy check-images error: failed to apply policy check-images rules [call-aws-signer-extension: failed to load context: failed to fetch data for APICall: HTTP 406 Not Acceptable: failed to verify attestatations: failed to verify attestations: failed to verify conditions 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1 sha256:352c1a77a9c635b9b2e9bc6b26ccedd9321088e103668ab935a0b6aa3a622aa4: failed to evaluate conditions: invalid license version
Conclusion
In this post, we showed you the details of kyverno-notation-aws, a Nirmata extension service that executes the AWS Signer plugin for Notation to verify image signatures and attestations. The Nirmata extension service is invoked from a Kyverno policy which then integrates with AWS Signer to provide image signature and attestation verification and digest mutation.
While the extension is ready for production use, we will continue to enhance and expand integrations for this extension, and we welcome your feedback. Please visit kyverno-notation-aws repository on GitHub to check on progress, and please open an issue to tell us about changes you’d like us to work on next.
If you are at AWS re:Invent 2023, lets connect there!
Sorry, the comment form is closed at this time.