Kyverno v1.7.0 - Mutate and Generate Existing Policy Resources

Kyverno 1.7.0 added the ability to mutate existing resources. A mutate rule with a targets list can patch resources already in the cluster, or a different resource than the one that triggered it, and mutateExistingOnPolicyUpdate re-applies the change when the policy is updated. The release also added generate rules for existing resources and multi-attestor image verification.

The Kyverno team shipped initial implementations of perhaps the most-requested features since the project began: mutation and generation for existing resources. There’s far more in this release than those two, though. Below are the main features, with example policies, followed by the changes to check before upgrading. For the releases on either side, see Kyverno v1.6.0 and Kyverno v1.8.0.

What’s in Kyverno v1.7.0

Area	What Kyverno 1.7.0 added or changed
Mutate existing resources	`mutate.targets[]` selects existing resources to patch; `mutateExistingOnPolicyUpdate` re-mutates them when the policy changes
Generate for existing resources	`generateExistingOnPolicyUpdate` fires generate rules for resources that already exist
GitOps	Auto-generated Pod controller rules computed and stored internally (beta, `--autogenInternals=true`)
Image verification	Multi-attestor support with AND/OR groups; global tag-to-digest and digest enforcement; global signing enforcement
Helm installation	Install message pointing to guidance on cluster and node failure trade-offs
Validate rules	In-line context variables; chained context entries; full image metadata preserved
Foreach	Iteration over a simple array of strings
JMESPath	New `items` and `objects_from_lists` filters
CLI	Tests for policies containing subjects, users and groups
Sample policies	More than twenty new policies, including ArgoCD, Flux and Kasten K10 by Veeam
Breaking changes	Multi-line YAML literal-block strings removed; minimum Kubernetes version raised to 1.21

Mutation for Existing Resources

Kyverno has long had mutation abilities, and its no-code policy declarations have solved a large number of use cases. In release 1.7.0 the team delivered one of the most requested features: the ability to mutate existing resources. With this new ability, Kyverno will be able to perform the same type of mutations on resources that already exist in addition to still giving you the option of applying mutations during admission controls, upon item creation and update. Additionally, it’ll allow you to update your Kyverno policy and have mutations performed once again. The policy below sets a label called mykey on all existing Namespaces.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: label-namespaces
spec:
  mutateExistingOnPolicyUpdate: true
  rules:
  - name: label-existing-namespaces
    match:
      any:
      - resources:
          kinds:
          - Namespace
    mutate:
      targets:
        - apiVersion: v1
          kind: Namespace
      patchStrategicMerge:
        metadata:
          labels:
            mykey: myvalue

There’s now a new mutate.targets[] section which allows specifying the existing resource you wish to mutate. The other capability this allows is specifying a different resource you wish to mutate rather than the one triggering the mutation. For example, if you wish to annotate a Deployment when a Secret consumed by the Deployment changes. And, the mutateExistingOnPolicyUpdate field, when set to true, allows you to update this policy later–for example to change the value of the label–and have Kyverno re-mutate those resources.

Other popular use cases for this mutation for existing resources feature of the new Kyverno version include:

Concatenating ConfigMaps
Scaling a Deployment to zero if Pods crash too often
Restart Deployment if one of its Secrets changes
Update image tag on sidecar containers for Deployments based on policy update

Generation for Existing Resources

Just like the new feature that enables mutation of existing resources, Kyverno’s popular generation capabilities were enhanced to extend to existing resources. One of the pain points we heard from the DevOps community was that users didn’t necessarily want to go back and “touch” every Namespace (for example) just to get a generate rule to fire. In 1.7.0, a policy can do this for you.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-resources
spec:
  generateExistingOnPolicyUpdate: true
  rules:
  - name: generate-existing-networkpolicy
    match:
      any:
      - resources:
          kinds:
          - Namespace
    generate:
      kind: NetworkPolicy
      name: default-deny
      namespace: "{{request.object.metadata.name}}"
      synchronize: true
      data:
        metadata:
          labels:
            created-by: kyverno
        spec:
          podSelector: {}
          policyTypes:
          - Egress

The above sample will generate a new NetworkPolicy called default-deny in every Namespace, denying all egress traffic. It also adds the label created-by=kyverno for easier identification.

Some other interesting things you can do with this enhanced generate ability are:

Generate a new ServiceAccount if an existing one is found
Generate PodDisruptionBudgets for existing Deployments
Generate a Custom Resource for all existing Namespaces

Better GitOps Support with Kyverno v 1.7.0

Another nice enhancement is a change to how Kyverno’s auto-generation works. For those not familiar, rule auto-generation is Kyverno’s ability to intelligently and automatically translate Pod policy for Kubernetes Pod controllers such as Deployments and Jobs. This has the benefit of validating Pod controllers rather than letting them succeed yet blocking only their Pods. As of Kyverno 1.7.0, these automatically-generated rules will no longer be written to the spec portion of a policy but will be computed and stored internal to Kyverno. This change will allow Kyverno to play much nicer with your GitOps tool of choice. To enable this ability, which is disabled by default for now as it’s a beta feature, change the flag on the main Kyverno container to –autogenInternals=true.

Enhancing Software Supply Chain Security with Image Verification

Kyverno is the most capable policy engine out there when it comes to ensuring software supply chain security through image verification, and these abilities saw a number of enhancements in Kyverno 1.7.0. Some of these improvements include:

Multi-attestor support using keys, certificates, and keyless signing with AND/OR groups
Global tag-to-digest and digest enforcement, independent of signing
Global signing enforcement (e.g. “all images must be signed”)

Here is one of them in detail: requiring multiple attestors with multiple keys.

In the below policy, you can require that an image with a certain name or pattern must be signed by multiple keys, for example, perhaps an organization-wide key as well as a team-specific key. This gives greater control and, with Kyverno’s existing ability to easily source data from other resources, greater flexibility at minimal cost.

The image verification policy shown here ensures that any image coming from the ghcr.io/myrepo/app repository is signed by a production key and a second key specific to that Namespace–all sourced from a ConfigMap.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image
spec:
  rules:
    - name: check-signatures
      match:
        any:
        - resources:
            kinds:
              - Pod
      context:
      - name: keys
        configMap:
          name: keys
          namespace: default
      verifyImages:
      - imageReferences:
        - "ghcr.io/myrepo/app*"
        attestors:
        - count: 2
          entries:
          - keys:
              publicKeys: "{{ keys.data.production }}"
          - keys:
              publicKeys: "{{ keys.data.{{request.namespace}} }}"

Helm Installation

One thing we’re making clearer with Helm installs is guidance on how Kyverno handles cluster/node failure scenarios. Starting with Kyverno 1.7.0, a Helm install will produce a message pointing you to the documentation which explains the trade-off between security and operability and how to decide which is best for you based upon your needs, requirements, and way of operating. With those in mind, you may choose to exclude the Kyverno Namespace favoring operability; or you may choose to leave the default behavior as-is favoring security.

Other Additions and Enhancements

Validate rules received some nice updates to context variables in that now in-line variables are supported. Context variables can also be chained together where one sets a context and another context can refer back to the first. And, images will have all their metadata preserved when wishing to verify any of it, useful in checking for metadata added by Docker BuildKit, for example.

Foreach rules can now iterate over a simple array of strings rather than needing to be a full object, helpful in situations where you have simpler inputs.

On the JMESPath side, we have two new filters called items and objects_from_lists which allow more manipulation of object type data.

On the CLI, Kyverno now has the ability to test policies which contain subjects, users, and groups.

And on the sample policy library, more than twenty new policies have been added including many for ecosystem tools like ArgoCD, Flux, Kasten K10 by Veeam, and others.

Potentially Breaking Changes

A couple things of which to be aware prior to upgrading. First, multi-line YAML strings with literal blocks, while easier to write than JSON-encoded arrays, were causing issues with being able to represent full Kubernetes manifests and having Kyverno work properly with them, so they have been removed. Anywhere you need to read in a list of strings from a ConfigMap, you’ll want to ensure those are converted to JSON format as outlined here.

Second thing is, the minimum version of Kubernetes has been bumped up to 1.21. We needed to take advantage of some of the newer capabilities and fix some issues, so starting with Kyverno 1.7.0 you should ensure you only install or upgrade to this version if you’re on at least 1.21.

Closing

Kyverno v1.7.0 closed over 200 issues. The maintainers and contributors spent the past few months on it, and we hope you find it useful. For an overview of the open source project, see Kyverno OSS. As always, we love the community and hope you engage with us on any one of our outlets.

Discover more about Kyverno v 1.7.0 for GitOps and DevOps needs here, and thanks for reading! You can also sign-up for a Free Trial to see what all the excitement is about.

Image by Christine Aubé from Pixabay

Frequently asked questions

Can a Kyverno 1.7 mutate rule change a resource other than the one that triggered it?

Yes. The resource listed under mutate.targets does not have to be the resource that matched the rule. A change to one object can trigger a patch on another, for example annotating a Deployment when a Secret it consumes changes. Restarting a Deployment when one of its Secrets changes is one of the use cases the release called out. The match block decides when the rule fires; targets decides what gets patched.

Does mutating existing resources replace mutation at admission time?

No. Kyverno 1.7.0 keeps mutation during admission control, when a resource is created or updated, and adds mutation for resources that already exist as a separate option. You can use both in the same cluster. Admission-time rules shape new objects as they arrive, while mutate-existing rules bring objects that were created earlier into line without anyone re-applying them.

What do I need to check before upgrading to Kyverno 1.7.0?

Two things. The cluster must run Kubernetes 1.21 or later, because 1.7.0 raised the minimum version. And multi-line YAML strings written as literal blocks are no longer supported, so any list of strings read from a ConfigMap has to be converted to JSON format first. Also note that the autogenInternals behaviour is a beta feature and stays off unless you set the flag.

Are the Kyverno 1.7 policy fields shown here still current?

Not all of them. This post describes Kyverno 1.7.0 as released in 2022. Later releases changed several policy-level settings; Kyverno 1.13, for example, deprecated spec.mutateExistingOnPolicyUpdate in favour of a per-rule mutate.mutateExisting field. Before you copy these examples, check the current documentation and release notes on kyverno.io for the version you run.