ChatGPT and generative AI are all the buzz these days so we started exploring what that means for us at Nirmata and for the broader Kyverno community. Are there ways to use or incorporate this groundbreaking technology to help our users? Can it reduce the learning curve? One area we started exploring is using natural language processing to help write Kyverno policies. In the past, we have explored options to build low-code/no-code designers to help create Kyverno policies so exploring ChatGPT as an alternative sounded interesting.

To run an experiment with ChatGPT, we asked it to write the most basic of Kyverno policies: Require Labels – This policy validates if a specific label exists on pods. 

Generate the “Require Labels” Kyverno policy

Me: Write a Kyverno policy to require that a label `app.kubernetes.io/name` exists on every pod

(Note: This is the exact same policy from the example above taken from the Kyverno website)

Here is the full policy that was generated and below is the screenshot of the full response:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-app-label
spec:
  background: false
  rules:
  - name: check-app-label
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Pods must have the app.kubernetes.io/name label."
      pattern:
        metadata:
          labels:
            app.kubernetes.io/name: "?*"

Notice that the Kyverno policy that was generated is identical to the example available on the kyverno.io website with a few differences:

• validationFailureAction is not specified – This is not a huge issue since by default validationFailureAction is set to audit
• background is set to false – This would be set to true in most cases but not a huge issue
• Rule name is different – Looks like it automatically generated the rule name instead of using the one in the example policy

Overall, not bad! We got what we asked for. We applied this policy to a cluster and verified that it worked as expected!

Generate the “Require Labels” Kyverno policy with a slight variation

Now, let’s take a look at another request to create the same “require labels” policy but with a different label. 

Me: Write a Kyverno policy to add a label `org-name` exists on every pod

(Note: In this case, we request “org-name” as the label which is different from the previous request.)

Here is the full policy that was generated and below is the screenshot of the full response:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-org-label
spec:
  background: false
  rules:
  - name: check-org-label
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Pods must have the org-name label."
      pattern:
        metadata:
          labels:
            org-name: "?*"

This policy is exactly identical to the one created previously with some minor differences:

• Rule name is different – indicating that is generated based on the name of the label being checked
• The label being checked is different – this is what we requested.

Again, not bad! We applied this policy to a cluster and verified that it worked as expected.

Generate the “Require Labels” Kyverno policy to block resources

Now, let’s complicate things a bit more. We ask ChatGPT to require a policy that blocks resources if the label is not present.

Me: Write a Kyverno policy that checks for a label called `org-name` exists on every pod. It should block the pod from being created if the label does not exist.

Here is the full policy that was generated and below is the screenshot of the full response:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-org-label
spec:
  background: false
  rules:
  - name: check-org-label
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Pods must have the org-name label."
      pattern:
        metadata:
          labels:
            org-name: "?*"
    deny:
      conditions:
      - key: metadata.labels.org-name
        operator: DoesNotExist

Looking at the policy, it seems to do what we asked to do but here are some observations

• The validationFailureAction is missing – In order to block resources that don’t meet the criteria, validateFailureAction should be set to ‘enforce’
• There is a ‘deny:’ block which seems out of place.

When I try to apply this policy to my cluster, I get this error message:

kubectl create -f check-org-label-block.yaml
error: error validating "check-org-label-block.yaml": error validating data: ValidationError(ClusterPolicy.spec.rules[0]): unknown field "deny" in io.kyverno.v1.ClusterPolicy.spec.rules; if you choose to ignore these errors, turn validation off with --validate=false

So, ChatGPT generated an incorrect policy that does not even match the schema. The actual valid policy is very simple. All you need to do is use the same policy from the previous test and add the validationFailureAction to ‘enforce’ to block the misconfigured resource from being created.

Correct policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-org-label
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: check-org-label
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Pods must have the org-name label."
      pattern:
        metadata:
          labels:
            org-name: "?*"

Generate the “Require Labels” Kyverno policy to block resources with variation

Let’s try it again with some variation in the request. 

Me: Write a Kyverno policy that enforces that a label called `org-name` exists on every pod.

Here is the full policy that was generated and below is the screenshot of the full response:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-org-name-label
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: check-org-name-label
    match:
      kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    validate:
      message: "Pods must have an 'org-name' label"
      pattern:
        metadata:
          labels:
            org-name: "<*>"

This time it comes close! We can see that the validationFailureAction is set to enforce. But the match and pattern blocks are different from our previous results. Unfortunately, this policy also does not seem correct. Applying this policy results in a validation error.

kubectl create -f enforce-org-label.yaml
error: error validating "enforce-org-label.yaml": error validating data: ValidationError(ClusterPolicy.spec.rules[0].match): unknown field "kinds" in io.kyverno.v1.ClusterPolicy.spec.rules.match; if you choose to ignore these errors, turn validation off with --validate=false

Summary

To summarize, while ChatGPT can write some simple policies that are similar to Kyverno policies in the example repository, it struggles to write even slightly more complex policies. Also, the generated policy heavily depends on the words used in the request. Getting the policy generated correctly requires trial and error. We have seen some users start using ChatGPT to generate Kyverno policies but recommend running validation checks before using them or posting them on community slack for debugging. But as ChatGPT evolves, there is no doubt that it will be possible to write even more complex policies in the future. Also, it could be interesting to understand what would be required to improve the accuracy of ChatGPT. But for now, we still need to do things the old-fashioned way! 

Some tips to write Kyverno policies:

• If using Kyverno CLI, you can use ‘kubectl explain’ command to understand the various fields in the policy
• If using Visual Studio code, you can use the Kubernetes plugin. Since Kyverno schema is based on OpenAPI v3, help is available inside the editor
• You can also use the policy studio built into Nirmata Control Hub. The studio provides AI enabled policy generation and testing.

Here are some useful links to learn more about Kyverno:

• Kyverno website
• Kyverno Github page
• Kyverno Overview videos
• Kyverno Docs
• Kyverno Policies
• Kyverno Certification (Coming Soon)
• Join the Kyverno community Slack at https://slack.k8s.io/and then search for the #kyverno
• Ebook – Policy-based security and governance for Kubernetes

And please contact us for any help with Kyverno or Kubernetes policy management.