Generating Kubernetes ValidatingAdmissionPolicies from Kyverno Policies

November 22, 2023 In Engineering, Kyverno By Mariam Fahmy

Generating Kubernetes ValidatingAdmissionPolicies from Kyverno Policies

In the previous blog post, we discussed writing Common Expression Language (CEL) expressions in Kyverno policies for resource validation. CEL was first introduced to Kubernetes for the Validation rules for CustomResourceDefinitions, and then it was used by Kubernetes ValidatingAdmissionPolicies in 1.26.

ValidatingAdmissionPolicies offer a declarative, in-process alternative to validating admission webhooks. They use the Common Expression Language (CEL) to declare the validation rules of a policy. They are highly configurable, enabling policy authors to define policies that can be parameterized and scoped to resources as needed by cluster administrators.

This post will show you how to generate Kubernetes ValidatingAdmissionPolicies and their bindings from Kyverno policies.

Prior posts in this series:

Prerequisite

Generating Kubernetes ValidatingAdmissionPolicies require the following:

  1. A cluster with Kubernetes 1.26 or higher.
  2. Enable the ValidatingAdmissionPolicy feature gate.
  3. Enable the admissionregistration.k8s.io/v1beta1 API for v1.28.
    OR
    Enable the admissionregistration.k8s.io/v1alpha1 API for v1.26 and v1.27.
  4. Set the –generateValidatingAdmissionPolicy flag in the Kyverno admission controller.
  5. Grant the admission controller service account the required permissions to generate ValidatingAdmissionPolicies and their bindings.

In this post, we will use the beta version of Kubernetes 1.28.

Installation & Setup

  • Create a local cluster:
kind create cluster --image "kindest/node:v1.28.0" --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  ValidatingAdmissionPolicy: true
runtimeConfig:
  admissionregistration.k8s.io/v1beta1: true
  admissionregistration.k8s.io/v1alpha1: true
nodes:
  - role: control-plane
  - role: worker
EOF
  • Add the Kyverno Helm repository:
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
  • Create a new file that overrides the values in the chart.
cat << EOF > new-values.yaml
features:
  generateValidatingAdmissionPolicy:
    enabled: true
admissionController:
  rbac:
    clusterRole:
      extraResources:
      - apiGroups:
          - admissionregistration.k8s.io
        resources:
          - validatingadmissionpolicies
          - validatingadmissionpolicybindings
        verbs:
          - create
          - update
          - delete
          - list
EOF
  • Deploy Kyverno:
helm install kyverno kyverno/kyverno -n kyverno --create-namespace --values new-values.yaml

We are now ready to generate Kubernetes ValidatingAdmissionPolicies from Kyverno policies.

Generating Kubernetes ValidatingAdmissionPolicies

In this section, we will create a Kyverno policy that ensures no hostPath volumes are in use for deployments, and then we will have a look at the generated ValidatingAdmissionPolicy and its binding. Finally, we will create a deployment that violates the policy.

Let’s start with creating the Kyverno policy.

kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-host-path
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: host-path
      match:
        any:
        - resources:
            kinds:
              - Deployment
      validate:
        cel:
          expressions:
            - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
              message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."
EOF

You can check whether a ValidatingAdmissionPolicy is generated or not from the Kyverno policy status.

$ kubectl get cpol disallow-host-path -o jsonpath='{.status}'
{
   "autogen":{},
   "conditions":[
      {
         "lastTransitionTime":"2023-09-12T11:42:13Z",
         "message":"Ready",
         "reason":"Succeeded",
         "status":"True",
         "type":"Ready"
      }
   ],
   "ready":true,
   "rulecount":{
      "generate":0,
      "mutate":0,
      "validate":1,
      "verifyimages":0
   },
   "validatingadmissionpolicy":{
      "generated":true,
      "message":""
   }
}

Let’s try getting the ValidatingAdmissionPolicy and its binding.

$ kubectl get validatingadmissionpolicy
NAME                                 VALIDATIONS   PARAMKIND   AGE
disallow-host-path               1                         <unset>          8m12s

$ kubectl get validatingadmissionpolicybindings
NAME                                                   POLICYNAME                    PARAMREF   AGE
disallow-host-path-binding                   disallow-host-path               <unset>           8m30s

You may notice that the ValidatingAdmissionPolicy and the ValidatingAdmissionPolicyBinding share the same name as the Kyverno policy they originate from, with the binding having a “-binding” suffix.

Let’s have a look at the ValidatingAdmissionPolicy and its binding in detail.

$ kubectl get validatingadmissionpolicy disallow-host-path -o yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
metadata:
  creationTimestamp: "2023-09-12T11:42:13Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: kyverno
  name: disallow-host-path
  ownerReferences:
  - apiVersion: kyverno.io/v1
    kind: ClusterPolicy
    name: disallow-host-path
    uid: e540d96b-c683-4380-a84f-13411384241a
  resourceVersion: "11294"
  uid: 9f3e0161-d010-4a6f-bd28-bf9c87151795
spec:
  failurePolicy: Fail
  matchConstraints:
    matchPolicy: Equivalent
    namespaceSelector: {}
    objectSelector: {}
    resourceRules:
    - apiGroups:
      - apps
      apiVersions:
      - v1
      operations:
      - CREATE
      - UPDATE
      resources:
      - deployments
      scope: '*'
  validations:
  - expression: '!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))'
    message: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset.
  variables: null
status:
  observedGeneration: 1
  typeChecking: {}
$ kubectl get validatingadmissionpolicybindings disallow-host-path-binding -o yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicyBinding
metadata:
  creationTimestamp: "2023-09-12T11:42:13Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: kyverno
  name: disallow-host-path-binding
  ownerReferences:
  - apiVersion: kyverno.io/v1
    kind: ClusterPolicy
    name: disallow-host-path
    uid: e540d96b-c683-4380-a84f-13411384241a
  resourceVersion: "11292"
  uid: 2fec35c3-8a8c-42a7-8a02-a75e8882a01e
spec:
  policyName: disallow-host-path
  validationActions:
  - Deny

Now, let’s try deploying an app that uses a hostPath:

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-server
        image: nginx
        volumeMounts:
          - name: udev
            mountPath: /data
      volumes:
      - name: udev
        hostPath:
          path: /etc/udev
EOF

As expected, the deployment creation is rejected by the API server and not by the Kyverno admission controller.

The deployments "nginx" is invalid:  ValidatingAdmissionPolicy 'disallow-host-path' with binding 'disallow-host-path-binding' denied request: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset.

If either the ValidatingAdmissionPolicy or the binding is deleted/updated for some reason, the controller is responsible for reverting it.

Let’s try deleting the ValidatingAdmissionPolicy.

$ kubectl delete validatingadmissionpolicy disallow-host-path 
validatingadmissionpolicy.admissionregistration.k8s.io "disallow-host-path" deleted

$ kubectl get validatingadmissionpolicy 
NAME                                  VALIDATIONS   PARAMKIND   AGE
disallow-host-path                   1                        <unset>        11s

In addition, you can update the Kyverno policy, and the controller will re-generate the ValidatingAdmissionPolicy accordingly. For example, you can change the Kyverno policy to match statefulsets too.

$ kubectl edit cpol disallow-host-path
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-host-path
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: host-path
      match:
        any:
        - resources:
            kinds:
              - Deployment
              - StatefulSet
      validate:
        cel:
          expressions:
            - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
              message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."

The ValidatingAdmissionPolicy will be updated to match StatefulSets too.

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicy
metadata:
  creationTimestamp: "2023-09-12T12:54:48Z"
  generation: 2
  labels:
    app.kubernetes.io/managed-by: kyverno
  name: disallow-host-path
  ownerReferences:
  - apiVersion: kyverno.io/v1
    kind: ClusterPolicy
    name: disallow-host-path
    uid: e540d96b-c683-4380-a84f-13411384241a
  resourceVersion: "29208"
  uid: 9325e2b7-9131-4ff4-9e56-244129cb625e
spec:
  failurePolicy: Fail
  matchConstraints:
    matchPolicy: Equivalent
    namespaceSelector: {}
    objectSelector: {}
    resourceRules:
    - apiGroups:
      - apps
      apiVersions:
      - v1
      operations:
      - CREATE
      - UPDATE
      resources:
      - deployments
      - statefulsets
      scope: '*'
  validations:
  - expression: '!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))'
    message: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset.
  variables: null
status:
  observedGeneration: 2
  typeChecking: {}

Conclusion

This blog discussed how to generate Kubernetes ValidatingAdmissionPolicies from Kyverno policies. You can use CEL expressions in Kyverno policies to validate resources through either the Kyverno engine or the API server. In the next blog, we will discuss how to generate BackgroundScanReports for ValidatingAdmissionPolicies.

Learn more about Kyverno at: https://kyverno.io/.

For help with Kyverno or getting started with Cloud Native Policy and Governance, schedule a meeting at: https://nirmata.com/contact-us/.

Why & How Wayfair Migrated from OPA to Kyverno
Verifying images and attestations using AWS Signer, Notation and Kyverno.
Tags:
No Comments

Sorry, the comment form is closed at this time.