Red Hat® OpenShift is a widely adopted containerization platform powered by Kubernetes. As the enterprise adoption of OpenShift grows, operators are often faced with the need to automatically update or generate configuration as well as ensure security and enforce best practices. Essentially they are looking to provide guardrails so that developers can continue to use OpenShift without impacting other applications or introducing security vulnerabilities via misconfigurations. Kyverno, a Kubernetes-native policy engine, is perfect for this task and is often being used to address the above-mentioned challenges. In this post, I will discuss how you can get started with Kyverno on the OpenShift Container Platform.

Red Hat OpenShift

Red Hat® OpenShift® Container Platform is the industry-leading hybrid cloud platform powered by containerization and Kubernetes. Using the OpenShift Container Platform simplifies and accelerates the development, delivery, and lifecycle management of a hybrid mix of applications – consistently anywhere across on-premises, public clouds, and edge. OpenShift Container Platform is designed to deliver continuous innovation and speed at any scale, helping organizations to be ready for today and build for the future.

Kyverno

Kyverno is the ideal DevOps solution to enable automation, governance and security for any Kubernetes-based platform including OpenShift Container Platform. Kyverno runs as a dynamic admission controller in the cluster. It receives validating and mutating admission webhook HTTP callbacks from the kube-apiserver and applies matching policies to return results that enforce admission policies or reject requests. Kyverno policies are written in Kubernetes-native YAML, significantly reducing the learning curve required to write custom policies. Kyverno policies can match resources using the resource kind, name, and label selectors to trigger actions such as validate, mutate, generate and image verification for container signing and software supply chain attestations.

Getting Started

In order to get started, you will need the following:

• OpenShift Container Platform 4.8 or higher installed
• Helm version 3.2 or greater installed and configured to access your OpenShift cluster
• Kubectl installed and configured to access your OpenShift cluster

Once you have all the components, you can get started with the following steps:

• Installing Kyverno
• Installing Kyverno policies
• Viewing Policy Violation Report

 

Installing Kyverno

You will need cluster-admin permissions to install Kyverno. The latest instructions to install Kyverno can be found here.

First, add the Kyverno helm repository and update it.

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

Next, install Kyverno to your OpenShift cluster. Note that the namespace ‘kyverno’ will automatically be created.

helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

Once the helm chart is installed, check if the Kyverno pod is running.

kubectl get pods -n kyverno

Note: Depending on the size of your OpenShift cluster, i.e. the number of resources in your cluster it may be necessary to increase the memory and cpu limits for the Kyverno deployed. You should also increase the number of replicas to 2 so that Kyverno is deployed in high availability mode.

On OpenShift clusters, if you want to prevent the scanning and validation of the resources in the system namespaces (the ones starting with openshift), you can update the kyverno config map to include the following entry:

webhooks: '[{"namespaceSelector":{"matchExpressions":[{"key":"openshift.io/run-level","operator":"NotIn",
    "values": ["0","1"]}]}}]'

Once Kyverno pod is running, it will automatically create the necessary admission webhooks. You can also check the CRDs that are installed for Kyverno using this command:

kubectl get crds |grep kyverno

 

Installing Kyverno Policies

Now that Kyverno is installed, you can install the policies. When installing policies for the first time, it is recommended that the policies are configured to run in ‘audit’ mode so that none of the include requests being made to your OpenShift cluster are blocked. You can check if a policy is configured as ‘audit’ by checking the validationFailureAction property in the policy manifest. 

Install sample policies using the command: 

helm install kyverno-policies kyverno/kyverno-policies --namespace kyverno

Next, you can check if the policies are installed using the command:

kubectl get clusterpolicies

The output will look like this:

NAME                             BACKGROUND   ACTION   READY
deny-privilege-escalation        true         audit    true
disallow-add-capabilities        true         audit    true
disallow-host-namespaces         true         audit    true
disallow-host-path               true         audit    true
disallow-host-ports              true         audit    true
disallow-privileged-containers   true         audit    true
disallow-selinux                 true         audit    true
require-default-proc-mount       true         audit    true
require-non-root-groups          true         audit    true
require-run-as-non-root          true         audit    true
restrict-apparmor-profiles       true         audit    true
restrict-seccomp                 true         audit    true
restrict-sysctls                 true         audit    true
restrict-volume-types            true         audit    true

Note that the policy state READY indicates that the policy is ready to process any incoming requests or perform background tasks.

 

Viewing Policy Violation Report

Once the policies are installed and ready, they should start generating policy violations. Policy violations can be viewed by fetching the policy reports. To fetch the policy reports for all namespaces, use the command:

kubectl get policyreports -A

To fetch the policy violations at the cluster scope, use the command:

kubectl get clusterpolicyreports

You can also view detailed policy results using the ‘kubectl describe’ command.

Issues and Troubleshooting

 

Kyverno pod constantly crashes

Check if the crash is caused due to the pod not getting enough memory. Increase the memory limit.

 

Policies are not applied

Check if the validating and mutating webooks are created correctly.

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

You should see:

NAME                                                                                                  WEBHOOKS   AGE
 validatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-policy-validating-webhook-cfg     1          46m
 validatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-resource-validating-webhook-cfg   1          46m
validatingwebhookconfiguration.admissionregistration.k8s.io/autoscaling.openshift.io   2          17d
validatingwebhookconfiguration.admissionregistration.k8s.io/multus.openshift.io        1          17d


 NAME                                                                                              WEBHOOKS   AGE
 mutatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-policy-mutating-webhook-cfg     1          46m mutatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-resource-mutating-webhook-cfg   1          46m
mutatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-verify-mutating-webhook-cfg     1          46m

Also, check if the Kyverno service is configured correctly.

kubectl get services -n kyverno

You should see:

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kyverno-svc           ClusterIP   172.30.97.254    <none>        443/TCP    13d
kyverno-svc-metrics   ClusterIP   172.30.110.252   <none>        8000/TCP   13d

For other troubleshooting, refer to the Kyverno documentation.

Summary

As you can see, it is extremely easy to get started using Kyverno on your OpenShift cluster. Once Kyverno is installed and policies are being applied, you can learn how to write new policies for your deployment. OpenShift installation includes several custom resource definitions and so in case you need to validate any custom resources, a Kyverno policy can be written. You can also find several policies contributed by the Kyverno community and apply them to your clusters.  

If you have several Kyverno OpenShift clusters or if you are using Kyverno OpenShift clusters along with EKS, GKE, AKS or any other managed cluster service, you can manage Kyverno policies centrally using the Nirmata Control Hub. Nirmata Control Hub allows you to deploy policies across your clusters using GitOps and also lets you visually remediate any policy violations from a central console.

Get more information on Openshift and Kyverno here. Please contact Nirmata if you have further questions on Kubernetes management with Kyverno!

---