Red Hat® OpenShift is a widely adopted containerization platform powered by Kubernetes. As the enterprise adoption of OpenShift grows, operators are often faced with the need to automatically update or generate configuration as well as ensure security and enforce best practices. Essentially they are looking to provide guardrails so that developers can continue to use OpenShift without impacting other applications or introducing security vulnerabilities via misconfigurations. Kyverno, a Kubernetes-native policy engine, is perfect for this task and is often being used to address the above-mentioned challenges. In this post, I will discuss how you can get started with Kyverno on the OpenShift Container Platform.
Red Hat OpenShift
Red Hat® OpenShift® Container Platform is the industry-leading hybrid cloud platform powered by containerization and Kubernetes. Using the OpenShift Container Platform simplifies and accelerates the development, delivery, and lifecycle management of a hybrid mix of applications – consistently anywhere across on-premises, public clouds, and edge. OpenShift Container Platform is designed to deliver continuous innovation and speed at any scale, helping organizations to be ready for today and build for the future.
Kyverno is the ideal DevOps solution to enable automation, governance and security for any Kubernetes-based platform including OpenShift Container Platform. Kyverno runs as a dynamic admission controller in the cluster. It receives validating and mutating admission webhook HTTP callbacks from the kube-apiserver and applies matching policies to return results that enforce admission policies or reject requests. Kyverno policies are written in Kubernetes-native YAML, significantly reducing the learning curve required to write custom policies. Kyverno policies can match resources using the resource kind, name, and label selectors to trigger actions such as validate, mutate, generate and image verification for container signing and software supply chain attestations.
In order to get started, you will need the following:
- OpenShift Container Platform 4.8 or higher installed
- Helm version 3.2 or greater installed and configured to access your OpenShift cluster
- Kubectl installed and configured to access your OpenShift cluster
Once you have all the components, you can get started with the following steps:
- Installing Kyverno
- Installing Kyverno policies
- Viewing Policy Violation Report
You will need cluster-admin permissions to install Kyverno. The latest instructions to install Kyverno can be found here.
First, add the Kyverno helm repository and update it.
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
Next, install Kyverno to your OpenShift cluster. Note that the namespace ‘kyverno’ will automatically be created.
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace
Once the helm chart is installed, check if the Kyverno pod is running.
kubectl get pods -n kyverno
Note: Depending on the size of your OpenShift cluster, i.e. the number of resources in your cluster it may be necessary to increase the memory and cpu limits for the Kyverno deployed. You should also increase the number of replicas to 2 so that Kyverno is deployed in high availability mode.
On OpenShift clusters, if you want to prevent the scanning and validation of the resources in the system namespaces (the ones starting with openshift), you can update the kyverno config map to include the following entry:
Once Kyverno pod is running, it will automatically create the necessary admission webhooks. You can also check the CRDs that are installed for Kyverno using this command:
kubectl get crds |grep kyverno
Installing Kyverno Policies
Now that Kyverno is installed, you can install the policies. When installing policies for the first time, it is recommended that the policies are configured to run in ‘audit’ mode so that none of the include requests being made to your OpenShift cluster are blocked. You can check if a policy is configured as ‘audit’ by checking the validationFailureAction property in the policy manifest.
Install sample policies using the command:
helm install kyverno-policies kyverno/kyverno-policies --namespace kyverno
Next, you can check if the policies are installed using the command:
kubectl get clusterpolicies
The output will look like this:
NAME BACKGROUND ACTION READY
deny-privilege-escalation true audit true
disallow-add-capabilities true audit true
disallow-host-namespaces true audit true
disallow-host-path true audit true
disallow-host-ports true audit true
disallow-privileged-containers true audit true
disallow-selinux true audit true
require-default-proc-mount true audit true
require-non-root-groups true audit true
require-run-as-non-root true audit true
restrict-apparmor-profiles true audit true
restrict-seccomp true audit true
restrict-sysctls true audit true
restrict-volume-types true audit true
Note that the policy state READY indicates that the policy is ready to process any incoming requests or perform background tasks.
Viewing Policy Violation Report
Once the policies are installed and ready, they should start generating policy violations. Policy violations can be viewed by fetching the policy reports. To fetch the policy reports for all namespaces, use the command:
kubectl get policyreports -A
To fetch the policy violations at the cluster scope, use the command:
kubectl get clusterpolicyreports
You can also view detailed policy results using the ‘kubectl describe’ command.
Issues and Troubleshooting
Kyverno pod constantly crashes
Check if the crash is caused due to the pod not getting enough memory. Increase the memory limit.
Policies are not applied
Check if the validating and mutating webooks are created correctly.
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
You should see:
NAME WEBHOOKS AGE
validatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-policy-validating-webhook-cfg 1 46m
validatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-resource-validating-webhook-cfg 1 46m
validatingwebhookconfiguration.admissionregistration.k8s.io/autoscaling.openshift.io 2 17d
validatingwebhookconfiguration.admissionregistration.k8s.io/multus.openshift.io 1 17d
NAME WEBHOOKS AGE
mutatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-policy-mutating-webhook-cfg 1 46m mutatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-resource-mutating-webhook-cfg 1 46m
mutatingwebhookconfiguration.admissionregistration.k8s.io/kyverno-verify-mutating-webhook-cfg 1 46m
Also, check if the Kyverno service is configured correctly.
kubectl get services -n kyverno
You should see:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kyverno-svc ClusterIP 172.30.97.254 <none> 443/TCP 13d
kyverno-svc-metrics ClusterIP 172.30.110.252 <none> 8000/TCP 13d
For other troubleshooting, refer to the Kyverno documentation.
As you can see, it is extremely easy to get started using Kyverno on your OpenShift cluster. Once Kyverno is installed and policies are being applied, you can learn how to write new policies for your deployment. OpenShift installation includes several custom resource definitions and so in case you need to validate any custom resources, a Kyverno policy can be written. You can also find several policies contributed by the Kyverno community and apply them to your clusters.
If you have several Kyverno OpenShift clusters or if you are using Kyverno OpenShift clusters along with EKS, GKE, AKS or any other managed cluster service, you can manage Kyverno policies centrally using the Nirmata Policy Manager. Nirmata Policy Manager allows you to deploy policies across your clusters using GitOps and also lets you visually remediate any policy violations from a central console.
Get more information on Openshift and Kyverno here. Please contact Nirmata if you have further questions on Kubernetes management with Kyverno!