Optimizing Kyverno Policy Enforcement with Global Context Entry and Projections

Introduction

In today’s fast-paced cloud-native world, Kubernetes has become the cornerstone of modern application deployment. Google Kubernetes Engine (GKE) Autopilot simplifies cluster management and reduces operational overhead. However, even in a managed environment like Autopilot, maintaining robust security and policy enforcement remains crucial. This blog post guides you through setting up and using Kyverno and Nirmata Control Hub to supercharge GKE Autopilot security.

Understanding the Technologies

GKE Autopilot

• GKE Autopilot is designed to provide a “hands-off” Kubernetes experience. Google manages the control plane, scales nodes automatically, and enforces best practices. This simplifies operations but introduces limitations on direct control.
• Security is a core focus, with built-in Pod Security Standards and enforced configurations.
• Refer to GKE Autopilot security measures

Kyverno

• Kyverno is a policy engine for Kubernetes that operates declaratively using Kubernetes manifests.
• It allows the definition of policies for validating, mutating, generating, and verifying images, all without writing code.
• This simplifies the implementation of complex security and compliance rules.

Nirmata Control Hub

• Nirmata Control Hub provides centralized policy management for Kyverno.
• It simplifies the deployment and management of policies across multiple Kubernetes clusters, including GKE Autopilot.
• It offers a user-friendly interface for creating, deploying, and monitoring Kyverno policies, and provides centralized compliance reporting.

Setting Up Kyverno on GKE Autopilot

Installation

Enterprise Kyverno can be installed using Helm.

helm repo add nirmata https://nirmata.github.io/kyverno-charts/
helm repo update nirmata
helm install kyverno nirmatakyverno/kyverno -n kyverno --create-namespace

Note: GKE Autopilot has limitations on modifying system namespaces. Therefore, installing Kyverno in its own namespace is crucial.

Verification

Verify that Kyverno is running correctly by checking the pod status:

kubectl get pods -n kyverno

Example Policy

Consider this policy that requires all containers in a pod to specify requests and limits:

apiVersion: kyverno.io/v1
    kind: ClusterPolicy
    metadata:
    name: require-requests-limits
    annotations:
    policies.kyverno.io/title: Require Limits and Requests
    spec:
    validationFailureAction: Audit
    background: true
    rules:
    - name: validate-resources
    match:
    any:
    - resources:
    kinds:
    - Pod
    validate:
    message: "CPU and memory resource requests and memory limits are required for containers."
    pattern:
    spec:
    containers:
    - resources:
    requests:
    memory: "?*"
    cpu: "?*"
    limits:
    memory: "?*"
    =(initContainers):
    - resources:
    requests:
    memory: "?*"
    cpu: "?*"
    limits:
    memory: "?*"
    =(ephemeralContainers):
    - resources:
    requests:
    memory: "?*"
    cpu: "?*"
    limits:
    memory: "?*"

Apply this policy using

kubectl apply -f require-requests-limits.yaml

Now consider this deployment without any resource requests or limits.

apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx-deployment
    labels:
    app: nginx
    spec:
    replicas: 3
    selector:
    matchLabels:
    app: nginx
    template:
    metadata:
    labels:
    app: nginx
    spec:
    containers:
    - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80

What happens when you apply this deployment manifest?

Autopilot adds default resource requests if they are missing (more details here). But it is also important to specify the pod CPU and memory limits. This can be enforced by the Kyverno policy. Even though Autopilot adds the default resource request, our deployment is still missing the resource limit. If the Kyverno policy is in Enforce mode, then it will block the request. If it is in Audit mode, then a policy report will be generated for the resource.

As stated in the GKE Autopilot documentation, it is recommended to add resource requests based on your application needs. While adding some default is still better than not specifying anything at all, it may not be sufficient. Kyverno policies can also be used to mutate resource requests based on the application needs.

Integrating GKE Autopilot with Nirmata Control Hub

Registering the Cluster

helm install enterprise-kyverno-operator nirmata/enterprise-kyverno-operator -n nirmata --create-namespace
helm install nirmata-kube-controller nirmata/nirmata-kube-controller -n nirmata --create-namespace  --set cluster.name=my-gke-autopilot --set namespace=nirmata --set readWriteMode=true --set apiToken=<api-token>

View Policy & Compliance Reports

Nirmata Control Hub provides centralized compliance reports, allowing monitoring of policy effectiveness across clusters.

Real-World Use Cases and Benefits

• Enforcing Security Best Practices: Prevent privileged containers, enforce resource limits, and restrict network access.
• Ensuring Compliance: Meet industry standards like PCI DSS or HIPAA by implementing specific policies.
• Automating Configuration Management: Automatically add labels or annotations to resources based on defined rules.
• Centralized Auditing: Gain visibility into policy violations and compliance status across GKE Autopilot clusters.
• Control plane Stability: Be aware of Google’s recommendations for webhooks – Ensure control plane stability when using webhooks

FAQ: Kyverno in GKE Autopilot

Are admission controllers allowed in GKE Autopilot?

Yes, admission controllers that utilize validating and mutating webhooks are permitted in GKE Autopilot. However, Autopilot imposes restrictions to maintain control plane stability and security. The default Enterprise Kyverno settings adhere to these requirements, therefore can be run within GKE Autopilot environments.

What are the restrictions on admission controllers in Autopilot?

Autopilot modifies validating and mutating webhook objects to exclude system namespaces and critical system resources. This prevents user-defined admission controllers from interfering with core cluster operations. In Enterprise Kyverno, system-namespaces are excluded by default. In addition to that, you can customize the excluded namespaces at the webhook level.

Can I run Kyverno in GKE Autopilot?

Yes, Kyverno can run on GKE Autopilot. However, ensure that your Kyverno policies adhere to Autopilot’s restrictions. Policies that attempt to modify system namespaces or critical resources will be blocked.

How can I ensure that my Kyverno installation is compatible with GKE Autopilot?

If you are an Enterprise Kyverno customer, then you can straight away run your installation on GKE Autopilot. If not, then test your Kyverno installation in a GKE Autopilot cluster. Review your Kyverno configuration and ensure it does not attempt to modify system namespaces or critical resources. Consult the GKE Autopilot documentation for detailed information on webhook restrictions. Reach out to us if you need expert advice on Kyverno.

Conclusion

By combining Kyverno and Nirmata Control Hub, robust policy enforcement can be achieved on GKE Autopilot without sacrificing its simplicity. This combination allows for improved security, compliance, and operational efficiency.

Ready to experience effortless policy management on GKE Autopilot?

• Try Nirmata Control Hub today: Sign up for a 15-day free trial. No credit card required!
• Have questions or need assistance? Reach out to our team for a personalized demo and to discuss how we can help you secure your Kubernetes environment.