Recently, CNCF’s Cloud Native Live hosted an insightful session titled “Optimizing Cost, Performance, And Security In K8s With Policy-As-Code” featuring Anusha Hegde, Senior Product Manager, and Sachin Agarwal, Senior Solutions Architect, both from Nirmata. The session dove deep into the challenges of managing resources in Kubernetes and showcased how open-source tools like KYVERNO and Vertical Pod Autoscaler (VPA), combined with a policy-as-code approach, can provide significant improvements. Matt, CIO, a CNCF Ambassador from the American Red Cross, hosted the session and highlighted his personal interest in compliance automation.

The Core Challenges of Resource Optimization in Kubernetes

Anusha kicked off the presentation by outlining the inherent complexities and dynamic nature of Kubernetes environments. She emphasized several key challenges:

• Dynamic Workloads: Modern containerized applications have varying and unpredictable resource needs, making static allocation ineffective. What’s optimal today might not be tomorrow.
• Misconfiguration Risks: Kubernetes’s flexibility comes with the risk of misconfigurations, which can lead to security issues and inefficient resource utilization through over- or under-provisioning. Teams might overprovision “just to be sure,” leading to wasted resources.
• Conflicting Needs: Different teams involved in application deployment (e.g., application, platform, cloud ops, security) often have conflicting priorities. Application teams prioritize agility and performance, potentially leading to generous resource allocation, which can clash with platform teams’ goals of stability and fair resource allocation and cloud ops’ focus on cost reduction.
• Impact of Dynamic Workloads: Traditional static resource allocation struggles to cope with the scalability and event-driven nature of cloud-native applications, potentially leading to performance issues during peak loads or resource wastage during off-peak times.

Anusha argued that the solution lies in automation and adaptability, requiring systems that can intelligently adjust resource allocation based on changing workloads.

Introducing KYVERNO and Vertical Pod Autoscaler (VPA)

• KYVERNO: Described as the “most loved CNCF policy engine,” KYVERNO operates as a Kubernetes admission controller (both mutating and validating). It uses custom Kubernetes resources (CRDs) and controllers, allowing users to define policies in plain YAML. Key use cases include pod security, workload security, resource optimization, multi-tenancy, micro-segmentation, and supply chain security (e.g., verifying container image signatures). Notably, KYVERNO can automate tasks based on triggers, such as creating resource quotas and network policies upon namespace creation.
• Vertical Pod Autoscaler (VPA): A SIG Autoscaling project, VPA focuses on the vertical scaling of pods by intelligently recommending and optionally automatically adjusting CPU and memory requests and limits. It prevents over and under-provisioning by analyzing historical workload behavior. VPA consists of three main components:
• Recommender: Analyzes historical data (from the metrics server) to suggest optimal CPU and memory values.
• Updater: This can automatically update the pod specifications based on the recommendations (though this can also be managed via KYVERNO).
• Admission Plugin: A mutating admission plugin that can modify incoming pod requests based on VPA recommendations.

The Power of Synergy: KYVERNO and VPA in Action

Anusha explained how KYVERNO and VPA can work together to create a comprehensive resource optimization solution.

• Automated VPA Creation: KYVERNO’s generated policies can automatically create a corresponding VPA resource for every new pod controller (such as Deployments, StatefulSets, and DaemonSets).
• Policy-Driven Validation: KYVERNO’s validating policies can then monitor the recommendations provided by VPA. For example, a policy can flag deployments where the actual resource allocation deviates significantly (e.g., more than a certain percentage) from VPA’s recommendations, indicating potential over- or under-provisioning. These violations are reported through KYVERNO’s policy reports, accessible via kubectl get policyreport.
• Automated Updates (Alternative to VPA Updater): While VPA has an updater, KYVERNO’s mutating capabilities can also automatically apply VPA recommendations to pod specifications.

Live Demo and Real-world Benefits

Sachin presented a live demonstration on an AWS EKS cluster. He showcased:

• Existing Policies: Two KYVERNO policies were highlighted: one to automatically generate VPA resources and another to check for resource over or under-provisioning based on VPA recommendations.
• VPA in Action: A running demo application (“payment” in the “banking-app” namespace) with an associated VPA resource showing optimal resource recommendations.
• Policy Violation: By manually increasing the deployment’s memory limit, Sachin demonstrated how KYVERNO immediately detected the overprovisioning and generated a policy violation report, highlighting the discrepancy and the VPA-recommended value.
• Integration with OpenCost: The demo also showed integration with OpenCost, providing potential cost savings associated with right-sizing the resources based on VPA recommendations.
• Load Generation: Sachin also initiated a load generation script to show how VPA recommendations would dynamically adjust based on increased CPU consumption.
• Dashboard Visualization: Nirmata’s dashboard was briefly shown, illustrating how these policy reports can be visualized and used to track resource optimization efforts.

Sachin emphasized the benefits of this solution:

• Continuously right-sizing applications based on VPA recommendations.
• Reducing overall inefficiency in resource allocation, leading to cost benefits.
• Improved budget tracking by integrating with cost monitoring tools.
• Combining VPA recommendations with KYVERNO’s mutating policies or tools like Carpenter has the potential to fully automate resource optimization.

Q&A Highlights

The session concluded with a Q&A session, addressing several viewer questions:

• In-place Resource Updates: Kubernetes version 1.27 introduced in-place resource updates, allowing CPU and memory adjustments without requiring a pod restart (though considerations for Java applications with Heap memory were mentioned).
• VPA Uncapped Target: The uncapped target in VPA represents a raw recommendation without upper bound constraints, useful when initial resource requirements are unknown.
• Many vs. Few Policies: The best approach depends on code organization and use cases. It’s generally better to group rules within a policy based on a specific resource type and use case (e.g., security policies for deployments vs. resource optimization policies for deployments). Cluster policies with namespace selectors are preferable to writing the same policy multiple times for different namespaces.
• EKS Compatibility: The demonstrated solution works on Amazon EKS.
• Managing ImagePullBackOff and CrashLoopBackOff: KYVERNO policies can be implemented to check pod status and potentially descale or scale down pods in such error states to prevent unnecessary node provisioning.
• Handling Stale Replicas: KYVERNO’s “cleanup rules” (cluster cleanup policies) can be used to identify and delete stale replica sets or pods (e.g., those without a parent deployment reference), which can hinder node de-provisioning.
• Implementing RBAC in Multi-Cloud: KYVERNO can enforce RBAC best practices through policies, and in multi-tenancy environments, it can automate the creation and auditing of roles and cluster roles.

Conclusion

The CNCF Live session provided a compelling overview of how policy as code, leveraging KYVERNO and VPA, can significantly improve resource optimization, security, and cost efficiency in Kubernetes environments. The demonstration highlighted the practical application of these tools, offering valuable insights for platform teams looking to automate and intelligently manage their Kubernetes resources. As Anusha aptly put it,” policy as code is a powerful approach for various Kubernetes challenges, and tools like KYVERNO are invaluable assets in this journey. The policies and demo setup are even open-sourced, encouraging the community to explore and adopt these practices.”

If you have questions on this material, please reach us here to start a conversation.

You can request a complimentary demo to see for yourself how Nirmata can help your DevOps to become more efficient and secure.