Introduction

In the world of cloud computing, Kubernetes has become the de facto standard for container orchestration. However, as organizations scale their Kubernetes deployments, they often face challenges in managing and optimizing costs. Cost allocation, the process of attributing costs to specific resources or users, is a crucial aspect of cost management in Kubernetes. 

In this blog post, we will explore how OpenCost and Kyverno can be used together to effectively allocate costs in Kubernetes, enabling organizations to gain better visibility and control over their cloud expenses.

Cost Allocation in Kubernetes

In Kubernetes, cost allocation involves associating resource usage with the responsible entity, whether it’s an individual or a team or an application (namespace). This helps organizations track and analyze the costs associated with different deployments, namespaces, or even specific pods. By implementing effective cost allocation strategies, organizations can identify cost optimization opportunities and make informed decisions to optimize their cloud spend.

OpenCost

OpenCost is an open-source project designed to address the challenges of cost allocation in Kubernetes. It provides a framework for tracking and allocating costs to various resources within a Kubernetes cluster. 

Key Features

Resource Mapping: OpenCost allows you to define resource mappings, enabling you to associate costs with specific Kubernetes resources such as namespaces, deployments, pods, or even individual containers.

Cost Metrics: It provides a mechanism to define and collect cost metrics, enabling organizations to measure the cost associated with each resource or entity accurately.

Reporting and Visualization: OpenCost offers reporting and visualization capabilities, allowing organizations to generate cost reports and visualize cost allocation patterns across their Kubernetes cluster.

Kyverno

Kyverno is a Kubernetes-native policy engine that provides a declarative approach to defining and enforcing policies within a cluster. Kyverno helps enforce policies related to security, compliance, automation, and best practices. 

Kyverno policies are managed as Kubernetes resources and no new language is required to write policies. This allows using familiar tools such as kubectl, git, and kustomize to deploy and manage policies. 

You can learn more about Kyverno features here.

Using Kyverno with OpenCost

By combining Kyverno with OpenCost, you can implement policy-based cost allocation, enabling you to enforce specific cost allocation rules across your Kubernetes resources.

Defining Cost Allocation Policies: With Kyverno, you can define cost allocation policies as Kubernetes resources. These policies can specify rules to allocate costs based on various factors such as labels, annotations, or specific resource usage metrics.

Dynamic Cost Allocation: Kyverno’s flexibility allows you to dynamically allocate costs based on the changing characteristics of your Kubernetes resources. For example, you can create policies to allocate costs based on CPU or memory usage thresholds, ensuring that costs are distributed based on resource consumption.

Enforcement and Auditing: Kyverno enforces cost allocation policies in real-time, ensuring that costs are allocated according to the defined rules. It also provides auditing capabilities, allowing you to track and monitor policy enforcement across your Kubernetes cluster.

Cost Tracking Example

Let’s take a look at an example of how Kyverno policies can be used to detect if any namespace is exceeding its cost allocation. In this example, we will use Kyverno 1.10 since includes the ability to call ANY Kubernetes Service as part of processing the policy

Steps

1. 1. Install opencost on your cluster by following its instructions. Note that for this example, opencost is installed in the opencost namespace.
   2. Install Kyverno 1.10 on your cluster by following the instructions here
   3. Verify that OpenCost is installed correctly by launching the OpenCost UI and verifying that cost information is being reported
   4. Next, create a configmap in a namespace (e.g. nirmata) and provide the cost allocation information for your namespaces. In the example below, we have allocated cost of $1 per day for namespace ‘argocd’namespace-quota-cm.yaml

      apiVersion: v1
      data:
        "argocd" : "1"
      kind: ConfigMap
      metadata:
        name: namespace-quota-cm
        namespace: nirmata
      

      > kubectl create -f namespace-quota-cm.yaml -n nirmata
      

   5. Now, let’s write a policy to check the daily cost of all the namespaces in the cluster and compare it with the allocated cost specified in the configmap. If the namespace it not added in the namespace-quota-cm configmap, it will be marked as ‘passed’check-namespace-costs-cm.yaml

      
      apiVersion: kyverno.io/v1
      kind: ClusterPolicy
      metadata:
        creationTimestamp: "2023-03-07T00:24:33Z"
        generation: 1
        name: check-namespace-costs-cm
        resourceVersion: "1741990"
        uid: bdbbf92e-6948-4553-a5af-c74f97436b77
        annotations:
          policies.kyverno.io/title: Namespace Cost Exceeded
          policies.kyverno.io/minversion: 1.10.0
          policies.kyverno.io/category: Cost Management
          policies.kyverno.io/severity: medium
          policies.kyverno.io/subject: Namespace
          policies.kyverno.io/description: >-
            This policy checks for namespace costs and generates a violation if the cost exceeds the allocated cost specified in the namespace-quota-cm config map.
      spec:
        background: true
        rules:
        - context:
          - apiCall:
              method: GET
              service:
                url: http://opencost.opencost:9090/model/allocation/compute?window=1d&aggregate=namespace&step=1d&accumulate=false
            name: result
          - name: totalCost
            variable:
              value: '{{ result.data[0].["{{ request.object.metadata.name }}"][0].totalCost
                || ''0'' }}'
          - name: costConfigMap
            configMap:
              name: namespace-quota-cm
              namespace: nirmata
          - name: allocatedCost
            variable:
              value: '{{ costConfigMap.data."{{ request.object.metadata.name }}" || ''0'' }}'
          match:
            any:
            - resources:
                kinds:
                - Namespace
          name: check-namespace-cost
          validate:
            deny:
              conditions:
                all:
                - key: '{{ allocatedCost }}'
                  operator: GreaterThan
                  value: '0'
                - key: '{{ totalCost }}'
                  operator: GreaterThan
                  value: '{{ allocatedCost }}'
            message: namespace {{request.object.metadata.name}} cost {{ totalCost }} exceeds
              maximum cost threshold {{ allocatedCost }}
        validationFailureAction: Audit

      Now, let’s take a look at the various sections of the policy:

      1. 1. 1. 1. • This policy will run whenever a namespace is created and periodically in background
                  • The apiCall is configured to get daily cost for the namespace using http GET and store the response JSON in ‘result’
                  • Next, the totalCost is extracted from the result data
                  • The costConfigMap is loaded and the allocatedCost for the namespace is retrieved.
                  • In the validation rule, there are two checks:
                    • First we check if allocatedCost was provided
                    • Next we check if the totalCost exceeds the allocatedCost
                  • If both the conditions are true, the policy check is marked as failed.

       

      Here are the summary and detailed outputs when the policy is applied to my cluster.

      > kubectl get cpolr cpol-check-namespace-costs-cm
      NAME                            PASS   FAIL   WARN   ERROR   SKIP   AGE
      cpol-check-namespace-costs-cm   20     1      0      0       0      78m
      
      > kubectl get cpolr cpol-check-namespace-costs-cm -o yaml
      …
      - category: Cost Management
        message: namespace argocd cost 1.37498 exceeds maximum cost threshold 1
        policy: check-namespace-costs-cm
        resources:
        - apiVersion: v1
          kind: Namespace
          name: argocd
          uid: 44b0b760-34f1-45f4-adfe-2522102e53d0
        result: fail
        rule: check-namespace-cost
        scored: true
        severity: medium
        source: kyverno
        timestamp:
          nanos: 0
          seconds: 1684622236
      …
      

      The policy violation clearly states that “namespace argocd cost 1.37498 exceeds maximum cost threshold 1” providing information on which namespaces exceed the allocated cost threshold. This policy violation can be further used to trigger additional automation such as preventing additional resources from being provisioned in the namespace, notifying namespace owners and even automatically scaling down resources to reduce costs.

      Benefits of OpenCost and Kyverno Integration

      As seen in the above example, OpenCost and Kyverno can be used for cost management and optimization. Here are the benefits of the integration:

      Granular Cost Allocation: The integration of OpenCost and Kyverno enables organizations to allocate costs at a granular level, from namespaces down to individual containers. This level of detail provides better visibility into resource utilization and cost attribution.

      Policy-Based Automation: By leveraging Kyverno’s policy engine, organizations can automate the enforcement of cost allocation rules. This eliminates the need for manual intervention, reducing the risk of human error and ensuring consistent cost allocation across the cluster.

      Identifying Cost Optimization Opportunities: The combined power of OpenCost and Kyverno allows organizations to identify cost optimization opportunities. By analyzing cost allocation patterns and resource usage metrics, organizations can make data-driven decisions to optimize their Kubernetes deployments and reduce unnecessary expenses.

      Conclusion

      Effective cost allocation is essential for managing and optimizing cloud costs in Kubernetes deployments. OpenCost and Kyverno provide a powerful combination to tackle cost allocation challenges by offering granular cost attribution, policy-based automation, and cost optimization opportunities. By implementing these tools and practices, organizations can gain better visibility and control over their Kubernetes costs, enabling them to optimize resource utilization and drive cost savings effectively.

      Explore a complete Kubernetes policy and governance solution at: https://try.nirmata.io. If you have questions on Kyverno, you can also contact Nirmata for further discussion.