Do policies that mutate or generate resources violate GitOps principles?

In this blog post, I will show you how policy-based resource management can be complementary to GitOps, what benefits it provides, and how to use Kyverno to mutate and generate rules with popular GitOps tools like Flux and ArgoCD.

Kubernetes Control Loops

The Controller pattern is a foundational concept in Kubernetes. Controllers are driven by declarative configurations that define the system’s desired state and run a control loop reconciling the current state with the desired state.

Kubernetes has several built-in controllers, like pod controllers. Kubernetes is extensible and supports custom controllers that can run inside of Kubernetes.

The GitOps Control Loop 

The GitOps pattern is another form of the controller pattern, where the desired state is versioned in Git, and the GitOps controller reconciles resources in clusters with resources declared in Git. 

The size of the control loops matters, and the tighter and more self-contained the loop, the better.  Tighter control loops are less error prone and more efficient, with fewer moving pieces. Tighter control loops can also offer greater security, as the attack surface is reduced when there are fewer dependencies on external systems.

The Policy Control Loop

Kubernetes offers many forms of policies, including API resources. Policy management solutions, like Kyverno and OPA/Gatekeeper, use dynamic admission controls to intercept API requests.

A Kubernetes policy controller is also a form of a control loop where the desired state is defined as a declarative policy. A policy controller applies policies to resource configurations in near real-time. Policies can be used to validate and block or flag insecure and misconfigured resources. Policies can also mutate, generate, and delete (cleanup) resources based on customizable criteria. 

Maintaining State in Git

“I want all cluster states to be stored in git” is often stated as a reason for not using mutation or generation policies. 

The goal of GitOps is to make the cluster state fully reproducible from Git and to store just enough to recover and track the state entirely.

For example, it does not make sense to store each Pod declaration in git as Kubernetes pod controllers like Deployments are used to manage the lifecycle of pods in a cluster. Hence, only pod controller declarations are stored in Git.

Similarly, policies can also be stored in Git and applied to clusters. A controller then watches the policy states, dynamically enforces required policies, and manages configurations.

The Advantages of Using Policies

Managing configurations using policies can bring numerous advantages. For instance, policies can simplify the configuration management process, enforce security measures, and promote best practices. In addition to these benefits, there are other advantages to using policies for configuration management.:

1. Just in Time Resource Management: policies can apply missing defaults and generate complete resources for security, isolation, multi-tenancy, or other concerns. Since this happens directly inside the cluster, it works with kubectl or any other Kubernetes client tool.

2. Tighter Security Controls: policy controllers, like Kyverno, register as dynamic admission controllers and monitor each API request. Since attackers will directly exploit security weaknesses within a cluster and bypass the CI/CD pipelines, admission controllers act as an essential line of defense. 

3. Separation of Concerns: policies allow cleanly separating concerns across development, operations, and security roles. For example, some policies may be mandated by the security team, whereas the operations team may manage other policies. A policy controller consolidates and applies policies across all. 

Avoiding Sync Errors with Server Side Apply (SSA)

Kubernetes Server Side Apply (SSA) is a feature that allows multiple controllers to collaborate on changes to a resource. 

Both FluxCD and ArgoCD can leverage SSA to coordinate changes with policy engines and other mutating admission controllers.

Kyverno and Flux

Flux enables SSA by default, so there is nothing special to configure.

This git repository, from Stefan Prodan, contains a detailed demo of how to use Kyverno policies to mutate a pod security context:

https://github.com/stefanprodan/gitops-kyverno 

Kyverno and ArgoCD

ArgoCD has added support for server-side diff in version 2.10. Here are the details:

https://argo-cd.readthedocs.io/en/latest/user-guide/diff-strategies/#server-side-diff

To enable SSA based diffs, an annotation must be specified on the application or globally via the “argocd-cmd-params-cm” config map:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true 
...

Demo

Here is a complete demonstration of how to use ArgoCD and Kyverno together for mutating resource configurations using policies:

Create a kind Cluster

Install kind and create a cluster:

kind create cluster --name=kyverno-argocd

Install Kyverno

kubectl create -f https://github.com/kyverno/kyverno/raw/main/config/install-latest-testing.yaml

Install Kyverno Policies

This command will install three Kyverno policies described below:

kubectl apply -f https://raw.githubusercontent.com/nirmata/demo-argocd-kyverno/main/config/kyverno-policies.yaml

This policy replaces all image tags with digests, which are immutable and more secure. Note that the policy does not perform any other form of image verification, like checking signatures or attestations:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: image-digest
 annotations:
   policies.kyverno.io/title: Convert tags to digests
   policies.kyverno.io/category: Supply Chain Security
   policies.kyverno.io/subject: Pod
   policies.kyverno.io/minversion: 1.11.0
   policies.kyverno.io/description: >-
     Image tags are mutable and can be spoofed. This policy resolves
     image tags to digests which are immutable.
spec:
 validationFailureAction: Enforce
 rules:
   - name: replace-tag
     match:
       any:
       - resources:
           kinds:
             - Pod
     exclude:
       all:
       - resources:
           namespaces:
             - argocd     
     verifyImages:
     - imageReferences:
       - "*"
       required: false
       verifyDigest: true
       mutateDigest: true

This policy allows the Kubernetes cluster auto-scaler to evict pods that use emptyDir by adding the annotation `cluster-autoscaler.kubernetes.io/safe-to-evict: true` if a pod contains an emptyDir volume.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: add-safe-to-evict
 annotations:
   policies.kyverno.io/title: Add Safe To Evict
   policies.kyverno.io/category: Other
   policies.kyverno.io/subject: Pod
   policies.kyverno.io/minversion: 1.6.0
   policies.kyverno.io/description: >-
     The Kubernetes cluster autoscaler does not evict pods that
     use hostPath or emptyDir volumes. To allow eviction of these
     pods the annotation cluster-autoscaler.kubernetes.io/safe-to-evict
     must be set to `true`.    
spec:
 rules:
 - name: annotate-empty-dir
   match:
     any:
     - resources:
         kinds:
         - Pod
   exclude:
     all:
     - resources:
         namespaces:
         - argocd
   mutate:
     patchStrategicMerge:
       metadata:
         annotations:
           +(cluster-autoscaler.kubernetes.io/safe-to-evict): "true"
       spec:         
         volumes:
         - <(emptyDir): {}

This policy turns off the auto-mount of service account tokens:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: service-account
 annotations:
   policies.kyverno.io/title: Disable Auto-Mount of Service Accounts
   policies.kyverno.io/category: Other
   policies.kyverno.io/subject: Pod
   policies.kyverno.io/minversion: 1.6.0
   policies.kyverno.io/description: >-
     Kubernetes automatically mounts ServiceAccount credentials in each Pod.
     The ServiceAccount may be assigned roles allowing Pods to access API resources.
     Blocking this ability is an extension of the least privilege best practice and should
     be followed if pods do not need to speak to the API server to function.
     This policy ensures that mounting of these ServiceAccount tokens is blocked.
spec:
 rules:
   - name: disable-automount-sa
     match:
       any:
         resources:
           kinds:
           - Pod
     exclude:
       all:
       - resources:
           namespaces:
           - argocd
     mutate:
       patchStrategicMerge:
         spec:
           +(automountServiceAccountToken): false

Install ArgoCD

Note that a version greater than 2.10 is required:

kubectl create ns argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.0-rc1/manifests/install.yaml

Create the ArgoCD Application

Install the ArgoCD application:

kubectl apply -f https://raw.githubusercontent.com/nirmata/demo-argocd-kyverno/main/config/argocd-app/application.yaml

The Application manifest looks like this – note the required annotation:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
  name: demo-app
  namespace: argocd
spec:
  destination:
    namespace: demo
    server: https://kubernetes.default.svc
  project: default
  source:
    path: config/kubernetes/
    repoURL: https://github.com/nirmata/demo-argocd
    targetRevision: HEAD
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

Once the application is deployed, we can check whether the policies have been applied. 

Check the running application for image tags:

kubectl -n demo get pod -o yaml | grep "image:"

This should show that images are references using digests:

    - image: ghcr.io/nirmata/demo-argocd:v1@sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105
      image: sha256:f9d5de0795395db6c50cb1ac82ebed1bd8eb3eefcebb1aa724e01239594e937b

Check the running application for service accounts:

This command will check for any volumes and if automountServiceAccountToken is disabled.

kubectl -n demo get pod -o yaml | grep -i "serviceAccountToken"

The output should only contain instructions to turn off the auto-mounting of service account tokens:

    automountServiceAccountToken: false

Check the running application for cluster autoscaler annotations:

This command will check the pod annotations:

kubectl -n demo get pod -o yaml | grep -i "annotations" -A1

The output should show the “cluster-autoscaler.kubernetes.io/safe-to-evict: true” annotation:

    annotations:
      cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

You can also view the pod and deployment details using ArgoCD:

Since Kyverno policy reports are namespaced Kubernetes resources, you can also check the compliance of each resource directly in the ArgoCD UI!

Conclusion

This blog post highlights the importance of mutating policies in securing and operating clusters and provides a tutorial on using Kyverno policies with FluxCD and ArgoCD. The post offers practical examples, such as adding pod security context defaults, necessary annotations, and converting image tags to digests during admission controls, all of which make sense to centralize using policies. If you’re interested in exploring other mutation policies, you can find them at https://kyverno.io/policies/?policytypes=mutate. 

Moreover, Kyverno also supports policies that generate resources, which can simplify self-service use cases for multi-tenancy and application isolation. To view examples of such policies, check out https://kyverno.io/policies/?policytypes=generate. 

Managing configurations using policies can bring numerous advantages. For instance, policies can simplify the configuration management process, enforce security measures, and promote best practices, while maintaining separation of concerns across developers, operations, and security teams.

For those looking to maximize the value of running Kyverno in production, Nirmata Control Hub offers a robust solution and a free trial! Simply head to https://try.nirmata.io/ to get started.