Argo CD is a name almost everyone knows at this point in time. As both one of the leading GitOps controllers and a newly-minted graduate of the CNCF, Argo CD has proven itself to be a highly-capable, feature-rich, and successful choice for implementing GitOps principles in a Kubernetes environment. Kyverno has also shown to be highly successful and has become the preferred policy engine of choice for Kubernetes on account of its prolific features and, perhaps most beloved of all, the promise of being purpose-built and complexity free for users. It’s natural and quite common, therefore, for both of these CNCF projects to be used together. In this post, we’ll talk about some ways these two are used together and share a few recommendations on how to go about doing it the best way.

Deploying Kyverno with Argo CD

Kyverno ships as a Helm chart as the most recommended route to a production install, and Argo CD very easily accommodates this form factor as a source. With Argo’s Application Custom Resource, one can easily point to Kyverno’s chart and be done very simply. However, here are a couple of recommendations we can offer after working with community members to successfully deploy Kyverno with Argo CD.

1. Use “Replace” as a sync option. Argo CD uses the kubectl program under the covers and the command applies to most of its manifests. While this works in most cases, for some large resources like Custom Resource Definitions, the apply command sometimes runs into troubles because of an annotation it uses to store the last-applied configuration. When deploying Kyverno through Argo CD, the first thing we’d like to recommend is to set Replace=true under your syncOptions section. Another option here would be to use Server-Side Apply.

2. Ignore RBAC changes made by AggregateRoles. In Kubernetes, Aggregated Roles are a very convenient feature that allows one to grant additional privileges to a given principal such as a ServiceAccount by creating a new ClusterRole and not modifying an existing one. This can be convenient especially for users of GitOps practices because it doesn’t involve changing resources that might be “owned” by your controller. Kyverno makes use of these aggregated roles as a way to make it easier for you, the user, to grant such additional privileges when use cases such as generation or cleanup are desired. In order to make Argo CD happiest, the second recommendation we have is to instruct Argo CD to ignore these aggregated role changes. You can do this by setting ignoreAggregatedRoles to a value of true in Argo CD’s ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  resource.compareoptions: |
    # disables status field diffing in specified resource types
    ignoreAggregatedRoles: true

3. Select a different tracking mechanism. In Kubernetes, metadata like labels are extremely commonly used to track and associate pieces and parts of an application to make them easier to be found, upgraded, deleted, etc. One of the most commonly-used labels for this purpose is app.kubernetes.io/instance. Argo CD has chosen this label but, in addition, so has Helm! Because of this conflict, Argo CD may get confused about the true owner of the application. Because Kyverno is packaged as a Helm chart, the third recommendation we’d suggest is to instruct Argo CD to use a different tracking mechanism, for example, the annotation+label option. Having this separation will ensure both are happy.

Although, in general, Argo CD happily deploys and manages Kyverno, these three recommendations above should help ensure you have a smoother and more enjoyable experience. All of these, by the way, can also be found in the official Kyverno documentation here.

Deploying Kyverno Policies with Argo CD

The second most popular use of Argo CD plus Kyverno we see is users deploying Kyverno policies. Kyverno policies are standard YAML files with no surprise block scalar into which some programming language is crammed. As a result, Kyverno policies can be manipulated with common tools such as Kustomize, ytt, and others. Also, since Kyverno policies are flexible as well as efficient, a single policy can be represented by just a single document and not multiple. Policies are “containers” for rules, and rules govern the type of policy behavior being applied. Therefore, Kyverno policies can be deployed simply by tools such as Argo CD without having to worry about tracking multiple files and dependencies.

A common choice for packaging these policies tends to be Helm and, as discussed earlier, Argo CD is fully capable of deploying Helm charts as well as bare manifests in a git repository. When choosing to deploy Kyverno policies as a Helm chart through Argo CD, because Kyverno uses JMESPath for more advanced handling, and JMESPath variables in Kyverno are wrapped in curly brackets just like Helm variables, you may need to escape these with Helm. Shown below is an example of how you can perform this escaping prior to Helm packaging so they are compatible with Argo CD.

{{`{{ request.userInfo.username }}`}}

In the example above, the Kyverno variable is {{ request.userInfo.username }} and the enclosing {{` and `}} characters will be stripped away by Helm upon installation of the policies.

And that’s about it. Very few modifications are needed and these are all designed to make Argo CD and Kyverno operate better together! If you liked this blog, check out its counterpart, “Enforcing Kubernetes Best Practices using Kyverno and Argo CD” on the Akuity blog.

If you are interested in scaling and automating Kubernetes using policies, check out our new eBook: The Ultimate Guide to Policy-based Governance, Security & Compliance for Kubernetes.