Velero is a Kubernetes-native open-source tool that allows users to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes. Velero is typically deployed and used by cluster administrators or storage administrators. So whenever developers want an application or namespace to be backed up, they have to reach out to the admin to set it up. Wouldn’t it be better if developers could enable backups for their applications themselves without having to bother admins? In fact, what if admins could specify the necessary configuration for backups such as the location, schedule and developers could simply indicate whether to enable backups or not. This “developer self-service” workflow is not currently supported natively by Velero as indicated in this Github issue. In this post, I will describe how to enable developer self-service backups with Velero using Kyverno, a new CNCF sandbox project.

Introducing Kyverno

Kyverno (which means “governance” in Greek) is a policy engine designed for Kubernetes. With Kyverno, policies are managed as Kubernetes resources, and no new language is required to write policies. This allows cluster administrators to use familiar tools such as kubectl, Git, and kustomize to manage policies. Kyverno policies can be used to validate, mutate, and generate Kubernetes resources. Using Kyverno, admins can define policies to ensure that applications deployed in the cluster are compliant and follow security and configuration best practices.

In order to enable self-service backups with Velero, we will use the “generate” capabilities of Kyverno to automatically generate a Schedule to backup the resources and data when a specific label, nirmata.io/auto-backup=enabled is added to a namespace. 

Velero Setup

First, install Velero in a Kubernetes cluster. Once Velero is installed, create the Backup Storage Location:

velero backup-location create default \
    --provider aws \
    --bucket velero-demo-backup-dir` \
    --config region=us-west-1

At this time, there are no backup schedules created so Velero is not backing up any resources.

Kyverno Setup

Next, install Kyverno by following the instructions here. Once Kyverno is installed, we will create a policy that can generate a Velero schedule whenever a label nirmata.io/auto-backup=enabled is added to a namespace, a backup schedule is created for that namespace.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: autobackup-policy
spec:
  background: false
  rules:
  - name: "add-velero-autobackup-policy"
    match:
        resources:
          kinds:
            - Namespace
          selector:
            matchLabels:
              nirmata.io/auto-backup: enabled
    generate:
        kind: Schedule
        name: "{{request.object.metadata.name}}-auto-schedule"
        namespace: velero
        apiVersion: velero.io/v1
        synchronize: true
        data:
          metadata:
            labels:
              nirmata.io/backup.type: auto
              nirmata.io/namespace: '{{request.object.metadata.name}}'
          spec:
            schedule: 0 1 * * *
            template:
              includedNamespaces:
                - "{{request.object.metadata.name}}"
              snapshotVolumes: false
              storageLocation: default
              ttl: 168h0m0s
              volumeSnapshotLocations:
                - default

The above policy is triggered when the label nirmata.io/auto-backup=enabled is added to a namespaces. The YAML for Velero backup schedule will be generated based on the settings in the “generate” section of the policy. The schedule, TTL and the backup location can be configured here.

Also, we need to allow Kyverno to generate the Velero Schedule resource updating the kyverno:generatecontroller cluster role.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:generatecontroller
rules:
- apiGroups:
  - '*'
  resources:
  - namespaces
  - networkpolicies
  - secrets
  - configmaps
  - resourcequotas
  - limitranges
  - clusterroles
  - rolebindings
  - clusterrolebindings
  - schedules
  verbs:
  - create
  - update
  - delete
  - list
  - get
- apiGroups:
  - '*'
  resources:
  - namespaces
  verbs:
  - watch

Self-service Backup

Once the policy is created and the cluster role is updated, you can verify if automated backups are working by adding the label nirmata.io/auto-backup=enabled to any namespace.

kubectl label namespace/nginx nirmata.io/auto-backup=enabled

Now check if the backup schedule is generated

kubectl get schedule -n velero

You can check the configuration for the schedule that was generated.

kubectl get schedule -n velero nginx-auto-schedule -o yaml

apiVersion: velero.io/v1
kind: Schedule
metadata:
  creationTimestamp: "2021-01-24T02:22:18Z"
  generation: 5
  labels:
    app.kubernetes.io/managed-by: kyverno
    kyverno.io/generated-by: Namespace--nginx
    policy.kyverno.io/gr-name: gr-gj8mz
    policy.kyverno.io/policy-name: autobackup-policy
    policy.kyverno.io/synchronize: enable
    nirmata.io/backup.type: auto
    nirmata.io/namespace: nginx
  name: nginx-auto-schedule
  namespace: velero
  resourceVersion: "14783286"
  selfLink: /apis/velero.io/v1/namespaces/velero/schedules/nginx-auto-schedule
  uid: ae8d6582-466c-470a-81f7-89941c94eab4
spec:
  schedule: 0 1 * * *
  template:
    includedNamespaces:
    - nginx
    snapshotVolumes: false
    storageLocation: default
    ttl: 168h0m0s
    volumeSnapshotLocations:
    - default

As you can see, it is really easy to enable automated self-service Velero backups with Kyverno. You can also use different labels to trigger different policies and generate Velero schedules with different configurations. Kyverno is very versatile and can be used for automating other workflows. You can check out the Resources page on the Kyverno website for more posts on how Kyverno can be used for Kubernetes security, governance, and automation.