Nirmata exists to eliminate friction with the enterprise adoption of open source and cloud-native technologies. A common challenge is that enterprise developers want to follow security best practices, but often fall short as securing applications can be complex, cumbersome, involves manual processes, and often also increases vendor lock-in.

A tool that helps address this problem is Vault by Hashicorp. Clearly, there are several benefits to decoupling sensitive data from applications and automating management. However, achieving this decoupling is not always easy without imposing new burdens on application developers.

In this post, I will show you how Nirmata simplifies the use of Vault for existing and new applications, without imposing any additional complexities.

Vault by HashiCorp

Vault is a tool for managing sensitive data (a.k.a. secrets) like passwords, access keys, and certificates. Vault allows us to decouple secrets from applications. Historically, sensitive data has been a pain to manage and has required manual requests and coordination across application development, operations, and security teams. Vault helps eliminate most of this hassle, by fully automating the processes around requesting and managing secrets.

Beyond the basics of securing data in transit and at rest, audit logs, and access controls, here are some of the advanced features that Vault offers:

Dynamic secrets: Vault can create secrets in a “just-in-time” manner i.e. only when an application requests them. This ensures that secrets are unique to the usage (application component) and environment (development, staging, production, etc.) For example, each instance of the application in production can have a different password. When an application instance is deleted its password can also be immediately deleted.

Leases: Vault provides the ability to associate a lease with dynamic secrets, hence ensuring that secrets expire and are rotated, or deleted if no longer needed.

When used with Kubernetes, the basic idea is to utilize Vault’s Kubernetes Authentication method to provide secrets for Kubernetes applications.

Running Vault

Typically, a single centralized Vault service will serve multiple Kubernetes clusters. Vault can be run in Kubernetes, or directly on virtual or physical servers.

For development and test systems, you can run Vault in dev-mode or use this image designed to run Vault in Kubernetes as a single instance which uses a Persistent Volume Claim (PVC) and file system based storage.

Accessing  Secrets from Kubernetes

Vault has built-in support for Kubernetes and can use Kubernetes APIs to verify the identity of an application.  The way this works is that when your application talks to Vault, it uses a JSON Web Token (JWT) from a Service Account to authenticate itself and request a named Role. Vault must be pre-configured to talk back to Kubernetes, using credentials that allow it to use the Token Reviewer role and validate the JWT supplied by the application pod. The application supplied JWT contains information Service Account name and the Namespace name for the application pod. Vault verifies that the JWT is valid and that the requested role is configured to allow access for the Service Account name and Namespace name. If all is good, Vault returns an access token which can then be used by the application to request secrets that are allowed for that role, via one or more Policies.

Setup Vault:

1. Enable the Kubernetes Authentication method for each cluster. This requires the cluster API Server address, the public keys, and an JSON Web Token (JWT) that provides access to the Kubernetes Token Reviewer role.
2. Define a policy for your application that provides access to the secrets your application requires. This policy can be reused access Kubernetes clusters.
3. Define a role for your application. The role definition includes the set of allowed Kubernetes Service Account names and allowed namespaces and a mapping to the policy. The role is specific to a cluster.

You can obtain the cluster API server address and the public keys from your kubectl config file. The API server address is found under clusters/cluster/server and the certificate is under clusters/cluster/certificate-authority-data. The certificate data must be base64 decoded to enter into Vault.

Here are the detailed instructions:

Create a new Service Account that will be used by Vault to access the cluster:

kubectl -n kube-system create sa vault-token-review

Add a Cluster Role for the Service Account using this YAML:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-vault-token-review-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-token-review
  namespace: kube-system

Extract the JWT using the following commands:

SECRET_NAME=$(kubectl -n kube-system get serviceaccount vault-token-review -o jsonpath='{.secrets[0].name}')
kubectl -n kube-system get secret ${SECRET_NAME} -o jsonpath='{.data.token}' | base64 --decode

You can then configure the Vault Kubernetes authentication method using the cluster address, certificate, and the token review role JWT.

In Kubernetes:

4. Run the application pod in an allowed namespace using an allowed Service Account.
5. Use the Service Account JWT to authenticate with Vault and get an access token.
6. Use the access token to fetch secrets.

Lets see how Nirmata simplifies this:

How Nirmata enables Vault for Kubernetes

Applications can directly integrate with Vault via its APIs. However, this tightly couples the application and the Secrets engine and is not considered a best practice. With Kubernetes, we can avoid this by using init containers to fetch secrets for applications. Kubernetes init containers run before other containers. Since they are in the same pod, Kubernetes init containers can share volumes with other containers in the pod.

The steps above may sound complex, but Nirmata makes this easy! Nirmata provides three things to enable integration between Vault and Kubernetes applications:

1. Simplified configuration of Vault settings for each cluster
2. Flexible policy-based injection of an init container to fetch secrets for the application
3. An open source init container implementation that does the work for fetching secrets from Vault.

To enable the integration, the cluster administrator creates a Vault Settings entry. A Vault Settings specifies how which Vault Service and Kubernetes Authentication path to use. Next, the administrator sets up one or more Secrets Policy entries, with the type set to Vault, the service account to use when authenticating with Vault, a list of secret entries to fetch and how to store these entries in the pod.

Vault Settings

You will need to create a Vault Settings instance for each Kubernetes cluster. Each Vault Settings specifies connection information for the Vault service, and the Kubernetes Authentication path to use for that cluster.

Here is an example of the Vault Settings in Nirmata:

Vault Secrets Policy

The Vault Secrets Policy contains the following:

• A pod selector. This can be used to filter based on applications, environments, or any label.
• The Service Account name
• The Vault Kubernetes Authentication Role name
• The secrets to be retrieved from Vault
• A file location to store the secrets

Here is an example of the Vault Secrets Policy in Nirmata:

Running the application pod

When an application is deployed, and a configured Vault Secrets Policy is matched, Nirmata will automatically inject an init container into the application pod. The init container is a light-weight, secure, and open source implementation of a custom vault client that uses the Vault APIs to authenticate using the Service Account JWT and configured role, and then fetch the secrets.

You can view the source and contribute at: https://github.com/nirmata/kube-vault-client.

In addition Nirmata also sets up the Service Account, Volume Mappings for each pod and matching a Volume for the pod, and an environment variable with the location of the secrets file. Here is an example of a deployment with the init container injected and settings to access secrets:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ghost
  namespace: ghost-aws-demo-prod
spec:
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      nirmata.io/component: ghost
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        nirmata.io/component: ghost
    spec:
      containers:
      - env:
        - name: VAULT_SECRETS_FILEPATH
          value: /var/run/vault
        image: ghost:0.11.9-alpine
        imagePullPolicy: Always
        name: ghost
        ports:
        - containerPort: 2368
          protocol: TCP
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run
          name: vault-secrets
      dnsPolicy: ClusterFirst
      initContainers:
      - args:
        - -kubeAuthRole
        - ghost-prod-role
        - -out
        - /var/run/vault
        - -secrets
        - secret/ghost/prod, secret/ghost/prod#password#MY_PASSWORD, secret/ghost/prod#token#MY_TOKEN
        - -url
        - https://vault-devtest2.nirmata.io/
        - -kubeAuthPath
        - kubernetes/prod/aws-demo/
        image: docker.io/nirmata/kube-vault-client:latest
        name: vault-init-secrets
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run
          name: vault-secrets
      serviceAccount: ghost-prod
      serviceAccountName: ghost-prod
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/run
          type: ""
        name: vault-secrets

Accessing secrets in the pod

Your application can now simply read secrets obtained form Vault at the configured location.

Optionally, you can “source” the secrets file to create an environment variable for each secret by modifying the run command of our application container.

Summary

Vault is a powerful security tool that provides secrets management, certificate management, encryption and other features. Vault helps automate security best practices by providing dynamic secrets and leases.

However using Vault can be a challenge, especially for existing applications. Nirmata’s integration with Vault makes it easy for cluster operators to setup and configure policies for Kubernetes clusters, and makes it seamless for enterprise developers to access secrets from Vault.

Visit our BrightTalk channel for more information and video demo of this feature.