Kyverno, the Kubernetes-native policy engine, is widely used with Amazon EKS for pod security, software supply chain security, and several other security and operational automation use cases. However, policy decisions often need access to external data which is not available in the cluster. Today, we are happy to announce the open-source release of the Nirmata AWS Adapter for Kyverno which makes key cloud configuration information available for policy enforcement and helps fully automate security and compliance of Amazon EKS clusters. 

Enabling Data Driven Policy Decisions

Policies require data to make decisions.  For example, the Nirmata Enterprise for Kyverno (available on the AWS Marketplace), which is the enterprise ready distribution with long term support of Kyverno from Nirmata, includes an adapter for reporting CIS Kubernetes Benchmarks for security issues with the cluster control plane configuration. When using Amazon EKS, the CIS Benchmark adapter, can only report issues with the worker node configuration as there is no access to the cluster control plane configuration. Amazon EKS control plane is managed by AWS and users don’t have direct access to the configuration but certain settings can be configured using the Amazon EKS APIs. Obviously not having access to the cluster configuration to detect security violations limits effectiveness of CIS Benchmark reporting and potentially requires other tools to detect cluster misconfigurations. We discussed this limitation with the AWS and Kyverno communities and explored different options to address it. 

Enabling Fast Policy Decisions 

At admission controls policy decisions need to execute reliably in a few seconds or less. The CNCF paper on Kubernetes CNCF Policy Management defines a Policy Information Point (PIP) as the component that provides data to enable policy decisions. 

Kyverno leverages the Kubernetes API server, with caching for fast policy decisions. One approach discussed was to add the ability in Kyverno to call external APIs such as the AWS API to fetch cluster configuration information. This poses some security and scalability challenges. Kyverno acts as an admission controller and making any external API calls can introduce latency in processing admission request. Also, adding cloud provider specific APIs in Kyverno can potentially make Kyverno codebase more complex as additional cloud provider support may be added in the future. 

A recommended approach is to decouple fetching of data from external sources such as Amazon EKS API by using data adapters. An adapter runs independently of Kyverno and fetches necessary data from the source (e.g. AWS API) and makes it available to Kyverno in the cluster using Customer Resource Definitions (CRDs) or ConfigMaps. 

Following this approach, today we are introducing the Nirmata AWS Adapter for Kyverno, an open source adapter that enables Kyverno policies to be created for Amazon EKS cluster configuration! 

Nirmata AWS Adapter for Kyverno

Nirmata AWS Adapter for Kyverno is deployed on clusters alongside Kyverno. It securely fetches the EKS cluster configuration information from the AWS API and makes it available in the cluster via a CRD, AWSAdapterConfig. 

The adapter is packaged as a helm chart and can be deployed to any cluster that has Kyverno running. Detailed instructions to install the adapter can be found here. Prior to deploying the adapter, the following should be configured:

• An IAM role for the service account to be used by the AWS adapter pod. This IAM role should have read access to EKS APIs.
• The values file in the Helm chart should be updated to include the EKS cluster name, the region, and the IAM Role ARN.

Once the adapter is installed, it will periodically fetch the configuration of the cluster and update the ‘status’ of the AWSAdapterConfig custom resource. 

apiVersion: security.nirmata.io/v1alpha1
kind: AWSAdapterConfig
metadata:
 name: eks-dev-cluster-config
 namespace: nirmata-aws-adapter
spec:
 name: eks-dev-cluster
 region: us-west-1
status:
 eksCluster:
   addons:
   - coredns
   - kube-proxy
   - nirmata_kyverno
   - vpc-cni
   arn: arn:aws:eks:us-west-1:XXXXXXXX:cluster/eks-dev-cluster
   certificate: ==REDACTED==
   compute:
     nodeGroups:
     - amiReleaseVersion: 1.23.13-20221112
       amiType: AL2_x86_64
       capacityType: ON_DEMAND
       createdAt: 2022-11-22 18:08:56.065 +0000 UTC
       diskSize: 60
       instanceTypes:
       - t3.medium
       launchTemplate: {}
       name: demo-ng
       nodeGroupArn: arn:aws:eks:us-west-1:XXXXXX:nodegroup/eks-dev-cluster/demo-ng/XXXXXX
       nodeRole: arn:aws:iam::XXXXXXX:role/Node-IAM-Role
       remoteAccessConfig: {}
       resources:
         autoScalingGroups:
         - eks-demo-ng-84c25043-aad4-f230-76d7-7ef665b79793
       scalingConfig:
         desiredSize: 2
         maxSize: 2
         minSize: 2
       status: ACTIVE
       subnets:
       - subnet-0dff3fad15acc9153
       - subnet-0ed4ba827d3066137
       tags:
         DoNotDelete: ""
       updateConfig:
         maxUnavailable: 1
   createdAt: 2022-10-13 18:52:27.886 +0000 UTC
   encryptionConfig:
   - keyARN: arn:aws:kms:us-west-1:XXXXXX:key/afa7691b-cbfd-4996-bfe9-9f264b434431
     resources:
     - secrets
   endpoint: https://657213779885F5257C0C35E7FC73E9DD.yl4.us-west-1.eks.amazonaws.com
   kubernetesVersion: "1.23"
   logging:
     apiServer: false
     audit: false
     authenticator: false
     controllerManager: false
     scheduler: false
   name:  eks-dev-cluster
   networking:
     ipFamily: ipv4
     serviceIPv4CIDR: 172.20.0.0/16
     vpc:
       clusterSecurityGroupID: sg-0902e1b9d22ef8ebd
       endpointPublicAccess: true
       publicAccessCIDRs:
       - 0.0.0.0/0
       subnetIDs:
       - subnet-0dff3fad15acc9154
       - subnet-0ed4ba827d3066144
       vpcID: vpc-01a1c8d10eb2bb2bb
   platformVersion: eks.2
   region: us-west-1
   roleArn: arn:aws:iam::XXXXXXXX:role/eks-role
   status: ACTIVE
   tags:
 lastPollInfo:
   status: success
   timestamp: "2022-11-30T18:14:48Z"
 lastUpdatedTimestamp: "2022-11-30T18:14:48Z"

Now, you can easily write Kyverno policies to detect and report on any configurations for the cluster. Here is an example policy which detects if the cluster is configured with a public endpoint.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: check-cluster-endpoint
 annotations:
   policies.kyverno.io/title: Check Public Endpoint
   policies.kyverno.io/category: EKS Best Practices
   policies.kyverno.io/severity: medium
   policies.kyverno.io/subject: Cluster
   policies.kyverno.io/description: >-
     Cluster endpoint should not be public.
spec:
 validationFailureAction: audit
 background: true
 rules:
   - name: check-cluster-endpoint
     match:
       any:
       - resources:
           kinds:
             - AWSAdapterConfig
     validate:
       message: "The cluster endpoint should not be public. The field status.eksCluster.networking.vpc.endpointPublicAccess must equal false."
       pattern:
         status:
           eksCluster:
             networking:
               vpc:
                 endpointPublicAccess: "false"

Now, if there is a policy violation, it will be reported and can be retrieved as part of the policy report.

apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
metadata:
 creationTimestamp: "2022-11-30T18:26:35Z"
 generation: 8
 labels:
   managed-by: kyverno
 name: polr-ns-nirmata-aws-adapter
 namespace: nirmata-aws-adapter
 ownerReferences:
 - apiVersion: v1
   controller: true
   kind: Namespace
   name: kyverno
   uid: 27d93209-af08-4bea-abeb-55ed2dbb24a4
 resourceVersion: "13342781"
 uid: b401d23c-0e68-48b6-81c3-18879dce71ca
results:
- category: EKS Best Practices
 message: 'validation error: The cluster endpoint should not be public. The field
   status.eksCluster.networking.vpc.endpointPublicAccess must equal false. Rule check-cluster-endpoint
   failed at path /status/eksCluster/networking/vpc/endpointPublicAccess/'
 policy: check-cluster-endpoint
 resources:
 - apiVersion: security.nirmata.io/v1alpha1
   kind: AWSAdapterConfig
   name: eks-dev-cluster-config
   namespace: nirmata-aws-adapter
   uid: 3128ddc2-e351-442a-8e7b-488f06dae224
 result: fail
 rule: check-cluster-endpoint
 scored: true
 severity: medium
 source: Kyverno
 timestamp:
   nanos: 0
   seconds: 1669832953

If you are using the Nirmata Policy Manager, these policy violations will also be reported in the Policy Reports dashboard. You can forward these violations to an appropriate Slack channel or create a Jira ticket to get the issue resolved.

Watch this video for the live demo:

Besides the AWS Adapter, the following adapters are available in Nirmata Enterprise Kyverno:

• Kube-bench adapter – this adapter detects and reports CIS Benchmark violations
• Venafi adapter – this adapter automates the retrieval of certificates from Venafi CodeSign Protect for container image verification

What’s Next

The Nirmata AWS Adapter for Kyverno currently fetches the Amazon EKS cluster using the AWS API. This information can be leveraged to provide feedback to the users on how they are doing with regard to Amazon EKS Best Practices. A comprehensive set of policies will be provided to detect and/or enforce Amazon EKS Best Practices.

AWS users often use the AWS Security Hub to check and centralize security alerts. The AWS Adapter will include the capability to forward policy violations to the AWS Security Hub so that security teams can get this information along with other findings in their cloud infrastructure. 

Nirmata AWS Adapter for Kyverno is available to try for free and will be included in Nirmata Enterprise for Kyverno and Nirmata Policy Manager. Both these products are available in the AWS Marketplace. If you have any other use cases that can be addressed by the AWS Adapter, please share them with us on the GitHub repo. 

Besides the AWS Adapter, the Nirmata team is also working on other adapters that can help address Kubernetes security, automation and cloud governance challenges as well as adapters for other cloud providers such as Microsoft Azure and Google Cloud. Stay tuned and subscribe to our newsletter to be notified – scroll to the bottom of the linked page to sign-up for our free newsletter.

Download this free ebook: Policy-based security and governance for Kubernetes

If you have further questions or queries, we’re here for you! Contact us to start a conversation, please.