Kyverno, a policy engine for Kubernetes, is increasingly becoming the defacto standard for how to apply policy in a Kubernetes environment as a result of it being specifically designed for Kubernetes. Since it does not require either policy authors or policy readers to learn any programming language, it’s a perfect fit for established tooling and has become something of a community favorite, leading the pack of Kubernetes policy engines with over 4,100 stars and 2 billion downloads as of this writing. People love Kyverno it seems pretty clear. But we’ve also heard increasingly that people who love Kyverno for this simplicity would like it extended to generic or non-Kubernetes JSON processing similar to what Open Policy Agent (OPA) sought to achieve from the outset. Even though validation for generic JSON is a Kyverno road map item, I wanted to see how users might accomplish this goal right now in case they couldn’t wait. In this post, my fellow Kyverno maintainer, developer extraordinaire, and author of the Policy Reporter and co-author of the Kyverno Playground, Frank Jogeleit, and I will show you an experimental method for how you can leverage Kyverno to process and validate any JSON you want TODAY.
Background
Kyverno was built from the very outset specifically for Kubernetes. It was never intended for non-Kubernetes applications unlike other policy engines like OPA. As a result of this decision, it had access and opportunity to be very complimentary to how Kubernetes resources are managed using declarative APIs, defining resources as YAML documents, and leveraging common Kubernetes libraries and components among many other benefits. But, really, at the end of the day it’s just (at least from a validation standpoint) a JSON processing engine wrapped in Kubernetes garments.
The Kubernetes API is a JSON API. Although we commonly interact with it using YAML, these files are serialized into JSON, persisted as JSON, and retrieved in JSON. YAML is really just a convenience for us humans, both in authoring and display. When policy engines such as Kyverno are employed as admission controllers in a Kubernetes environment, the “language” they speak back and forth to each other is just JSON over HTTP. Not binary, not YAML, and not any other type of proprietary format or protocol. The API server communicates with the engine using a JSON-formatted document called an AdmissionReview, the engine churns over it, and then responds back to the API server with more JSON. This flow and what these contents look like are outlined in the Kyverno documentation here.
Figure 1: Request flow from API server to admission controllers.
All Kubernetes resources are subject to this flow and format irrespective of whether they’re stock (like Pods) or custom (like Kyverno’s policy resources). When it comes to their contents, the portion most of interest in the Kubernetes world are those under the spec field or, for the resources which don’t define spec such as ConfigMaps and Secrets, something like data. This field holds the data proper while others such as apiVersion, kind, metadata, and status have metadata (from Greek, literally “data about data”) and so forth which are used to further describe and contextualize the data proper. When a resource is sent to admission controllers, this resource state is wrapped in an AdmissionReview, which contains further data enrichment such as user information, what the old resource looked like (ex., in an update), and more.
Figure 2: A depiction of how data to be processed is wrapped by the Kubernetes API server in order to be presented to admission controllers for consumption.
Using this combined data, the admission controller is able to then make a decision as to what should be done (in the case of validation) and inform the API server of its decision.
This process is specific to Kubernetes and outside of those walls this additional wrapping and enrichment is often not performed. Systems communicate with one another using the raw JSON emitted by the previous, and this raw JSON is what must be validated.
Any JSON that needs to be validated by Kyverno needs to be masqueraded as a Kubernetes resource since that is what it is trained to process. But, as you’ll see, it really isn’t that difficult to put on a charade yourself. And, as stated earlier, this is really just a temporary measure. In the future, there are plans to support raw JSON directly.
As for use cases of generic JSON processing, there are two main ones:
- Send Kyverno JSON data “online” (as a service) from a running application
- Use Kyverno to statically validate “offline” JSON data like in a pipeline
As a Service
In this first use case, you have some app running someplace (we’ll assume this is inside the same Kubernetes cluster where Kyverno runs) and want it to send its JSON data to Kyverno for validation. The app is responsible for producing the request, sending it to Kyverno, and processing Kyverno’s response. Kyverno may also be processing requests from the Kubernetes API server as well.
Figure 3: A logical diagram for the service use case.
The JSON you want to validate is considered your “interesting data” as shown in figure 2. For example, this might look something like the following.
{
"color": "red",
"pet": "dog",
"foo": "bar"
}In order for Kyverno validate this, it needs to be represented as a Kubernetes resource. To do that, we have to add those boilerplate fields that all Kubernetes resources have, particularly apiVersion, kind, metadata, and spec which is where we’ll store the interesting data. In YAML, this would be defined as the following.
apiVersion: testing.io/v1
kind: MyJson
metadata:
name: testing
namespace: default
spec:
color: red
pet: dog
foo: bar
Notice here that the kind field is defined as MyJson. This is just some made up Custom Resource name I created which can be anything and in any apiVersion. But one important thing is required in this use case: a Custom Resource Definition (CRD). Kyverno requires, when running in a cluster, that a matching CRD be found for any policy which matches on a given Custom Resource like our MyJson one. However, one need only create a very simple one to get the job done. It might look like the following.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: myjsons.testing.io
spec:
group: testing.io
names:
kind: MyJson
plural: myjsons
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: This is a boilerplate custom resource used for testing of MyJson resources.
properties:
spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
served: true
storage: trueYou can see here that the only contents being specified here are a field called spec which allows anything underneath through use of the x-kubernetes-preserve-unknown-fields: true field. This is the simplest way to have any type of JSON data you want to be processed just keeping in mind it could make Kyverno policies a bit more verbose to author.
We then need some policy to validate our interesting data. Let’s just say, to keep things simple, we want to ensure the value of the foo field is always set to `bar` and that it is required to be present (not optional). This intent would be expressed like any other Kyverno policy just in the context of a MyJson resource kind since this is the “wrapper” used to present it.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: json-test
spec:
background: false
validationFailureAction: Enforce
rules:
- name: test
match:
any:
- resources:
kinds:
- MyJson
validate:
message: The value of `foo` must be set to `bar`.
pattern:
spec:
foo: barNow that our “interesting data” has been wrapped in a resource, the final layer is to present that in its final form: an AdmissionReview. While an AdmissionReview has more fields still, the main resource is represented under the `request.object` structure. A representation of such an AdmissionReview might look like below.
{
"kind": "AdmissionReview",
"apiVersion": "admission.k8s.io/v1",
"request": {
"uid": "ffffffff-ffff-ffff-ffff-ffffffffffff",
"kind": {
"group": "testing.io",
"version": "v1",
"kind": "MyJson"
},
"resource": {
"group": "testing.io",
"version": "v1",
"resource": "myjsons"
},
"requestKind": {
"group": "testing.io",
"version": "v1",
"kind": "MyJson"
},
"requestResource": {
"group": "testing.io",
"version": "v1",
"resource": "myjsons"
},
"name": "testing",
"namespace": "default",
"operation": "CREATE",
"userInfo": null,
"roles": null,
"clusterRoles": null,
"object": {
"apiVersion": "testing.io/v1",
"kind": "MyJson",
"metadata": {
"name": "testing",
"namespace": "default"
},
"spec": {
"color": "red",
"pet": "dog",
"foo": "bar"
}
},
"oldObject": null,
"dryRun": false,
"options": null
},
"oldObject": null,
"dryRun": false,
"options": null
}Now that this AdmissionReview has been created, all that’s left to do is send it to Kyverno.
When it comes to communicating with Kyverno, there’s nothing special with respect to how that’s done. Kyverno accepts JSON data over HTTP-REST from any source (provided the network allows it) and not just call backs from the Kubernetes API server. You will need to know where to call, and that can be gathered from inspecting Kyverno’s resource ValidatingWebhookConfiguration. Once the policy from above has been created, inspecting the webhook would show something like the following.
$ kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg -o yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
creationTimestamp: "2023-05-30T12:03:53Z"
generation: 262
labels:
webhook.kyverno.io/managed-by: kyverno
name: kyverno-resource-validating-webhook-cfg
resourceVersion: "12121845"
uid: c1b8d54b-60aa-49a7-9e45-ab52f6676bc9
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0t<snip>
service:
name: kyverno-svc
namespace: kyverno
path: /validate/fail
port: 443
failurePolicy: Fail
matchPolicy: Equivalent
name: validate.kyverno.svc-fail
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- kyverno
objectSelector: {}
rules:
- apiGroups:
- testing.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
- DELETE
- CONNECT
resources:
- myjsons
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10As you can see, the service getting called is kyverno-svc in the Namespace kyverno and at the path /validate/fail over port 443. Putting this together, when addressed from another location inside the same cluster, would be https://kyverno-svc.kyverno:443/validate/fail. You can even test this out by curling to it from some test Pod in your environment, assuming you created output.json with the AdmissionReview contents shown earlier.
curl -k -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' \
https://kyverno-svc.kyverno:443/validate/fail --data-binary "@output.json"The output from this command might look like the following.
{
"kind": "AdmissionReview",
"apiVersion": "admission.k8s.io/v1",
"request": {
"uid": "ffffffff-ffff-ffff-ffff-ffffffffffff",
"kind": {
"group": "testing.io",
"version": "v1",
"kind": "MyJson"
},
"resource": {
"group": "testing.io",
"version": "v1",
"resource": "myjsons"
},
"requestKind": {
"group": "testing.io",
"version": "v1",
"kind": "MyJson"
},
"requestResource": {
"group": "testing.io",
"version": "v1",
"resource": "myjsons"
},
"name": "testing",
"namespace": "default",
"operation": "CREATE",
"userInfo": {},
"object": {
"apiVersion": "testing.io/v1",
"kind": "MyJson",
"metadata": {
"name": "testing",
"namespace": "default"
},
"spec": {
"color": "red",
"pet": "dog",
"foo": "bar"
}
},
"oldObject": null,
"dryRun": false,
"options": null
},
"response": {
"uid": "ffffffff-ffff-ffff-ffff-ffffffffffff",
"allowed": true
}
}See here how Kyverno responded back to you and added a request.response field? It also set allowed to a value of true indicating that Kyverno allowed the request.
Modify your contents of output.json to produce a failure by setting foo to something else other than bar which would go against the policy and submit it again. This time you’ll see Kyverno blocked it since the policy was in Enforce mode.
{
"kind": "AdmissionReview",
"apiVersion": "admission.k8s.io/v1",
"request": {
"uid": "ffffffff-ffff-ffff-ffff-ffffffffffff",
"kind": {
"group": "testing.io",
"version": "v1",
"kind": "MyJson"
},
"resource": {
"group": "testing.io",
"version": "v1",
"resource": "myjsons"
},
"requestKind": {
"group": "testing.io",
"version": "v1",
"kind": "MyJson"
},
"requestResource": {
"group": "testing.io",
"version": "v1",
"resource": "myjsons"
},
"name": "testing",
"namespace": "default",
"operation": "CREATE",
"userInfo": {},
"object": {
"apiVersion": "testing.io/v1",
"kind": "MyJson",
"metadata": {
"name": "testing",
"namespace": "default"
},
"spec": {
"color": "red",
"pet": "dog",
"foo": "junk"
}
},
"oldObject": null,
"dryRun": false,
"options": null
},
"response": {
"uid": "ffffffff-ffff-ffff-ffff-ffffffffffff",
"allowed": false,
"status": {
"metadata": {},
"status": "Failure",
"message": "resource MyJson/default/testing was blocked due to the following policies json-test: test: validation error: The foo field must be set to bar. rule test failed at path /spec/foo/"
}
}
}Not only did Kyverno block it, but this time the request.response object has a status.message field which contains why Kyverno rejected it. This is the same message you’d see in a cluster if this bogus MyJson resource was actually submitted to the API server.
So, we’ve established that this whole process works just fine by emulating how the Kubernetes API server would handle things. All that’s left is for you to make a similar call from whatever service in your cluster you like and present it with the same information. Rather than stop here, I wanted to go a bit further this time and illustrate two things. First, an easy way for folks to see and play with this for themselves in a hands-on fashion. And, second, provide some sample code for how you might implement such a call if you couldn’t wait and wanted this right now.
Kyverno json-validator
Thanks to the great work of my cohort, Frank Jogeleit, he has kindly built a small demo app complete with UI you can use to test this whole flow as a simulation of what your app may need to do. You can find this (experimental) demo app under the Kyverno organization on GitHub in a repo called json-validator.
Figure 4: Screenshot of the UI of Kyverno json-validator, a small demo application used to send any JSON document direct to Kyverno.
The json-validator app deploys as a Helm chart in the cluster where Kyverno is running and contains a single Pod running the UI and the code needed to perform this double wrapping before sending it on to Kyverno. Installation instructions are here and quite simple, but let’s walk through it together.
Add the Helm repository and scan for updates.
helm repo add kyverno-json-validator https://kyverno.github.io/json-validator/
helm repo updateInstall the chart. I’m doing this in a test cluster where Kyverno is installed in the Kyverno Namespace and so the Service name is at its default. With no other values the chart will create a ClusterIP Service, but it’s also possible to expose it using other types including with an Ingress.
helm install kyverno-json-validator -n kyverno kyverno-json-validator/kyverno-json-validatorAfter successful installation, the chart will print out access instructions. The MyJson dummy Custom Resource is already installed for you as part of this chart.
Port forward to that Service.
kubectl -n kyverno port-forward service/kyverno-json-validator 8080Now just pull up the UI in a browser by navigating to http://127.0.0.1:8080. You’ll be presented with an editor in which some starter JSON is present and can copy-and-paste whatever you want to test.
Figure 5: Showing the json-validator in action.
Clicking the “Validate” button will cause the app to do the following.
- Take the contents of the editor and insert it under the spec object of the scaffolding MyJson resource which is already packaged in an AdmissionReview.
- POST the full AdmissionReview to Kyverno.
- Parse the response and display a nice success or failure message which includes the message returned by Kyverno.
Let’s test this out but with a real-wold use case.
Let’s say you wanted to perhaps use Kyverno to validate that a Terraform plan conformed to some guardrails, for example to deny creations of new AWS Autoscaling Groups (ASGs) but still allow updates and deletions to any existing ones.
First, we’ll create the Kyverno policy that enforces these guardrails.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: terraform
spec:
background: false
validationFailureAction: Enforce
rules:
- name: prevent-asg-creation
match:
any:
- resources:
kinds:
- MyJson
validate:
message: AWS Autoscaling groups in Terraform plans may not be created, only updated or deleted.
pattern:
spec:
resource_changes:
- (type): aws_autoscaling_group
change:
actions:
- "!create"Here, we’re matching on the MyJson Custom Resource installed as part of the chart, and performing a validation that, if under the resource_changes object, there is a type which is equal to aws_autoscaling_group that the action request associated is not a create.
Install the policy and make sure it’s ready.
Now, go to the json-validator UI and paste in the JSON representing the plan. A sample plan you can use for testing is shown below. Converting a Terraform plan to JSON is out of scope for this article, but see the docs here for more information.
{
"format_version": "0.1",
"terraform_version": "0.12.6",
"planned_values": {
"root_module": {
"resources": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managedA",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_name": "aws",
"schema_version": 0,
"values": {
"availability_zones": [
"us-west-1a"
],
"desired_capacity": 4,
"enabled_metrics": null,
"force_delete": true,
"health_check_grace_period": 300,
"health_check_type": "ELB",
"initial_lifecycle_hook": [],
"launch_configuration": "my_web_config",
"launch_template": [],
"max_size": 5,
"metrics_granularity": "1Minute",
"min_elb_capacity": null,
"min_size": 1,
"mixed_instances_policy": [],
"name": "my_asg",
"name_prefix": null,
"placement_group": null,
"protect_from_scale_in": false,
"suspended_processes": null,
"tag": [],
"tags": null,
"termination_policies": null,
"timeouts": null,
"wait_for_capacity_timeout": "10m",
"wait_for_elb_capacity": null
}
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_name": "aws",
"schema_version": 1,
"values": {
"ami": "ami-09b4b74c",
"credit_specification": [],
"disable_api_termination": null,
"ebs_optimized": null,
"get_password_data": false,
"iam_instance_profile": null,
"instance_initiated_shutdown_behavior": null,
"instance_type": "t2.micro",
"monitoring": null,
"source_dest_check": true,
"tags": null,
"timeouts": null,
"user_data": null,
"user_data_base64": null
}
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_name": "aws",
"schema_version": 0,
"values": {
"associate_public_ip_address": false,
"enable_monitoring": true,
"ephemeral_block_device": [],
"iam_instance_profile": null,
"image_id": "ami-09b4b74c",
"instance_type": "t2.micro",
"name": "my_web_config",
"name_prefix": null,
"placement_tenancy": null,
"security_groups": null,
"spot_price": null,
"user_data": null,
"user_data_base64": null,
"vpc_classic_link_id": null,
"vpc_classic_link_security_groups": null
}
}
]
}
},
"resource_changes": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managed",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_name": "aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"availability_zones": [
"us-west-1a"
],
"desired_capacity": 4,
"enabled_metrics": null,
"force_delete": true,
"health_check_grace_period": 300,
"health_check_type": "ELB",
"initial_lifecycle_hook": [],
"launch_configuration": "my_web_config",
"launch_template": [],
"max_size": 5,
"metrics_granularity": "1Minute",
"min_elb_capacity": null,
"min_size": 1,
"mixed_instances_policy": [],
"name": "my_asg",
"name_prefix": null,
"placement_group": null,
"protect_from_scale_in": false,
"suspended_processes": null,
"tag": [],
"tags": null,
"termination_policies": null,
"timeouts": null,
"wait_for_capacity_timeout": "10m",
"wait_for_elb_capacity": null
},
"after_unknown": {
"arn": true,
"availability_zones": [
false
],
"default_cooldown": true,
"id": true,
"initial_lifecycle_hook": [],
"launch_template": [],
"load_balancers": true,
"mixed_instances_policy": [],
"service_linked_role_arn": true,
"tag": [],
"target_group_arns": true,
"vpc_zone_identifier": true
}
}
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_name": "aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"ami": "ami-09b4b74c",
"credit_specification": [],
"disable_api_termination": null,
"ebs_optimized": null,
"get_password_data": false,
"iam_instance_profile": null,
"instance_initiated_shutdown_behavior": null,
"instance_type": "t2.micro",
"monitoring": null,
"source_dest_check": true,
"tags": null,
"timeouts": null,
"user_data": null,
"user_data_base64": null
},
"after_unknown": {
"arn": true,
"associate_public_ip_address": true,
"availability_zone": true,
"cpu_core_count": true,
"cpu_threads_per_core": true,
"credit_specification": [],
"ebs_block_device": true,
"ephemeral_block_device": true,
"host_id": true,
"id": true,
"instance_state": true,
"ipv6_address_count": true,
"ipv6_addresses": true,
"key_name": true,
"network_interface": true,
"network_interface_id": true,
"password_data": true,
"placement_group": true,
"primary_network_interface_id": true,
"private_dns": true,
"private_ip": true,
"public_dns": true,
"public_ip": true,
"root_block_device": true,
"security_groups": true,
"subnet_id": true,
"tenancy": true,
"volume_tags": true,
"vpc_security_group_ids": true
}
}
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_name": "aws",
"change": {
"actions": [
"create"
],
"before": null,
"after": {
"associate_public_ip_address": false,
"enable_monitoring": true,
"ephemeral_block_device": [],
"iam_instance_profile": null,
"image_id": "ami-09b4b74c",
"instance_type": "t2.micro",
"name": "my_web_config",
"name_prefix": null,
"placement_tenancy": null,
"security_groups": null,
"spot_price": null,
"user_data": null,
"user_data_base64": null,
"vpc_classic_link_id": null,
"vpc_classic_link_security_groups": null
},
"after_unknown": {
"ebs_block_device": true,
"ebs_optimized": true,
"ephemeral_block_device": [],
"id": true,
"key_name": true,
"root_block_device": true
}
}
}
],
"configuration": {
"provider_config": {
"aws": {
"name": "aws",
"expressions": {
"region": {
"constant_value": "us-west-1"
}
}
}
},
"root_module": {
"resources": [
{
"address": "aws_autoscaling_group.my_asg",
"mode": "managed",
"type": "aws_autoscaling_group",
"name": "my_asg",
"provider_config_key": "aws",
"expressions": {
"availability_zones": {
"constant_value": [
"us-west-1a"
]
},
"desired_capacity": {
"constant_value": 4
},
"force_delete": {
"constant_value": true
},
"health_check_grace_period": {
"constant_value": 300
},
"health_check_type": {
"constant_value": "ELB"
},
"launch_configuration": {
"constant_value": "my_web_config"
},
"max_size": {
"constant_value": 5
},
"min_size": {
"constant_value": 1
},
"name": {
"constant_value": "my_asg"
}
},
"schema_version": 0
},
{
"address": "aws_instance.web",
"mode": "managed",
"type": "aws_instance",
"name": "web",
"provider_config_key": "aws",
"expressions": {
"ami": {
"constant_value": "ami-09b4b74c"
},
"instance_type": {
"constant_value": "t2.micro"
}
},
"schema_version": 1
},
{
"address": "aws_launch_configuration.my_web_config",
"mode": "managed",
"type": "aws_launch_configuration",
"name": "my_web_config",
"provider_config_key": "aws",
"expressions": {
"image_id": {
"constant_value": "ami-09b4b74c"
},
"instance_type": {
"constant_value": "t2.micro"
},
"name": {
"constant_value": "my_web_config"
}
},
"schema_version": 0
}
]
}
}
}Click the “Validate” button and see this fails with the message contents as written in the Kyverno policy.
Figure 6: Failure when performing a validation in the json-validator UI results in relaying the message written in the matching Kyverno policy.
Now, scroll down to line 105 and change create to update and try to validate once more. This should succeed and return green.
Figure 7: A successful validation in the json-validator UI.
Super!
Let’s do one more real use case.
Suppose you were using Amazon ECS and wanted to validate Tasks as part of a delivery pipeline. Specifically, you wanted to ensure that a given Role was used based upon some other criteria.
We’ll create the following policy which requires that the executionRoleArn be set to `arn:aws:iam::*:role/ecsTaskExecutionRole` if the networkMode in use is `awsvpc`. The wildcard allows Kyverno to accept any value between the two other colon characters making this very flexible for IAM usage.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: ecs
spec:
background: false
validationFailureAction: Enforce
rules:
- name: task-awsvpc-use-ecstaskexecutionrole
match:
any:
- resources:
kinds:
- MyJson
validate:
message: If awsvpc mode is used, executionRoleArn must be ecsTaskExecutionRole
pattern:
spec:
(networkMode): awsvpc
executionRoleArn: arn:aws:iam::*:role/ecsTaskExecutionRoleNow we’ll drop some JSON for an ECS Task in the UI.
{
"containerDefinitions": [
{
"command": [
"New-Item -Path C:\\inetpub\\wwwroot\\index.html -Type file -Value '<html> <head> <title>Amazon ECS Sample App</title> <style>body {margin-top: 40px; background-color: #333;} </style> </head><body> <div style=color:white;text-align:center> <h1>Amazon ECS Sample App</h1> <h2>Congratulations!</h2> <p>Your application is now running on a container in Amazon ECS.</p>'; C:\\ServiceMonitor.exe w3svc"
],
"entryPoint": [
"powershell",
"-Command"
],
"essential": true,
"cpu": 2048,
"memory": 4096,
"image": "mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/fargate-windows-task-definition",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
},
"name": "sample_windows_app",
"portMappings": [
{
"hostPort": 80,
"containerPort": 80,
"protocol": "tcp"
}
]
}
],
"memory": "4096",
"cpu": "2048",
"networkMode": "awsvpc",
"family": "windows-simple-iis-2019-core",
"executionRoleArn": "arn:aws:iam::012345678910:role/ecsTaskExecutionRole",
"runtimePlatform": {
"operatingSystemFamily": "WINDOWS_SERVER_2019_CORE"
},
"requiresCompatibilities": [
"FARGATE"
]
}Click the “Validate” button and you should see this is successful.
Now produce a failure by changing the value of the executionRoleArn to something like arn:aws:iam::012345678910:role/bobCustomRole. Click “Validate” again and this will return the message below.
Validation failed: resource MyJson/default/testing was blocked due to the following policies ecs: task-awsvpc-use-ecstaskexecutionrole: 'validation error: If awsvpc mode is used, executionRoleArn must be ecsTaskExecutionRole. rule task-awsvpc-use-ecstaskexecutionrole failed at path /spec/executionRoleArn/'Not sure about you but I think that’s pretty cool!
It’s also possible using this pattern to fetch additional data, both from within Kubernetes as well as outside, to factor into the decision making process. For example, envision that you wanted to use Kyverno for generic JSON validation of some cloud resource corresponding to a firewall rule. In order to allow removal, you wanted to check if that rule was in use anywhere in your cloud or Kubernetes environment by ensuring it wasn’t applicable to Ingress resources. Using the service call feature of Kyverno you could totally do this.
Hopefully you can see how powerful this is and that it can be incorporated into something like your internal developer platform (IDP) to automate similar types of validations.
But that’s not all that is possible. You can also use the Kyverno CLI right now to perform these types of “offline” validations in CI pipelines.
In a Pipeline
The Kyverno CLI is the Kyverno engine packaged in a slightly different form factor so as to allow testing of static resources against policies away from a cluster. It has two primary commands: test and apply. The latter is used to test a specific policy against one or more resources to determine its disposition. Since we’ve already done the work above, we can go ahead with testing our MyJson policy against a matching MyJson resource. The best part is, the CLI doesn’t even require a CRD and so as long as your generic JSON is in a Kubernetes Custom Resource suit, it’s ready to go.
For reference, here’s a basic ClusterPolicy once again.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: json-test
spec:
background: false
validationFailureAction: Enforce
rules:
- name: test
match:
any:
- resources:
kinds:
- MyJson
validate:
message: The value of `foo` must be set to `bar`.
pattern:
spec:
foo: barAnd a sample resource which matches the policy.
apiVersion: testing.io/v1
kind: MyJson
metadata:
name: testing
namespace: default
spec:
color: red
pet: dog
foo: badApply the policy to the resource.
$ kubectl kyverno apply cpol.yaml -r myjson.yaml
Applying 1 policy rule to 1 resource...
policy json-test -> resource default/MyJson/testing failed:
1. test: validation error: The value of `foo` must be set to `bar`. rule test failed at path /spec/foo/
pass: 0, fail: 1, warn: 0, error: 0, skip: 0Now fix the resource so foo is equal to bar as required by the policy. Apply once more.
$ kubectl kyverno apply cpol.yaml -r myjson.yaml
Applying 1 policy rule to 1 resource...
pass: 1, fail: 0, warn: 0, error: 0, skip: 0You can also output the results to a Policy Report and send that off for other processing should you desire. The full array of CLI arguments are at your fingertips.
Closing and Notes
Although the Kyverno project has intentions for expanding into other use cases such as using Kyverno as a generic JSON processor, as Frank and I have demonstrated it is entirely possible to achieve this behavior today with a little bit of abstraction and a smidge of trickery. Using Frank’s handy dandy json-validator, you have all the pieces you need to begin building the pieces out in your own automation system should you choose to go down that path. And, rest assured, this will get significantly easier in the future, it’ll just take some time.
With all that said, I wanted to cover a few notes with you as well, particularly as they pertain to the first use case of running validations against the Kyverno service co-located in a Kubernetes cluster.
It’s totally possible to expose Kyverno for processing by outside applications as well just by changing its Service type. But Kyverno must still run inside _some_ Kubernetes cluster currently. It also isn’t required that it must provide admission control services for the cluster in which it’s installed. It can be installed in a minimum fashion by just including the admission controller and not any of the others such as background, reports, or cleanup.
Since these requests/responses do not travel between the Kubernetes API server, there is no risk of “good” resources being persisted into the cluster as what normally happens in admission mode. But what will be persisted (unless overridden by a flag) are Kubernetes Events. Kyverno generates these when it detects a violation which has a blocking action. Disabling them could be done by setting the flag –maxQueuedEvents=0 but this will take effect for everything sent to that Kyverno instance.
Lastly, in this proof-of-concept we are essentially bypassing TLS verification on the Kyverno Service. If you wanted to do this “for real” just be mindful that Kyverno will present a TLS certificate and so you’d probably want to establish trust to ensure your payload was sent to the intended receiver.
That’s it from me. This was a really fun experiment and I’m glad Frank and I were not only successful but managed to give you something not just workable but fairly easy to deploy and operate. If you thought this was helpful (or at least entertaining) we’d love to hear your feedback. Reach us both on Twitter at Frank and Chip.
A heady experiment indeed. If you want more information on Kyverno as developed by Nirmata, you can contact us for a conversation. Or sign-up for a demo. Thanks for reading.
Sorry, the comment form is closed at this time.