PolicyReport API

The Policy Report API was developed under the Kubernetes Policy Working Group and provides an API that can be used by any policy engine, security scanner, or other security and compliance tool that wants to produce or consume policy results, security findings, or other reports for cluster resources. 

The Problem: Overwhelmed API Server and etcd

These policy reports are produced and consumed by various tools (see the Adopters list), with Kyverno being a prominent adopter. Kyverno generates detailed policy reports for every policy and rule applicable to each matching resource in a namespace. These reports are regenerated whenever resources are updated or recreated, and several intermediate ephemeral reports are also generated during this process.

In busy clusters, frequenting and updating policy reports can significantly load on etcd and the API server. Despite the cleaning up ephemeral reports, the sheer volume of data can overwhelm these critical components, causing performance degradation and, in extreme cases, bringing the cluster to a halt.

The Solution: Reports Server

The Reports Server is a new component designed to handle policy reports more efficiently and scale seamlessly with your cluster’s needs. The Reports Server offloads the storage of policy reports from etcd to an external storage solution, such as a Postgres database. This alleviates the burden on the API server and ensures the cluster remains responsive and reliable. The Reports Server provides a scalable solution for managing policy and cluster policy reports by storing reports in a relational database. This architecture enhances system performance and supports more efficient report consumer workflows. Analytical users, who often require complex queries on aggregate data, can leverage the robust query capabilities of PostgreSQL. This shift from API-based data retrieval to direct database queries significantly improves query performance and scalability, reducing latency and improving user experience.

Note: Reports Server is available with the latest release of Enterprise Kyverno (1.12).

Key Features of Reports Server

The Reports Server is a separate, dedicated component that offers flexibility in its deployment modes to suit different environments and use cases.

Deployment Modes

• In-Cluster PostgreSQL: Suitable for smaller clusters or test environments where simplicity and ease of setup are paramount.
• In-Memory Database: This mode is ideal for development and sandbox environments. It is fast and requires minimal configuration, making it perfect for quick iterations and testing.
• External PostgreSQL (Cloud Provider Managed): Recommended for production environments. By leveraging a managed PostgreSQL service, you benefit from enhanced reliability, automatic backups, and scaling capabilities without the need to manage the database infrastructure yourself.

Our Recommendation

We recommend configuring the reports server with an externally managed Postgres database in a production environment. This provides several benefits, such as –

• Scalability: Easily handle growing volumes of policy reports without compromising performance.
• Reliability: Managed services typically offer high availability and automatic failover, ensuring continuous operation.
• Maintenance: Offload routine database maintenance tasks such as backups, updates, and patches to the cloud provider.
• Performance: Optimized configurations and performance tuning by the provider enhance the overall efficiency.

In a dev/sandbox environment, we recommend configuring the reports server to run an in-memory database. This provides a lot of flexibility for the developers. Some benefits include –

• Speed: In-memory databases offer high-speed data access, making them ideal for rapid development and testing cycles.
• Simplicity: Minimal setup and teardown overhead, allowing developers to focus on building and testing features.
• Flexibility: Easily reset and modify data states, facilitating a more dynamic development process.

Scale Testing Results

We conducted extensive scale testing to ensure the Reports Server meets the demands of large-scale clusters. Here are some key metrics:

Key Metric	Observation
Cluster Size	108 Nodes
Policies Deployed	17
Policy Reports Generated	10000+
Etcd Load	Reduced by ~70%

A detailed comparison of how Reports Server affects etcd consumption can be inferred from the below graph.

These results demonstrate that the Reports Server can handle high volumes of policy reports without compromising the cluster’s performance or stability. Read more about the scale tests here.

What’s Next

Nirmata is committed to continuously improving the Reports Server. Our next steps include supporting more database options to provide greater flexibility and meet diverse user needs. We invite you to share your preferences and requirements regarding additional database support. Whether it’s MySQL, MongoDB, or another database, your feedback will guide our development priorities.

Conclusion

Reports Server represents a significant advancement in managing policy reports within your cluster. By transferring the storage responsibility from etcd to an external solution, the reactivity of your API server is upheld, thus ensuring the smooth operation of your cluster. With diverse deployment modes tailored to different environments and compelling scale testing outcomes, the Reports Server is poised to enhance your operational efficiency and dependability. We look forward to sharing further updates and encourage you to indicate your preferred databases for future support.

Nirmata’s Enterprise Kyverno distribution (N4K) integrates the advanced Reports Server, boosting system performance and scalability while enhancing policy report management. With powerful query capabilities, Kyverno streamlines data analysis and reporting, improving reliability and optimizing resources. Elevate your cluster management with Nirmata’s Enterprise Kyverno.

You can sign up for a 15-day free trial here. To set up the Reports Server in your environment, refer to the detailed documentation or contact us at support@nirmata.com to get a demo of this feature.