Skip to content

All posts

KyvernoKubernetesEngineering

How AT&T Turns Signed Evidence into Deployment Decisions with Kyverno

Sagar Kundral

Kyverno: How AT&T Turns Signed Evidence into Deployment Decisions with Kyverno

AT&T uses a Kyverno ImageValidatingPolicy, written in CEL, to decide at admission whether an image may run in a given environment. It verifies the image’s signed evidence, such as SLSA provenance and SBOM attestations from GitHub Actions, then checks those claims against the destination’s rules. The same image can be admitted to development and denied in production.

Chad Swenson, Senior Principal Software Engineer at AT&T, and Sagar Kundral, Solutions Architect at Nirmata, presented this design at KubeCon + CloudNativeCon North America 2026 in Salt Lake City. The talk was called “Signed, Sealed, Denied: Attestation-Driven Admission with Kyverno + CEL at AT&T”. This post covers the problem, the architecture, the policy, the performance numbers and the lessons from that talk.

Why isn’t a signature enough?

A few years ago, Chad noticed that many open source projects, including Kubernetes, Flux and Kyverno, were publishing their SLSA implementations. Most of that conversation was about consumers trusting a project’s builds. Chad saw a different opportunity: could verified build evidence stop every app team from proving the same things to auditors again and again?

Another team at AT&T was already designing standardized build pipelines. Chad brought SLSA into that work, and signing and attestations were added to the reusable workflows. That raised the next question: now that we have cryptographically verifiable evidence, what do we do with it?

The answer the talk opened with: a valid signature is not deployment approval.

Step What it tells you
Signature A trusted signer, and that the artifact hasn’t changed
Provenance Where and how the artifact was built
Signed claims Evidence the policy can evaluate
Destination rules Whether that evidence is enough for this environment

Signatures and provenance are useful, and AT&T keeps them. But neither is a deployment decision. The platform still has to ask whether the trusted claims meet the requirements of the place the image is going.

Same image, different destination, different decision

Here is the example from the talk. One image digest has signed provenance but no SBOM attestation.

Environment Requires Result
Development Verified provenance Admit
Production Provenance and a signed SBOM Deny: SBOM missing

Nothing about the image changed. The signature isn’t broken; the evidence just isn’t enough for production. Development keeps the bar lower to move fast, and production asks for more.

Why not just enforce at promotion?

Many pipelines check policy when an artifact is promoted to a production registry, often recording the result as a Verification Summary Attestation (VSA). AT&T does that too. Promotion and admission work better together, but they answer different questions:

  • The destination matters. The same image can face different requirements in different environments.
  • Requirements change. Yesterday’s approval is not today’s policy decision.
  • Evidence goes stale. Some attestations are time sensitive, such as a requirement for a passing vulnerability scan in the last two weeks.
  • Not everything goes through promotion. Requests can come from GitOps, a pipeline or someone running kubectl.

As the slide put it: promotion approves an artifact; admission evaluates this request against the policy that applies now.

What AT&T needed from the platform

AT&T’s platform team had three design requirements:

  1. Shared platform controls. App teams shouldn’t each rebuild the same compliance controls. They should be on by default.
  2. Multiple trust roots. Internal builds and approved upstream sources arrive with different signing identities.
  3. Retained decision evidence. Connect the policy, the evidence and the result, so a decision can be explained later.

They also had engineering constraints: measured latency, high availability, deliberate behavior when dependencies fail, and compatibility with the pipelines teams already used.

AT&T had used Kyverno for years for its other policy needs. The team looked at a few image verification controllers, but didn’t want to add a second policy engine for one problem. They preferred declarative rules and CEL in an engine their teams already knew.

The three building blocks

Building block Role
Sigstore Signing identity and trust material, with keyless verification. Public repositories use the public Sigstore instance; private repositories use GitHub’s private instance, which has no public transparency log.
GitHub Artifact Attestations Signed in-toto claims about a build: SLSA build provenance, SBOM attestations when the pipeline generates them, and custom test and quality attestations. Each is a Sigstore bundle pushed to an OCI registry.
Kyverno and CEL An ImageValidatingPolicy verifies the evidence and evaluates the claims against the destination’s rules, using the same Common Expression Language Kubernetes uses for CRD validation and ValidatingAdmissionPolicy.

Provenance says how an image was built, the SBOM says what’s inside it, and CEL expresses the policy.

The end-to-end flow: your trip to KubeCon

Chad explained the flow with a travel analogy (credited to his AT&T colleague Max Benjamin). You are the artifact: one exact, digest-pinned you.

Step Kubernetes Your trip to KubeCon
1. Build Build the image and pin its digest Getting ready at home
2. Sign and attest Publish signed build evidence Your ID and KubeCon registration: a trusted identity and a signed claim
3. Promote An earlier policy check, often captured as a VSA Airport security at home: checked earlier, somewhere else
4. Distribute Keep the image and its evidence together Your registration travels with you
5. Admit and record Verify the evidence and apply the destination’s policy Badge check at the venue, and a 21+ wristband for the receptions
6. Run and scale Replicas scale up and down Sessions and happy hours

A few points stand out. Your registration only helps if the venue can find it, so the evidence must stay discoverable as images are copied between registries; the OCI referrers API helps, but the tooling has to preserve the link. Some of you drove instead of flying, and the venue still checks your badge: admission is the one decision point for every covered request. And a valid badge isn’t an all-access pass. The 21+ wristband is an extra rule at some events, just as production asks for more evidence than development.

How the policy decides

Sagar walked through the admission step, where Kyverno has to allow or deny the Pod. The real question isn’t whether the image was signed. It’s who signed it, what they claim, and whether that is good enough for where it’s going. The policy answers in three steps.

1. Verify the evidence before reading it. The policy only reads an attestation’s payload after its signature has been verified against an approved attestor. If verification fails, the result is null and the contents are never used:

images.containers.map(image,
  verifyAttestationSignatures(
    image, attestations.slsa,
    [attestors.github]
  ) > 0
  ? extractPayload(image, attestations.slsa)
  : dyn(null))

The attestor checks the attestation against the central reusable workflow and GitHub’s private Sigstore root. You never want a decision based on a claim you haven’t verified.

2. Evaluate the verified claims. A properly signed claim can still describe a build you don’t accept. The policy checks the image was built by AT&T’s own GitHub organization, using the organization’s immutable numeric owner ID rather than its name, because names can be changed and re-registered.

3. Apply the destination’s rules. In the demo, development required provenance, an SBOM and a vulnerability scan. Production also required at least two approving code reviews. The same image was admitted to development and denied in production with the message “Image requires at least 2 approving code reviews”.

One policy engine, multiple trust roots

Not everything AT&T runs is built by AT&T. Alongside internal builds there are approved upstream images and some vendor images, each with its own signing identity and trust material. One engine handles them all, with an explicit image scope and approved attestors for each source.

Source Trust
Internal builds GitHub Actions signing identity, GitHub private Sigstore trust material, an approved workflow and required claims
Approved upstream images The public Sigstore instance or vendor trust, with an explicitly approved signer

Verifying images signed with GitHub’s private Sigstore instance hit a real limitation: Kyverno only knew the public Sigstore root. Chad raised it with Jim Bugwadia, Kyverno’s lead maintainer and Nirmata’s CEO, at the previous KubeCon + CloudNativeCon, and Jim wrote the fix himself (kyverno/kyverno#16666). It shipped in Kyverno 1.19. As Sagar said on stage, without that PR the talk would have had very different slides.

What changed in Kyverno for this use case

Release Change
1.19.0 Private Sigstore support: give Kyverno GitHub’s private trust material through cosign.trustedRoot
1.19.1 Autogen for custom workload resources, such as JobSet, so policies follow Pods wherever they’re created. A security fix (GHSA-5cjf-wwfg-pj4c) so an exception for one image can’t exempt other images in the same Pod.
1.19.2 A lighter admission path: the request context is built once per admission instead of once per matching policy
1.20 Compiled CEL programs are reused across admissions, a cached verification result skips the registry, and matching ImageValidatingPolicies run in parallel

If you run an ImageValidatingPolicy in Enforce mode, the talk’s advice was to use 1.19.2 or later.

Performance before and after

These are Kyverno benchmark figures, not AT&T production numbers. Sagar measured the effect on one laptop, using the same setup for every run and changing only the Kyverno version. The setup was a kind cluster with fake nodes, one ImageValidatingPolicy, one signed image, a warm cache and a registry with a fixed 25 ms delay.

Metric Kyverno 1.18.2 Kyverno 1.19.1 main (1.20-dev)
Admission latency p50 / p99 990 / 1170 ms 312 / 333 ms 96 / 107 ms
Admissions per second 1.0 3.2 10.4
Registry calls per admission 14 10 3
CPU / memory per replica 44 mC / 607 MiB 67 mC / 463 MiB 34 mC / 65 MiB

The milliseconds depend on the machine, but the ratios travel. The number Sagar cares most about is registry calls per admission, which fell from 14 to 3, because fewer round trips to your registry help everyone. CPU is the honest row: 1.19.1 used a little more CPU than 1.18.2 but did about three times the work. Measure with your own policies and your own registry.

Operating it in production

Getting a policy to work is one thing. Living with it in production is another. The talk named three choices to make up front, not during your first incident:

  • Exceptions. There will always be a team with a valid reason for one. Scope each exception narrowly to a policy, resource and image, give it an owner and a reason, and enforce an expiry date so it doesn’t become a permanent way around the policy.
  • Failure behavior. A registry that can’t be reached is very different from a policy that rejects an image. Decide whether to fail open or fail closed when verification can’t complete, and whether that differs between development, staging and production. Then test it.
  • Coverage and availability. Know which requests your policies cover. Run the admission path with high availability, and watch its latency and errors.

Make the answer a query, not a project

Eventually your security team will ask why an image was rejected in production last Tuesday. Without the right evidence, answering becomes a project: digging through logs, working out which policy version ran and what evidence existed at the time.

Instead, capture everything behind a decision together: the image digest, the destination cluster and namespace, the policy and its revision, the evidence found, the result and a correlation ID for the original request. In the talk’s example, the record shows provenance verified, SBOM absent and the image denied.

Kyverno policy reports show the current state of policy evaluation; they are not a history of every admission request. Forwarding report changes, for example with Kyverno Policy Reporter, isn’t the same thing either: it sends results as they change, so repeated admissions with the same result may not appear. If you need a per-request history, and most enterprises do, capture Kubernetes API server audit events together with the admission decision details, and retain them in your SIEM. Record every outcome: allowed, denied, and requests where verification couldn’t complete.

Roll out in phases, not in a panic

  1. Audit. Turn violations into a backlog while measuring latency and errors.
  2. Remediate. Fix problems in the shared workflows before filing the same ticket for every team.
  3. Enforce by tier. Start with an agreed, lower-risk scope.
  4. Expand coverage. Grow deliberately and own the exceptions.

You’re ready when coverage is tested, exceptions are owned, and latency and failure behavior are measured. Clean reports alone aren’t the whole test. Looking ahead, as tools adopt the SLSA 1.2 Source track, AT&T wants policy to check how source changes were reviewed and accepted, not just how the image was built.

What AT&T would do again

  1. Make the right way easy. Build evidence into reusable workflows, so no team has to become a signing expert before it can ship.
  2. Use the engine you know. Make controls a shared platform capability in the tools your teams already run, not a new compliance project for every application.
  3. Bring maintainers a real problem. The private Sigstore fix shows the feedback loop working: a concrete use case, a hands-on maintainer and upstream code.

The talk closed with three things to take home: verify the evidence, enforce the requirements and retain the decision. Start with one image class, one destination and one evidence requirement. A signature is the starting point; an explainable deployment decision is the outcome.

Applying the pattern

To set up the same building blocks, see supply chain security with GitHub artifact attestations and Kyverno. For exceptions an auditor can follow, read how to document policy exceptions as audit evidence. What’s new in Kyverno 1.20 covers the latest image verification changes, and our KubeCon + CloudNativeCon North America 2026 recap covers the rest of the week. To run image verification and policy reporting across many clusters, see software supply chain security with Nirmata.

Frequently asked questions

Why isn't a valid image signature enough to approve a deployment?

A signature tells you who built an image and that it hasn't changed. It doesn't tell you whether that image meets the rules of the environment it is being deployed to. AT&T verifies the signature and the signed claims first, then checks them against the destination's requirements, so the same image digest can be admitted to development and denied in production.

Why enforce at admission if images are already checked at promotion?

Promotion approves an artifact earlier and somewhere else. Admission evaluates the specific request against the policy that applies now. The destination may need extra evidence, policy may have changed since promotion, and some attestations are time sensitive. Admission also covers requests that never went through promotion, such as a kubectl apply.

Can Kyverno verify images signed with GitHub's private Sigstore instance?

Yes. AT&T needed this for images built from private repositories, and Kyverno maintainer Jim Bugwadia wrote the fix (kyverno/kyverno#16666). It shipped in Kyverno 1.19, which lets you give Kyverno GitHub's private trust material directly.

How fast is Kyverno image verification at admission?

In a single-laptop benchmark presented at KubeCon + CloudNativeCon North America 2026, p50 admission latency for one ImageValidatingPolicy fell from about 990 ms on Kyverno 1.18.2 to about 312 ms on 1.19.1 and about 96 ms on the 1.20 development branch. Registry calls per admission fell from 14 to 3. The milliseconds depend on the machine, so measure with your own policies and registry.

How should you roll out image verification policies?

In phases. Start in Audit mode to see violations, latency and errors, fix problems in shared build workflows, enforce for an agreed lower-risk tier first, then expand coverage. Decide up front how verification failures and registry outages are handled in each environment, and keep exceptions narrow, owned and time limited.

Software Supply Chain SecuritySignature and SBOM checks before an image runs.
Share

Stay up-to-date with the latest in AI, governance, and infrastructure

Subscribe to our free monthly newsletter to get industry news, DevOps tips, and articles right to your inbox on Kyverno, containerization, Kubernetes deployment & management and more.