Skip to content

Case study

How AT&T Turns Signed Evidence into Deployment Decisions with Kyverno

AT&T uses a Kyverno ImageValidatingPolicy, written in CEL, to decide at admission whether an image may run in a given environment. It verifies the image’s signed evidence, such as SLSA provenance and SBOM attestations from GitHub Actions, then checks those claims against the destination’s rules.

AT&T by the numbers

p50 admission latency (Kyverno benchmark, one laptop)
990 → 96 ms
Registry calls per admission (Kyverno benchmark, one laptop)
14 → 3
Admissions per second (Kyverno benchmark, one laptop)
1.0 → 10.4
Memory per replica (Kyverno benchmark, one laptop)
607 → 65 MiB

Challenge

Why isn’t a signature enough?

AT&T had added signing and attestations to its standardized, reusable build workflows. That raised the next question: now that we have cryptographically verifiable evidence, what do we do with it? The answer: a valid signature is not deployment approval. A signature says who built an image and that it hasn’t changed. It doesn’t say whether the image meets the rules of the place it is going.

EnvironmentRequiresResult
DevelopmentVerified provenanceAdmit
ProductionProvenance and a signed SBOMDeny: SBOM missing

Checking at promotion isn’t enough on its own. The destination may need extra evidence, requirements change, some attestations are time sensitive, and requests can come from GitOps, a pipeline or kubectl. Promotion approves an artifact; admission evaluates this request against the policy that applies now.

The platform team wanted shared controls that app teams don’t each rebuild, support for more than one trust root, and enough retained evidence to explain any decision later, with measured latency, high availability and deliberate failure behavior.

Solution

One policy engine for evidence and destination rules

AT&T had used Kyverno for years and didn’t want a second policy engine for one problem. The design has three building blocks:

Building blockRole
SigstoreSigning identity and trust material, public or GitHub’s private instance
GitHub Artifact AttestationsSLSA build provenance, SBOM attestations and custom test and quality attestations
Kyverno and CELAn ImageValidatingPolicy verifies the evidence and evaluates the claims against the destination’s rules

The policy decides in three steps:

  1. Verify the evidence before reading it. The payload is only read after the attestation’s signature is verified against an approved attestor.
  2. Evaluate the verified claims. The image must be built by AT&T’s own GitHub organization, matched on its immutable numeric owner ID rather than its name.
  3. Apply the destination’s rules. Development requires provenance, an SBOM and a vulnerability scan. Production also requires at least two approving code reviews.

Internal builds and approved upstream images are handled by the same engine, each with an explicit image scope and approved attestors. Verifying images signed with GitHub’s private Sigstore instance needed a Kyverno change: Chad raised it with Jim Bugwadia, who wrote the fix himself (kyverno/kyverno#16666). It shipped in Kyverno 1.19.

Results

Faster admission with each Kyverno release

These figures are a Kyverno benchmark, not AT&T production numbers. Sagar measured image verification on one laptop, from Kyverno 1.18.2 to the 1.20 development branch, changing only the Kyverno version: a kind cluster, one ImageValidatingPolicy, one signed image, a warm cache and a registry with a fixed 25 ms delay.

MetricKyverno 1.18.2Kyverno 1.19.1main (1.20-dev)
Admission latency p50 / p99990 / 1170 ms312 / 333 ms96 / 107 ms
Admissions per second1.03.210.4
Registry calls per admission14103
CPU / memory per replica44 mC / 607 MiB67 mC / 463 MiB34 mC / 65 MiB

The milliseconds depend on the machine, but the ratios travel. Fewer registry round trips help everyone. Measure with your own policies and your own registry.

Lessons

What AT&T would do again

  1. Make the right way easy. Build evidence into reusable workflows, so no team has to become a signing expert before it can ship.
  2. Use the engine you know. Make controls a shared platform capability, not a new compliance project for every application.
  3. Bring maintainers a real problem. A concrete use case and a hands-on maintainer turned into upstream code.

Roll out in phases: audit, remediate shared workflows, enforce by tier, then expand coverage. Keep exceptions narrow, owned and time limited, decide up front how verification failures are handled, and retain every decision so “why was this image denied last Tuesday?” is a query, not a project.

More case studies

All case studies

Want to fast-track security and compliance in your Kubernetes environment?