How to build deterministic AI workflows on Kubernetes

A deterministic AI workflow on Kubernetes keeps every step that can be a query, a rule or an API call as ordinary code, and gives a model one bounded judgement step. Each step runs as its own Job under its own identity, and the model sees only the fields the workflow passes it.

Why determinism is the property to design for

An AI workflow fails in a different way from a script. A script that runs the same commands on the same cluster state produces the same output, so a reviewer can read the code and predict its behaviour. A model step does not work that way. Two runs on identical evidence can produce different wording, different priorities, or a different conclusion.

The practical answer is not to remove the model. It is to make sure the model is the only part of the workflow that can be non-deterministic. Collection, gating and publishing should stay in code that a reviewer can diff, rerun and test. The rest of this guide shows how to split a workflow that way using Kubernetes primitives you already run.

Split the workflow into collect, analyze and publish

Most operational loops, such as a cost review, a pod triage or a hygiene sweep, have three stages. Only one of them needs judgement.

Stage	What runs	Model involved	How it fails
Collect	Queries against the Kubernetes API or metrics, written to an evidence file	No	The Job retries under its backoff settings
Analyze	One model call that reads only the evidence file	Yes	A CEL gate decides whether the step runs at all, and a failed step is retried or stopped
Publish	Writes the result somewhere a reviewer can read it	No	The Job retries under its backoff settings

Keep the analyze stage as small as the task allows. If a condition can be written as an expression, write it as an expression. The model should see the cases a rule cannot decide, not the whole cluster.

Run each deterministic step as a Job

Each collect and publish step runs as a Kubernetes Job. A Job gives you retries, a deadline and cleanup without any custom controller. The steps below set the fields that matter for a workflow. Checked against the Kubernetes Job and ServiceAccount documentation as fetched on 2026-11-09.

1. Create one Job per step, so each one has its own pod template, service account and retry budget.
2. Set backoffLimit explicitly. Failed pods count against it, and the default is 6.
3. Set activeDeadlineSeconds to cap the total time for the step. It applies to the whole Job, not to each pod.
4. Set ttlSecondsAfterFinished so finished steps are cleaned up and the evidence is kept in a place you chose.

apiVersion: batch/v1
kind: Job
metadata:
  name: collect-evidence
spec:
  backoffLimit: 5
  activeDeadlineSeconds: 100
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      containers:
      - name: collect
        image: perl:5.34.0
        command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never

The Job controller recreates failed pods with an exponential delay of 10 seconds, then 20, then 40, capped at ten minutes. A deadline also overrides the retry count: once activeDeadlineSeconds is reached, the Job stops creating pods even if backoffLimit has not been reached.

Keep the model out of the API

The analyze step should not need a Kubernetes API token. It reads the evidence file its workflow hands it and writes a result. Give it no API access at all.

1. Create a dedicated service account for each step that needs one. Do not let the analyze step run as default.
2. Set automountServiceAccountToken: false on the analyze pod.
3. Do not bind any Role to the analyze step’s service account.

Kubernetes creates a default service account in every active namespace, so a pod that names no account gets that one. The token admission behaviour is the reason step 2 matters: when the field is false, the ServiceAccount admission controller does not add the token volume or its mount to the pod.

apiVersion: batch/v1
kind: Job
metadata:
  name: analyze-evidence
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 300
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      serviceAccountName: workflow-analyze
      automountServiceAccountToken: false
      containers:
      - name: analyze
        image: perl:5.34.0
        command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never

The image and command are the docs’ sample. Swap in your model-calling container. The workflow-analyze account exists only so the pod names an account that has no bindings. Create it before the Job runs: the ServiceAccount admission controller rejects an incoming pod whose ServiceAccount does not exist in its namespace, and that check applies to default too.

Gate the judgement step with CEL

Before the analyze step runs, evaluate a CEL expression over the evidence. The gate decides whether a model call is needed at all. A CEL program is side-effect-free and terminating, so the gate cannot change the cluster while it runs, and a given evaluation environment always yields the same value or the same error.

Two properties need handling in the runner. CEL has no in-language representation of errors, so a program cannot catch one. Treat any evaluation error as a failed gate and stop the workflow rather than running the model on data you could not check.

evidence.restarts > 3 && evidence.oomKilled

Only when this expression evaluates to true should the runner start the analyze Job. Otherwise the workflow publishes the collected evidence without a model call.

Bound retries with a pod failure policy

The default retry rule counts every pod failure against backoffLimit. For a step where some exit codes mean a bug in the workflow, not a transient fault, a pod failure policy lets you fail the Job immediately. The rules are evaluated in order, and the first match wins.

apiVersion: batch/v1
kind: Job
metadata:
  name: collect-with-failure-policy
spec:
  completions: 12
  parallelism: 3
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: main
        image: docker.io/library/bash:5
        command: ["bash"]
        args:
        - -c
        - echo "Hello world!" && sleep 5 && exit 42
  backoffLimit: 6
  podFailurePolicy:
    rules:
    - action: FailJob
      onExitCodes:
        containerName: main
        operator: In
        values: [42]
    - action: Ignore
      onPodConditions:
      - type: DisruptionTarget

The docs say the Ignore action on a DisruptionTarget condition excludes pod disruptions from the backoffLimit count. Use that for node drains and preemptions, which are not faults in your workflow.

Setting	Default in the docs	What it controls
backoffLimit	6	Failed pods counted toward the retry limit
activeDeadlineSeconds	Not set	Total run time for the Job; takes precedence over backoffLimit
ttlSecondsAfterFinished	Not set	Automatic deletion of the finished Job and its pods
podFailurePolicy	Not set	Exit-code and pod-condition rules, evaluated in order

Publish the result and keep the run record

The publish step is also a Job with no model in it. It writes the evidence, the gate decision and, when the analyze step ran, the model’s output to a destination a reviewer reads. Store the gate expression and the evidence file together with the result, so a reviewer can replay the gate against the same inputs and confirm why the model was or was not called.

Keep the record for longer than the Job. ttlSecondsAfterFinished deletes the Job and its pods, so anything you need later must already be in your publish destination.

Checklist for a deterministic workflow

1. Every stage that can be a query, a rule or an API call is a Job or a CEL expression.
2. Exactly one stage calls a model, and it reads only the evidence file.
3. Each step has its own service account, and the model step has none bound.
4. backoffLimit, activeDeadlineSeconds and ttlSecondsAfterFinished are set on every Job in the YAML.
5. A gate error stops the workflow rather than running the model.
6. The evidence, gate expression and output are stored together for review.

Where to go next

For the identity side of the model step, read RBAC for AI agents in Kubernetes. For isolating the step’s container, read how to sandbox AI agents on Kubernetes. For recording what each step did, read how to audit AI agent actions in Kubernetes. The earlier introduction to the OttoFlow engine is OttoFlow: AI workflows for Kubernetes, and its project page is at /ottoflow-oss/. For the wider set of controls on AI agents running in clusters, start with the governing AI agents overview.

Frequently asked questions

What makes a Kubernetes AI workflow deterministic?

Only the judgement step calls a model. Every other step is a Kubernetes Job or a CEL expression, so the same inputs produce the same evidence and the same gate decision on every run. The model's output is still probabilistic, which is why it sits behind a gate and a fixed publish step instead of deciding what runs next.

Should an AI agent step use a Kubernetes service account?

Give it a named service account only if it must call the API. For a model step that only reads the evidence file its workflow hands it, set automountServiceAccountToken to false on its pod so no API token is mounted. Kubernetes then adds no token volume, and the step has no API access to misuse.

How many times does Kubernetes retry a failed Job step?

By default, the Job controller counts failed pods against spec.backoffLimit, which defaults to 6. It recreates failed pods with an exponential delay of 10, 20 and 40 seconds, capped at ten minutes. Set backoffLimit explicitly on each workflow step so a retry budget is reviewed in YAML rather than inherited from a default.

Can a CEL expression call an API or change state?

No. The CEL specification describes a program as side-effect-free and terminating, and says a CEL expression deterministically evaluates to either a value or an error for a given environment. That makes it safe as a gate between a collect step and an analyze step, because evaluating it cannot change the cluster.