Deploying an AI gateway in front of or behind your existing API gateway

Put an AI gateway in front of your API gateway when every agent call must pass the AI gateway’s checks before API policies run. Put it behind when the API gateway should stay the single entry point. Either way, agents need no route to model providers that skips both.
Checked against the Envoy and Kubernetes documentation on their main branches, fetched 2026-11-14. The config below uses the Gateway API v1 resource version shown on the Kubernetes page. This post does not test it on a running cluster.
Two placements, two different first hops
The placement decides which component sees an agent’s request first. That component is the only one that can read the prompt, the tool name and its arguments before anything else acts on them.
If you are still deciding whether you need a gateway at all, the AI gateway buyer’s guide covers what a gateway can and cannot govern. For the difference between a gateway and a policy decision point, read MCP gateway vs policy enforcement. The placement question below assumes you have already decided a gateway belongs in the path.
AI gateway in front
The AI gateway is the first hop for agent traffic. Model calls and MCP calls reach it before the API gateway, so it can apply content and tool rules, then forward what remains. The API gateway sees only the requests the AI gateway sends on, which means identity has to travel with them.
Envoy’s external authorization filter shows the mechanism. The filter calls an external service, returns a 403 when that service denies the request, and ingests metadata from the authorization response. If your AI gateway does not forward a per-agent or per-user identity, the API gateway sees one caller for every agent.
AI gateway behind
The API gateway stays the entry point. It terminates TLS, applies the routes and limits it already runs, and forwards model traffic to an internal AI gateway service. Your existing rules run first, and the AI gateway adds model-level rules afterwards.
The cost is depth. If the API gateway routes on path and headers only, prompt and tool-argument rules run one hop later, and only for requests the API gateway chose to forward. Check exactly what the first hop passes on before you rely on the second hop for content rules.
Comparison
| Question | AI gateway in front | AI gateway behind |
|---|---|---|
| First component to see an agent call | The AI gateway | The API gateway |
| Content rules (prompts, tool arguments) run | At the first hop | At the second hop, after API routing |
| Identity the API gateway’s rules see | Whatever the AI gateway forwards | The API gateway’s own authentication of the caller |
| Existing API policies cover | Only the calls the AI gateway sends on | Every call, before the AI gateway |
| Identity must be carried from | AI gateway to API gateway | API gateway to AI gateway |
| Bypass to test first | Calls that reach the API gateway without the AI gateway | Calls that reach the AI gateway without the API gateway’s rules |
Placement fails open without egress rules
A proxy only governs traffic that goes through it. A workload that can reach a model provider directly can skip either placement. In Kubernetes, that is the default for pods. A pod is non-isolated for egress unless a policy selects it, and then “all outbound connections are allowed.”
A pod becomes isolated for egress when a NetworkPolicy both selects it and lists Egress in its policyTypes. Once isolated, it can reach only what some egress rule allows. The rule takes effect only if the cluster’s network plugin enforces NetworkPolicy, which the Kubernetes page states as a requirement. Test that enforcement on a cluster before you count on it.
Where Gateway API fits
Kubernetes recommends Gateway API for new routing. The Ingress API is frozen, and it exposes only HTTP and HTTPS. Gateway API splits the job into a GatewayClass (which names the controller), a Gateway (the listener and address) and routes such as HTTPRoute. A Gateway accepts routes only from its own namespace unless its allowedRoutes setting says otherwise. That default matters when the AI gateway’s Service lives in a different namespace from the API gateway.
The design is role-oriented. Gateway API names three roles: an infrastructure provider, a cluster operator who manages network access and policies, and an application developer who composes Services. A Gateway may stand for a cloud load balancer or an in-cluster proxy, so the same kinds can describe either placement. Ingress needs a controller to do anything, and creating an Ingress resource alone has no effect. Gateway API also expresses header matching and traffic weighting directly, where Ingress needed custom annotations. That is why the Kubernetes project points new routing at Gateway API, and why the route definitions in this post use it.
Steps: place the gateway and test for bypass
- List every egress path agents use: model provider hosts, MCP server endpoints, and any local model servers, per cluster and on developer machines.
- Choose the placement from the comparison table. Write down the identity each hop receives and where it comes from.
- Attach the AI gateway’s route to a Gateway using
parentRefs. The Kubernetes example below routes one path on one hostname to a Service. Replace the names, the hostname and the Service port with your AI gateway’s values. Also replace the/loginpath prefix. It matches none of the paths model APIs use, so add one match for each model and MCP endpoint your agents call, and check each one. - If the route and the Gateway are in different namespaces, set
allowedRouteson the Gateway deliberately. Keepfrom: Sameif the route lives beside the Gateway.allowedRoutescontrols which namespaces can attach routes to the Gateway. It does not decide where a backend Service lives, so check that each backend reference resolves before you deploy. - Isolate agent pods for egress with a NetworkPolicy whose
egresslist allows only the gateway and cluster DNS. The Kubernetes page notes that a default deny-all egress policy also blocks DNS, so without a DNS rule the gateway hostname will not resolve. Confirm the cluster’s network plugin enforces the policy. - Test the bypass from an agent pod. Call a model provider’s host directly and expect the connection to fail. Then call it through the gateway and confirm the request is logged with the agent’s identity.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: example-httproute
spec:
parentRefs:
- name: example-gateway
hostnames:
- "www.example.com"
rules:
- matches:
- path:
type: PathPrefix
value: /login
backendRefs:
- name: example-svc
port: 8080
Fine print that breaks the design
Three details from the proxy documentation change how much a placement can enforce.
Routing can depend on request headers. Envoy’s router can match on path, prefix, arbitrary headers and cookies. A client controls its own headers, so do not let a header an agent can set decide whether its traffic reaches the gateway. Match on the route, and on an identity the gateway verifies itself. For the Claude Code side of this, see how to route Claude Code through an AI gateway.
External processing can see headers and bodies, but only if you configure body forwarding. Envoy’s processing mode defaults both the request and response body modes to NONE, so enabling the filter alone can leave prompts and tool arguments uninspected. Set the body modes explicitly and verify them. The processor can then change headers and bodies or return a new response. Its statistics include failure_mode_allowed, a count of errors that were ignored because of configuration. Alert on that counter. An ignored error is a request that was not checked.
External authorization has the same counter. Its failure_mode_allow setting lets error responses through when set to true, and the failure_mode_allowed statistic counts them. The fetched overview does not state the default for either setting, so check the reference page for your version before you deploy.
Enforcing this with AIControls
AIControls governs Claude Code through an AI gateway, for both the CLI and VS Code. It issues per-user tokens and sets per-user and per-session cost and budget limits, so the identity your placement carries can map to a person and a budget. The placement is a design choice that the steps above help you make, and the bypass test in step 6 applies whichever placement you pick. For the runtime side of agent controls, see admission control vs runtime enforcement for AI agents. See the AIControls platform page for the product, and the governing AI agents and MCP security pages for the wider controls.
Sources
- Kubernetes: Gateway API
- Kubernetes: Ingress
- Kubernetes: Network Policies
- Envoy: HTTP routing
- Envoy: external authorization filter
- Envoy: external processing filter
Frequently asked questions
Should an AI gateway sit in front of or behind an existing API gateway?
Put it in front when every agent call must pass the AI gateway's checks before the API gateway's policies run. Put it behind when the API gateway must stay the single entry point and route model traffic to an internal AI gateway service. Choose on the question of which hop must see the prompt and tool arguments, and which hop must carry the agent's identity.
What happens to agent identity when the AI gateway is in front?
The API gateway sees whatever the AI gateway forwards. If the AI gateway calls the API gateway with a shared service credential, every agent looks the same to the API gateway's own rules. Pass the per-agent or per-user identity as a header or token the API gateway checks, and test that it arrives on every route.
Does an AI gateway matter if agents can still reach model providers directly?
Not for enforcement. An agent that can reach a provider's API without passing the gateway is not governed by the gateway, whatever the placement. In Kubernetes, pods can send outbound traffic anywhere by default. Isolate agent pods for egress with a NetworkPolicy, and confirm the cluster's network plugin enforces NetworkPolicy.
Should new routes use Ingress or Gateway API?
Use Gateway API. The Kubernetes project recommends Gateway over Ingress, and the Ingress API is frozen with no further changes planned. Ingress only exposes HTTP and HTTPS. Gateway API resources such as Gateway and HTTPRoute let you attach routes from other namespaces, but only when the Gateway's allowedRoutes setting allows it.