What Is an LLM Gateway? Architecture, Cost Control, and When You Need One

An LLM gateway is a service that sits between your applications and the model providers behind them, exposing one API in place of many. Instead of every application holding its own OpenAI, Anthropic, and Google credentials and its own retry logic, calls go to the gateway, and the gateway decides which model answers, what it costs, and what gets recorded.

That is the whole idea. Everything else in this guide is a consequence of it.

The term shows up interchangeably with AI gateway, and in most 2026 product marketing the two mean the same thing. Where a distinction exists, "AI gateway" tends to be the broader label covering embeddings, image models, and speech alongside text generation, while "LLM gateway" points specifically at text-generation traffic.

Why does an LLM gateway exist at all?

The first model integration is easy. You install a provider SDK, put a key in an environment variable, and ship.

The problems start at the second one. Providers do not agree on request shape, authentication, streaming format, or error semantics. As one architecture write-up puts it, LLM calls "stream responses, bill per token instead of per request, throw provider-specific errors" (Collin Wilkins). A traditional API gateway was not designed for any of that. It counts requests, not tokens, and it has no opinion about which model should answer.

So teams write their own adapter layer. Then they write it again in the next service. Six months later there are four copies with different retry behaviour, and nobody can answer what the company spent on inference last month or which team spent it.

The gateway is the consolidation of that layer into one place. TrueFoundry frames the core of it as "a single API in front of every model provider, so you stop maintaining separate integrations for each one."

The five problems it actually solves

One interface across providers. Applications call one endpoint with a model identifier, the same principle behind no-code agent builders that abstract the provider away. Swapping the underlying model becomes a configuration change rather than a code change. Inworld's teardown notes the reason plainly: OpenAI, Anthropic, and Google "use different authentication methods, request formats, and response structures."

Routing and failover. When a provider degrades or rate-limits you, the gateway can retry the same request against a different model without the calling application knowing. This is the reliability argument, and for anything customer-facing it is usually the one that gets the project approved.

Cost control. Per-team keys, per-key budgets, and hard spend caps live at the gateway. So does caching, which serves repeat requests without paying for them twice. The wider cost-savings picture usually involves more than routing.

Observability. One place where every prompt, completion, token count, latency figure, and error is recorded, which is the same reasoning behind tracking agent performance metrics. Without it, per-provider dashboards have to be reconciled by hand, and they do not share a request identifier.

Governance. Key management, redaction of sensitive fields, and policy checks apply uniformly instead of being reimplemented per service. For regulated teams this connects directly to data privacy compliance.

How does an LLM gateway work?

The request path is short and worth understanding precisely, because most of the value sits in steps three and six.

1. The application calls the gateway using one schema. Most gateways expose an OpenAI-compatible endpoint, because that shape is the closest thing the ecosystem has to a default.
2. The gateway authenticates the caller using a gateway-issued key, not a provider key. Provider credentials stay in the gateway.
3. A routing rule selects the target model. This can be static, or conditional on the requested model name, the calling team, cost ceilings, prompt length, or current provider health.
4. The request is translated into the target provider's format.
5. The response streams back, translated into the caller's expected shape.
6. Everything is recorded: tokens in and out, computed cost, latency, cache status, and outcome.

On failure, the gateway can retry against a fallback chain rather than surfacing the error. That is the difference between a provider incident being an internal event and a customer-visible outage.

LLM gateway compared with a traditional API gateway

The row that matters most is billing. A traditional gateway can tell you a service made 40,000 calls. It cannot tell you that 3% of them consumed 60% of the budget because one prompt template grew a large context window. That question is the reason the category exists.

When do you actually need one?

The honest answer is that a single application calling a single model does not need a gateway. The abstraction costs you a network hop and an operational dependency, and buys little.

The trigger points are specific:

A second model reaches production. This is the clearest one. The moment two providers are live, somebody is maintaining translation logic.

Spend stops being attributable. When finance asks which team drove last month's bill and the answer takes a week, the logging problem has become a governance problem.

An outage becomes customer-visible. If a provider incident takes your feature down and there was a capable substitute model available, the failover argument pays for the project on its own.

Compliance enters scope. Covered in more depth in the enterprise AI buyer's checklist. Redaction, retention, residency, and audit logging are far cheaper to implement once at a chokepoint than in every service.

Prompt caching becomes material. At high volume with repetitive prompts, cache hit rates translate directly into money.

If none of those apply, a provider SDK and a thin internal wrapper is a reasonable answer, and you can add a gateway later. The migration is not especially painful, precisely because the gateway's job is to look like the thing you were already calling.

What should you evaluate in an LLM gateway?

Buyer guides in this category tend to compare feature checklists. Feature lists converge quickly, so the differences that survive contact with production are narrower than the marketing suggests.

Latency overhead. The gateway is on the critical path of every call. Vendors publish single-digit or low-double-digit millisecond figures, and those numbers are worth verifying against your own traffic rather than taking on trust, particularly at the tail.

Failure behaviour when the gateway itself is down. Ask what happens when the control plane is unreachable. Some designs fail open to a direct provider call, some fail closed. Both are defensible, and you need to know which you bought.

Streaming fidelity. Token-level streaming through a translation layer is where subtle bugs live. Test it with your actual client, not a curl command.

Cost attribution model. Per-key, per-team, per-user, or per-request tagging. If the granularity does not match how your finance team allocates cost, the observability is decorative.

Self-hosted or managed. Self-hosted options keep prompts inside your perimeter and remove a vendor from the critical path, at the cost of running it. Managed services remove the operational burden and add a dependency and, usually, a data-handling question.

Model coverage that matches your roadmap. Coverage counts are a weak signal, the same way connector counts are. What matters is whether the three or four models you actually intend to run are first-class, including their newer parameters.

Governance depth. Redaction, prompt logging controls, retention windows, and model allowlists per team.

What does the 2026 landscape look like?

There is a real ecosystem here across three shapes. Pricing and feature claims in this category go stale within weeks, so rather than restate figures, the links below go to each project's own documentation. Read the primary source and check the pricing page on the day you evaluate.

Self-hosted and open source

You run it, prompts stay inside your perimeter, and there is no third party on the critical path. The trade is that you own the upgrades.

LiteLLM is the most widely deployed option in this shape, exposing an OpenAI-compatible proxy in front of a large provider set with per-key budgets and logging. Helicone approaches the same problem observability-first, and can be self-hosted or used as a managed service.

Managed commercial gateways

Someone else runs the control plane. You get the features sooner and accept a dependency plus a data-handling question worth reading carefully.

Portkey is the most feature-complete commercial option in this shape, combining routing, caching, fallbacks, and budgets. Cloudflare AI Gateway sits on their edge network and is compelling if your traffic already transits Cloudflare. Vercel AI Gateway targets teams already deployed there. OpenRouter is the aggregator-shaped variant, unifying billing across a very large model catalogue, which makes it strong for breadth of model access and weaker as an enterprise governance layer.

Cloud-provider and API-native routing

If your models already come from one cloud, some of the gateway's job is available without adding a component. Amazon Bedrock and Google Vertex AI both expose multiple model families behind one interface with the cloud's own IAM and billing attached. Kong extends a conventional API gateway with AI-specific plugins, which is the natural path if Kong is already your ingress. Provider APIs like the Claude API remain the thing underneath all of it.

The trap with cloud-native routing is model coverage. It is excellent inside one vendor's catalogue and, by design, does not help you fail over to a competitor's model, which is often the specific reliability scenario that motivated the project.

How to actually choose

Shortlist on the six criteria above, then run a two-week trial with production-shaped traffic, measuring added latency at p50 and p99, cache hit rate on your real prompt mix, and whether the cost attribution reconciles with the provider invoice. That last check catches more problems than any feature matrix.

What routing strategies do gateways actually offer?

"Routing" covers several different mechanisms that get bundled under one word, and they have very different operational profiles.

Static mapping. The caller asks for a model alias and the gateway resolves it to a specific provider model. This is the least interesting form and the most used, because it delivers the main benefit: changing the underlying model without redeploying applications.

Fallback chains. An ordered list of models. If the first returns an error, times out, or hits a rate limit, the gateway tries the next. The design question is what counts as a failure worth falling back on. A 429 clearly qualifies. A 500 usually does. A slow but successful response is ambiguous, and gateways differ on whether a latency ceiling triggers a fallback or just gets recorded.

Cost-based routing. Send cheaper or shorter requests to a smaller model, and reserve the expensive model for requests that need it. The hard part is not the routing, it is the classification. Deciding which requests "need" the larger model is a quality judgement, and getting it wrong shows up as degraded output rather than an error, which makes it hard to detect.

Load balancing across keys. Spreading traffic over multiple provider accounts to raise effective rate limits. Useful at scale, and worth confirming it does not violate your provider agreement.

Health-aware routing. The gateway tracks error rates and latency per provider and shifts traffic away from a degrading one before requests start failing. This is the most valuable form and the least common, because it requires the gateway to hold real state about provider behaviour.

A practical warning about fallback chains: they change your cost profile silently. If your primary is a cheap model and your fallback is an expensive one, a provider incident can multiply your bill while the application reports everything as healthy. Alert on fallback rate, not just on error rate.

How does caching work in an LLM gateway?

Caching is where the cost argument usually gets made, and it is worth being precise about what is being cached, because two very different mechanisms share the name.

Exact-match caching hashes the request, including the model and parameters, and returns a stored response on a repeat. It is safe, predictable, and only helps when your traffic genuinely repeats. For a chat product where every prompt is unique, the hit rate approaches zero. For a classification or extraction pipeline running over a bounded set of inputs, it can be very high.

Semantic caching embeds the prompt and returns a stored response when a new prompt is sufficiently similar. The hit rate is far higher, and so is the risk. Two prompts can be semantically close and still require different answers, and the failure mode is a confidently wrong response rather than an error. If you enable it, set the similarity threshold conservatively and sample the hits for correctness before trusting it in a customer-facing path.

There is also provider-side prompt caching, which is a different thing again. Several providers now discount repeated prefixes, which rewards structuring prompts with the stable content first. A gateway can help here by keeping prompt construction consistent, but the discount comes from the provider, not the gateway.

The honest summary is that caching saves real money on repetitive workloads and very little on conversational ones. Measure your prompt uniqueness before building a business case on it.

What does good observability look like?

Every gateway claims observability. The useful test is whether it can answer these five questions without an export and a spreadsheet:

1. What did each team spend last month, and on which models?
2. Which prompt templates consume the most tokens per call, and has that changed?
3. What is the p99 latency per model, separated from the gateway's own overhead?
4. What proportion of requests fell back, and to what?
5. Which requests failed, with the full prompt and the provider's error, retained long enough to debug?

Question two is the one that catches the most expensive surprises. Context windows grow quietly as features get added to a prompt template, and per-call token counts drift upward without anyone deciding to spend more.

Question five carries a governance tension worth resolving deliberately. Retaining full prompts and completions makes debugging tractable and creates a data-protection surface, particularly if user content is included. Most gateways offer redaction or logging controls; decide the policy before you turn logging on rather than after.

Should you build or buy?

Building a thin internal gateway is genuinely reasonable for a small number of models and one team. The first version is a few hundred lines: a request translation layer, a key store, and structured logging.

What makes teams regret building is rarely the first version. It is the accumulation of the rest: streaming edge cases per provider, retry semantics that differ by error type, per-team budgets, a cache with an invalidation story, provider parameter changes as models are deprecated and replaced, and a dashboard somebody has to maintain. None of it is hard, and all of it is ongoing.

A defensible rule: build if the gateway is a routing convenience, buy if it is becoming a control plane. The moment finance, security, or compliance has a requirement of it, the maintenance burden stops being an engineering side project.

If you do buy, self-hosting an open-source gateway is a middle path that keeps prompts inside your perimeter without you owning the code.

How do you adopt one without a risky migration?

The migration is usually less disruptive than expected, because the gateway's job is to imitate the interface you already call.

A sequence that works:

1. Stand it up in shadow. Route a copy of production traffic through the gateway without using its responses. Compare outputs and measure the added latency.
2. Move one low-risk service. An internal tool or a batch job, not the customer-facing path.
3. Turn on logging and attribution first, before routing or caching. The observability alone usually justifies the project, and it is the lowest-risk feature.
4. Add fallback chains next, with alerting on the fallback rate so you learn what your real provider reliability looks like.
5. Introduce cost routing and caching last, because they are the two features that can change output quality rather than just availability.

Teams that reverse this order, starting with semantic caching and cost routing because those have the clearest savings story, tend to spend their first month debugging quality regressions and lose confidence in the whole layer.

Where an LLM gateway stops being enough

A gateway solves the plumbing between an application and a model. It does not solve what the model knows.

This distinction gets lost when teams evaluate the category. A gateway routes a request to Claude or GPT and records what it cost. It does not give that model your Salesforce pipeline, your Jira history, your Slack decisions, or the context of who is asking and what they were working on last week. Those are separate problems, and the second one is usually the reason an AI project fails to produce value even when the infrastructure works.

Two capabilities sit above the gateway layer:

Connection to systems of record. The model needs the data. Model Context Protocol has become the common standard for this, and gateways and MCP are complements rather than substitutes. One handles which model answers, the other handles what the model can reach.

Organizational context. Beyond raw access, an agent needs to know what your company already decided, who owns what, and how work actually flows. That is a memory problem, not a routing problem, and it is why enterprise AI knowledge management is a separate category from infrastructure.

Coworker AI works at that second layer. It connects to 50+ tools, keeps organizational memory across them, and runs agents that act on what they find rather than only retrieving it. Plans are Pro at $29.99 per user per month, Max at $149.99, and Enterprise pricing on request. If you already run a gateway, the two sit at different points in the stack and do not conflict.

Book a demo if you want to see how organizational context changes what agents can do.

Frequently asked questions

What is an LLM gateway?

An LLM gateway is a service that sits between your applications and the model providers you use, exposing a single API in place of many. It handles routing between models, failover when a provider degrades, spend limits, caching, and logging of tokens and cost.

What is the difference between an LLM gateway and an API gateway?

A traditional API gateway routes on path and host, bills by request, and retries the same upstream on failure. An LLM gateway routes on model and cost, understands token-based billing and streamed responses, and can fail over to a different model entirely. It also records prompts, completions, and computed spend, which a conventional gateway has no concept of.

Is an AI gateway the same thing as an LLM gateway?

In most 2026 product marketing they are used interchangeably. Where a distinction is drawn, "AI gateway" is the broader term including embeddings, image, and speech models, while "LLM gateway" refers specifically to text-generation traffic.

Do I need an LLM gateway for a single model?

Usually not. One application calling one provider gains little from the extra hop and the operational dependency. The case becomes clear once a second model reaches production, once spend needs attributing per team, or once a provider outage would be visible to customers.

Does an LLM gateway reduce AI costs?

It can, through three mechanisms: caching repeated requests, routing cheaper requests to smaller models, and enforcing hard spend caps per team. The size of the saving depends entirely on your prompt mix. Highly repetitive workloads benefit most, and workloads with unique long-context prompts benefit least.

Can an LLM gateway increase latency?

Yes. It adds a network hop and a translation step to every call. Well-engineered gateways keep the overhead small, but you should measure it on your own traffic at p99 rather than relying on published figures, because tail latency is where the difference shows.

Does an LLM gateway give models access to company data?

No. A gateway governs which model answers and what the call costs. It does not connect the model to your systems of record. That is a separate layer, usually handled through Model Context Protocol servers or a platform that maintains organizational context.

How does an LLM gateway relate to agent orchestration?

They solve adjacent problems. A gateway governs a single model call: which model, what it cost, whether it succeeded. Agent orchestration governs a sequence of calls and tool uses working toward a goal, including state between steps. Orchestration layers usually sit above a gateway and call through it.

Should the gateway or the application handle prompt templates?

Keep templates in the application. A gateway that owns prompt content becomes hard to version alongside the code that depends on it, and it blurs the line between infrastructure and product logic. The gateway should govern the call, not compose it. The exception is guardrail text injected for policy reasons, which belongs at the chokepoint precisely because it must apply uniformly.

What is the fastest way to reduce model spend without a gateway?

Three things, in order of effect: shorten system prompts, which are paid for on every call; move repetitive extraction and classification work to a smaller model; and check whether your provider offers prefix caching that your prompt structure is failing to trigger. None of these need new infrastructure, and they are worth doing before you add a component.

Related reading

• ChatGPT MCP: How to Connect MCP Servers to ChatGPT
• 12 Claude Code MCP Servers Every AI Team Needs
• Claude API Pricing: A Complete Guide for AI Teams
• OpenRouter Pricing: Cost-Saving Tips for AI Teams
• DeepSeek API Pricing: The Full Cost Breakdown
• The Cheapest Way to Access Claude, GPT, and Gemini
• AI Agent Orchestration Platforms, and How to Choose
• Agent Workflows Explained
• Enterprise AI Platform: The Complete Buyer's Guide
• Enterprise AI Buyer's Checklist: 10 Questions