LLM Gateway vs LLM Router

What an LLM gateway does

An LLM gateway is a single entry point between your applications and the model providers they use. Instead of every service holding its own provider keys and client-library quirks, everything calls the gateway, and the gateway forwards each request to the provider.

The gateway's job is mostly operational. Typical responsibilities include:

• One endpoint and one API format. Applications speak a common format, often OpenAI-compatible, and the gateway translates for each provider.
• Authentication and keys. Provider credentials stay in one place; teams and services get their own gateway keys instead.
• Rate limits and budgets. Limits per key, team or user stop one service from exhausting a shared quota or budget.
• Logging and usage tracking. One place to see which service called which model, how many tokens it used and what it cost.
• Retries and fallbacks. When a provider errors or times out, the gateway retries or sends the request to a backup.
• Caching. Exact repeats can be answered from a cache, and provider prompt-caching hints can be passed through.

A gateway answers the question "how do requests reach models safely and consistently?" It doesn't necessarily decide which model is right for a given request: often the caller still names the model.

What an LLM router does

An LLM router decides which model serves each request. The caller might send a generic routing model name, and the router picks the underlying model based on rules, price, latency or the task itself. What is an LLM router? covers the approaches in detail.

A router answers a different question: "which model should handle this request?" Its value shows up in quality, speed and cost per request, not in key management or access control.

Gateway vs router, side by side

The overlap is real. Fallback from one model to another is both a reliability feature and a routing decision, and caching affects both cost and which model is worth calling.

Most products combine both

In practice, few tools are only one or the other. A few well-known examples, described from their own documentation:

• LiteLLM is an open-source proxy and Python library that gives one OpenAI-format interface to many providers. Its proxy adds virtual keys, budgets, rate limits and spend tracking, and its router offers strategies such as weighted shuffling, latency-based and cost-based selection, with fallbacks. It also documents an Auto Router add-on, labelled beta. It is designed primarily to be self-hosted by platform teams.
• Portkey describes itself as an AI gateway with observability, guardrails and prompt management. Routing is set through configs with fallback, load-balancing and conditional modes, where conditions use request metadata, parameters or the URL path. Portkey is now part of Palo Alto Networks' Prisma AIRS.
• OpenRouter is a hosted unified API for models from many providers, with provider routing, model fallbacks and an Auto Router that classifies the prompt before choosing a model.

Each puts the emphasis in a different place: self-hosted control, enterprise governance or catalog breadth. For side-by-side notes with AirLLM, see the comparisons hub.

AI gateway, LLM gateway, model gateway: the terms

These names are used loosely and often interchangeably. LLM gateway usually means a gateway for text and chat models. AI gateway tends to be broader and may also cover embeddings, images, audio, and sometimes security features such as guardrails and data-loss checks. Model gateway is a generic synonym.

When you evaluate a product, ignore the label and check the features: which API formats it accepts, whether it chooses models or only forwards them, how it handles failures, and what it logs.

Gateways, routers and the control plane

Larger setups often split the system into a data plane and a control plane. The data plane carries the requests themselves: the gateway and router handling traffic. The control plane is where people set the rules: which models are allowed, budgets, keys, routing policies and the dashboards that show what happened.

Keeping these separate means you can change a budget or a routing policy without redeploying applications. AirLLM's platform overview shows how a coding agent, the routing layer, model compute and the control plane fit together.

Which do you need?

• You mainly need control. Many teams share provider access and you want one place for keys, limits and logs: start with gateway features, and keep model choice explicit.
• You mainly need cost and quality per request. Requests vary widely in difficulty, as they do inside agents: you need routing, because a gateway that forwards a fixed model name won't change what each request costs.
• You run coding agents or other multi-step agents. You need both, plus routing that understands sessions: steps that belong to one task, prompt caches that are expensive to rebuild, and loops that can run up a bill. See what an AI coding agent router does.
• You need strict data residency or air-gapped hosting. Look at self-hosted options first, and check where prompts and logs are stored.

What to check before you choose

Whether you pick a gateway, a router or a product that does both, the same practical questions decide whether it fits your stack:

• API formats. Does it accept what your tools already send, such as OpenAI Chat Completions, the OpenAI Responses API or Anthropic Messages, with streaming and tool calls intact?
• Keys and teams. Can you issue separate keys per service or team, and revoke one without touching the others?
• Data handling. What is logged, where, and for how long? Can you turn content logging off? The provider that finally answers a request processes it under its own terms, so check that too.
• Failure behaviour. What happens on provider errors, timeouts and rate limits, and can you see when a fallback was used?
• Visibility. Can you see which model served each request and what it cost, and export that data?
• Overhead and hosting. How much latency does it add, and do you run it yourself or use a hosted service?
• Portability. If you leave, how much configuration has to change? Standard API formats keep that to the base URL, the key and the model name.

Where AirLLM fits

Air Router is AirLLM's intelligent routing layer, and it also covers the gateway basics a routing layer needs. Coding tools, agents and apps connect to one endpoint with one AirLLM API key, using the OpenAI-compatible endpoint https://api.airllm.ai/v1 or, for Claude Code, Anthropic's Messages format. Usage shows in the AirLLM dashboard, and budgets help keep spending in check.

The routing side is where Air Router focuses: it matches each request or agent step to a suitable model, considers existing prompt caches before switching models, reduces repeated context and helps stop runaway agent loops, with fallback to cloud model providers when needed. Pick a routing model such as airllm-auto to route across supported providers.

Frequently asked questions

Related

Sources

• LiteLLM — project README
• LiteLLM docs — Routing
• LiteLLM docs — Virtual keys and budgets
• Portkey docs — Configs
• Portkey docs — Conditional routing
• Palo Alto Networks — Acquisition of Portkey completed (May 29, 2026)
• OpenRouter docs — FAQ
• OpenRouter docs — Auto Router