> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Virtual Models
> Route requests across multiple model providers with load balancing, failover, and retries using a single model name.
A **virtual model** is a named entry in TrueFoundry AI Gateway that your application calls like any other model (for example `my-group/production-chat`). Behind that name, you configure **one routing strategy** and **one or more real target models** (for example `azure/gpt-4o` and `openai/gpt-4o`). The AI Gateway handles load balancing, health-aware routing, retries, and fallbacks automatically so you do not hard-code provider details in every service.
Virtual models only apply to **synchronous** API calls (chat completions, completions, embeddings, responses, rerank, image/audio, etc.). The [Batch API](/docs/ai-gateway/batch-predictions-with-truefoundry-llm-gateway) (`/batches`) does **not** support virtual models — batch jobs run on a single provider asynchronously, so the AI Gateway has no opportunity to fallback to another target if a request inside the batch fails. Send batch requests directly to a real catalog model (for example `openai-main/gpt-4o`).
You can set [Budget Limiting](/docs/ai-gateway/budget-limiting-v2) rules on a virtual model id in **Models** so spend is capped for the facade name clients call. See [Virtual models in when.models](/docs/ai-gateway/budget-limiting-v2#virtual-models-in-whenmodels).
```mermaid theme={"dark"}
flowchart LR
App[Your application] --> VM["model: my-group/production-chat"]
VM --> GW[AI Gateway]
GW --> T1["azure/gpt-4o"]
GW --> T2["openai/gpt-4o"]
GW --> T3["anthropic/claude-3"]
```
## Why route across multiple targets?
Production LLM traffic benefits when the AI Gateway can choose or failover among more than one backend. Common drivers:
Model providers experience outages and downtimes. For e.g. here's a screenshot of OpenAI's and Anthropic's status page from Feb to May 2025.
To avoid the downtime of your applications when models go down, a lot of organizations use multiple model providers, and configure load balancing to route to the healthy model in case one of the models goes down, hence avoiding any downtime of their applications for their users.
Latency and performance varies on time, region, model and provider. Here's a graph of the latency variance of a few models over a course of a month.
We want to be able to route dynamically to the model with the lowest latency at any point in time.
A lot of the LLM providers enforce strict rate limits on API usage. Here's a screenshot of Azure OpenAI's rate limits:
When these limits are exceeded, requests begin to fail and we want to be able to route to other models to keep our application running.
Testing new models or updates in production carries significant risks. Dynamic load balancing can be used to route a small percentage of traffic to the new model, and monitor the performance before routing all the traffic to the new model.
## Why Virtual Models?
* **Stable API surface** — Your apps pass one model identifier; you change targets, weights, or providers in the AI Gateway without redeploying clients.
* **Resilience** — Retries and fallback status codes route around rate limits and transient errors across targets.
* **Governance** — Virtual model provider groups support [collaborator roles](/docs/ai-gateway/gateway-access-control) so teams can **use** or **manage** routing separately.
## Supported API Types
A virtual model is configured with one or more **model types** (set under **Model types** when you create it). The model type controls which API operations the virtual model can serve — and every target in the virtual model must support the operation you call.
TrueFoundry AI Gateway supports the following model types, each callable on the matching OpenAI-compatible endpoint:
| Model type | Endpoint(s) | Description |
| --------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `chat` | `POST /chat/completions`, `POST /messages` | Chat completions and the Anthropic [Messages API](/docs/ai-gateway/messages-overview) |
| `responses` | `POST /responses` | OpenAI [Responses API](/docs/ai-gateway/responses-api) |
| `completion` | `POST /completions` | Legacy text completions |
| `embedding` | `POST /embeddings` | Text embeddings |
| `rerank` | `POST /v2/rerank` | Document [reranking](/docs/ai-gateway/rerank) |
| `moderation` | `POST /moderations` | Content [moderation](/docs/ai-gateway/moderation) |
| `image` | `POST /images/generations`, `POST /images/edits`, `POST /images/variations` | Image generation, edits, and variations |
| `text_to_speech` | `POST /audio/speech` | [Text-to-speech](/docs/ai-gateway/text-to-speech) |
| `audio_transcription` | `POST /audio/transcriptions` | Speech-to-text |
| `audio_translation` | `POST /audio/translations` | [Audio translation](/docs/ai-gateway/audio-translation) |
\*\*`messages` can be served by a virtual model with `chat`.
A single virtual model can enable **multiple** model types (for example `chat` and `embedding`), as long as every target supports each operation you intend to call.
The `responses` type behaves differently from the rest: conversations are pinned to the backend that started them, and every target must support the Responses API natively. See [Responses API conversations](#responses-api-conversations) in the FAQ.
## Routing Strategies
When you create a virtual model, you choose **one** routing strategy. Each strategy uses the same list of targets — the difference is how the AI Gateway picks among healthy targets for each request.
| Strategy | How it works? | Best for |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| [**Weight-based**](#weight-based-routing) | Distributes traffic by assigned weights (e.g. 80/20). Also supports [sticky routing](#sticky-routing-weight-based-only) to pin sessions to a target. | Canary rollouts, fixed capacity splits, A/B allocation. |
| [**Priority-based**](#priority-based-routing) | Routes to the highest-priority healthy target (0 = highest). Falls back to the next priority model on failure. Supports [SLA cutoff](#sla-cutoff). | Primary + backup topologies, cost optimization. |
| [**Latency-based**](#latency-based-routing) | Per-caller deterministic selection weighted by recent latency, sticky for 10-minute windows. No weights needed. | Performance chasing across regions or providers with prompt-cache stability. |
| [**Complexity-based**](/docs/ai-gateway/complexity-based-routing) | Classifies each request as `simple`, `medium`, or `complex` and routes it to the target configured for that tier. | Cost optimization — serving easy requests on a cheaper model without changing your application. |
### Weight-based routing
You assign a weight to each target. The AI Gateway distributes incoming requests in proportion to those weights. For example, 90% to `azure/gpt-4o` and 10% to `openai/gpt-4o`.
#### Sticky routing (weight-based only)
Sticky routing pins requests that share the same session key to the same target model for a configurable time window (`ttl_seconds`). Within that window, every request carrying the same session identifier is routed to the same model. When the window expires, the session is re-evaluated and may land on a different model. Useful for multi-turn conversations, prompt cache efficiency, and consistent user experience.
Add a `sticky_routing` block inside your weight-based routing configuration. Two fields are required: `ttl_seconds` and at least one entry in `session_identifiers`.
```yaml theme={"dark"}
routing_config:
type: weight-based-routing
sticky_routing:
ttl_seconds: 3600
session_identifiers:
- key: x-user-id
source: headers
load_balance_targets:
- target: provider-a/model-a
weight: 70
fallback_candidate: true
- target: provider-b/model-b
weight: 30
fallback_candidate: true
```
**Session identifiers** tell the AI Gateway which fields to read from the request to identify a session. All configured identifiers are combined into a single session key — you can mix headers and metadata fields.
* **`key`** — The header or metadata field name to read.
* **`source`** — `headers` to read from HTTP request headers, or `metadata` to read from request metadata.
```yaml theme={"dark"}
# Pin by a combination of user and conversation
session_identifiers:
- key: x-user-id
source: headers
- key: x-conversation-id
source: headers
# Pin using request metadata
session_identifiers:
- key: tenant-id
source: metadata
- key: user-id
source: metadata
```
If a configured identifier is missing from the request, it contributes an empty string to the session key. All requests missing that field will be treated as the same session. Make sure your clients always send the identifier fields you configure.
**TTL window:** `ttl_seconds` defines how long a session stays pinned. For chatbots, `3600` (1 hour) is a common starting point. For longer workflows, consider `86400` (24 hours).
**Fallback during a sticky session:** If the pinned model fails, the remaining healthy targets are tried in sequence (this fallback order is **not weight-based**). Only targets with `fallback_candidate: true` are eligible. After a successful fallback, subsequent requests for that session in the same TTL window are routed to the working target — not back to the one that failed.
### Priority-based routing
Each target has a priority number. The AI Gateway routes to the highest priority target (**0 is highest**) that is healthy. If that target fails or is unavailable, the AI Gateway falls back to the next priority model.
Priority-based routing supports **SLA cutoff** to automatically mark models as unhealthy when they breach performance thresholds. You can configure either or both of:
* **Time Per Output Token (TPOT)** threshold per target using `sla_cutoff.time_per_output_token_ms`
* **Time To First Token (TTFT)** threshold per target using `sla_cutoff.time_to_first_token_ms`
How the check works?
* The AI Gateway monitors the rolling average of each configured metric over a **3-minute window** (up to 10 samples, minimum 3 required)
* For TTFT, only streaming requests record a sample — non-streaming requests are skipped from the TTFT average
* For TPOT, only requests with at least **30 output tokens** count as valid samples — shorter responses are excluded because their per-token latency is dominated by TTFT rather than steady-state generation speed
* If either metric exceeds its threshold, the target is marked unhealthy and moved to the end of the list
* Recovery is automatic when metrics improve or older data ages out
SLA cutoff is only available for priority-based routing, not weight or latency based routing.
### Latency-based routing
You do not set weights. The AI Gateway picks a target for each caller **deterministically**, and keeps the same target for the duration of a short time window. This gives every caller a stable behaviour for prompt-cache reuse and reduces routing variance for agents and long conversations. Across many independent callers, traffic distributes in inverse proportion to each target's measured latency — faster targets get a larger share of overall traffic, but every healthy target still receives some.
**Stickiness is built-in.** Latency-based routing is automatically sticky-per-caller-per-epoch — you do **not** need to add a `sticky_routing` block to enable this. The behaviour applies as soon as you choose `latency-based-routing` as the rule type.
The selector runs three steps on every request:
1. **Measure each target's recent latency.** For each target, the AI Gateway looks at recent successful requests over the **last 20 minutes** and computes the **Time Per Output Token (TPOT)** — total response time divided by the number of output tokens. TPOT folds time-to-first-token and inter-token latency into a single, output-length-independent number. Targets that don't have recent samples are treated as average so they aren't penalised before they've had a chance to be measured.
2. **Pick a target per caller, sticky for the epoch.** Selection is based on the caller's identity and the current **10-minute epoch**. Lower-latency targets (lower TPOT) are more likely to be picked, but every healthy target retains some share. Because the selection inputs are stable for the duration of an epoch, the **same caller routes to the same target for up to 10 minutes** — this is the sticky part of the algorithm, and is what gives you prompt-cache reuse and lower routing variance within a session.
3. **Order the remaining targets for fallback.** Lower-latency targets come first in the fallback chain. If the primary target fails or returns a fallback-status-code response, the AI Gateway tries the next target in the chain.
When the epoch rolls over (every 10 minutes), the same caller may be routed to a different target — so the assignment is stable for long enough to benefit caching, but not so long that the routing decision becomes stale.
### Complexity-based routing
Each target is assigned a complexity tier — `simple`, `medium`, or `complex` — and the AI Gateway classifies every incoming request into one of those tiers before choosing a target. Easy requests are served by your cheapest model and hard ones by your most capable, without your application changing anything.
Classification is either an in-process heuristic (default, no added latency or cost) or a call to a fast classifier model you nominate. Requests can also be pinned to a tier for the duration of a session.
This is the one strategy that reads the request body to make its decision, so it applies only to virtual models with `chat`, `completion`, or `responses` model types, and only to virtual models — not the tenant-level routing config.
See [Complexity-based Routing](/docs/ai-gateway/complexity-based-routing) for tiers, classifier options, escalation, sticky routing, and the full configuration reference.
## Configuration structure
The following YAML shows the complete shape of a virtual model's routing configuration with all available fields. In the dashboard UI, the same fields are set through the form editor. For the complexity-based shape — which uses per-target tiers and a classification strategy instead of weights or priorities — see the [Complexity-based Routing reference](/docs/ai-gateway/complexity-based-routing#configuration-reference).
```yaml YAML theme={"dark"}
routing_config:
type: weight-based-routing | latency-based-routing | priority-based-routing
# Sticky routing (weight-based only)
sticky_routing:
ttl_seconds: integer # how long a session stays pinned (seconds)
session_identifiers:
- key: string # header or metadata field name
source: headers | metadata # where to read the session key from
load_balance_targets:
- target: string # model identifier in the AI Gateway (e.g. azure/gpt-4o)
# Routing strategy fields (mutually exclusive)
weight: integer # 0–100, sum to 100 (weight-based only)
priority: integer # lower = higher priority (priority-based only)
# Retry configuration
retry_config:
attempts: integer # retries on the SAME target; default: 0
delay: integer # ms between retries; default: 100
on_status_codes: string[] # codes that trigger retry; default: ["429","500","502","503"]
# Fallback configuration
fallback_status_codes: string[] # codes that trigger trying a DIFFERENT target
# default: ["401","403","404","408","429","500","502","503"]
fallback_candidate: boolean # eligible to receive fallback traffic from other targets?
# default: true
# SLA cutoff (priority-based only). Configure either or both fields.
sla_cutoff:
time_per_output_token_ms: integer # TPOT threshold; target marked unhealthy when exceeded
time_to_first_token_ms: integer # TTFT threshold; target marked unhealthy when exceeded
# (averaged across streaming requests only)
# Metadata-based target filtering
metadata_match: # key-value pairs that must match resolved request metadata
key1: value1 # all pairs must match (AND logic)
key2: value2
# Header overrides
headers_override:
set: # headers to add or overwrite on the outgoing request
header-name: header-value
remove: # headers to strip from the outgoing request
- header-name
# Override parameters
override_params:
temperature: number # per-target temperature override
max_tokens: integer # per-target max_tokens override
prompt_version_fqn: string # per-target prompt version for hydration
```
### Key fields
**`type`** — The routing strategy for this virtual model:
* **`weight-based-routing`** — Distribute traffic by assigned weights that sum to 100.
* **`latency-based-routing`** — Per-caller deterministic selection weighted by recent latency, sticky for 10-minute windows so the same caller stays on the same target within that window. No weights needed.
* **`priority-based-routing`** — Route to the highest priority (lowest number) healthy target, falling back to the next on failure.
* **`complexity-based-routing`** — Classify each request as `simple`, `medium`, or `complex` and route it to the target configured for that tier. See [Complexity-based Routing](/docs/ai-gateway/complexity-based-routing).
**`load_balance_targets`** — The list of real models eligible for routing. Each target and its configuration options are described in detail in the [Per-target configuration](#per-target-configuration) section below.
## Per-target configuration
Regardless of which routing strategy you choose, each target in the virtual model supports several options that control what happens when a request is routed to that target.
### Retries and fallbacks
Each target can define how the AI Gateway should handle failures before giving up or moving to another target:
* **Retry configuration** — Number of attempts, delay between retries, and which status codes trigger a retry on the **same** target. Defaults: 0 attempts, 100 ms delay, retry on `429`, `500`, `502`, `503`.
* **Fallback status codes** — Which status codes cause the AI Gateway to stop retrying this target and try a **different** target instead. Default: `401`, `403`, `404`, `408`, `429`, `500`, `502`, `503`.
* **Fallback candidate** — Whether this target is eligible to receive traffic when **another** target fails. Default: `true`. Set to `false` when you want a target to be used only as a primary and never receive fallback traffic from other targets.
```yaml theme={"dark"}
routing_config:
type: priority-based-routing
load_balance_targets:
- target: azure/gpt-4o
priority: 0
retry_config:
attempts: 3
delay: 200
on_status_codes: ["429", "500", "503"]
fallback_status_codes: ["429", "500", "502", "503"]
fallback_candidate: true
- target: openai/gpt-4o
priority: 1
retry_config:
attempts: 2
delay: 100
fallback_status_codes: ["429"]
fallback_candidate: true
- target: anthropic/claude-sonnet
priority: 2
fallback_candidate: false # only used as primary, never receives fallback traffic
```
In this configuration:
* A request first goes to `azure/gpt-4o` (priority 0). If it returns `429`, the AI Gateway retries up to 3 times with 200 ms delay. If retries are exhausted or a fallback status code is returned, it falls back to the next target.
* `openai/gpt-4o` (priority 1) is tried next with its own retry config.
* `anthropic/claude-sonnet` (priority 2) has `fallback_candidate: false`, so it is **never** tried as a fallback for the other two targets — it is only used when it is itself the highest-priority healthy target.
### Header overrides
You can inject or remove HTTP headers on a per-target basis, applied just before the request is sent to that model. This is useful when a specific target requires headers that the others don't — for example, a region identifier, a deployment ID, or an API version header expected by one provider but not the rest.
Add a `headers_override` block to any target in your `load_balance_targets` list:
```yaml theme={"dark"}
routing_config:
type: weight-based-routing
load_balance_targets:
- target: provider-a/model-a
weight: 80
headers_override:
set:
x-region: us-east-1
remove:
- x-internal-debug
- target: provider-b/model-b
weight: 20
headers_override:
set:
x-region: eu-west-1
```
* **`set`** — Key-value pairs of headers to add or overwrite on the outgoing request.
* **`remove`** — List of header keys to strip from the outgoing request.
Header keys are **case-insensitive** — `X-Custom-Auth` and `x-custom-auth` refer to the same header. The AI Gateway normalises all keys to lowercase before applying overrides. Header overrides are applied **last**, after parameter overrides, so they reflect the final outgoing headers sent to the provider.
`x-tfy-anthropic-beta` is an exception: it is consumed into the Anthropic beta pipeline and is not forwarded upstream. When that header is present on the request (or set here), it **replaces** every other client beta source and is not merged. Setting or removing `anthropic-beta` in `headers_override` does not override it — to change the betas, set `x-tfy-anthropic-beta`.
### Metadata-based target filtering
You can constrain a target to only receive traffic when request metadata matches specific key-value pairs using `metadata_match`.
The AI Gateway evaluates **resolved metadata** (not just raw request headers). Metadata can come from:
* **Request metadata header** — `x-tfy-metadata` (JSON object with string keys and values)
* **Virtual account tags** — when using a virtual account, its tags are included in metadata
* **Default gateway metadata** — configured at gateway level (commonly used in self-hosted setups)
* **SaaS gateway location metadata** — `tfy_gateway_region` and `tfy_gateway_zone` are automatically added by the SaaS gateway based on which region handled the request
For the full list of SaaS gateway location metadata keys and their values, see [Metadata Keys](/docs/ai-gateway/globally-distributed-saas#metadata-keys).
When the same key appears in multiple sources, precedence is:
1. request metadata
2. default gateway metadata (overrides request value for overlapping keys)
3. virtual account tags (highest precedence)
For each target:
* If `metadata_match` is not set (or empty), that target always stays eligible.
* If `metadata_match` is set, **all** configured pairs must match exactly (`AND` semantics).
* Filtering happens before load-balancing order and sticky routing are computed, so only matching targets participate in routing for that request.
```yaml theme={"dark"}
routing_config:
type: weight-based-routing
load_balance_targets:
- target: azure/gpt-4o
weight: 70
metadata_match:
region: us
tier: enterprise
- target: openai/gpt-4o
weight: 30
```
For a request with:
```http theme={"dark"}
x-tfy-metadata: {"region":"us","tier":"enterprise"}
```
both targets are eligible. For:
```http theme={"dark"}
x-tfy-metadata: {"region":"eu","tier":"enterprise"}
```
only `openai/gpt-4o` remains eligible because the first target does not match.
On the [SaaS gateway](/docs/ai-gateway/globally-distributed-saas), every request is automatically tagged with `tfy_gateway_region` and `tfy_gateway_zone` based on which gateway handled it. You can use `metadata_match` to route traffic from specific regions to region-appropriate model deployments — without the client needing to send any metadata.
```yaml theme={"dark"}
routing_config:
type: priority-based-routing
load_balance_targets:
# US gateway traffic → Azure US East deployment
- target: azure-us/gpt-4o
priority: 0
metadata_match:
tfy_gateway_region: US
# EU gateway traffic → Azure EU West deployment
- target: azure-eu/gpt-4o
priority: 0
metadata_match:
tfy_gateway_region: EU
# India gateway traffic → Azure India deployment
- target: azure-in/gpt-4o
priority: 0
metadata_match:
tfy_gateway_region: IN
# Catch-all fallback for any other region
- target: openai/gpt-4o
priority: 1
```
In this setup:
* A user in the US hits `gateway.truefoundry.ai`, which routes to the nearest US gateway. The AI Gateway automatically sets `tfy_gateway_region: US` in resolved metadata, so the request matches `azure-us/gpt-4o`.
* A user in Europe hits the nearest EU gateway (`tfy_gateway_region: EU`), matching `azure-eu/gpt-4o`.
* A user in India hits the nearest India gateway (`tfy_gateway_region: IN`), matching `azure-in/gpt-4o`.
* Users from any other region (e.g. Australia, South America) don't match any `metadata_match` rule, so they fall through to `openai/gpt-4o` (priority 1) which has no `metadata_match` and acts as the default.
You can also filter by zone for finer-grained control:
```yaml theme={"dark"}
metadata_match:
tfy_gateway_zone: SFO # only match traffic from the San Francisco gateway
```
If no target matches the request metadata, the AI Gateway returns `404` with an error indicating that none of the configured targets matched `metadata_match` conditions. Always include at least one target without `metadata_match` as a catch-all, or ensure your metadata rules cover all possible values.
### Model-specific prompt overrides
When a virtual model sends traffic to targets from different model families, you may need different prompt versions per provider. Configure `prompt_version_fqn` in override parameters on each target. When a request is routed to a target, the AI Gateway uses that target's prompt version for hydration.
This is useful when:
* Different models require different prompt formats or structures
* You want to optimize prompts for specific model capabilities
* You need to maintain model-specific prompt versions behind one virtual model name
```yaml theme={"dark"}
routing_config:
type: weight-based-routing
load_balance_targets:
- target: openai/gpt-4o
weight: 70
override_params:
prompt_version_fqn: chat_prompt:internal/my-app/gpt4-optimized-prompt:1
- target: anthropic/claude-sonnet
weight: 30
override_params:
prompt_version_fqn: chat_prompt:internal/my-app/claude-optimized-prompt:1
```
`prompt_version_fqn` override does not work with agents (when using MCP/tools). It is supported for standard chat completion requests.
## Unhealthy target detection
All the routing strategies described above only consider **healthy** targets when deciding where to send a request. The AI Gateway continuously monitors every target and automatically marks targets as unhealthy when they start failing or breaching performance thresholds. This section explains how that health tracking works.
When a target is marked unhealthy, healthy targets are always tried first. Unhealthy targets are moved to the end of the list and only used as a last resort if all healthy targets fail. Recovery is automatic once errors age out of the evaluation window.
Health-based demotion does **not** apply to [complexity-based routing](/docs/ai-gateway/complexity-based-routing#escalation-and-fallback), where target order is determined by tier and escalation. Per-target retries and fallback status codes still apply there.
The AI Gateway tracks error responses for each target and marks a target unhealthy when failures cross a threshold in a recent time window.
* Error responses considered: `5xx`, `429`, `401`, and `403`
* Default failure threshold: `2` or more failures
* Default evaluation window: last `2` minutes (rolling window)
* Recovery: automatic, once failures age out of the window
For priority-based routing, you can also configure per-target latency thresholds via `sla_cutoff`:
* `time_per_output_token_ms` — average TPOT over a 3-minute rolling window (only requests with at least **30 output tokens** are counted; shorter responses are excluded as their per-token latency is dominated by TTFT rather than steady-state generation speed)
* `time_to_first_token_ms` — average TTFT over a 3-minute rolling window (streaming requests only)
If either configured metric exceeds its threshold (with at least 3 samples in the window), the target is marked unhealthy. See [SLA cutoff](#sla-cutoff) above.
## FAQ
Yes. Updates apply to new requests immediately; in-flight requests keep their current routing.
The AI Gateway returns the actual model used in the `x-tfy-resolved-model` [response header](/docs/ai-gateway/headers#response-headers). This may differ from the virtual model you requested due to load balancing or fallbacks. You can also view per-target traffic, success rates, and latency in the AI Gateway dashboard.
Yes, if you enable multiple model types on the virtual model. The supported model types are `chat`, `responses`, `completion`, `embedding`, `rerank`, `moderation`, `image`, and the audio types — `text_to_speech`, `audio_transcription` (STT), and `audio_translation`. Every target must support the operation you call. See [Supported API types](#supported-api-types) for the full endpoint mapping.
No. Virtual models are designed for **synchronous** requests where the AI Gateway can observe the first attempt and, if it fails, retry or fall back to another target. The Batch API (`/batches`) is asynchronous — requests inside a batch are processed by a single provider hours later, so there is no live response for the AI Gateway to act on, and no opportunity to fail over.
Use a real catalog model identifier (e.g. `openai-main/gpt-4o`) when creating batch jobs. For synchronous traffic that needs resilience across providers, virtual models are the right tool.
After retries and fallbacks are exhausted, the request fails with an error. Add enough fallback candidates for critical paths.
No. Allowing virtual models as targets could lead to recursive or deeply nested routing chains that are hard to reason about and debug. To keep routing predictable, targets must be real catalog models. If you need to split traffic further, create separate virtual models and have your client choose between them.
Explicit `sticky_routing` configuration isn't supported for latency-based or priority-based routing — and it's typically not needed.
* **Priority-based routing** sends every request to the highest-priority healthy target, so a session naturally stays on the same target until that target becomes unhealthy.
* **Latency-based routing** is **inherently sticky-per-caller-per-epoch**: the target is picked deterministically from a hash of the caller's identity and the current 10-minute epoch, so the same caller stays on the same target for up to 10 minutes at a time. This gives you prompt-cache benefits and consistent behaviour for agents and long conversations without any extra configuration.
* **Weight-based routing** is the one strategy that deliberately spreads traffic across targets on every request — so `sticky_routing` exists there for cases where you want to override that and pin a session to a single target.
* **Complexity-based routing** has its own `sticky_routing` block with a shorter TTL range (5 minutes to 1 hour), because a pin there holds a session on a particular price tier. See [Sticky routing for complexity-based routing](/docs/ai-gateway/complexity-based-routing#sticky-routing).
No — only within the configured `ttl_seconds` window. Once the window expires, the session is re-evaluated and may land on a different model. Size `ttl_seconds` to match your typical session duration.
Yes. Every gateway pod uses the same configuration and the same time-based window, so all pods independently arrive at the same assignment for the same session. When a fallback occurs mid-window, the update is propagated across all pods automatically.
Header overrides are strictly **per-target** — they only apply when a request is dispatched to that specific target. Other targets in the same virtual model are not affected.
`metadata_match` uses **all-keys-must-match** logic for a target. If you configure multiple keys, every key-value pair must match request metadata exactly for that target to be eligible.
Targets without `metadata_match` remain eligible for all requests and act as a default path. Targets with `metadata_match` are included only when their conditions match.
Metadata filtering runs first. Sticky routing and fallback then operate only within the filtered target set for that request.
### Responses API conversations
Calling [`/responses`](/docs/ai-gateway/responses-api) on a virtual model works differently from the other model types, because a conversation belongs to the backend that created it.
A Responses API conversation is stored on the backend that created it. If a follow-up turn were load-balanced to a different target, that target would not recognise the `previous_response_id` and the request would fail. The AI Gateway prevents this automatically: **the first turn load-balances normally, and every follow-up is pinned to whichever target served the first turn.**
To do this, the AI Gateway returns a response `id` of its own — prefixed `resp_tfyv1-` — that encodes which target served the turn alongside the provider's real response id. When you send that id back as `previous_response_id` (or in the path of `GET`/`DELETE /responses/{id}`), the AI Gateway decodes it, routes only to that target, and swaps in the provider's native id before forwarding.
```mermaid theme={"dark"}
sequenceDiagram
participant App as Your application
participant GW as AI Gateway
participant A as azure/gpt-4o
participant B as openai/gpt-4o
App->>GW: POST /responses on my-group/prod-responses
GW->>B: load-balanced to openai/gpt-4o
B-->>GW: id resp_abc
GW-->>App: id resp_tfyv1-… encodes openai/gpt-4o plus resp_abc
App->>GW: POST /responses with previous_response_id resp_tfyv1-…
Note over GW,A: azure/gpt-4o is skipped — no load balancing, no fallback
GW->>B: pinned, previous_response_id resp_abc
B-->>GW: id resp_def
GW-->>App: id resp_tfyv1-… encodes openai/gpt-4o plus resp_def
```
There is nothing to configure. The routing information travels inside the id itself rather than a shared cache, so pinning works across gateway pods and survives restarts.
The AI Gateway returns its own `resp_tfyv1-…` id that encodes which target served the turn, so it can pin follow-ups to that same backend. Pass the id back verbatim — it is opaque, and the provider's raw id is not exposed. Do not parse it or assume it matches the provider's id format.
The format does not depend on `store`, so a `store: false` turn returns a gateway-encoded id too; it just has nothing on the provider side to chain to.
For a pinned follow-up, the origin binding overrides everything else:
* **Load balancing is skipped.** Weights, priorities, and latency scores do not apply — the request goes to the origin target only.
* **Sticky routing is skipped.** The origin binding is authoritative and must not rotate when a `ttl_seconds` window expires.
* **Fallback is disabled.** Falling back to a sibling target would silently drop the conversation history, so the AI Gateway fails the request instead.
* **Retries still apply.** Only cross-target fallback is removed — retries against the pinned target itself work as configured.
Turns that are not pinned — a first turn, or a stateless turn replaying no gateway-minted reasoning content — participate in normal routing.
Send `store: false` and the AI Gateway does not track conversation state for you — you carry the full history in `input` yourself, and there is no `previous_response_id` to chain on.
```python lines theme={"dark"}
response = client.responses.create(
model="my-group/prod-responses",
input=[{"role": "user", "content": "Give me a two-word tagline."}],
store=False,
)
```
**A stateless first turn load-balances normally.** From the second turn onward, whether it is pinned depends on what you replay:
* If your `input` carries reasoning items with `encrypted_content` that the AI Gateway minted, that turn is **pinned to the backend that produced it**. Reasoning ciphertext is only valid at the model that created it, so replaying it elsewhere would fail.
* If your `input` carries no such content — plain text history only — the turn load-balances like a chat completion.
Pinning is therefore acquired by replaying content, not declared by `store`. It applies to `store: false` and `store: true` turns alike.
The AI Gateway marks its own reasoning ciphertext so it can recognise it later, and strips that marking before forwarding, so the provider only ever sees its own value. Treat `encrypted_content` as opaque and round-trip it verbatim — rebuilding or truncating it loses the pin. Ciphertext minted by a **direct** (non-virtual) model call is not marked, so replaying it at a virtual model is not pinned and can still be rejected by the provider.
This is also the right mode for targets whose Responses API is stateless — [OpenRouter](/docs/ai-gateway/openrouter#supported-apis), for example, rejects `store: true` outright.
Every target of a virtual model you call `/responses` on must support the Responses API **natively**. Translating a Responses request into a chat completion would lose conversation state, so the AI Gateway rejects it rather than falling back.
See the provider table on the [Responses API](/docs/ai-gateway/responses-api#provider-capabilities) page for which providers qualify.
| Status | When | What to do |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `409` | The target that started the conversation is no longer a target of the virtual model. | Start a new conversation. |
| `409` | The origin target is currently unavailable — it returned a `5xx` or no response at all. | Retry shortly, or start a new conversation. The AI Gateway deliberately does not fall back to another target. |
| `409` | The `previous_response_id` looks like a gateway-minted id but cannot be decoded. | Start a new conversation. |
| `400` | A target does not support the Responses API natively, so serving the request would mean translating it to chat completions and losing conversation state. | Point the virtual model at targets that support `/responses` natively. This error is terminal — the AI Gateway does not fall back. |
Errors returned by the provider itself (any other `4xx`) pass through unchanged. A pinned target that is rate limited returns its own `429` — the AI Gateway does not reroute around it.
If you replay reasoning `encrypted_content` that the AI Gateway did not mint — for example content from a direct model call, or content your client rebuilt — the turn is not pinned and the provider may reject the ciphertext with an error of its own.
## Next steps
Ready to set up a virtual model? See [Create a virtual model](/docs/ai-gateway/virtual-model-advanced) for the step-by-step walkthrough and common configuration patterns.
