Virtual Models

Route requests across multiple model providers with load balancing, failover, and retries using a single model name.

A virtual model is a named entry in TrueFoundry AI Gateway that your application calls like any other model (for example my-group/production-chat). Behind that name, you configure one routing strategy and one or more real target models (for example azure/gpt-4o and openai/gpt-4o). The AI Gateway handles load balancing, health-aware routing, retries, and fallbacks automatically so you do not hard-code provider details in every service.

Virtual models only apply to synchronous API calls (chat completions, completions, embeddings, responses, rerank, image/audio, etc.). The Batch API (/batches) does not support virtual models — batch jobs run on a single provider asynchronously, so the AI Gateway has no opportunity to fallback to another target if a request inside the batch fails. Send batch requests directly to a real catalog model (for example openai-main/gpt-4o).

You can set Budget Limiting rules on a virtual model id in Models so spend is capped for the facade name clients call. See Virtual models in when.models.

Why route across multiple targets?

Production LLM traffic benefits when the AI Gateway can choose or failover among more than one backend. Common drivers:

Service outages and downtime

Model providers experience outages and downtimes. For e.g. here’s a screenshot of OpenAI’s and Anthropic’s status page from Feb to May 2025.

OpenAI Status Page

Anthropic Status Page

To avoid the downtime of your applications when models go down, a lot of organizations use multiple model providers, and configure load balancing to route to the healthy model in case one of the models goes down, hence avoiding any downtime of their applications for their users.

Latency variance among models

Latency and performance varies on time, region, model and provider. Here’s a graph of the latency variance of a few models over a course of a month.

Latency variance of models over a course of a month

We want to be able to route dynamically to the model with the lowest latency at any point in time.

Rate limits of models

A lot of the LLM providers enforce strict rate limits on API usage. Here’s a screenshot of Azure OpenAI’s rate limits:

Azure OpenAI Rate Limits

When these limits are exceeded, requests begin to fail and we want to be able to route to other models to keep our application running.

Canary testing

Testing new models or updates in production carries significant risks. Dynamic load balancing can be used to route a small percentage of traffic to the new model, and monitor the performance before routing all the traffic to the new model.

Why Virtual Models?

• Stable API surface — Your apps pass one model identifier; you change targets, weights, or providers in the AI Gateway without redeploying clients.
• Resilience — Retries and fallback status codes route around rate limits and transient errors across targets.
• Governance — Virtual model provider groups support collaborator roles so teams can use or manage routing separately.

Supported API Types

A virtual model is configured with one or more model types (set under Model types when you create it). The model type controls which API operations the virtual model can serve — and every target in the virtual model must support the operation you call.

TrueFoundry AI Gateway supports the following model types, each callable on the matching OpenAI-compatible endpoint:

Model type	Endpoint(s)	Description
chat	POST /chat/completions, POST /messages	Chat completions and the Anthropic Messages API
responses	POST /responses	OpenAI Responses API
completion	POST /completions	Legacy text completions
embedding	POST /embeddings	Text embeddings
rerank	POST /v2/rerank	Document reranking
moderation	POST /moderations	Content moderation
image	POST /images/generations, POST /images/edits, POST /images/variations	Image generation, edits, and variations
text_to_speech	POST /audio/speech	Text-to-speech
audio_transcription	POST /audio/transcriptions	Speech-to-text
audio_translation	POST /audio/translations	Audio translation

**messages can be served by a virtual model with chat.

A single virtual model can enable multiple model types (for example chat and embedding), as long as every target supports each operation you intend to call.

The responses type behaves differently from the rest: conversations are pinned to the backend that started them, and every target must support the Responses API natively. See Responses API conversations in the FAQ.

Routing Strategies

When you create a virtual model, you choose one routing strategy. Each strategy uses the same list of targets — the difference is how the AI Gateway picks among healthy targets for each request.

Strategy	How it works?	Best for
Weight-based	Distributes traffic by assigned weights (e.g. 80/20). Also supports sticky routing to pin sessions to a target.	Canary rollouts, fixed capacity splits, A/B allocation.
Priority-based	Routes to the highest-priority healthy target (0 = highest). Falls back to the next priority model on failure. Supports SLA cutoff.	Primary + backup topologies, cost optimization.
Latency-based	Per-caller deterministic selection weighted by recent latency, sticky for 10-minute windows. No weights needed.	Performance chasing across regions or providers with prompt-cache stability.
Complexity-based	Classifies each request as simple, medium, or complex and routes it to the target configured for that tier.	Cost optimization — serving easy requests on a cheaper model without changing your application.

Weight-based routing

You assign a weight to each target. The AI Gateway distributes incoming requests in proportion to those weights. For example, 90% to azure/gpt-4o and 10% to openai/gpt-4o.

Sticky routing (weight-based only)

Sticky routing pins requests that share the same session key to the same target model for a configurable time window (ttl_seconds). Within that window, every request carrying the same session identifier is routed to the same model. When the window expires, the session is re-evaluated and may land on a different model. Useful for multi-turn conversations, prompt cache efficiency, and consistent user experience.

Configuring sticky routing

Add a sticky_routing block inside your weight-based routing configuration. Two fields are required: ttl_seconds and at least one entry in session_identifiers.

routing_config:
  type: weight-based-routing
  sticky_routing:
    ttl_seconds: 3600
    session_identifiers:
      - key: x-user-id
        source: headers
  load_balance_targets:
    - target: provider-a/model-a
      weight: 70
      fallback_candidate: true
    - target: provider-b/model-b
      weight: 30
      fallback_candidate: true

Sticky routing configuration in the Virtual Model dashboard

Session identifiers tell the AI Gateway which fields to read from the request to identify a session. All configured identifiers are combined into a single session key — you can mix headers and metadata fields.

• key — The header or metadata field name to read.
• source — headers to read from HTTP request headers, or metadata to read from request metadata.

# Pin by a combination of user and conversation
session_identifiers:
  - key: x-user-id
    source: headers
  - key: x-conversation-id
    source: headers

# Pin using request metadata
session_identifiers:
  - key: tenant-id
    source: metadata
  - key: user-id
    source: metadata

If a configured identifier is missing from the request, it contributes an empty string to the session key. All requests missing that field will be treated as the same session. Make sure your clients always send the identifier fields you configure.

TTL window: ttl_seconds defines how long a session stays pinned. For chatbots, 3600 (1 hour) is a common starting point. For longer workflows, consider 86400 (24 hours).

Fallback during a sticky session: If the pinned model fails, the remaining healthy targets are tried in sequence (this fallback order is not weight-based). Only targets with fallback_candidate: true are eligible. After a successful fallback, subsequent requests for that session in the same TTL window are routed to the working target — not back to the one that failed.

Priority-based routing

Each target has a priority number. The AI Gateway routes to the highest priority target (0 is highest) that is healthy. If that target fails or is unavailable, the AI Gateway falls back to the next priority model.

SLA cutoff

Priority-based routing supports SLA cutoff to automatically mark models as unhealthy when they breach performance thresholds. You can configure either or both of:

• Time Per Output Token (TPOT) threshold per target using sla_cutoff.time_per_output_token_ms
• Time To First Token (TTFT) threshold per target using sla_cutoff.time_to_first_token_ms

How the check works?

• The AI Gateway monitors the rolling average of each configured metric over a 3-minute window (up to 10 samples, minimum 3 required)
• For TTFT, only streaming requests record a sample — non-streaming requests are skipped from the TTFT average
• For TPOT, only requests with at least 30 output tokens count as valid samples — shorter responses are excluded because their per-token latency is dominated by TTFT rather than steady-state generation speed
• If either metric exceeds its threshold, the target is marked unhealthy and moved to the end of the list
• Recovery is automatic when metrics improve or older data ages out

SLA cutoff is only available for priority-based routing, not weight or latency based routing.

Latency-based routing

You do not set weights. The AI Gateway picks a target for each caller deterministically, and keeps the same target for the duration of a short time window. This gives every caller a stable behaviour for prompt-cache reuse and reduces routing variance for agents and long conversations. Across many independent callers, traffic distributes in inverse proportion to each target’s measured latency — faster targets get a larger share of overall traffic, but every healthy target still receives some.

Stickiness is built-in. Latency-based routing is automatically sticky-per-caller-per-epoch — you do not need to add a sticky_routing block to enable this. The behaviour applies as soon as you choose latency-based-routing as the rule type.

How the target is selected?

The selector runs three steps on every request:

1. Measure each target’s recent latency. For each target, the AI Gateway looks at recent successful requests over the last 20 minutes and computes the Time Per Output Token (TPOT) — total response time divided by the number of output tokens. TPOT folds time-to-first-token and inter-token latency into a single, output-length-independent number. Targets that don’t have recent samples are treated as average so they aren’t penalised before they’ve had a chance to be measured.

2. Pick a target per caller, sticky for the epoch. Selection is based on the caller’s identity and the current 10-minute epoch. Lower-latency targets (lower TPOT) are more likely to be picked, but every healthy target retains some share. Because the selection inputs are stable for the duration of an epoch, the same caller routes to the same target for up to 10 minutes — this is the sticky part of the algorithm, and is what gives you prompt-cache reuse and lower routing variance within a session.

3. Order the remaining targets for fallback. Lower-latency targets come first in the fallback chain. If the primary target fails or returns a fallback-status-code response, the AI Gateway tries the next target in the chain.

When the epoch rolls over (every 10 minutes), the same caller may be routed to a different target — so the assignment is stable for long enough to benefit caching, but not so long that the routing decision becomes stale.

Complexity-based routing

Each target is assigned a complexity tier — simple, medium, or complex — and the AI Gateway classifies every incoming request into one of those tiers before choosing a target. Easy requests are served by your cheapest model and hard ones by your most capable, without your application changing anything.

Classification is either an in-process heuristic (default, no added latency or cost) or a call to a fast classifier model you nominate. Requests can also be pinned to a tier for the duration of a session.

This is the one strategy that reads the request body to make its decision, so it applies only to virtual models with chat, completion, or responses model types, and only to virtual models — not the tenant-level routing config.

See Complexity-based Routing for tiers, classifier options, escalation, sticky routing, and the full configuration reference.

Configuration structure

The following YAML shows the complete shape of a virtual model’s routing configuration with all available fields. In the dashboard UI, the same fields are set through the form editor. For the complexity-based shape — which uses per-target tiers and a classification strategy instead of weights or priorities — see the Complexity-based Routing reference.

YAML

routing_config:
  type: weight-based-routing | latency-based-routing | priority-based-routing

  # Sticky routing (weight-based only)
  sticky_routing:
    ttl_seconds: integer              # how long a session stays pinned (seconds)
    session_identifiers:
      - key: string                   # header or metadata field name
        source: headers | metadata    # where to read the session key from

  load_balance_targets:
    - target: string                  # model identifier in the AI Gateway (e.g. azure/gpt-4o)

      # Routing strategy fields (mutually exclusive)
      weight: integer                 # 0–100, sum to 100 (weight-based only)
      priority: integer               # lower = higher priority (priority-based only)

      # Retry configuration
      retry_config:
        attempts: integer             # retries on the SAME target; default: 0
        delay: integer                # ms between retries; default: 100
        on_status_codes: string[]     # codes that trigger retry; default: ["429","500","502","503"]

      # Fallback configuration
      fallback_status_codes: string[] # codes that trigger trying a DIFFERENT target
                                      # default: ["401","403","404","408","429","500","502","503"]
      fallback_candidate: boolean     # eligible to receive fallback traffic from other targets?
                                      # default: true

      # SLA cutoff (priority-based only). Configure either or both fields.
      sla_cutoff:
        time_per_output_token_ms: integer  # TPOT threshold; target marked unhealthy when exceeded
        time_to_first_token_ms: integer    # TTFT threshold; target marked unhealthy when exceeded
                                           # (averaged across streaming requests only)

      # Metadata-based target filtering
      metadata_match:                 # key-value pairs that must match resolved request metadata
        key1: value1                  # all pairs must match (AND logic)
        key2: value2

      # Header overrides
      headers_override:
        set:                          # headers to add or overwrite on the outgoing request
          header-name: header-value
        remove:                       # headers to strip from the outgoing request
          - header-name

      # Override parameters
      override_params:
        temperature: number           # per-target temperature override
        max_tokens: integer           # per-target max_tokens override
        prompt_version_fqn: string    # per-target prompt version for hydration

Key fields

type — The routing strategy for this virtual model:

• weight-based-routing — Distribute traffic by assigned weights that sum to 100.
• latency-based-routing — Per-caller deterministic selection weighted by recent latency, sticky for 10-minute windows so the same caller stays on the same target within that window. No weights needed.
• priority-based-routing — Route to the highest priority (lowest number) healthy target, falling back to the next on failure.
• complexity-based-routing — Classify each request as simple, medium, or complex and route it to the target configured for that tier. See Complexity-based Routing.

load_balance_targets — The list of real models eligible for routing. Each target and its configuration options are described in detail in the Per-target configuration section below.

Per-target configuration

Regardless of which routing strategy you choose, each target in the virtual model supports several options that control what happens when a request is routed to that target.

Retries and fallbacks

Each target can define how the AI Gateway should handle failures before giving up or moving to another target:

• Retry configuration — Number of attempts, delay between retries, and which status codes trigger a retry on the same target. Defaults: 0 attempts, 100 ms delay, retry on 429, 500, 502, 503.
• Fallback status codes — Which status codes cause the AI Gateway to stop retrying this target and try a different target instead. Default: 401, 403, 404, 408, 429, 500, 502, 503.
• Fallback candidate — Whether this target is eligible to receive traffic when another target fails. Default: true. Set to false when you want a target to be used only as a primary and never receive fallback traffic from other targets.

Retry and fallback example

routing_config:
  type: priority-based-routing
  load_balance_targets:
    - target: azure/gpt-4o
      priority: 0
      retry_config:
        attempts: 3
        delay: 200
        on_status_codes: ["429", "500", "503"]
      fallback_status_codes: ["429", "500", "502", "503"]
      fallback_candidate: true
    - target: openai/gpt-4o
      priority: 1
      retry_config:
        attempts: 2
        delay: 100
      fallback_status_codes: ["429"]
      fallback_candidate: true
    - target: anthropic/claude-sonnet
      priority: 2
      fallback_candidate: false   # only used as primary, never receives fallback traffic

In this configuration:

• A request first goes to azure/gpt-4o (priority 0). If it returns 429, the AI Gateway retries up to 3 times with 200 ms delay. If retries are exhausted or a fallback status code is returned, it falls back to the next target.
• openai/gpt-4o (priority 1) is tried next with its own retry config.
• anthropic/claude-sonnet (priority 2) has fallback_candidate: false, so it is never tried as a fallback for the other two targets — it is only used when it is itself the highest-priority healthy target.

Header overrides

You can inject or remove HTTP headers on a per-target basis, applied just before the request is sent to that model. This is useful when a specific target requires headers that the others don’t — for example, a region identifier, a deployment ID, or an API version header expected by one provider but not the rest.

Configuration and examples

Add a headers_override block to any target in your load_balance_targets list:

routing_config:
  type: weight-based-routing
  load_balance_targets:
    - target: provider-a/model-a
      weight: 80
      headers_override:
        set:
          x-region: us-east-1
        remove:
          - x-internal-debug
    - target: provider-b/model-b
      weight: 20
      headers_override:
        set:
          x-region: eu-west-1

Header overrides configuration per target in the Virtual Model dashboard

• set — Key-value pairs of headers to add or overwrite on the outgoing request.
• remove — List of header keys to strip from the outgoing request.

Header keys are case-insensitive — X-Custom-Auth and x-custom-auth refer to the same header. The AI Gateway normalises all keys to lowercase before applying overrides. Header overrides are applied last, after parameter overrides, so they reflect the final outgoing headers sent to the provider.

x-tfy-anthropic-beta is an exception: it is consumed into the Anthropic beta pipeline and is not forwarded upstream. When that header is present on the request (or set here), it replaces every other client beta source and is not merged. Setting or removing anthropic-beta in headers_override does not override it — to change the betas, set x-tfy-anthropic-beta.

Metadata-based target filtering

You can constrain a target to only receive traffic when request metadata matches specific key-value pairs using metadata_match.

The AI Gateway evaluates resolved metadata (not just raw request headers). Metadata can come from:

• Request metadata header — x-tfy-metadata (JSON object with string keys and values)
• Virtual account tags — when using a virtual account, its tags are included in metadata
• Default gateway metadata — configured at gateway level (commonly used in self-hosted setups)
• SaaS gateway location metadata — tfy_gateway_region and tfy_gateway_zone are automatically added by the SaaS gateway based on which region handled the request

For the full list of SaaS gateway location metadata keys and their values, see Metadata Keys.

When the same key appears in multiple sources, precedence is:

1. request metadata
2. default gateway metadata (overrides request value for overlapping keys)
3. virtual account tags (highest precedence)

For each target:

• If metadata_match is not set (or empty), that target always stays eligible.
• If metadata_match is set, all configured pairs must match exactly (AND semantics).
• Filtering happens before load-balancing order and sticky routing are computed, so only matching targets participate in routing for that request.

Basic metadata filtering

routing_config:
  type: weight-based-routing
  load_balance_targets:
    - target: azure/gpt-4o
      weight: 70
      metadata_match:
        region: us
        tier: enterprise
    - target: openai/gpt-4o
      weight: 30

For a request with:

x-tfy-metadata: {"region":"us","tier":"enterprise"}

both targets are eligible. For:

x-tfy-metadata: {"region":"eu","tier":"enterprise"}

only openai/gpt-4o remains eligible because the first target does not match.

SaaS gateway region-based routing

On the SaaS gateway, every request is automatically tagged with tfy_gateway_region and tfy_gateway_zone based on which gateway handled it. You can use metadata_match to route traffic from specific regions to region-appropriate model deployments — without the client needing to send any metadata.

routing_config:
  type: priority-based-routing
  load_balance_targets:
    # US gateway traffic → Azure US East deployment
    - target: azure-us/gpt-4o
      priority: 0
      metadata_match:
        tfy_gateway_region: US
    # EU gateway traffic → Azure EU West deployment
    - target: azure-eu/gpt-4o
      priority: 0
      metadata_match:
        tfy_gateway_region: EU
    # India gateway traffic → Azure India deployment
    - target: azure-in/gpt-4o
      priority: 0
      metadata_match:
        tfy_gateway_region: IN
    # Catch-all fallback for any other region
    - target: openai/gpt-4o
      priority: 1

In this setup:

• A user in the US hits gateway.truefoundry.ai, which routes to the nearest US gateway. The AI Gateway automatically sets tfy_gateway_region: US in resolved metadata, so the request matches azure-us/gpt-4o.
• A user in Europe hits the nearest EU gateway (tfy_gateway_region: EU), matching azure-eu/gpt-4o.
• A user in India hits the nearest India gateway (tfy_gateway_region: IN), matching azure-in/gpt-4o.
• Users from any other region (e.g. Australia, South America) don’t match any metadata_match rule, so they fall through to openai/gpt-4o (priority 1) which has no metadata_match and acts as the default.

You can also filter by zone for finer-grained control:

metadata_match:
  tfy_gateway_zone: SFO   # only match traffic from the San Francisco gateway

If no target matches the request metadata, the AI Gateway returns 404 with an error indicating that none of the configured targets matched metadata_match conditions. Always include at least one target without metadata_match as a catch-all, or ensure your metadata rules cover all possible values.

Model-specific prompt overrides

When a virtual model sends traffic to targets from different model families, you may need different prompt versions per provider. Configure prompt_version_fqn in override parameters on each target. When a request is routed to a target, the AI Gateway uses that target’s prompt version for hydration.

When and how to use prompt overrides

This is useful when:

• Different models require different prompt formats or structures
• You want to optimize prompts for specific model capabilities
• You need to maintain model-specific prompt versions behind one virtual model name

routing_config:
  type: weight-based-routing
  load_balance_targets:
    - target: openai/gpt-4o
      weight: 70
      override_params:
        prompt_version_fqn: chat_prompt:internal/my-app/gpt4-optimized-prompt:1
    - target: anthropic/claude-sonnet
      weight: 30
      override_params:
        prompt_version_fqn: chat_prompt:internal/my-app/claude-optimized-prompt:1

prompt_version_fqn override does not work with agents (when using MCP/tools). It is supported for standard chat completion requests.

Unhealthy target detection

All the routing strategies described above only consider healthy targets when deciding where to send a request. The AI Gateway continuously monitors every target and automatically marks targets as unhealthy when they start failing or breaching performance thresholds. This section explains how that health tracking works.

When a target is marked unhealthy, healthy targets are always tried first. Unhealthy targets are moved to the end of the list and only used as a last resort if all healthy targets fail. Recovery is automatic once errors age out of the evaluation window.

Health-based demotion does not apply to complexity-based routing, where target order is determined by tier and escalation. Per-target retries and fallback status codes still apply there.

Failure-based cooldown (all routing types)

The AI Gateway tracks error responses for each target and marks a target unhealthy when failures cross a threshold in a recent time window.

• Error responses considered: 5xx, 429, 401, and 403
• Default failure threshold: 2 or more failures
• Default evaluation window: last 2 minutes (rolling window)
• Recovery: automatic, once failures age out of the window

SLA-based cooldown (priority-based routing only)

For priority-based routing, you can also configure per-target latency thresholds via sla_cutoff:

• time_per_output_token_ms — average TPOT over a 3-minute rolling window (only requests with at least 30 output tokens are counted; shorter responses are excluded as their per-token latency is dominated by TTFT rather than steady-state generation speed)
• time_to_first_token_ms — average TTFT over a 3-minute rolling window (streaming requests only)

If either configured metric exceeds its threshold (with at least 3 samples in the window), the target is marked unhealthy. See SLA cutoff above.

FAQ

Can I change the routing strategy after creating a virtual model?

Yes. Updates apply to new requests immediately; in-flight requests keep their current routing.

How do I know which target handled a request?

The AI Gateway returns the actual model used in the x-tfy-resolved-model response header. This may differ from the virtual model you requested due to load balancing or fallbacks. You can also view per-target traffic, success rates, and latency in the AI Gateway dashboard.

Can one virtual model support several API types (chat, embedding, audio, etc.)?

Yes, if you enable multiple model types on the virtual model. The supported model types are chat, responses, completion, embedding, rerank, moderation, image, and the audio types — text_to_speech, audio_transcription (STT), and audio_translation. Every target must support the operation you call. See Supported API types for the full endpoint mapping.

Can I use a virtual model with the Batch API?

No. Virtual models are designed for synchronous requests where the AI Gateway can observe the first attempt and, if it fails, retry or fall back to another target. The Batch API (/batches) is asynchronous — requests inside a batch are processed by a single provider hours later, so there is no live response for the AI Gateway to act on, and no opportunity to fail over.

Use a real catalog model identifier (e.g. openai-main/gpt-4o) when creating batch jobs. For synchronous traffic that needs resilience across providers, virtual models are the right tool.

What if every target fails?

After retries and fallbacks are exhausted, the request fails with an error. Add enough fallback candidates for critical paths.

Can a virtual model point to another virtual model as a target?

No. Allowing virtual models as targets could lead to recursive or deeply nested routing chains that are hard to reason about and debug. To keep routing predictable, targets must be real catalog models. If you need to split traffic further, create separate virtual models and have your client choose between them.

Can I use sticky routing with latency-based or priority-based routing?

Explicit sticky_routing configuration isn’t supported for latency-based or priority-based routing — and it’s typically not needed.

• Priority-based routing sends every request to the highest-priority healthy target, so a session naturally stays on the same target until that target becomes unhealthy.
• Latency-based routing is inherently sticky-per-caller-per-epoch: the target is picked deterministically from a hash of the caller’s identity and the current 10-minute epoch, so the same caller stays on the same target for up to 10 minutes at a time. This gives you prompt-cache benefits and consistent behaviour for agents and long conversations without any extra configuration.
• Weight-based routing is the one strategy that deliberately spreads traffic across targets on every request — so sticky_routing exists there for cases where you want to override that and pin a session to a single target.
• Complexity-based routing has its own sticky_routing block with a shorter TTL range (5 minutes to 1 hour), because a pin there holds a session on a particular price tier. See Sticky routing for complexity-based routing.

Will the same session always go to the same model forever?

No — only within the configured ttl_seconds window. Once the window expires, the session is re-evaluated and may land on a different model. Size ttl_seconds to match your typical session duration.

Will sticky routing work if my gateway has multiple pods?

Yes. Every gateway pod uses the same configuration and the same time-based window, so all pods independently arrive at the same assignment for the same session. When a fallback occurs mid-window, the update is propagated across all pods automatically.

Do header overrides affect all targets or just the one I configure them on?

Header overrides are strictly per-target — they only apply when a request is dispatched to that specific target. Other targets in the same virtual model are not affected.

How does metadata matching work across multiple keys?

metadata_match uses all-keys-must-match logic for a target. If you configure multiple keys, every key-value pair must match request metadata exactly for that target to be eligible.

What if I set metadata rules on some targets but not others?

Targets without metadata_match remain eligible for all requests and act as a default path. Targets with metadata_match are included only when their conditions match.

How does metadata filtering interact with sticky routing and fallback?

Metadata filtering runs first. Sticky routing and fallback then operate only within the filtered target set for that request.

Responses API conversations

Calling /responses on a virtual model works differently from the other model types, because a conversation belongs to the backend that created it.

How does the AI Gateway keep a conversation on one target?

A Responses API conversation is stored on the backend that created it. If a follow-up turn were load-balanced to a different target, that target would not recognise the previous_response_id and the request would fail. The AI Gateway prevents this automatically: the first turn load-balances normally, and every follow-up is pinned to whichever target served the first turn.

To do this, the AI Gateway returns a response id of its own — prefixed resp_tfyv1- — that encodes which target served the turn alongside the provider’s real response id. When you send that id back as previous_response_id (or in the path of GET/DELETE /responses/{id}), the AI Gateway decodes it, routes only to that target, and swaps in the provider’s native id before forwarding.

There is nothing to configure. The routing information travels inside the id itself rather than a shared cache, so pinning works across gateway pods and survives restarts.

Why does my Responses API id look different from the provider's?

The AI Gateway returns its own resp_tfyv1-… id that encodes which target served the turn, so it can pin follow-ups to that same backend. Pass the id back verbatim — it is opaque, and the provider’s raw id is not exposed. Do not parse it or assume it matches the provider’s id format.

The format does not depend on store, so a store: false turn returns a gateway-encoded id too; it just has nothing on the provider side to chain to.

What does pinning change about routing?

For a pinned follow-up, the origin binding overrides everything else:

• Load balancing is skipped. Weights, priorities, and latency scores do not apply — the request goes to the origin target only.
• Sticky routing is skipped. The origin binding is authoritative and must not rotate when a ttl_seconds window expires.
• Fallback is disabled. Falling back to a sibling target would silently drop the conversation history, so the AI Gateway fails the request instead.
• Retries still apply. Only cross-target fallback is removed — retries against the pinned target itself work as configured.

Turns that are not pinned — a first turn, or a stateless turn replaying no gateway-minted reasoning content — participate in normal routing.

How do stateless (`store: false`) conversations behave?

Send store: false and the AI Gateway does not track conversation state for you — you carry the full history in input yourself, and there is no previous_response_id to chain on.

response = client.responses.create(
    model="my-group/prod-responses",
    input=[{"role": "user", "content": "Give me a two-word tagline."}],
    store=False,
)

A stateless first turn load-balances normally. From the second turn onward, whether it is pinned depends on what you replay:

• If your input carries reasoning items with encrypted_content that the AI Gateway minted, that turn is pinned to the backend that produced it. Reasoning ciphertext is only valid at the model that created it, so replaying it elsewhere would fail.
• If your input carries no such content — plain text history only — the turn load-balances like a chat completion.

Pinning is therefore acquired by replaying content, not declared by store. It applies to store: false and store: true turns alike.

The AI Gateway marks its own reasoning ciphertext so it can recognise it later, and strips that marking before forwarding, so the provider only ever sees its own value. Treat encrypted_content as opaque and round-trip it verbatim — rebuilding or truncating it loses the pin. Ciphertext minted by a direct (non-virtual) model call is not marked, so replaying it at a virtual model is not pinned and can still be rejected by the provider.

This is also the right mode for targets whose Responses API is stateless — OpenRouter, for example, rejects store: true outright.

Which targets can serve `/responses`?

Every target of a virtual model you call /responses on must support the Responses API natively. Translating a Responses request into a chat completion would lose conversation state, so the AI Gateway rejects it rather than falling back.

See the provider table on the Responses API page for which providers qualify.

What errors can Responses pinning return?

Status	When	What to do
409	The target that started the conversation is no longer a target of the virtual model.	Start a new conversation.
409	The origin target is currently unavailable — it returned a 5xx or no response at all.	Retry shortly, or start a new conversation. The AI Gateway deliberately does not fall back to another target.
409	The previous_response_id looks like a gateway-minted id but cannot be decoded.	Start a new conversation.
400	A target does not support the Responses API natively, so serving the request would mean translating it to chat completions and losing conversation state.	Point the virtual model at targets that support /responses natively. This error is terminal — the AI Gateway does not fall back.

Errors returned by the provider itself (any other 4xx) pass through unchanged. A pinned target that is rate limited returns its own 429 — the AI Gateway does not reroute around it.

If you replay reasoning encrypted_content that the AI Gateway did not mint — for example content from a direct model call, or content your client rebuilt — the turn is not pinned and the provider may reject the ciphertext with an error of its own.

Next steps

Ready to set up a virtual model? See Create a virtual model for the step-by-step walkthrough and common configuration patterns.