Skip to main content
When Anthropic is overloaded, the provider can return an overloaded_error in streaming flows. A common failure mode is receiving HTTP 200 but an error payload in the early stream events.

​
TrueFoundry behavior

For every Anthropic and AWS Claude Platform streaming request, TrueFoundry AI Gateway waits until it sees the first non-empty stream chunk before opening the response to your client. Claude models on the AWS Bedrock integration do not use this check.
  • If that first meaningful chunk indicates an Anthropic overloaded_error, the AI Gateway marks the attempt as failed with HTTP 503. With a routing config or virtual model, it then falls back to the next eligible target.
  • If a normal first meaningful chunk arrives, streaming continues on the same target.
This helps avoid “successful but unusable” first responses during Anthropic overload events. This check is one case of the AI Gateway’s first chunk gate, which also covers time-to-first-token timeouts and in-band errors on the Responses API.

​
Why this matters?

In streaming mode, some provider-side overload failures can appear as an error payload in early stream events instead of a non-2xx HTTP status. The first-chunk check ensures these requests fail fast and route to the next eligible target.

​
Practical recommendation

  • If you use Anthropic with streaming in TrueFoundry, configure fallback targets on a Virtual Model.
  • Keep fallback chains short and provider-diverse (for example Anthropic primary, OpenAI or Bedrock secondary).
  • Continue using retries and rate limits to reduce pressure during provider-side overload windows.