Responses API (/responses)

Learn how to use OpenAI’s Responses API through TrueFoundry AI Gateway for creating, retrieving, and managing text and multimodal completions.

API Reference: POST /responses

Provider capabilities

The table below summarizes gateway support for this endpoint by provider.

Legend:

• ✅ Supported by provider and TrueFoundry
• Provided by provider, but not by TrueFoundry
• Provider does not support this feature

Provider	Model Response
OpenAI	✅
Azure OpenAI	✅
Anthropic
Bedrock
Vertex
Cohere
Gemini
Groq
Cerebras
Together-AI
xAI
DeepInfra
OpenRouter	✅
Databricks	✅

For every gateway endpoint and provider, see Supported APIs.

This guide explains how to use the OpenAI client to interact with TrueFoundry’s responses endpoint for making inference requests. Any provider marked supported above serves this endpoint natively — the request is forwarded as-is rather than translated into a chat completion.

On Databricks, native support covers the databricks-gpt-5* foundation model endpoints. Other Databricks endpoints accept /responses but are translated into a chat completion — see Databricks Responses API.

The AI Gateway also uses the Responses API in the opposite direction, serving some Chat Completions requests through it upstream — see Chat completions served through the Responses API below.

Authentication

You’ll need a TrueFoundry API key to authenticate your requests. You can find authentication details from here.

You’ll also need to set the x-tfy-provider-name header to the name of the provider integration you’re using.

For example, if you’re using a provider integration named my-openai-provider, you’ll set the x-tfy-provider-name header to my-openai-provider.

from openai import OpenAI

client = OpenAI(
    base_url="{GATEWAY_BASE_URL}",
    api_key="your_truefoundry_api_key",
    default_headers={"x-tfy-provider-name": "my-openai-provider"}
)

Text Completion

To get a text completion response:

response = client.responses.create(
    model="your_truefoundry_model_name", // tfy model name of a provider that supports /responses
    input=[{"role": "user", "content": "Your prompt here"}]
)

print(response)

Image Inputs

For tasks involving images:

response = client.responses.create(
    model="your_truefoundry_model_name", // tfy model name of a provider that supports /responses
    input=[
        {"role": "user", "content": "Your prompt here"},
        {
            "role": "user",
            "content": [
                {
                    "type": "input_image",
                    "image_url": "your_image_url"
                }
            ]
        }
    ]
)
response_id = response.id

Exprected Response

{
  "id": "resp_6847fa2670f8819887e2d14c08bdc5a305d8dc9b22b6f0dd",
  "created_at": 1749547558,
  "error": null,
  "incomplete_details": null,
  "instructions": null,
  "metadata": {},
  "model": "gpt-4o-mini-2024-07-18",
  "object": "response",
  "output": [
    {
      "id": "msg_6847fa35e8c48198a5120ee3d38c353105d8dc9b22b6f0dd",
      "content": [
        {
          "annotations": [],
          "text": "It seems you might be looking to start a discussion or ask a question! How can I assist you today?",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "parallel_tool_calls": true,
  "temperature": 1,
  "tool_choice": "auto",
  "tools": [],
  "top_p": 1,
  "max_output_tokens": null,
  "previous_response_id": null,
  "reasoning": {
    "effort": null,
    "generate_summary": null,
    "summary": null
  },
  "status": "completed",
  "text": {
    "format": {
      "type": "text"
    }
  },
  "truncation": "disabled",
  "usage": {
    "input_tokens": 10,
    "input_tokens_details": {
      "cached_tokens": 0
    },
    "output_tokens": 23,
    "output_tokens_details": {
      "reasoning_tokens": 0
    },
    "total_tokens": 33
  },
  "user": null,
  "background": false,
  "service_tier": "default",
  "store": true,
  "provider": "openai"
}

Managing Responses

Retrieve Response

Retrieve a response by id.

response = client.responses.retrieve(
	response_id=response_id
)

print(response)

Expected Output

{
  "id": "resp_6847fa2670f8819887e2d14c08bdc5a305d8dc9b22b6f0dd",
  "created_at": 1749547558,
  "error": null,
  "incomplete_details": null,
  "instructions": null,
  "metadata": {},
  "model": "gpt-4o-mini-2024-07-18",
  "object": "response",
  "output": [
    {
      "id": "msg_6847fa35e8c48198a5120ee3d38c353105d8dc9b22b6f0dd",
      "content": [
        {
          "annotations": [],
          "text": "It seems you might be looking to start a discussion or ask a question! How can I assist you today?",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "parallel_tool_calls": true,
  "temperature": 1,
  "tool_choice": "auto",
  "tools": [],
  "top_p": 1,
  "max_output_tokens": null,
  "previous_response_id": null,
  "reasoning": {
    "effort": null,
    "generate_summary": null,
    "summary": null
  },
  "status": "completed",
  "text": {
    "format": {
      "type": "text"
    }
  },
  "truncation": "disabled",
  "usage": {
    "input_tokens": 10,
    "input_tokens_details": {
      "cached_tokens": 0
    },
    "output_tokens": 23,
    "output_tokens_details": {
      "reasoning_tokens": 0
    },
    "total_tokens": 33
  },
  "user": null,
  "background": false,
  "service_tier": "default",
  "store": true,
  "provider": "openai"
}

Delete Response

Delete a response permanently

delete_result = client.responses.delete(
	response_id=response_id
)

Retrieve and delete are only available for providers that store responses server-side. Providers with a stateless Responses API — OpenRouter, for example — do not expose them.

Chat completions served through the Responses API

Everything above covers requests you send to /responses. The AI Gateway also uses the Responses API in the other direction: a Chat Completions request to OpenAI or Azure OpenAI can be served through the provider’s Responses API upstream and converted back before it reaches you.

This exists because OpenAI exposes a model’s reasoning only through its Responses API — a plain Chat Completions call to gpt-5.x cannot return it. Your client stays on the Chat Completions contract, so your code does not change.

Provided the model supports the Responses API, any one of these turns it on:

Trigger	Example
The request asks for reasoning	reasoning_effort set to anything other than none, or a thinking field
You opt in with a header	x-tfy-openai-use-responses-api: true
The model only speaks Responses	codex-mini, gpt-5.5-pro

That last row is worth calling out: responses-only models used to be rejected on /chat/completions. They now work, so you can reach them through the same OpenAI SDK client as every other model.

The model’s thinking comes back as message.thinking_blocks, alongside a plain-text message.reasoning_content summary. Replay the assistant message unchanged on the next turn to continue the reasoning chain.

Backend pinning does not apply on this path. Unlike /responses, a Chat Completions request that replays thinking_blocks to a virtual model is load-balanced normally, so the replay can land on a different backend than the one that produced the reasoning — and that backend will reject the ciphertext. Point reasoning traffic at a single model, or use /responses, if you need the reasoning chain preserved across turns.

Some Chat Completions parameters have no Responses equivalent — including stop, seed, logprobs, and the sampling penalties — and are dropped when a request is served this way. Azure OpenAI also requires an api-version of 2025-03-01-preview or newer.

For the full behaviour — the response shape, multi-turn replay, the complete list of dropped parameters, and the Azure requirements — see Reasoning on OpenAI and Azure OpenAI.

Virtual Models

You can call /responses on a virtual model and let the AI Gateway load-balance across several backends. Because a conversation is stored on the backend that created it, the AI Gateway pins every follow-up turn to that same backend automatically — the first turn load-balances, and anything you chain off it with previous_response_id goes back to where it started.

The one thing to know as a client: on a virtual model the AI Gateway returns its own response id prefixed with resp_tfyv1-, which encodes the origin backend. Pass it back verbatim and treat it as opaque; the provider’s raw id is not exposed.

Send store: false if you’d rather manage history yourself. Note that this does not guarantee load balancing on every turn: if your input replays reasoning items carrying encrypted_content the AI Gateway minted, that turn is still pinned to the backend that produced it, because reasoning ciphertext is only valid at the model that created it.

For the full behaviour — what pinning changes about routing, how stateless turns behave, and the errors you may see — see Responses API conversations.