Native SDK Support

Use OpenAI, AzureOpenAI, Google Gen AI, Anthropic, and boto3 SDKs with the TrueFoundry AI Gateway.

You can use these native provider SDKs with the TrueFoundry AI Gateway to send requests to your configured models. Tracing, cost tracking, and rate budget limits are supported for all of them.

SDK	Models supported	Cost tracking	Rate/Budget limits	Routing configuration
OpenAI	All models	Yes	Yes	Yes
Google Gen AI	Gemini, Vertex AI models	Yes	Yes	No
Legacy Vertex AI	Vertex AI Gemini models	Yes	Yes	Yes
Anthropic	Anthropic models only	Yes	Yes	Yes
AzureOpenAI	Azure OpenAI models only	Yes	Yes	Yes
boto3 (Bedrock Runtime)	Bedrock models	Yes	Yes	No
langchain_aws	Bedrock models	Yes	Yes	No

Copy these examples from this page. For chat models, the Playground’s Code Snippet panel only generates OpenAI-compatible snippets — OpenAI Library, LangChain, LangGraph, LlamaIndex, LiteLLM, curl, NodeJS, Go, Rust, and similar. It does not generate the Google Gen AI, Anthropic, boto3, or langchain_aws setups shown below.

The panel also has no snippet for the Responses API (/responses) or for /completions; use the examples in those guides instead.

Realtime and audio models are the exception: the Playground does generate native snippets for those, including Google Gen AI, OpenAI Realtime, Azure Voice Live, ElevenLabs, Deepgram, and Cartesia.

OpenAI SDK

Use the standard OpenAI client with the AI Gateway base URL and your TrueFoundry API key. See Chat Completions - Getting Started for the full example:

from openai import OpenAI

client = OpenAI(
    api_key="your_truefoundry_api_key",
    base_url="{GATEWAY_BASE_URL}"
)

response = client.chat.completions.create(
    model="openai-main/gpt-4o-mini", # tfy model name
    messages=[{"role": "user", "content": "Hello, how are you?"}]
)

print(response.choices[0].message.content)

Google Gen AI SDK

The Google Gen AI SDK can be used with Gemini and Vertex AI via the AI Gateway by setting a custom base URL.

Model ID must match the model name: When using the Google Gen AI SDK with the AI Gateway, the model’s display name in TrueFoundry must match the model ID, because the AI Gateway authorizes your API key against the model. You only need to pass the actual model ID (e.g. gemini-2.5-flash) in your request.

Routing configuration is not supported when using the Google Gen AI SDK with the AI Gateway.

from google import genai
from google.genai import types

client = genai.Client(
    api_key="TFY_API_KEY",
    http_options=types.HttpOptions(
        base_url="{GATEWAY_BASE_URL}/gemini/{providerAccountName}/proxy"
    )
)

response = client.models.generate_content(
    model="gemini-2.5-flash",  # actual model id
    contents="How does AI work?",
)

print(response.text)

Replace {providerAccountName} with your Gemini or Vertex AI model account name from the AI Gateway.

Legacy Vertex AI SDK

The legacy vertexai.generative_models Python SDK can call Vertex AI models through the AI Gateway. Pin a pre-removal google-cloud-aiplatform version when using it. New integrations should use google-genai instead: Google deprecated the legacy generative AI modules and announced their removal for June 24, 2026.

Required gateway headers: pass the TFY model FQN through x-tfy-model-name and the TFY credential through x-goog-api-key in request_metadata. The SDK’s model value remains the native Vertex model ID.

The gateway’s configured provider account supplies the upstream Google credentials, so this example intentionally uses AnonymousCredentials().

import vertexai
from google.auth.credentials import AnonymousCredentials
from vertexai.generative_models import GenerativeModel

tfy_model_name = "tfy-ai-vertex/google-gemini-3.5-flash-lite"
tfy_api_key = "your_truefoundry_api_key"

vertexai.init(
    project="your-gcp-project",
    location="us-central1",
    # The gateway authenticates upstream with its configured provider account.
    credentials=AnonymousCredentials(),
    # No https://. The SDK adds it.
    api_endpoint="{GATEWAY_HOST}/api/llm/vertex-ai",
    api_transport="rest",
    request_metadata=[
        ("x-goog-api-key", tfy_api_key),
        ("x-tfy-model-name", tfy_model_name),
    ],
)

model = GenerativeModel("google-gemini-3.5-flash-lite")

response = model.generate_content("Hello from Vertex AI.")

print(response.text)

The gateway uses the provider account and model from x-tfy-model-name for routing, credentials, and upstream project/region configuration. The project and location passed to the SDK are required to construct its native request path, but do not override the gateway’s configured provider account. Install the legacy client with a pinned pre-removal version: python3 -m pip install "google-cloud-aiplatform<2".

The SDK appends this native Vertex path below api_endpoint:

/v1/projects/{project}/locations/{location}/publishers/google/models/{model}:generateContent

The gateway replaces the SDK-supplied project, location, and native model path with the configured provider-account values selected by x-tfy-model-name.

Supported methods include:

• GenerativeModel.generate_content
• GenerativeModel.generate_content(..., stream=True)
• GenerativeModel.start_chat().send_message
• Function calling and multimodal content
• GenerativeModel.count_tokens

For this gateway proxy mode, AnonymousCredentials() avoids a local Google credential lookup: the gateway’s configured Vertex provider account authenticates the upstream request. If you call Google directly instead of through the gateway, use ADC, GOOGLE_APPLICATION_CREDENTIALS, or explicit service-account credentials.

Anthropic SDK

Use the Anthropic client with the AI Gateway base URL and your TrueFoundry API key to call Claude models:

from anthropic import Anthropic

BASE_URL = "{GATEWAY_BASE_URL}"
API_KEY = "your-truefoundry-api-key"

client = Anthropic(
    api_key=API_KEY,
    base_url=BASE_URL,
    default_headers={
        "Authorization": f"Bearer {API_KEY}"
    }
)

def generate_response():
    response = client.messages.create(
        model="anthropic/claude-3-5",  # tfy model name
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": "Hello, Claude! Please explain quantum computing in simple terms."
            }
        ]
    )

    print(response.content)

generate_response()

AzureOpenAI SDK

Use the Python AzureOpenAI client (from the openai package) with the AI Gateway to call Azure OpenAI models with the legacy, deployment-scoped Azure surface.

Authentication: Pass your TrueFoundry API key as api_key. The AzureOpenAI client sends it via the api-key header (not Authorization: Bearer) — the AI Gateway recognizes this natively, so no extra headers are needed. Your actual Azure credentials (API key, Entra ID, workload identity, etc.) stay configured on the Azure provider account in TrueFoundry and are never sent by the client.

Model is the TFY model FQN: Pass the TrueFoundry model FQN (provider-account/model-name, for example azure-openai-main/gpt-5-3-chat) as model — this is what the AzureOpenAI client calls the “deployment name”. The AI Gateway uses it as the routing identity and resolves the actual Azure deployment from your provider account configuration.

Configure AzureOpenAI with azure_endpoint pointing to the AI Gateway’s dedicated AzureOpenAI sub-app: append /azure-openai to the AI Gateway base URL (which already includes /api/llm for the standard TrueFoundry proxy):

from openai import AzureOpenAI

client = AzureOpenAI(
    api_key="your_truefoundry_api_key",
    azure_endpoint="{GATEWAY_BASE_URL}/azure-openai",
    api_version="2025-04-01-preview",
)

response = client.chat.completions.create(
    model="azure-openai-main/gpt-5-3-chat", # tfy model name
    messages=[{"role": "user", "content": "Hello, how are you?"}],
)

print(response.choices[0].message.content)

Why /azure-openai: the AzureOpenAI SDK appends /openai to azure_endpoint itself — do not append /openai manually. Requests land on {GATEWAY_BASE_URL}/azure-openai/openai/..., which the AI Gateway routes through a sub-app dedicated to Azure-SDK-compatible paths, keeping them isolated from the shared, provider-neutral endpoints (like the canonical /responses route) used by every other SDK on this page.

Use azure_endpoint, not base_url. The gateway supports the AzureOpenAI deployment-scoped paths generated by azure_endpoint; base_url mode is not part of the supported AzureOpenAI surface.

Supported operations

• Chat Completions (client.chat.completions.create) — including streaming and tool calls
• Completions (client.completions.create)
• Audio speech (client.audio.speech.create)
• Audio transcription (client.audio.transcriptions.create)
• Audio translation (client.audio.translations.create)
• Image generation (client.images.generate)
• Image edits (client.images.edit)
• Responses (client.responses.create) — including streaming
• Embeddings (client.embeddings.create)

Files and batches are not currently supported through this deployment-scoped AzureOpenAI compatibility surface. They are account-level operations and do not carry the TFY model identity needed for gateway routing.

api_version

The api_version you pass to the client is used as-is for the upstream Azure request. If you omit it, the AI Gateway falls back to the api_version configured on the model in TrueFoundry. Azure still rejects versions it doesn’t support for a given operation (for example, the Responses API requires 2025-03-01-preview or later) regardless of which side supplied the version.

Deferred: the modern OpenAI(base_url="{GATEWAY_BASE_URL}/openai/v1/") client (Microsoft’s recommended non-deployment-scoped surface) is not yet covered by this guide — use the AzureOpenAI client above.

boto3 SDK (AWS Bedrock Runtime)

Use boto3 with the Bedrock Runtime client when you want Bedrock-native APIs (converse, converse_stream, invoke_model, invoke_model_with_response_stream) through the AI Gateway.

Set your endpoint to the AI Gateway Bedrock proxy and pass your TrueFoundry API key as AWS_BEARER_TOKEN_BEDROCK.

import boto3
import os

os.environ["AWS_BEARER_TOKEN_BEDROCK"] = "your_truefoundry_api_key"

bedrock_runtime = boto3.client(
    service_name="bedrock-runtime",
    region_name=os.environ.get("AWS_REGION", "dummy"),
    endpoint_url="{GATEWAY_BASE_URL}/bedrock/proxy",
)

response = bedrock_runtime.converse(
    modelId="tfy-ai-bedrock/global-anthropic-claude-opus-4-5-20251101-v1-0",  # tfy model name
    messages=[
        {"role": "user", "content": [{"text": "Hello, how are you?"}]},
    ],
    inferenceConfig={"maxTokens": 256, "temperature": 0.7},
)

for block in response["output"]["message"]["content"]:
    if "text" in block:
        print(block["text"], end="")

Anthropic models with invoke_model*

For Anthropic models, Bedrock invoke_model and invoke_model_with_response_stream should use Anthropic Messages API fields:

• Add anthropic_version (for example, bedrock-2023-05-31)
• Add max_tokens
• Use typed content blocks: {"type": "text", "text": "..."}

Use typed blocks for content to avoid validation errors, for example: messages.0.content.0.type: Field required.

import json

response = bedrock_runtime.invoke_model(
    modelId="tfy-ai-bedrock/global-anthropic-claude-opus-4-5-20251101-v1-0",
    contentType="application/json",
    accept="application/json",
    body=json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 256,
        "messages": [
            {
                "role": "user",
                "content": [{"type": "text", "text": "Hello, how are you?"}],
            }
        ],
    }),
)

LangChain AWS SDK

You can also use langchain_aws with the TFY AI Gateway Bedrock endpoint:

from langchain_aws import ChatBedrockConverse

llm = ChatBedrockConverse(
    model="tfy-ai-bedrock/global-anthropic-claude-opus-4-5-20251101-v1-0",  # tfy model name
    region_name="dummy",
    endpoint_url="{GATEWAY_BASE_URL}/bedrock/proxy",
    api_key="your_truefoundry_api_key",
    temperature=0.2,
    max_tokens=256,
)

response = llm.invoke([("human", "Hello, how are you?")])
print(response.content)