Connect Private Models and MCP Servers

When your models or MCP servers run inside a private network — a VPC, on-prem cluster, or any environment unreachable from the public internet — the SAAS gateway needs a network path to reach them. This page describes the supported ways to set up that path, along with the trade-offs of each.

Choose an approach

Option	What it does	Best for	Plan
Reverse Tunnel Agent	A small agent inside your network opens an outbound-only tunnel that the AI Gateway uses to reach your private services.	Fast to set up. Works in any environment that can make outbound HTTPS.	Enterprise
Allowlist TrueFoundry IPs	Keep your service on a reachable address and restrict inbound traffic to TrueFoundry’s published outbound IPs.	Services that already sit behind a firewall, security group, or WAF you manage. No extra components to run.	All SaaS plans
Self-Host the AI Gateway	Run the AI Gateway plane inside your own infrastructure so it has direct network access to your services.	Zero cross-network traffic and full data residency control.	Enterprise

Option 1: Reverse Tunnel Agent

Instead of exposing your model or MCP server publicly, you run a cloudflared agent inside your network. The agent opens an outbound-only tunnel to a TrueFoundry-managed domain dedicated to your tenant. The AI Gateway then reaches your private services through that tunnel — no inbound firewall ports need to be opened.

This option is available on enterprise plans. Contact the TrueFoundry team to enable it for your tenant.

How it works

Two components run inside your network:

cloudflared agent — The tunnel client. It initiates outbound-only connections to Cloudflare’s edge and carries encrypted traffic from the AI Gateway back to your private services.
Caddy internal router — TrueFoundry’s routing component. It receives traffic arriving through the tunnel and forwards each request to the correct upstream service based on the URL.

Once the tunnel is active, any private service in your network is reachable at:

https://<tenant-name>.private-endpoints.truefoundry.ai/default/<service-host-and-path>

For example, an MCP server running at http://my-mcp-server.internal:8080/sse inside your VPC becomes reachable at:

https://acme.private-endpoints.truefoundry.ai/default/my-mcp-server.internal:8080/sse

Request flow

Diagram showing an authorized client sending HTTPS requests to the TrueFoundry AI Gateway, which routes through a tenant-scoped Cloudflare tunnel to the cloudflared agent and Caddy router inside the customer's private network, which then forward to MCP Services A, B, and C — Authorized client requests reach private MCP services through the TrueFoundry gateway and a tenant-scoped Cloudflare tunnel into the customer's network.

Resource specs

The tunnel stack has a small compute footprint. The Helm chart applies these per-replica requests and limits by default:

Component	CPU request	Memory request	CPU limit	Memory limit
cloudflared	100m	128Mi	200m	256Mi
Caddy	300m	256Mi	500m	512Mi

The chart also deploys 2 replicas of each component by default.

cloudflared can run on any Linux distribution and any instance type. There are no special OS or hardware requirements beyond the resource specs above.

Setup

Request a tunnel

Contact the TrueFoundry team to provision a new tunnel for your tenant. You will receive:

public_domain — Your tenant’s public domain, e.g., <tenant-name>.private-endpoints.truefoundry.ai.
auth_token — The token used to run cloudflared in your environment.

Deploy the tunnel stack

Run cloudflared and the Caddy internal router inside your network using the auth_token.

Kubernetes (Helm)
VM

helm repo add truefoundry https://truefoundry.github.io/infra-charts/
helm repo update
helm install tfy-cloudflared truefoundry/tfy-cloudflared \
  --set tunnel.token=<auth_token> \
  -n tfy-cloudflared --create-namespace

Confirm the pods are running:

kubectl get pods -n tfy-cloudflared

By default the router will proxy to any host reachable from its pod. Set caddy.allowedHosts to the exact services you want exposed — see Control what the agent can reach.

helm install tfy-cloudflared truefoundry/tfy-cloudflared \
  --set tunnel.token=<auth_token> \
  --set 'caddy.allowedHosts={my-mcp-server.internal,some-svc.my-namespace.svc.cluster.local}' \
  -n tfy-cloudflared --create-namespace

Run cloudflared and the Caddy router so that cloudflared can reach Caddy over the network. The example below runs both on one host for convenience, but they don’t have to be co-located.

Caddy listens on port 8080 in the config below. To use a different port, change the :8080 address in the Caddyfile (and the published port). Share the router’s address with the TrueFoundry team so the provisioned tunnel origin points to it.

Save the Caddyfile

Write the following to ./Caddyfile. It receives traffic from the tunnel and routes each request to the correct upstream service based on the URL.

Caddyfile

{
	admin 0.0.0.0:2019
	log {
		output stdout
		format json
		level INFO
	}
}

:8080 {
	encode zstd gzip
	log {
		output stdout
		format json
	}

	respond /healthz 200

	@direct_url_http path_regexp direct_url_http ^/[^/]+/http:/{1,2}([^/]+):([0-9]+)(?:/(.*))?$
	handle @direct_url_http {
		rewrite * /{re.direct_url_http.3}
		reverse_proxy {re.direct_url_http.1}:{re.direct_url_http.2} {
			header_up Host {re.direct_url_http.1}
			header_up X-Forwarded-Host {host}
			header_up X-Original-URI {uri}
			lb_try_duration 5s
			lb_try_interval 250ms
			fail_duration 10s
			max_fails 2
			transport http {
				dial_timeout 3s
				response_header_timeout 30s
				read_timeout 5m
				write_timeout 5m
			}
		}
	}

	@direct_url_https path_regexp direct_url_https ^/[^/]+/https:/{1,2}([^/]+):([0-9]+)(?:/(.*))?$
	handle @direct_url_https {
		rewrite * /{re.direct_url_https.3}
		reverse_proxy {re.direct_url_https.1}:{re.direct_url_https.2} {
			header_up Host {re.direct_url_https.1}
			header_up X-Forwarded-Host {host}
			header_up X-Original-URI {uri}
			lb_try_duration 5s
			lb_try_interval 250ms
			fail_duration 10s
			max_fails 2
			transport http {
				tls
				tls_server_name {re.direct_url_https.1}
				dial_timeout 3s
				response_header_timeout 30s
				read_timeout 5m
				write_timeout 5m
			}
		}
	}

	@direct_https path_regexp direct_https ^/[^/]+/https/([^/]+):([0-9]+)(?:/(.*))?$
	handle @direct_https {
		rewrite * /{re.direct_https.3}
		reverse_proxy {re.direct_https.1}:{re.direct_https.2} {
			header_up Host {re.direct_https.1}
			header_up X-Forwarded-Host {host}
			header_up X-Original-URI {uri}
			lb_try_duration 5s
			lb_try_interval 250ms
			fail_duration 10s
			max_fails 2
			transport http {
				tls
				tls_server_name {re.direct_https.1}
				dial_timeout 3s
				response_header_timeout 30s
				read_timeout 5m
				write_timeout 5m
			}
		}
	}

	@direct_http path_regexp direct_http ^/[^/]+/([^/]+):([0-9]+)(?:/(.*))?$
	handle @direct_http {
		rewrite * /{re.direct_http.3}
		reverse_proxy {re.direct_http.1}:{re.direct_http.2} {
			header_up Host {re.direct_http.1}
			header_up X-Forwarded-Host {host}
			header_up X-Original-URI {uri}
			lb_try_duration 5s
			lb_try_interval 250ms
			fail_duration 10s
			max_fails 2
			transport http {
				dial_timeout 3s
				response_header_timeout 30s
				read_timeout 5m
				write_timeout 5m
			}
		}
	}

	respond 404
}

This mirrors the Caddy config that the tfy-cloudflared Helm chart deploys. The canonical version lives at charts/tfy-cloudflared/templates/caddy/configmap.yaml — refer to it if you need the latest routing rules.

As written, the host capture group ([^/]+) in each matcher allows proxying to any host reachable from the Caddy host. This is the VM equivalent of leaving caddy.allowedHosts empty — see Control what the agent can reach.To restrict it, replace ([^/]+) in all four path_regexp matchers with an alternation of the hostnames you want to allow, escaping any dots:

@direct_url_http path_regexp direct_url_http ^/[^/]+/http:/{1,2}(my-mcp-server\.internal|some-svc\.internal):([0-9]+)(?:/(.*))?$

Requests to any other target fall through to the final respond 404. Restart the Caddy container after editing the file.

Run the tunnel stack

Save the following as docker-compose.yml next to your Caddyfile, then start both containers.

docker-compose.yml

services:
  caddy:
    image: public.ecr.aws/docker/library/caddy:2.6.3
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
  cloudflared:
    image: cloudflare/cloudflared:latest
    restart: unless-stopped
    command: tunnel --no-autoupdate run --token ${TUNNEL_TOKEN}
    network_mode: host   # so cloudflared reaches Caddy at localhost:8080

TUNNEL_TOKEN=<auth_token> docker compose up -d

Verify

Confirm Caddy is up:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/healthz

A 200 response means the router is running. Check docker compose logs cloudflared to confirm the tunnel registered successfully.

Once the tunnel is active, your private services are reachable at:

https://<tenant-name>.private-endpoints.truefoundry.ai/default/<service-host-and-path>

Use this URL as the endpoint when registering the model or MCP server in the AI Gateway. For MCP servers, see Getting started with MCP servers.

Security

No inbound exposure. All traffic from the AI Gateway to your services travels over the encrypted outbound tunnel. No inbound ports are opened on your network.
Gateway-only access to the tunnel. Each tenant’s tunnel endpoint is locked down to the TrueFoundry gateway’s IP addresses, so requests from any other source are rejected — even if they know the private endpoint URL.
Tenant isolation. MCP server URLs can only be registered if they begin with your tenant’s private endpoint domain (<tenant-name>.private-endpoints.truefoundry.ai), which prevents cross-tenant SSRF. Access is further restricted by a tenant-specific secret token.
Protect the auth_token. Treat it as a high-value secret — store it in Kubernetes Secrets or a dedicated secrets manager. If an attacker obtains it, they can impersonate your tunnel. Contact the TrueFoundry team immediately if you suspect it has been compromised.

Control what the agent can reach

The tunnel stack is not a cluster agent. The Helm chart creates no Role, ClusterRole, or RoleBinding, so neither component is granted any Kubernetes API permissions — it cannot list pods, read secrets, or enumerate anything in the cluster. Both components are HTTP proxies, so the only meaningful question is which HTTP endpoints they can reach. Three controls scope that, and you own all three.

Restrict upstream hosts with `caddy.allowedHosts`

This is the primary control. caddy.allowedHosts is an explicit allowlist of the upstream hostnames the Caddy router will proxy to. Requests to any other target return 404.

caddy:
  allowedHosts:
    - my-mcp-server.internal
    - some-svc.my-namespace.svc.cluster.local

Entries are hostnames only — do not include a port. The port still comes from the request path.
Hostnames are matched exactly, so svc.ns.svc.cluster.local will not match evil-svc.ns.svc.cluster.local.
Onboarding a new host is a values change followed by helm upgrade. The Caddy pods roll automatically to pick up the new allowlist.

The default is an empty list, which allows proxying to any host reachable from the Caddy pod’s network. Set caddy.allowedHosts before deploying into a shared cluster — otherwise the blast radius is everything the pod can reach, not just the services you intend to expose.

Network policies

The chart ships two NetworkPolicies, both enabled by default:

Policy	Effect
`cloudflared.networkPolicy.enabled`	cloudflared may egress only to the Caddy router on port `80`, DNS on port `53`, and Cloudflare’s documented edge IP ranges on port `7844`. It cannot reach any other pod in the cluster. Ingress is limited to the metrics port.
`caddy.networkPolicy.enabled`	Only cloudflared pods may reach Caddy on port `80`, so no other workload can use the router as an open proxy to internal targets.

Caddy’s own egress is deliberately left open at the network layer so that a cluster-wide default-deny-egress policy does not break it from reaching upstreams or DNS. Upstream targets are scoped at the application layer by caddy.allowedHosts. If you also want network-layer enforcement, deploy the chart into a dedicated namespace and apply your own egress policy — extraManifests can carry it alongside the release.NetworkPolicies are only enforced on clusters whose CNI supports them. They are inert otherwise.

Auditing

Caddy writes a structured JSON access log to stdout for every proxied request, including the upstream host it dialed, so you can audit exactly what the tunnel touched. cloudflared exposes Prometheus metrics on port 2000; set serviceMonitor.enabled=true to scrape them.

caddy.allowedHosts requires chart version 0.4.0 or later, and the NetworkPolicies require 0.5.0 (Caddy) and 0.6.0 (cloudflared). Run helm repo update before installing to pick up the latest chart.

Option 2: Allowlist TrueFoundry IP addresses

If your model or MCP server can be given an address TrueFoundry can reach — for example a public load balancer or API gateway in front of it — you can keep it closed to everyone else by allowing inbound traffic only from TrueFoundry’s outbound IP addresses. Traffic from TrueFoundry leaves through a fixed set of NAT addresses per control plane and gateway location, so a firewall rule, security group, or WAF allowlist built from those addresses admits TrueFoundry and rejects everything else. The addresses are published in the truefoundry/ip-ranges repository, in a machine-readable form intended for automation.

These addresses are shared by every tenant on a control plane, so an IP allowlist proves a request came from TrueFoundry, not that it came from your tenant. Keep authentication on the endpoint as well — an API key, OAuth, or mTLS — and configure those credentials in the AI Gateway when you register the model or MCP server.

Pick the list for your control plane

TrueFoundry runs two independent control planes, each with its own list:

Control plane	Your tenant URL looks like	Gateways	List
Global — hosted in Europe (Ireland)	`<tenant>.truefoundry.cloud`	Gateways worldwide	`global.json`
US	`<tenant>.us.truefoundry.cloud`	Gateways in the US	`us.json`

If you are on the US control plane, use only us.json. The US control plane is being rolled out, so its gateway addresses are added to us.json gradually as they come online — build your allowlist from the file rather than from a one-time copy. Both URLs always serve the latest list. Replace .json with .txt for one CIDR per line. Each GitHub release also attaches the lists, if you want to pin a specific version.

Choose which entries to allowlist

Each list contains one entry per hostname, and nat_ips on an entry is every address that hostname can send traffic from:

{
  "endpoints": [
    { "type": "control_plane", "hostname": "*.truefoundry.cloud", "nat_ips": ["..."] },
    { "type": "gateway", "hostname": "gateway.truefoundry.ai", "nat_ips": ["..."] },
    { "type": "gateway", "hostname": "us.gateway.truefoundry.ai", "nat_ips": ["..."] },
    { "type": "gateway", "airport_code": "orf", "hostname": "orf.gateway.truefoundry.ai", "nat_ips": ["..."] }
  ]
}

Always allowlist the control plane entry, since the control plane also connects to your services. Then add the gateway entry that matches the gateway endpoint your tenant uses:

Your tenant uses	Allowlist
The global endpoint, `gateway.truefoundry.ai` (default) — requests can be served by any gateway	Control plane + the `gateway.truefoundry.ai` entry, which contains every gateway’s addresses
A multi-regional endpoint such as `us.gateway.truefoundry.ai`	Control plane + that region’s entry, which already includes its fallback locations

For example, to build the allowlist for a tenant on the global control plane that uses the US multi-regional endpoint:

curl -s https://truefoundry.github.io/ip-ranges/global.json \
  | jq -r --arg gw us.gateway.truefoundry.ai \
      '.endpoints[] | select(.type == "control_plane" or .hostname == $gw) | .nat_ips[]' \
  | sort -u

Set gw to gateway.truefoundry.ai if you use the global endpoint.

Keep the allowlist up to date

TrueFoundry does not notify customers individually when these addresses change. Keeping your allowlist current is your responsibility — subscribe to the changes and automate your allowlist so it picks them up.

Every change to the addresses is published as a GitHub release of the repository:

New addresses are announced in a release and start carrying traffic 15 days after that release. Add them to your allowlist within those 15 days.
Removed addresses are announced in a new release. Remove them from your allowlist after that release.

To get notified, choose Watch → Custom → Releases on the repository, or subscribe to its release feed:

https://github.com/truefoundry/ip-ranges/releases.atom

For automation, run the extraction above on a schedule — daily is plenty given the 15-day window — and apply the result to your firewall, security group, or WAF. Because new addresses are announced 15 days before they carry traffic, a daily sync always has them in place before they are used.

Option 3: Self-Host the AI Gateway

If none of the above options meets your requirements, you can self-host the TrueFoundry gateway plane inside your own infrastructure. In this setup, the AI Gateway runs in the same network as your models and MCP servers and reaches them directly — no public exposure or cross-network connectivity is required. See Deploy the AI Gateway Plane for installation details.

Get Started

LLM Gateway

MCP Registry and Gateway

Skills Registry

Prompt Registry

Guardrails and Security

Observability

Deployment

Admin Guide

Chat

Messages

Embeddings

Rerank

Responses

Image

Audio

Batch

Files

Fine-tuning

Moderations

Video

Models

Connect Private Models and MCP Servers

Choose an approach

Option 1: Reverse Tunnel Agent

How it works

Request flow

Resource specs

Setup

Security

Control what the agent can reach

Restrict upstream hosts with `caddy.allowedHosts`

Network policies

Auditing

Option 2: Allowlist TrueFoundry IP addresses

Pick the list for your control plane

Choose which entries to allowlist

Keep the allowlist up to date

Option 3: Self-Host the AI Gateway

​Choose an approach

​Option 1: Reverse Tunnel Agent

​How it works

​Request flow

​Resource specs

​Setup

​Security

​Control what the agent can reach

​Restrict upstream hosts with caddy.allowedHosts

​Network policies

​Auditing

​Option 2: Allowlist TrueFoundry IP addresses

​Pick the list for your control plane

​Choose which entries to allowlist

​Keep the allowlist up to date

​Option 3: Self-Host the AI Gateway

Choose an approach

Option 1: Reverse Tunnel Agent

How it works

Request flow

Resource specs

Setup

Security

Control what the agent can reach

Restrict upstream hosts with `caddy.allowedHosts`

Network policies

Auditing

Option 2: Allowlist TrueFoundry IP addresses

Pick the list for your control plane

Choose which entries to allowlist

Keep the allowlist up to date

Option 3: Self-Host the AI Gateway