Deploy Gateway Plane

Installation guide of deploying the gateway plane on your own infrastructure.

This guide covers the deployment option 2 mentioned here: deployment-overview. In this mode of deployment, you deploy the gateway plane on your own infrastructure and the control-plane is hosted by TrueFoundry.

Prerequisites for Installation

1. Kubernetes Cluster: K8s cluster 1.27+.

2. Domain to map the ingress of the AI Gateway along with certificate for the domain. (also any ingress controller or istio gateway you are using)

   This Domain will be referred as Gateway Endpoint Host in our documentation.

3. TrueFoundry SaaS Account: Sign up and create a new tenant on TrueFoundry SaaS by logging in to TrueFoundry. You will receive an invitation on your email to verify your email and login.

   While signing up, you will sign up by entering your company name. This company name will be used as the tenant name. Your control plane URL will be https://<YOUR_COMPANY_NAME>.truefoundry.cloud. If your company name is example-company, then your control plane URL will be https://example-company.truefoundry.cloud.

4. Egress access to TrueFoundry SaaS (Control Plane URL is from Step 3):

   • Control Plane: https://<CONTROL_PLANE_URL> (e.g., https://example-company.truefoundry.cloud)
   • NATS: wss://<CONTROL_PLANE_URL>:443 (e.g., wss://example-company.truefoundry.cloud:443)
   • OpenTelemetry: https://<CONTROL_PLANE_URL>/api/otel/v1/traces (e.g., https://example-company.truefoundry.cloud/api/otel/v1/traces)
   CONTROL_PLANE_URL: Your control plane URL in the format company-name.truefoundry.cloud (without the https:// prefix). This is the same as your tenant name from Step 3. For example, if your company name is example-company, then your CONTROL_PLANE_URL is example-company.truefoundry.cloud.

5. Image Pull Secret: Image pull secret from TrueFoundry team to enable pulling the truefoundry images from the private registry.

Installation Instructions

1. Create Gateway Endpoint in TrueFoundry SaaS

   Create an entry for Gateway Endpoints by logging into the TrueFoundry SaaS platform.

   1. Navigate to Settings -> Gateway Endpoints

   

   2. Create a new gateway endpoint with a unique name
   3. Enter the URL where you will be deploying your gateway

   

   The gateway installation name you provide here will be used in step 3 to obtain the TFY_API_KEY.

2. Obtain the TFY_API_KEY

   To obtain the TFY_API_KEY, you need to run the following curl command:

   curl --location 'https://<CONTROL_PLANE_URL>/api/svc/v1/llm-gateway/installations/token?name=<GATEWAY_INSTALLATION_NAME>' \
   --header 'Authorization: Bearer <PERSONAL_ACCESS_TOKEN>'
   

   In this command:

   • CONTROL_PLANE_URL: Your control plane URL in the format company-name.truefoundry.cloud (without the https:// prefix). For example, if your company name is example-company, then use example-company.truefoundry.cloud. This is the same as your tenant name from Step 3.
   • PERSONAL_ACCESS_TOKEN: Your personal access token. Create a token by going to Access -> Personal Access Tokens

   

   Once created, please copy/download the token and save it in a secure location. The token will not be shown again.

   • GATEWAY_INSTALLATION_NAME: The gateway installation name that was given in step 1.

   The response will contain the TFY_API_KEY which you will use in the next step.

3. Create Kubernetes Secrets

   We will create two secrets in this step:

   1. Store the TFY_API_KEY
   2. Store the Image Pull Secret

   Create Kubernetes Secret for TFY_API_KEY

   We need to create a Kubernetes secret containing the TFY_API_KEY obtained in the previous step.

   truefoundry-creds.yaml

   apiVersion: v1
   kind: Secret
   metadata:
     name: truefoundry-creds
   type: Opaque
   stringData:
     TFY_API_KEY: <TFY_API_KEY> # Obtained from step 2
   

   Apply the secret to the Kubernetes cluster (Assuming you are installing the gateway plane in the truefoundry namespace)

   kubectl apply -f truefoundry-creds.yaml -n truefoundry
   

   Create Kubernetes Secret for Image Pull Secret

   We need to create a Image Pull Secret to enable pulling the truefoundry images from the private registry.

   truefoundry-image-pull-secret.yaml

   apiVersion: v1
   kind: Secret
   metadata:
     name: truefoundry-image-pull-secret
   type: kubernetes.io/dockerconfigjson
   data:
     .dockerconfigjson: <IMAGE_PULL_SECRET> # Provided by TrueFoundry team
   

   Apply the secret to the Kubernetes cluster (Assuming you are installing the gateway plane in the truefoundry namespace)

   kubectl apply -f truefoundry-image-pull-secret.yaml -n truefoundry
   

4. Create Helm Chart Values file

   Create a values file as given below and replace the following values:

   • CONTROL_PLANE_URL: Your control plane URL in the format company-name.truefoundry.cloud (without the https:// prefix). This is the same as your tenant name. For example, if your company name is example-company, then use example-company.truefoundry.cloud.
   • TENANT_NAME: The tenant name (same as the company name you used to create your account on TrueFoundry SaaS)
   • GATEWAY_ENDPOINT_HOST: The domain where you will expose the gateway endpoint (e.g., gateway.example.com)

   truefoundry-values.yaml

   global:
     # This is the reference to the secrets we created in the previous step
     imagePullSecrets:
       - name: "truefoundry-image-pull-secret"
   
     # Choose the resource tier as per your needs
     resourceTier: medium # or small or large
   
   env:
     # CONTROL_PLANE_URL format: company-name.truefoundry.cloud (e.g., example-company.truefoundry.cloud)
     CONTROL_PLANE_NATS_URL: wss://<CONTROL_PLANE_URL>:443
     TFY_API_KEY: ${k8s-secret/truefoundry-creds/TFY_API_KEY}
     # CONTROL_PLANE_URL format: company-name.truefoundry.cloud (e.g., example-company.truefoundry.cloud)
     SERVICEFOUNDRY_SERVER_URL: https://<CONTROL_PLANE_URL>/api/svc
     # CONTROL_PLANE_URL format: company-name.truefoundry.cloud (e.g., example-company.truefoundry.cloud)
     TFY_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: https://<CONTROL_PLANE_URL>/api/otel/v1/traces
     TENANT_NAME: <TENANT_NAME>
     RUN_IN_SINGLE_TENANT_MODE: 'true'
   
   # Optional: Istio configuration (if using Istio instead of standard ingress)
   # istio:
   #   virtualservice:
   #     hosts:
   #       - <GATEWAY_ENDPOINT_HOST>
   #     enabled: true
   #     retries:
   #       enabled: true
   #       retryOn: gateway-error
   #     gateways:
   #       - istio-system/tfy-wildcard
   #     annotations: {}
   

   You can find full list of supported values here.

5. Install the Helm Chart

   Install the TrueFoundry Helm chart using the values file created in the previous step:

   helm upgrade --install truefoundry oci://tfy.jfrog.io/tfy-helm/tfy-llm-gateway -n truefoundry --create-namespace -f truefoundry-values.yaml
   

6. Verify the Deployment

   Verify that all pods are running:

   kubectl get pods -n truefoundry
   

   You can verify the deployment by checking the health of the gateway endpoint by running the following command:

   curl -X GET "https://<GATEWAY_ENDPOINT_HOST>/health"
   

   A successful response indicates that the gateway plane is deployed and running correctly.

Accessing the Gateway

You can now access the gateway endpoint from the TrueFoundry SaaS UI by selecting the gateway endpoint from the dropdown in Playground as shown below:

The gateway endpoint will be available at https://<GATEWAY_ENDPOINT_HOST> and can be used to make LLM inference requests through the TrueFoundry AI Gateway.