Deployment Options
| Deployment Scenario | Installation | Remarks | Cost of Hosting | Pricing Tier |
|---|---|---|---|---|
| 1. AI Gateway SAAS only Gateway | No Installation Needed | Use managed AI Gateway for LLM requests, no compute plane or control plane hosting needed. | 0 | Starter |
| 2. Gateway Plane only Gateway | Deploy Gateway Plane | Host the Gateway plane for LLM requests and use control-plane hosted by Truefoundry. | ~$600/month | Enterprise |
| 3. Control Plane + Gateway Plane AI Gateway | Deploy Control Plane and Gateway Plane | Use only AI Gateway, full control, suitable if you don’t need compute plane/app deployment. | ~$800-1000/month | Enterprise |
| 4. Compute Plane + AI Gateway SAAS AI Deployment, AI Gateway | Deploy Compute Plane | Deploy models/services (Compute Plane) on your own infrastructure while using managed AI Gateway | Fixed cost of ~$200/month. Cost scales as compute plane scales. | Pro |
| 5. Compute Plane only AI Deployment | Deploy Compute Plane | Host the compute plane for model/services deployment, control-plane hosted by Truefoundry. | Fixed cost of ~$200/month. Cost scales as compute plane scales. | Pro |
| 6. Compute Plane + Control Plane AI Deployment | Deploy Control and Compute Plane | Complete self-hosted for AI Deployment, full control over infra, no Gateway product. | Fixed cost ~$600/month. Cost scales as compute plane scales. | Enterprise |
| 7. Control Plane + Gateway Plane + Compute Plane AI Deployment, AI Gateway | Deploy Control and Compute Plane | Full flexibility, full control; self-host the entire platform. | Fixed cost ~$1200/month. Cost scales as the compute plane scales and requests through gateway scales. | Enterprise |
Option 1: AI Gateway SAAS only
Option 1: AI Gateway SAAS only
This is a fully managed solution on Truefoundry’s secure cloud infrastructure with enterprise-grade features. You don’t need to deploy any infrastructure on your end. This includes usage of only the AI Gateway modle and not AI Deployment.
- Globally distributed gateway to minimize latency: Truefoundry gateway is deployed in multiple regions of the world across multiple zones and multiple cloud providers to provide low latency and high availability. Learn more about our globally distributed infrastructure.
- Zero Overhead of maintenance: There is no overhead of maintaining infrastructure and you can get access to the latest features and improvements.
- Data is encrypted at rest and in transit.
- Truefoundry Infrastructure is SOC2, ISO27001, GDPR, and HIPAA compliant
Option2: Gateway Plane only
Option2: Gateway Plane only
This is a deployment of the gateway plane on your own infrastructure. The control-plane is hosted by Truefoundry and it doesn’t include the AI Deployment module. The gateway exports the request-response data to an ingestion server running on the Truefoundry control plane. From the ingestion server, the data can be stored in a blob storage bucket either on the Truefoundry side or on your own infrastructure — depending on your data residency and compliance requirements. The control-plane stores the metrics and has access to the bucket containing the request-response data.
- LLM Traffic stays within your own premises: All LLM traffic stays within your own infrastructure and Truefoundry doesn’t come into the live path of a request to LLM.
- Flexible data storage: The request-response data flows from the gateway to the ingestion server on the Truefoundry control plane, and can then be stored in a bucket on the Truefoundry side or in a bucket on your own infrastructure. If stored on your end, you retain full ownership of the data. The data is stored in parquet format — so you can use it for analytics, debugging, and evaluation via Spark, DuckDB, Athena, or any tool of your choice.
- Management of Gateway: The gateway needs to be managed on your end.
- Truefoundry control plane has access to the bucket containing the data: Whether the data is stored on the Truefoundry side or on your infrastructure, the control plane needs access to the bucket to let you browse request logs on the Truefoundry dashboard. The control plane also compacts the logs and creates indexes to make querying faster.
If the bucket is on your infrastructure, you must grant the Truefoundry control plane access to it. Without this access, you will not be able to browse logs or use the dashboard features.
When the data is stored on your infrastructure and you browse request logs in the Truefoundry dashboard, the data is fetched from your blob storage — so you might incur egress charges from your cloud provider. The data might be cached temporarily in the control plane for faster queries. Egress charges also apply when the control plane compacts logs and creates indexes.
Option3: Control Plane + Gateway Plane
Option3: Control Plane + Gateway Plane
Host both the gateway plane and the control plane on your own infrastructure.In this case, everything except the authentication server and analytics server, everything is hosted on your own infrastructure.
The only data sent to authentication/licensing server are the emails of the employees using the platform and the count of the requests flowing through the gateway. To understand how SSO works with our central authentication server, refer to this page. This helps us keep track of the licenses and billing.
Option 4: Compute Plane + AI Gateway SAAS
Option 4: Compute Plane + AI Gateway SAAS
Deploy models/services (Compute Plane) on your own infrastructure while using managed AI Gateway.
Option 5: Compute Plane only
Option 5: Compute Plane only
Host the compute plane for model/services deployment, control-plane hosted by Truefoundry. This doesn’t include the AI Gateway product.
Option 6: Compute Plane + Control Plane
Option 6: Compute Plane + Control Plane
Complete self-hosted for AI Deployment, full control over infra, no Gateway product.
Option 7: Control Plane + Gateway Plane + Compute Plane
Option 7: Control Plane + Gateway Plane + Compute Plane
Self-host the control-plane, gateway plane and compute plane all in your own infrastructure.
Understanding the Installation Process
Truefoundry software ships as a combination of OpenTofu/Terraform code and helm charts. Your exact deployment process will depend on your current state of infrastructure and the modules you want to deploy. Here’s a brief overview of how the compute-plane, gateway-plane and control-plane are deployed.Compute Plane
This comprises of a Kubernetes cluster and few add-ons on the Kubernetes cluster as described here. If don’t have a Kubernetes cluster, Truefoundry can also provide the OpenTofu/Terraform code to provision the cluster on AWS, GCP or Azure. The OpenTofu/Terraform code brings up the cluster, installs the add-ons on the cluster, and creates roles to grant the control-plane permission to DockerRegistry, SecretsManager, BlobStorage and the Kubernetes cluster itself. Each of the addons is installed as an argo-application on the Kubernetes cluster. You can find the list of addons in the compute plane overview page. Once the addons are installed on the cluster, the management of the addons (upgrade, value changes) are done from the Truefoundry dashboard itself and terrraform is not required. OpenTofu/Terraform is only used to upgrade the cluster or make other changes in the infrastructure.
Gateway Plane
The gateway plane ships as a single helm-chart that can be deployed on any Kubernetes cluster. The gateway has no external dependencies and only needs to be able to connect to the control-plane via a secure WebSocket connection. The gateway plane is also stateless and has no database or storage attached to it. The Truefoundry gateway helm chart is available in this Github repository: tfy-llm-gatewayControl Plane
The control-plane ships as a single helm-chart that can be deployed on any Kubernetes cluster. It also requires a PostgreSQL database to store the data and connection to a blob storage to store the data. You can either bring your own Kubernetes cluster, Postgres and blobstorage or Truefoundry can help provision the same using our OpenTofu/Terraform code. The control-plane helm chart is available in this Github repository: truefoundry The control-plane helm chart includes the gateway helm chart as a dependency to make it easier to install both the control-plane and the gateway in one go.Overview of Helm Charts in Truefoundry
The key helm-charts in Truefoundry and their composition and usage is as follows:| Helm Chart | Component | Description |
|---|---|---|
| truefoundry | Control-Plane + Gateway (optionally) | The control-plane helm chart. You only need this chart if you are self-hosting the control-plane. |
| tfy-llm-gateway | Gateway Only | The gateway helm chart - you only need to install this if you are self-hosting the gateway. |
| tfy-k8s-aws-eks-inframold | AWS Compute-plane + Control-Plane(optionally) | The AWS Inframold helm chart that contains all the addons in compute-plane and also the truefoundry control-plane. You can disable the control-plane installation if you are only installing the compute-plane |
| tfy-k8s-gcp-gke-standard-inframold | GCP Compute-plane + Control-Plane(optionally) | The GCP Inframold helm chart that contains all the addons in compute-plane and also the truefoundry control-plane. You can disable the control-plane installation if you are only installing the compute-plane |
| tfy-k8s-azure-aks-inframold | Azure Compute-plane + Control-Plane(optionally) | The Azure Inframold helm chart that contains all the addons in compute-plane and also the truefoundry control-plane. You can disable the control-plane installation if you are only installing the compute-plane |
| tfy-k8s-generic-inframold | On-prem Compute-plane + Control-Plane(optionally) | The Generic Inframold helm chart that contains all the addons in compute-plane and also the truefoundry control-plane. You can disable the control-plane installation if you are only installing the compute-plane |