OTEL Buffer

Buffer AI Gateway traces on disk in your gateway plane to improve reliability of traces, and optionally send them to other OpenTelemetry collectors.

The OTEL buffer runs next to the AI Gateway in your gateway plane. It stores gateway traces on disk and forwards them to the TrueFoundry control plane, so traces are not lost when the control plane is briefly unreachable. It can also send the same traces to your own OpenTelemetry collectors.

Enable the OTEL buffer

Set enableOtelBuffer in your values file:

truefoundry-values.yaml

tfy-llm-gateway:
  enableOtelBuffer: true

The gateway now exports traces to the OTEL buffer, and the buffer forwards them to the control plane. Everything else is set up for you.

Send traces to other collectors

Add destinations under tfy-otel-buffer.additionalSinks. They receive the same traces as the control plane.

1. Store the collector token in a Secret

   kubectl create secret generic tfy-otel-buffer-sink-credentials -n truefoundry \
     --from-literal=COLLECTOR_TOKEN='<YOUR-TOKEN>'
   
2. Add the sink to your values file

   truefoundry-values.yaml

   tfy-llm-gateway:
     enableOtelBuffer: true
     tfy-otel-buffer:
       additionalSinks:
         my_collector:
           type: opentelemetry
           protocol:
             type: http
             uri: https://otel-collector.example.com/v1/traces
             method: post
             compression: gzip
             encoding:
               codec: otlp
             request:
               headers:
                 Authorization: "Bearer SECRET[additional_sink_credentials.COLLECTOR_TOKEN]"
   

   The traces now appear both in TrueFoundry and in your collector.

FAQ

Which collectors can I send traces to?

Any collector that accepts OTLP over HTTP. Use protocol.type: http and the collector’s /v1/traces endpoint. OTLP over gRPC is not supported in the current TrueFoundry release.

How do I add or rotate a credential?

Add one key per credential to the tfy-otel-buffer-sink-credentials Secret and reference it in the sink as SECRET[additional_sink_credentials.<KEY>], for example in a header.

After you change a key, restart the buffer so it picks up the new value:

kubectl rollout restart statefulset tfy-otel-buffer -n truefoundry

What happens when one of my collectors is down?

Each destination buffers and retries on its own, so an unreachable collector does not delay traces to the control plane or to your other collectors.

How much disk can additional sinks use?

The disk buffers of all additional sinks together can use up to 5 GiB per replica. This limit can be increased by increasing the buffer volume size.