> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Agent Metrics
> Query Gateway agent invocation metrics for framework, outcome, and latency analytics via API.
The **Gateway Agent Metrics Query API** provides a flexible way to query agent-framework invocations and their outcomes: which agent ran, on which framework, on which server type, and how long it took. You can retrieve either **distribution** (aggregated) or **timeseries** results with powerful filtering and grouping.
This page covers `datasource: "agentMetrics"`. For other datasources, see the sibling pages for [Model](/docs/ai-gateway/fetch-model-metrics), [MCP](/docs/ai-gateway/fetch-mcp-metrics), [Guardrail](/docs/ai-gateway/fetch-guardrail-metrics), [Cache](/docs/ai-gateway/fetch-cache-metrics), and [Routing](/docs/ai-gateway/fetch-routing-metrics) metrics.
All requests go to a single endpoint:
```
POST https://{your_control_plane_url}/api/svc/v1/llm-gateway/metrics/query
```
Send JSON with `Authorization: Bearer ` and `Content-Type: application/json`.
## Access control
Access to metrics is governed by the **data access rules configured by your tenant**. The server applies these rules automatically based on the caller's identity—you don't pass any RBAC or scoping fields in the request. What a caller can query (their own data, their team's data, or tenant-wide data) depends entirely on the rules an admin has set up.
See [Configure Data Access](/docs/ai-gateway/data-access) for how these rules are defined and evaluated.
## Authentication
Authenticate with your TrueFoundry API key. You can use either a Personal Access Token **(PAT)** or Virtual Account Token **(VAT)**.
1. **Personal Access Token (PAT)**: Go to Access → Personal Access Tokens in your TrueFoundry dashboard
2. **Virtual Account Token (VAT)**: Go to Access → Virtual Account Tokens (requires admin permissions)
For detailed authentication setup, see our [Authentication guide](/docs/ai-gateway/authentication).
## Quick start
By default, agent metrics include **every Gateway request**, not just agent invocations. Rows that didn't go through an agent will have `agentName`, `agentFramework`, and `agentServerType` set to `null` and surface in `groupBy` output as null buckets. The `IS_NULL` operator is **not** supported on these three fields; to scope a query to a known set of agents, filter with `{"fieldName": "agentName", "operator": "IN", "value": [""]}` or use `STRING_*` operators.
p50 and p99 latency per agent:
```python theme={"dark"}
import requests
response = requests.post(
"https://{your_control_plane_url}/api/svc/v1/llm-gateway/metrics/query",
headers={
"Authorization": "Bearer ",
"Content-Type": "application/json"
},
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"aggregations": [
{"type": "p50", "column": "latencyMs"},
{"type": "p99", "column": "latencyMs"}
],
"groupBy": ["agentName"]
}
)
print(response.json())
```
Hourly p99 latency per agent, restricted to failed invocations:
```python theme={"dark"}
import requests
response = requests.post(
"https://{your_control_plane_url}/api/svc/v1/llm-gateway/metrics/query",
headers={
"Authorization": "Bearer ",
"Content-Type": "application/json"
},
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "1 hour",
"aggregations": [
{"type": "p99", "column": "latencyMs"}
],
"groupBy": ["agentName"],
"filters": [
{"fieldName": "isFailure", "operator": "EQUAL", "value": true}
]
}
)
print(response.json())
```
## API reference
Post JSON to the endpoint above with `Authorization: Bearer ` and `Content-Type: application/json`.
### Request parameters
ISO 8601 timestamp marking the **inclusive** lower bound of the query window.
ISO 8601 timestamp marking the **exclusive** upper bound of the query window.
The data source to query. Use `"agentMetrics"` for Gateway agent metrics.
The type of query to execute:
* `"distribution"`: returns aggregated rows (one row per `groupBy` combination).
* `"timeseries"`: returns time-bucketed rows. Requires `interval`.
Array of `{ type, column }` objects describing the aggregations to compute. When omitted, only the implicit `total = COUNT(*)` is returned.
| Type | Description |
| ------------------------------------------------------------- | ------------------------------------------------------------- |
| `sum` | Sum of values |
| `count` | Non-null count of the column |
| `countDistinct` | Distinct count |
| `min` | Minimum value |
| `max` | Maximum value |
| `avg` | Average |
| `p5`, `p10`, `p25`, `p50`, `p75`, `p90`, `p95`, `p99`, `p999` | Percentiles (approximate) |
| `rateSum`, `rateAvg`, `rateMin`, `rateMax` | Rates normalised by the interval in seconds (timeseries only) |
| `ratePerMinute` | Value divided by the interval in minutes (timeseries only) |
| Column | Supported aggregation types | Notes |
| ----------- | ------------------------------ | ------------------------------ |
| `latencyMs` | all scalar and all percentiles | Agent invocation latency in ms |
Array of field names to group results by. Custom metadata keys are supported with a `metadata.` prefix.
| Field | Notes |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `agentName` | Agent identifier |
| `agentFramework` | Framework, e.g. `langgraph`, `crewai` |
| `agentServerType` | Transport, e.g. `sse`, `streamable-http` |
| `isFailure` | Boolean: `true` for failed invocations. GroupBy is permitted but rarely useful (only two values); filtering is the primary use. |
| `httpStatusCode` | Raw HTTP status code (NULL for rows without one). Frontends typically render NULL as "unknown" and numerics as `HTTP: `. |
| `userEmail` | Group by user (response key: `createdBySubjectSlug`) |
| `virtualaccount` | Group by virtual account (response key: `createdBySubjectSlug`) |
| `team` | Unnests the `Teams` array |
| `createdBySubjectType` | Distinguishes `user` vs `virtualaccount` |
| `metadata.` | Group by a custom metadata key |
When `groupBy` contains `userEmail` (without `virtualaccount`), the server auto-injects `WHERE CreatedBySubjectType = 'user'`. `virtualaccount` alone auto-injects `'virtualaccount'`. When both appear, scope it yourself with `createdBySubjectType` if needed.
Array of filter objects, AND-combined. See [Filtering](#filtering) below for the full operator reference and the per-field allow-list.
**Required for timeseries queries.** Bucket size as ` `, where `` is one of `second`, `minute`, `hour`, `day`, `week`, `month`, `year` (with or without a trailing `s`). Examples: `"30 second"`, `"5 minute"`, `"1 hour"`, `"1 day"`. Compound expressions like `"1 hour 30 minute"` are rejected.
**Deprecated alias for `interval`.** Accepts a positive integer number of seconds. Prefer `interval` in new code. If both are provided, `interval` wins.
## Filtering
Filters narrow down the rows that go into each aggregation and group. They are AND-combined; there is no OR-group support. The server enforces a per-field operator allow-list, so the exact subset of operators you can use depends on the field.
For standard datasource fields, use `fieldName`:
```json theme={"dark"}
{
"fieldName": "agentName",
"operator": "IN",
"value": ["support-bot", "research-agent"]
}
```
For custom request-metadata keys, use `metadataKey`. Works on every datasource:
```json theme={"dark"}
{
"metadataKey": "environment",
"operator": "IN",
"value": ["production"]
}
```
The agent-specific string fields (`agentName`, `agentFramework`, `agentServerType`) accept the narrow set `IN`, `NOT_IN`, `STRING_CONTAINS`, `STRING_STARTS_WITH`, `STRING_ENDS_WITH`. `IS_NULL` is **not** supported on these fields; sending it returns `400 Bad Request` with `Field "agentName" does not support operator "IS_NULL"`. `EQUAL` and `NOT_EQUAL` are only supported on subject fields (`userEmail`, `virtualAccount`) and `conversationID`.
| Field | Type | Allowed operators |
| -------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------- |
| `agentName` | string | `IN`, `NOT_IN`, `STRING_CONTAINS`, `STRING_STARTS_WITH`, `STRING_ENDS_WITH` |
| `agentFramework` | string | `IN`, `NOT_IN`, `STRING_CONTAINS`, `STRING_STARTS_WITH`, `STRING_ENDS_WITH` |
| `agentServerType` | string | `IN`, `NOT_IN`, `STRING_CONTAINS`, `STRING_STARTS_WITH`, `STRING_ENDS_WITH` |
| `isFailure` | boolean | `EQUAL` |
| `httpStatusCode` | number | `EQUAL`, `NOT_EQUAL`, `IN`, `NOT_IN`, `GREATER_THAN`, `LESS_THAN`, `GREATER_THAN_EQUAL`, `LESS_THAN_EQUAL`, `IS_NULL` |
| `userEmail` | string | full string operator set (no `IS_NULL`) |
| `virtualAccount` | string | full string operator set (no `IS_NULL`) |
| `team` | array | `ARRAY_HAS_ANY`, `ARRAY_HAS_NONE` |
| `latencyMs` | number | `GREATER_THAN`, `LESS_THAN`, `GREATER_THAN_EQUAL`, `LESS_THAN_EQUAL`, `BETWEEN` |
| `conversationID` | string | full string operator set |
| `metadataKey` / `metadata.` | string | full string operator set |
**String field operators**
| Operator | Description | Example value |
| ------------------------ | ---------------------------------------------------------------------------------- | -------------------------- |
| `EQUAL` | Exact match | `"alice@example.com"` |
| `NOT_EQUAL` | Not equal to value | `"bot@example.com"` |
| `IN` | Match any value in the list | `["langgraph", "crewai"]` |
| `NOT_IN` | Exclude values in the list | `["deprecated-framework"]` |
| `STRING_CONTAINS` | Contains substring | `"graph"` |
| `STRING_NOT_CONTAINS` | Does not contain substring | `"internal"` |
| `STRING_STARTS_WITH` | Starts with prefix | `"prod-"` |
| `STRING_NOT_STARTS_WITH` | Does not start with prefix | `"internal-"` |
| `STRING_ENDS_WITH` | Ends with suffix | `"-v1"` |
| `STRING_NOT_ENDS_WITH` | Does not end with suffix | `"-deprecated"` |
| `IS_NULL` | `true` matches rows where the field is unset; `false` matches rows where it is set | `true` |
**Numeric field operators**
| Operator | Description | Example value |
| -------------------- | ---------------------------------------------------------------------------------- | ----------------- |
| `EQUAL` | Exact match | `200` |
| `NOT_EQUAL` | Not equal to value | `200` |
| `IN` | Match any value in the list | `[200, 404, 500]` |
| `NOT_IN` | Exclude values in the list | `[200]` |
| `GREATER_THAN` | Strictly greater than | `499` |
| `LESS_THAN` | Strictly less than | `300` |
| `GREATER_THAN_EQUAL` | Greater than or equal to | `400` |
| `LESS_THAN_EQUAL` | Less than or equal to | `299` |
| `BETWEEN` | Between two values (inclusive) | `[500, 5000]` |
| `IS_NULL` | `true` matches rows where the field is unset; `false` matches rows where it is set | `true` |
**Boolean field operators**
| Operator | Description | Example value |
| -------- | ----------- | ------------- |
| `EQUAL` | Exact match | `true` |
**Array field operators (used by `team`)**
| Operator | Description | Example value |
| ---------------- | ---------------------------------------------- | ----------------------------- |
| `ARRAY_HAS_ANY` | Match if the array contains any of the values | `["team-alpha", "team-beta"]` |
| `ARRAY_HAS_NONE` | Match if the array contains none of the values | `["excluded-team"]` |
**Custom metadata filtering and grouping.** Every datasource supports filtering and grouping by custom request-metadata keys:
* **Filter:** `{ "metadataKey": "environment", "operator": "EQUAL", "value": "prod" }`
* **Group:** include `"metadata.environment"` in the `groupBy` array.
Metadata fields are treated as strings; use the string field operators above.
**Implicit team unnesting.** When `team` is in `groupBy` (or used as the column of an aggregation), the server transparently UNNESTs the `Teams` array CTE before applying RBAC. Callers don't need to do anything extra. Rows whose `Teams` array is NULL or empty drop out naturally.
**Combining multiple filters.** Filters are AND-combined:
```json theme={"dark"}
{
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"filters": [
{"fieldName": "agentFramework", "operator": "IN", "value": ["langgraph"]},
{"fieldName": "agentName", "operator": "IN", "value": ["support-bot", "research-agent"]},
{"fieldName": "latencyMs", "operator": "BETWEEN", "value": [100, 10000]},
{"fieldName": "team", "operator": "ARRAY_HAS_ANY", "value": ["team-alpha"]}
],
"groupBy": ["agentName", "isFailure"]
}
```
## Query examples
Every example posts a JSON body to the endpoint above. To keep the snippets short, only the `json` body is shown; the request wrapper is identical to the [Quick start](#quick-start).
Agent metrics include every Gateway request by default. Rows that didn't go through an agent will have `agentName`, `agentFramework`, and `agentServerType` set to `null` and show up as null buckets in `groupBy` output. `IS_NULL` is **not** supported on these three fields; to scope to specific known agents, use `agentName IN [...]` or one of the `STRING_*` operators.
### Distribution examples
Counts grouped by agent framework, useful for capacity planning across `langgraph`, `crewai`, etc.:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"groupBy": ["agentFramework"]
}
```
Counts grouped by agent and outcome. Surfaces flaky agents:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"groupBy": ["agentName", "isFailure"]
}
```
p50, p90, and p99 latency grouped by framework:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"aggregations": [
{"type": "p50", "column": "latencyMs"},
{"type": "p90", "column": "latencyMs"},
{"type": "p99", "column": "latencyMs"}
],
"groupBy": ["agentFramework"]
}
```
Group by raw `httpStatusCode`; rows without one show up as `null` (rendered as "unknown" in dashboards):
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"groupBy": ["httpStatusCode"]
}
```
Cross-tab of transport vs framework:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"groupBy": ["agentServerType", "agentFramework"]
}
```
Restrict to failed invocations:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"aggregations": [
{"type": "p99", "column": "latencyMs"}
],
"groupBy": ["agentName"],
"filters": [
{"fieldName": "isFailure", "operator": "EQUAL", "value": true}
]
}
```
Volume of invocations slower than 30 seconds, grouped by agent:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"aggregations": [
{"type": "count", "column": "agentName"}
],
"groupBy": ["agentName"],
"filters": [
{"fieldName": "latencyMs", "operator": "GREATER_THAN", "value": 30000}
]
}
```
Use `agentName IN [...]` to restrict the query to specific agents you care about:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "distribution",
"aggregations": [
{"type": "p99", "column": "latencyMs"}
],
"groupBy": ["agentName"],
"filters": [
{"fieldName": "agentName", "operator": "IN", "value": ["support-bot", "research-agent"]}
]
}
```
### Timeseries examples
Every timeseries query must include `interval` (or the deprecated `intervalInSeconds`). Buckets are expressed as ` ` strings like `"5 minute"`, `"1 hour"`, or `"1 day"`.
Per-agent invocation count over time:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "1 hour",
"groupBy": ["agentName"]
}
```
Watch latency regressions per agent:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "1 hour",
"aggregations": [
{"type": "p99", "column": "latencyMs"}
],
"groupBy": ["agentName"]
}
```
Track failures per agent bucket-by-bucket:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "1 hour",
"aggregations": [
{"type": "count", "column": "agentName"}
],
"groupBy": ["agentName"],
"filters": [
{"fieldName": "isFailure", "operator": "EQUAL", "value": true}
]
}
```
Per-framework volume over time:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "1 hour",
"groupBy": ["agentFramework"]
}
```
Fine-grained breakdown to investigate a regression:
```python theme={"dark"}
json={
"startTs": "2026-04-21T14:00:00.000Z",
"endTs": "2026-04-21T16:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "5 minute",
"aggregations": [
{"type": "p99", "column": "latencyMs"}
],
"groupBy": ["agentName"]
}
```
Daily agent traffic across a 7-day window:
```python theme={"dark"}
json={
"startTs": "2026-04-14T00:00:00.000Z",
"endTs": "2026-04-21T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "1 day",
"groupBy": ["agentName"]
}
```
Per-team agent usage over time:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "1 hour",
"groupBy": ["team"],
"filters": [
{"fieldName": "team", "operator": "ARRAY_HAS_ANY", "value": ["team-alpha", "team-beta"]}
]
}
```
Use `agentName IN [...]` to scope the timeseries to specific agents:
```python theme={"dark"}
json={
"startTs": "2026-04-21T00:00:00.000Z",
"endTs": "2026-04-22T00:00:00.000Z",
"datasource": "agentMetrics",
"type": "timeseries",
"interval": "1 hour",
"aggregations": [
{"type": "p99", "column": "latencyMs"}
],
"groupBy": ["agentName"],
"filters": [
{"fieldName": "agentName", "operator": "IN", "value": ["support-bot", "research-agent"]}
]
}
```
## Response format
Every successful response has the same outer shape:
```json theme={"dark"}
{
"data": {
"dataPoints": [
{
"startTimestamp": "2026-04-29T12:00:00.000Z",
"endTimestamp": "2026-04-29T13:00:00.000Z",
"total": 1234,
"": 0,
"": "value-or-null"
}
]
}
}
```
* **`total`**: implicit `COUNT(*)` for the row. Always present.
* **``**: one key per requested aggregation. The key is `` in camelCase (e.g. `countAgentName`, `p99LatencyMs`).
* **``**: one key per `groupBy` entry. The key is the lowerCamelCase form of the underlying column. Two special mappings:
* `userEmail` and `virtualaccount` both map to `createdBySubjectSlug` in the response.
* `team` maps to `team` (the value is a single unnested scalar, not an array).
All other `groupBy` keys preserve their lowerCamelCase name.
* **`startTimestamp`** / **`endTimestamp`**: present only for timeseries responses. Bucket start and end as ISO 8601 timestamp strings; `endTimestamp` equals the next bucket's `startTimestamp`. Distribution responses omit both.
```json theme={"dark"}
{
"data": {
"dataPoints": [
{
"agentName": "support-bot",
"total": 1240,
"p50LatencyMs": 1820.5,
"p99LatencyMs": 8240.0
},
{
"agentName": "research-agent",
"total": 380,
"p50LatencyMs": 4220.0,
"p99LatencyMs": 12480.0
}
]
}
}
```
```json theme={"dark"}
{
"data": {
"dataPoints": [
{
"startTimestamp": "2026-04-21T00:00:00.000Z",
"endTimestamp": "2026-04-21T01:00:00.000Z",
"agentName": "support-bot",
"total": 52,
"p99LatencyMs": 8100.0
},
{
"startTimestamp": "2026-04-21T01:00:00.000Z",
"endTimestamp": "2026-04-21T02:00:00.000Z",
"agentName": "support-bot",
"total": 61,
"p99LatencyMs": 8240.0
}
]
}
}
```
If `groupBy` is empty or omitted, the response collapses to a single row (or one row per timeseries bucket) summarising every agent invocation inside the window.
By default the response includes non-agent traffic (rows where `agentName` is null), since `IS_NULL` is not a valid operator on `agentName`. Most dashboards either render these as "unknown agent" or filter them out client-side. To scope server-side to specific known agents, use `{"fieldName": "agentName", "operator": "IN", "value": [...]}`.
### Error responses
A malformed query returns `400 Bad Request`:
```json theme={"dark"}
{
"statusCode": 400,
"message": "Invalid query",
"details": ["..."]
}
```
Common causes of `400`:
* Operator not allowed on this field, for example, `EQUAL` or `IS_NULL` on `agentName` (it supports only `IN`, `NOT_IN`, and the `STRING_*` operators).
* Missing required `value` (or wrong shape, e.g. scalar where array is expected for `IN` / `BETWEEN`).
* Unknown field name for the datasource.
* Invalid `interval` format (compound expressions, unrecognised unit, non-positive integer).
* Missing required `interval` for a timeseries query.
Other status codes:
* `401 Unauthorized`: missing or invalid bearer token.
* `403 Forbidden`: caller does not have permission for the requested scope.
* `500 Internal Server Error`: unexpected server error while executing the query.