Which Guardrail Goes on Which Hook
The same placement applies when you use an external provider for that type of check — for example, Azure Prompt Shield goes where Prompt Injection goes, and OpenAI Moderations goes where Content Moderation goes.
Why Each Guardrail Goes Where It Does
Prompt Injection — LLM Input and MCP Tool Post-Invoke
Prompt injection reaches the model in two ways, and each needs its own hook:- Direct injection and jailbreaks come from the user. On LLM Input, the guardrail blocks them before the model sees them. Because it runs in
validatemode, it runs in parallel with the model request and adds no time to first token when it passes. When it fails, the gateway cancels the model request. - Indirect injection comes from content the agent pulls in: a web page, an email, a file, a database row. That content enters through tool results. On MCP Tool Post-Invoke, the guardrail checks every tool result before the model reads it.
Content Moderation — LLM Input
Content Moderation stops harmful requests (hate, self-harm, sexual, violence) before you pay for a response. On LLM Input, it runs in parallel with the model request, so it adds no latency when it passes, and a failure cancels the model call. Leave it off the other hooks:- LLM Output: output guardrails don’t run on streamed responses (
"stream": true), which covers most chat and agent traffic. When they do run, the model has already been paid for. - MCP hooks: tool arguments and results are machine-generated data, not user-written content. Moderating every tool call adds latency to each hop of an agent loop without catching much.
PII / PHI Detection — LLM Input and MCP Tool Pre-Invoke
Put PII/PHI redaction at the points where data leaves your boundary:- LLM Input redacts PII/PHI from the prompt before it reaches the model provider. The model never sees the data, so it can’t repeat it in its response either.
- MCP Tool Pre-Invoke redacts PII/PHI from tool arguments before they reach the MCP server — for example, a search query or a ticket body that the agent sends to a third-party tool.
mutate operation so the request continues with the sensitive values masked, instead of being blocked. Mutate guardrails run before the model request, so their latency adds to time to first token — keep the entity categories to the ones you need.
Don’t add it on MCP Tool Post-Invoke by default. Tool results often need the identifiers they return (a CRM lookup returns the customer’s email so the agent can act on it), and redacting them breaks the workflow. Results are also large, so scanning them adds latency to every tool call. Anything in a tool result that the model later repeats is caught on the next LLM Input.
Secrets Detection — LLM Input and MCP Tool Post-Invoke
- LLM Input catches credentials that users paste into a prompt (a stack trace with an API key, a
.envfile) before they are sent to the model provider. - MCP Tool Post-Invoke catches credentials in tool results — file reads, environment dumps, config fetches, database rows — before the model reads them and can repeat them.
mutate to redact the secret and keep the request going. Add LLM Output as well only for non-streaming apps where the model could generate a credential from its own context.
Regex Pattern Matching — LLM Input and MCP Tool Pre-Invoke
Regex covers fixed-format identifiers that a model-based PII detector doesn’t know about — employee IDs, internal account numbers, project codenames. Place it with PII/PHI Detection, on LLM Input and MCP Tool Pre-Invoke, inmutate mode. It runs locally in the gateway, so it adds very little latency.
SQL Sanitizer — MCP Tool Pre-Invoke
SQL is dangerous when it runs, so check it before a database tool executes it. On MCP Tool Pre-Invoke, aDROP TABLE or a DELETE without WHERE is blocked and the tool never runs. Scope the policy to your database MCP servers or tools with the mcpServers / mcpTools targets.
Code Safety Linter — MCP Tool Pre-Invoke and LLM Output
- MCP Tool Pre-Invoke on code-execution and shell tools blocks
rm -rf,eval,os.system, and similar patterns in the tool’s arguments before the code runs. This is the hook that prevents damage. Scope it to those tools only. - LLM Output flags unsafe code in model responses when the code is shown to a user rather than run by a tool. It only applies to non-streaming requests.
Cedar / OPA — MCP Tool Pre-Invoke
Access policies decide whether a tool call is allowed — who can call which tool, with which arguments. That decision has to happen before the tool runs, so attach them to MCP Tool Pre-Invoke. OPA can also run on LLM Input when you need request-level rules (for example, which teams may call which model with which parameters).Metadata Validation — LLM Input
Metadata Validation checks theX-TFY-METADATA header of the request, so it only does anything on LLM Input. On every other hook it passes without checking.
Hallucination and response quality — LLM Output
Checks such as faithfulness or groundedness need the complete answer, so they belong on LLM Output. They don’t run on streamed responses — set"stream": false for the traffic you want checked.
Recommended Baseline Policy
A starting point that applies the placement above to all models and MCP servers. Replace the selectors with the guardrails you registered in AI Gateway → Guardrails:General Recommendations
Prefer earlier hooks
Prefer earlier hooks
A check on LLM Input or MCP Tool Pre-Invoke stops the problem before the model call or tool call happens. A check on LLM Output runs after the model has already been paid for, and a check on MCP Tool Post-Invoke runs after the tool’s side effects have already happened. If a risk can be caught before the call, catch it there.
Don't rely on LLM Output for streaming or agent traffic
Don't rely on LLM Output for streaming or agent traffic
LLM Output guardrails are skipped when
"stream": true. Chat UIs and coding agents almost always stream, so for that traffic put your checks on LLM Input and the MCP hooks. Use LLM Output only for non-streaming APIs.Use validate where you can, mutate where you must
Use validate where you can, mutate where you must
On LLM Input,
validate guardrails run in parallel with the model request; mutate guardrails run before it and add their latency to time to first token. Use mutate only for redaction (PII/PHI, Secrets, Regex) and validate for everything that only needs to block.Scope MCP guardrails to the tools that need them
Scope MCP guardrails to the tools that need them
MCP guardrails run on every tool call, and an agent may make dozens of calls per task. Attach SQL Sanitizer and Code Safety Linter only to the MCP servers or tools that run SQL or code, using
mcpServers / mcpTools targets.Tools that don't go through the gateway's MCP hooks
Tools that don't go through the gateway's MCP hooks
MCP hooks run only on tool calls that go through the AI Gateway (MCP servers registered on the gateway). Tools that the client runs itself — for example, a coding agent’s local shell or file tools — don’t trigger them; their results come back to the gateway as messages in the next LLM request. To check those results for indirect prompt injection, use a prompt-injection guardrail on LLM Input that scans the whole conversation, such as Azure Prompt Shield with your own credentials, together with the
X-TFY-GUARDRAILS-SCOPE header.Roll out in Audit first
Roll out in Audit first
Start every new guardrail with the Audit enforcing strategy and check AI Gateway → Monitor → Request Traces for what it would have blocked. Move to Enforce But Ignore On Error once false positives are acceptable, and to Enforce only for checks that must never be skipped. See Enforcement Strategy.
Self-hosted gateways
Self-hosted gateways
The TrueFoundry-managed Prompt Injection, Content Moderation, and PII/PHI Detection guardrails are available only when TrueFoundry hosts the gateway. On self-hosted and hybrid deployments, use the alternatives listed here on the same hooks.