Guardrails

Overview

Guardrails apply a content policy to traffic passing through the gateway. A request whose prompt violates the policy is refused with HTTP 403 before it reaches any provider; a response that violates it is refused before it reaches the caller. Enforcement runs as a built-in plugin ahead of the semantic cache, so a cached answer cannot serve a prompt that today’s policy forbids.

This page describes what the shipped gateway does. Guardrails have no managed moderation backends, no CEL rule language, no profiles, no dedicated REST API, no redaction and no per-virtual-key policy binding. Earlier revisions of this page described all of those; none of them exist. See What guardrails do not do.

What a policy is made of

A policy has two layers, both configured on the Guardrails page of the console. The default blocklist (the Rules tab) is a list of case-insensitive terms and a block message. It is the quickest way to ban a handful of words across all traffic. Providers (the Providers tab) are named, independently evaluated rules. Each has a type:

Type	What it matches
`keyword`	Case-insensitive substrings.
`regex`	RE2 regular expressions. RE2 has no backtracking, so a pattern cannot hang the gateway.
`pii`	Built-in patterns for `email`, `ssn`, `credit_card`, `phone` and `ip`.
`llm`	Sends the text to a chat-completions endpoint with a SAFE/UNSAFE instruction and blocks on UNSAFE.

The first enabled rule that matches blocks the request. Every rule carries an optional message that replaces the default one when that rule is the one that matched.

Choosing what a rule inspects

Every rule — the default blocklist and each provider — carries apply_to:

Value	Effect
`input`	Scans the prompt. The default.
`output`	Scans the model’s response.
`both`	Scans both.

apply_to defaults to input, so a policy written before response scanning existed keeps behaving exactly as it did. Guarding responses is deliberate, because a term that is right to ban from a prompt is not automatically right to ban from an answer — a support assistant may need to discuss a term it must never be asked about.

Responses and streaming

An output rule inspects the model’s message content, its refusal text and its reasoning trace. Reasoning is included on purpose: a rule banning a term should not be satisfiable by emitting it in a reasoning block. For streamed responses the gateway accumulates the text as chunks arrive and scans the accumulation each time, so a banned term split across a chunk boundary is still caught and the stream is cut at the first chunk that completes a violation.

Chunks already delivered cannot be recalled. A client that has begun reading a stream will have received the prefix that preceded the violation. Scanning as chunks arrive keeps that prefix as short as possible, but it cannot be zero — this is inherent to streaming, not a limitation of this implementation. Where that matters, do not stream.

The model-based rule

An llm provider needs an endpoint and a model, and optionally a bearer token. The endpoint is usually this gateway’s own /v1/chat/completions, which lets the classifier reuse a configured provider. It fails closed. If the classifier is unreachable, rate-limited, mis-keyed or returning errors, the request is blocked. A security control whose default posture is “disabled on failure” is indistinguishable from a working one right up until it matters. Set fail_open on the provider to prefer availability over enforcement during a classifier outage. It is off by default and turning it on is a deliberate choice. Recursion is prevented by an out-of-band per-process header, not by anything in the prompt, so a caller cannot craft text that exempts itself from moderation.

The API key is masked in API responses and preserved when the console saves an unchanged policy. It cannot yet be sourced from an environment variable or a secret manager the way provider keys can.

Configuring guardrails in config.json

Guardrails are configured as a plugin, in the plugins array. There is no guardrails_config top-level block — a config file containing one is ignored, and the gateway now says so at startup.

{
  "plugins": [
    {
      "name": "guardrails",
      "enabled": true,
      "config": {
        "blocked_keywords": ["internal-project-atlas"],
        "block_message": "This request was blocked by the content policy.",
        "apply_to": "both",
        "providers": [
          {
            "id": "pii-out",
            "name": "Outbound PII",
            "type": "pii",
            "enabled": true,
            "apply_to": "output",
            "categories": ["ssn", "credit_card"],
            "message": "The response was withheld because it contained personal data."
          },
          {
            "id": "judge",
            "name": "Model judge",
            "type": "llm",
            "enabled": true,
            "apply_to": "input",
            "endpoint": "http://127.0.0.1:8080/v1/chat/completions",
            "model": "openai/gpt-4o-mini"
          }
        ]
      }
    }
  ]
}

A configuration the gateway could not fully enforce is rejected with a 400 rather than saved and reported as successful — an invalid regular expression, an unknown PII category, a rule left with nothing to match, an llm rule missing its endpoint or model, or an unrecognised apply_to. If a stored policy contains such a fault (saved before that check existed), the plugin starts with the rest of the policy in force and reports status degraded, which the console renders as a red banner. A policy running on less than it was given never passes for a healthy one.

What is covered

Prompts are scanned for chat, responses, compaction, count-tokens, realtime turns, the WebSocket responses bridge, text completion, embeddings, rerank, image generation, image edit, speech, transcription prompts, video generation and video remix — and their streaming variants. The *_passthrough routes are scanned too. Because those forward an arbitrary provider-shaped body with no schema to read a prompt out of, every string value in the JSON body is scanned. This over-includes: model names, identifiers and enum values are scanned alongside the prompt, so an aggressive rule can match something that is not caller text. Narrow the rule if that happens — the alternative was that appending _passthrough to a path skipped the policy entirely. A request type the plugin does not classify is blocked, not forwarded. An unrecognised shape cannot be evaluated, and forwarding it while recording the call as policy-checked is the one outcome worse than refusing it.

Known gaps in coverage

These carry caller text and are not scanned:

Batch create, file upload, cached-content create and container file create — the payload is an uploaded file or provider-specific JSON this hook does not parse.
Passthrough bodies that are not JSON, or larger than 1 MiB.
MCP tool arguments and tool results.
OCR and image-variation requests, where the text is inside a binary or behind a URL and does not exist yet at request time.

What guardrails do not do

Stated plainly, because earlier documentation described all of it:

No managed moderation backends. There is no AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, CrowdStrike AIDR, GraySwan Cygnal or Patronus AI integration. The four types in the table above are the whole set.
No CEL rules, profiles, or /api/guardrails/* REST API. Guardrails are configured through the plugin config, in the console or in config.json.
No redaction or content modification. Blocking is the only outcome. There is no log-only or dry-run mode, so a new rule cannot be piloted without enforcing it.
No sampling. Every request is evaluated synchronously.
No per-virtual-key, per-team, per-customer or per-model binding. There is one policy for the whole gateway.
No prompt-injection or jailbreak detection. The built-in classifier prompt asks about violence, illegal activity, self-harm, hate and sexual content.
No per-rule timeout. The classifier is bounded by a shared 20-second client timeout and by the caller’s own context.

Observability

A block is recorded on the request’s log row as policy_decision=deny with policy_reason=guardrails_violation, for input and output blocks alike. Which rule matched is written to the process log only, and is not persisted or surfaced in the console; there is no per-rule hit counter and no metric.

Permissions

The guardrails admin surface is gated on the Guardrails RBAC resource, enforced server-side and not only in the console.

Getting Started

Moving from OSS

Release Cadence

Migration Guides

Features

Security

Overview

What a policy is made of

Choosing what a rule inspects

Responses and streaming

The model-based rule

Configuring guardrails in config.json

What is covered

Known gaps in coverage

What guardrails do not do

Observability

Permissions

​Overview

​What a policy is made of

​Choosing what a rule inspects

​Responses and streaming

​The model-based rule

​Configuring guardrails in config.json

​What is covered

​Known gaps in coverage

​What guardrails do not do

​Observability

​Permissions

Overview

What a policy is made of

Choosing what a rule inspects

Responses and streaming

The model-based rule

Configuring guardrails in config.json

What is covered

Known gaps in coverage

What guardrails do not do

Observability

Permissions