Skip to main content

​
Fiddler Guardrails: Frequently Asked Questions

​
Deployment & Latency

Q: Why am I getting much higher latencies than advertised? A: Fiddler Guardrails achieves its lowest latencies when requests are made to a Fiddler environment within the same network boundaries as your application. Network distance between your application and the Fiddler environment adds round-trip time. Latency spikes can also occur due to temporary usage surges, even though we provision for peak capacity. If you’re consistently experiencing higher latencies, please contact us on the #fiddler-guardrails-support channel. Q: What does this error code mean? How do I fix it? A: Please visit the Guardrails tutorials for detailed information about error codes and their solutions.

​
Service Features

Q: What’s included in Fiddler Guardrails? A: Fiddler Guardrails includes:
  • Real-time safety moderation across 11 dimensions
  • Faithfulness (hallucination) detection for RAG applications
  • PII/PHI detection and redaction
  • Integration options with NVIDIA NeMo Guardrails and LangChain
Q: Does Fiddler Guardrails support streaming responses? A: Fiddler Guardrails is designed for non-streaming (buffered) request/response calls, where it can reliably block or redact content before it reaches the user. On streamed responses, the model’s output is sent to the client token-by-token, so by the time a guardrail decision is available the content has typically already reached the user. This means block and redact cannot be reliably enforced mid-stream, and streamed responses are not guarded by default. Limited, opt-in mitigations exist, but each is off by default, covers only specific routes, and protects block more than redact:
  • LiteLLM — streaming_transform_mode: incremental_diff is designed to withhold chunks, scan the accumulated text, and re-emit guardrailed deltas, covering the OpenAI chat completions streaming path only (Anthropic /v1/messages and other surfaces fall back to unprotected behavior). This mode is not usable with Fiddler’s endpoint today: it relies on the guardrail returning a stream_holdback_chars value so a match is never split across chunks, which Fiddler does not yet return — in testing it applied a redaction but truncated the response and ended the stream with an error. Treat it as experimental until Fiddler’s endpoint returns that value; for reliable enforcement, disable streaming on that route. See LiteLLM guardrails — Failure Behavior.
  • AgentGateway — set guardrails.streaming: Enabled (AgentGateway v1.3.0+) to evaluate the response stream in batches. This enforces block (reject) only; redact (mask) gets no benefit. See AgentGateway guardrails — Failure Mode, Timeouts, and Streaming.
If you need reliable enforcement on streamed responses today, disable streaming on that route — it falls back to buffered mode, where guardrails work fully — or treat streamed-response guarding as best-effort.

​
General Information & Future Plans

Q: What are Fiddler Centor Models? A: Integral to the Fiddler AI Observability and Security platform, Fiddler Centor Models are a series of proprietary, fine-tuned small language models (SLMs) that enable high-quality LLM monitoring and scoring in live environments with the fastest guardrails in the industry. For added security, Centor Models can be deployed in VPC or air-gapped environments, ensuring enterprises maintain strict data control and safeguard LLM applications. Q: How does Fiddler Guardrails use Fiddler Centor Models? A: At <100ms latency, Fiddler Guardrails is the fastest in the industry. It leverages the scoring of Fiddler Centor Models to evaluate prompts and responses and moderate harmful outputs for hallucination, toxicity, and jailbreaks. Simply specify your desired metric thresholds and let Fiddler Guardrails handle the enforcement. Q: What languages are supported? A: Fiddler Guardrails currently only supports English. Based on user feedback, we’re evaluating support for additional languages. Q: How can I stay updated on service improvements? A: Stay informed about regular updates to Fiddler Guardrails by: Q: Where can I share feedback or request features? A: We welcome your input! Please reach out to our team on Slack in the #fiddler-guardrails-support channel. Your feedback helps us prioritize improvements. Q: Is Fiddler Guardrails an experimental or beta product? A: No, Fiddler Guardrails is a fully supported feature of the Fiddler AI Observability and Security platform.