# Fiddler Google ADK SDK
Source: https://docs.fiddler.ai/changelog/adk-sdk
Fiddler's Google ADK SDK release history. Contains ADK SDK release highlights, deprecation notices, and more.
[](https://pypi.org/project/fiddler-adk/)
## 1.0
* **Initial PyPI Release**
* First version of `fiddler-adk` published to PyPI (`pip install fiddler-adk`).
* `GoogleADKInstrumentor`: sets up an isolated Fiddler tracing pipeline via `FiddlerClient` and promotes it to the global tracer provider so ADK's `gcp.vertex.agent` tracer resolves to it.
* `ADKSpanProcessor`: backfills `gen_ai.conversation.id` from child spans onto the parent `invocation` root span, fixing orphan-root rendering in the Fiddler UI.
* Span-type classification, content extraction, and attribute normalization are delegated to the Fiddler backend's `GoogleADKSpanMapper`.
* Supported versions: `google-adk >= 1.34.2` (1.x and 2.x lines), `opentelemetry-api >= 1.37.0`, Python 3.10-3.14.
# Compatibility Matrix
Source: https://docs.fiddler.ai/changelog/compatibility-matrix
Compatibility guidance between the Fiddler platform and Python client. Find the recommended Python client version to use with your Fiddler platform version.
This table summarizes the compatibility between the Fiddler Python client and the Fiddler application. You can find:
* **Corresponding Fiddler Platform(s)**: Exactly the same features / API objects in both python client and the Fiddler Platform version
* **Lower Fiddler Platform**: Python client has features or API objects that may not be present in the Fiddler Platform, either due to that python client has additional new methods, or that the server has removed old APIs. However, everything they have in common (i.e., most APIs) will work.
* **Higher Fiddler Platform**: The Fiddler Platform has features the python client can't use, either due to the server has additional new APIs, or that python client has removed old API. However, everything they share in common (i.e., most APIs) will work.
This matrix maps Python **client** versions to Fiddler **platform** versions. To see which **Python language** versions Fiddler's SDKs support, see the [Python version support policy](/reference/python-support-policy).
| Client version | Corresponding Fiddler Platform(s) ✅ | Lower Fiddler Platform | Higher Fiddler Platform |
| -------------- | ------------------------------------------------------------------ | ----------------------- | ----------------------- |
| 3.13 | 26.13, 26.14, 26.15, 26.16 | Between 26.12 and 26.11 | 26.17 or above |
| 3.12 | 26.11, 26.12 | 26.10 | 26.13 or above |
| 3.11 | 25.22, 26.1, 26.2, 26.3, 26.4, 26.5, 26.6, 26.7, 26.8, 26.9, 26.10 | Between 25.21 and 25.13 | 26.11 or above |
| 3.10 | 25.13 | Between 25.12 and 25.9 | 25.14 or above |
| 3.9 | 25.9 | Between 25.8 and 25.2 | 25.10 or above |
| 3.8 | 25.2 | Between 25.1 and 24.19 | 25.3 or above |
| 3.7 | 24.18, 24.19, 25.0, 25.1 | Between 24.17 and 24.4 | 25.2 or above |
| 3.6 | 24.17 | Between 24.16 and 24.4 | 24.18 or above |
| 3.5 | 24.16 | Between 24.15 and 24.4 | 24.17 or above |
| 3.4 | 24.13, 24.14, 24.15 | Between 24.12 and 24.4 | 24.16 or above |
| 3.3 | 24.10, 24.11, 24.12 | Between 24.9 and 24.4 | 24.13 or above |
| 3.2 | 24.8, 24.9 | Between 24.7 and 24.4 | 24.10 or above |
| 3.1 | 24.5, 24.6, 24.7 | 24.4 | 24.8 or above |
| 3.0 | 24.4 | Not compatible | 24.5 or above |
# Fiddler Evals SDK
Source: https://docs.fiddler.ai/changelog/evals-sdk
Fiddler's Evals SDK release history. Contains Evals SDK release highlights, SDK deprecation notices, and more.
[](https://pypi.org/project/fiddler-evals/)
## 0.6
* **New Features**
* **Golden Datasets** (New): Build evaluation datasets from real production traffic by promoting spans into a dataset. `Application.get_span_fields()` discovers which attribute keys your spans carry, with coverage counts and any evaluator outputs recorded on them. `Dataset.add_items_from_spans()` resolves span references server-side, applies a `FieldMapping`, and writes one dataset item per span. `Dataset.get_schema()` introspects an existing dataset's fields so a new mapping stays consistent with it. Writes are all-or-nothing: if any span cannot be resolved, nothing is written and the call returns `422` with a `SpanNotFound` entry per unresolved span. Adds the `SpanReference`, `FieldMapping`, `SearchFilter`, and `SearchScope` models; structured span filtering reuses the existing `QueryCondition`, `QueryRule`, and `OperatorType` models.
* **Experiment Trace Capture** (New): `evaluate()` now captures the OpenTelemetry spans your task emits and links them to the experiment item that produced them, with no configuration. Evaluators opt in to reading them by declaring a `session` parameter, which receives the item's captured spans and enables scoring on execution — tool call counts, latency, retries, span exception events — rather than only on the output string. Capture is best-effort: if it is unavailable, `session` is `None` and the experiment runs normally.
* **Enhancements**
* **`fiddler-otel` is now a base dependency**: Required by trace capture, so it installs with the SDK rather than as an extra.
* **Pre-production application guard**: If an existing `FiddlerClient` is scoped to a different application than the dataset's, `evaluate()` raises `ValueError` instead of sending evaluation traces to that application.
* **Fixes**
* **`evaluate()` returns an up-to-date experiment**: `ExperimentResult.experiment` reflected `create()`-time state, so `status` read `PENDING` and `duration_ms` was `None` even after a completed run. The runner now reassigns the hydrated entity at each status transition.
## 0.5
* **Fixes**
* **Experiment runner threading**: Fixed an issue where the experiment runner could fail when executed inside a thread.
## 0.4
* **Breaking Changes**
* **Experiment result models renamed**: The experiment item data models were refactored. `ExperimentItem` is now `ExperimentResultItem` and `ExperimentItemResult` is now `ScoredExperimentItem` (with a new `ExperimentResultScore` model). `Experiment.get_items()` now returns `Iterator[ExperimentResultItem]`. Update imports from `fiddler_evals.pydantic_models.experiment` accordingly.
* **Enhancements**
* **CustomJudge alignment**: `CustomJudge` now aligns with Fiddler's internal PromptSpecV2 model for consistent custom LLM-as-a-Judge evaluation.
* **Fixes**
* Fixed a URL construction issue affecting API requests.
## 0.3
* **New Evaluators**
* **Context Relevance** (New): Measures whether retrieved documents are relevant to the user query. Ordinal scoring — High (1.0), Medium (0.5), Low (0.0) with detailed reasoning.
* **RAG Faithfulness** (New): LLM-as-a-Judge evaluator that assesses whether the response is grounded in the retrieved documents. Binary scoring — Yes (1.0) / No (0.0) with detailed reasoning.
* **CustomJudge** (New): Build custom LLM-as-a-Judge evaluators using `prompt_template` with Jinja `{{ placeholder }}` syntax and `output_fields` for structured evaluation results.
* **Enhancements**
* **Answer Relevance 2.0**: Upgraded from binary to ordinal scoring — High (1.0), Medium (0.5), Low (0.0) with detailed reasoning.
* **Ordinal Score Bounding**: Ordinal scores from the scoring API are now bounded to \[0, 1].
* **Enhancements**
* **Model and Credential Parameters**: `model` and `credential` are now parameters on LLM-as-a-Judge evaluators, enabling configuration of the LLM used for evaluation.
* **Evaluator-Level Score Function Mapping**: Evaluators now support `score_fn_kwargs_mapping` at the evaluator level for more flexible parameter binding.
* **Score Name Prefix**: Added support for custom score name prefixes on evaluators.
* **Evals API Error Handling**: Improved error handling and messaging for Evals API responses.
* **Coherence Prompt Input Required**: The `prompt` input for the Coherence evaluator is now required.
* **Removed Pandas Core Dependency**: Pandas moved from core to optional dependency, reducing install footprint.
* **Docstring Standardization**: Fixed docstring errors and standardized documentation format across all evaluators.
* **Removals**
* **Toxicity Evaluator Removed**: The Toxicity evaluator has been removed from the SDK.
* **Initial Release**
* Core SDK with HTTP client, entity management (Project, Application, Dataset, Experiment), and the `evaluate()` function for running experiments.
* **Evaluators**: AnswerRelevance, Coherence, Conciseness, Sentiment, TopicClassification, FTLPromptSafety, FTLResponseFaithfulness, RegexSearch, and support for user-defined function evaluators.
* **Data Input**: Load test cases from pandas DataFrames, CSV files, or JSONL files.
* **Concurrent Processing**: Parallel evaluation with ThreadPoolExecutor and tqdm progress tracking.
* **PyPI Publishing**: Available as `pip install fiddler-evals`.
# Changelog
Source: https://docs.fiddler.ai/changelog/index
Release notes for the Fiddler AI Observability Platform and supported SDKs.
Track new features, fixes, and deprecations across the Fiddler platform and
its SDKs. Each section below has its own page and its own RSS feed.
## Platform
Platform features, enhancements, and fixes for Fiddler AI Observability.
Recommended `fiddler-client` version for each Fiddler platform release.
## SDKs
Releases of `fiddler-client` — model onboarding, publishing,
baselines, alerts, and custom metrics.
Releases of `fiddler-evals` — built-in and custom LLM-as-a-judge
evaluators for experiments and regression tests.
Releases of `fiddler-langgraph` — auto-instrumentation and tracing
for LangGraph agents.
Releases of `fiddler-strands` — instrumentation and tracing for
Strands Agents.
Releases of `fiddler-langchain` — middleware-based instrumentation
for LangChain V1 agents.
Releases of `fiddler-otel` — the shared OpenTelemetry foundation
used by the agentic SDKs.
## Subscribe
Every changelog page exposes an RSS feed at the same URL with `/rss.xml`
appended — for example,
`https://docs.fiddler.ai/changelog/product-releases/rss.xml`. Look for the
RSS icon at the top of any release-notes page, or add the feed directly to
Slack, email, or your reader of choice.
# Fiddler LangChain SDK
Source: https://docs.fiddler.ai/changelog/langchain-sdk
Fiddler's LangChain SDK release history. Contains LangChain SDK release highlights, SDK deprecation notices, and more.
[](https://pypi.org/project/fiddler-langchain/)
## 1.1
### **1.1.2**
*May 25, 2026*
* **Bug Fixes**
* **Suppressed `gen_ai.llm.context` on routing-only LLM spans**: When an LLM span's output is purely `tool_calls` (a routing decision with no text content), the SDK now blanks `gen_ai.llm.context` on that span. Previously, RAG context set earlier in the same turn via `set_llm_context()` leaked onto these routing spans, triggering faithfulness evaluation on spans that produce no answer text.
***
### **1.1.1**
*May 12, 2026*
* **Bug Fixes**
* **Fixed `wrapt` 2.x compatibility**: `wrapt` 2.0 renamed the first parameter of `wrap_function_wrapper()` from `module` to `target`. The SDK passed it as a keyword argument (`module='...'`), which raised `TypeError: wrap_function_wrapper() got an unexpected keyword argument 'module'` on `wrapt >= 2.0`. Fixed with a version-detecting helper that dispatches the correct keyword argument name based on the installed `wrapt` major version (`module=` for 1.x, `target=` for 2.x).
* **Dependencies**
* Bumps `fiddler-otel` dependency to `1.2.0`.
***
### **1.1.0**
*April 14, 2026*
* **New Features**
* **`clear_llm_context()` function**: New convenience function to explicitly remove RAG context from an LLM instance, preventing faithfulness evaluation on subsequent non-RAG spans. In multi-step agent workflows, context set after a RAG retrieval step previously leaked into later non-RAG LLM calls (tool planning, routing, etc.), causing unintended faithfulness evaluation. `set_llm_context(llm, None)` also now clears context.
```python theme={null}
from fiddler_langchain import set_llm_context, clear_llm_context
# After RAG retrieval
set_llm_context(model, retrieved_documents)
response = model.invoke(rag_prompt) # faithfulness evaluated
# Before non-RAG steps
clear_llm_context(model)
plan = model.invoke(planning_prompt) # no faithfulness evaluation
```
* **Python 3.14 support**: Added Python 3.14 to CI test matrix and PyPI classifiers. All runtime dependencies have compatible wheels.
***
## 1.0
### **1.0.1**
*March 20, 2026*
* **Bug Fixes**
* **Fixed `fiddler-otel` dependency constraint**: Relaxed the `fiddler-otel` upper bound from `<1.0.0` to `<2.0.0`, resolving a dependency conflict when installing alongside `fiddler-otel 1.x`.
***
### **1.0.0**
*March 18, 2026*
Initial release of the Fiddler LangChain SDK.
* **Added**
* **`FiddlerLangChainInstrumentor`**: Auto-instrumentor for LangChain V1 agents. Monkey-patches `langchain.agents.create_agent` so every subsequent call automatically receives a `FiddlerAgentMiddleware`. Idempotent — calling `instrument()` multiple times is safe.
* `instrument()`: Enable automatic instrumentation of all `create_agent` calls.
* `uninstrument()`: Restore the original `create_agent` function. Does not affect already-created agents.
* `is_instrumented_by_opentelemetry`: Property indicating whether instrumentation is currently active.
* **`FiddlerAgentMiddleware`**: LangChain V1 `AgentMiddleware` that instruments agents with Fiddler tracing. Produces a clean, flat trace hierarchy: agent → LLM calls → tool calls, with no noisy Chain wrappers.
* `before_agent()`: Opens a root Agent span (`TYPE=agent`) at the start of each invocation.
* `after_agent()`: Closes the root Agent span.
* `wrap_model_call()` / `awrap_model_call()`: Wraps synchronous and asynchronous model calls with LLM spans (`TYPE=llm`). Captures model name, provider, system prompt, user prompt, full message history, output messages, token usage, LLM context, and tool definitions.
* `wrap_tool_call()` / `awrap_tool_call()`: Wraps synchronous and asynchronous tool calls with Tool spans (`TYPE=tool`). Captures tool name, input, and output.
* **`set_llm_context()`**: Attach a context string to a `BaseLanguageModel` or `RunnableBinding`. The middleware reads this at invocation time and sets `gen_ai.llm.context` on the LLM span.
* **`add_span_attributes(**kwargs)`**: Attach custom metadata to a specific LangChain component (model, tool, or retriever). Emitted as `fiddler.span.user.{key}` only on the span for that component — scoped, unlike `add_session_attributes`.
* **`set_conversation_id()`**: Re-exported from `fiddler_otel` for convenience. Propagates conversation ID to all spans in the current execution context.
* **`add_session_attributes()`**: Re-exported from `fiddler_otel`. Attaches session-level metadata to all spans in the current context.
* **Full async support**: `awrap_model_call` and `awrap_tool_call` hooks fully support `agent.ainvoke()`.
* **Error handling**: Failing LLM or tool spans are marked `StatusCode.ERROR` with the exception recorded. The root Agent span is always closed cleanly, preventing dangling open spans. Exceptions are re-raised so normal application error handling is unaffected.
* **Multi-agent single-trace nesting**: When a sub-agent is invoked from within a delegation tool, its root Agent span is automatically created as a child of the active tool span — the entire multi-agent flow appears in one trace. Top-level agents (no active parent span) start a new trace as usual.
* **Retriever-as-tool**: Retrievers wrapped with `@tool` are automatically traced as `TYPE=tool` spans.
* **JSONL local capture**: Inherited from `fiddler-otel`. Use `jsonl_capture_enabled=True` on `FiddlerClient`. The `FIDDLER_JSONL_FILE` environment variable overrides the output file path.
## 0.1
### **0.1.1**
*March 18, 2026*
Pre-release.
# Fiddler LangGraph SDK
Source: https://docs.fiddler.ai/changelog/langgraph-sdk
Fiddler's LangGraph SDK release history. Contains LangGraph SDK release highlights, SDK deprecation notices, and more.
[](https://pypi.org/project/fiddler-langgraph/)
## Deprecation notices
### Re-exported `fiddler-otel` symbols
Importing core `fiddler-otel` symbols directly from `fiddler_langgraph` is deprecated and will be removed in a future major release. This affects the following imports:
```python theme={null}
# Deprecated — will be removed in a future release
from fiddler_langgraph import FiddlerClient
from fiddler_langgraph import FiddlerSpan, FiddlerGeneration, FiddlerChain, FiddlerTool
from fiddler_langgraph import trace, get_current_span, get_client, set_conversation_id
```
**Recommended migration:** Import these symbols from `fiddler_otel` directly:
```python theme={null}
# Correct — import from fiddler_otel
from fiddler_otel import FiddlerClient
from fiddler_otel import FiddlerSpan, FiddlerGeneration, FiddlerChain, FiddlerTool
from fiddler_otel import trace, get_current_span, get_client, set_conversation_id
```
`LangGraphInstrumentor`, `add_session_attributes`, `add_span_attributes`, `set_llm_context`, and `clear_llm_context` remain in `fiddler_langgraph` and are not affected.
***
## 1.5
* **Bug Fixes**
* **Suppressed `gen_ai.llm.context` on routing-only LLM spans**: When an LLM span's output is purely `tool_calls` (a routing decision with no text content), the SDK now blanks `gen_ai.llm.context` on that span. Previously, RAG context set earlier in the same turn via `set_llm_context()` leaked onto these routing spans, triggering faithfulness evaluation on spans that produce no answer text.
* **Bug Fixes**
* **Fixed `wrapt` 2.x compatibility**: `wrapt` 2.0 renamed the first parameter of `wrap_function_wrapper()` from `module` to `target`. The SDK passed it as a keyword argument (`module='...'`), which raised `TypeError: wrap_function_wrapper() got an unexpected keyword argument 'module'` on `wrapt >= 2.0`. Fixed with a version-detecting helper that dispatches the correct keyword argument name based on the installed `wrapt` major version (`module=` for 1.x, `target=` for 2.x).
* **Dependencies**
* Bumps `fiddler-otel` dependency to `1.2.0`.
* **New Features**
* **`clear_llm_context()` function**: New convenience function to explicitly remove RAG context from an LLM instance, preventing faithfulness evaluation on subsequent non-RAG spans. In multi-step agent workflows, context set after a RAG retrieval step previously leaked into later non-RAG LLM calls (tool planning, routing, etc.), causing unintended faithfulness evaluation. `set_llm_context(llm, None)` also now clears context.
```python theme={null}
from fiddler_langgraph import set_llm_context, clear_llm_context
# After RAG retrieval
set_llm_context(llm, retrieved_documents)
response = llm.invoke(rag_prompt) # faithfulness evaluated
# Before non-RAG steps
clear_llm_context(llm)
plan = llm.invoke(planning_prompt) # no faithfulness evaluation
```
* **Python 3.14 support**: Added Python 3.14 to CI test matrix and PyPI classifiers. All runtime dependencies have compatible wheels. OpenTelemetry minimum bumped to `>= 1.28.0` (protobuf 4.x is incompatible with Python 3.14 due to metaclass `tp_new` changes; OTel `>= 1.28.0` resolves to protobuf 5.x).
* **Bug Fixes**
* **Fixed `@trace` nesting inside auto-instrumented LangGraph tools**: `@trace` decorated functions running inside LangGraph tools now correctly nest their spans under the callback handler's tool span instead of creating disconnected root traces. The root cause was that `_CallbackHandler` created OTel spans but never called `context.attach()`, making them invisible to Python's OTel context system. All start/end handlers now call `context.attach()`/`context.detach()` so that `start_as_current_span()` inside a tool finds the correct active parent span.
* **Improvements** (via `fiddler-otel 1.1.1`)
* **`add_session_attributes()` now accepts all OTel primitive value types**: Previously restricted to `str`, forcing callers to convert numeric metadata to strings. Now supports `str`, `bool`, `int`, `float`, and homogeneous sequences of these (per the OpenTelemetry attribute specification).
***
## 1.4
**Breaking change released as a patch version.** If you are upgrading from 1.4.1 or earlier, see the migration steps below.
* **Breaking Changes**
* **Module restructure — `fiddler_langgraph.tracing` removed and instrumentor class renamed**: The internal `tracing` sub-package has been removed and the instrumentor class has been renamed from `FiddlerLangChainInstrumentor` to `LangGraphInstrumentor`. All existing code using the old import path must be updated.
**Old import (1.4.1 and earlier):**
```python theme={null}
from fiddler_langgraph.tracing.instrumentation import FiddlerLangChainInstrumentor
instrumentor = FiddlerLangChainInstrumentor(client=client)
instrumentor.instrument()
```
**New import (1.4.2+):**
```python theme={null}
from fiddler_langgraph import LangGraphInstrumentor
instrumentor = LangGraphInstrumentor(client=client)
instrumentor.instrument()
```
**Migration:** Update the import path and rename `FiddlerLangChainInstrumentor` to `LangGraphInstrumentor`. The constructor signature and all methods (`instrument()`, `uninstrument()`) are identical — no other code changes required.
**Impact:** Upgrading to 1.4.2 without updating the import raises `ModuleNotFoundError: No module named 'fiddler_langgraph.tracing'`. If you cannot upgrade immediately, pin to `fiddler-langgraph==1.4.1`.
* **Other functions relocated from `fiddler_langgraph.tracing.instrumentation`**: The following utility functions were also in the removed `tracing` sub-package. They are now importable directly from `fiddler_langgraph`:
| Function | Old import (1.4.1 and earlier) | New import (1.4.2+) |
| ------------------------ | ------------------------------------------------------------------------------ | ------------------------------------------------------ |
| `add_session_attributes` | `from fiddler_langgraph.tracing.instrumentation import add_session_attributes` | `from fiddler_langgraph import add_session_attributes` |
| `add_span_attributes` | `from fiddler_langgraph.tracing.instrumentation import add_span_attributes` | `from fiddler_langgraph import add_span_attributes` |
| `set_llm_context` | `from fiddler_langgraph.tracing.instrumentation import set_llm_context` | `from fiddler_langgraph import set_llm_context` |
| `set_conversation_id` | `from fiddler_langgraph.tracing.instrumentation import set_conversation_id` | `from fiddler_otel import set_conversation_id` |
**Impact:** Any of these imports using the `fiddler_langgraph.tracing.instrumentation` path will raise `ModuleNotFoundError: No module named 'fiddler_langgraph.tracing'` when upgrading to 1.4.2.
* **`is_fiddler_span` utility relocated to `fiddler_otel.utils`**: The `is_fiddler_span()` function has moved from `fiddler_langgraph.core.utils` to `fiddler_otel.utils`. Update the import if you are using this utility directly.
**Old import (1.4.1 and earlier):**
```python theme={null}
from fiddler_langgraph.core.utils import is_fiddler_span
```
**New import (1.4.2+):**
```python theme={null}
from fiddler_otel.utils import is_fiddler_span
```
**Impact:** Upgrading to 1.4.2 without updating the import raises `ModuleNotFoundError: No module named 'fiddler_langgraph.core'`.
* **New features** (via `fiddler-otel 1.1.0`)
* **Offline / S3 routing mode**: `FiddlerClient` now accepts two new parameters for writing traces to local files instead of sending them directly to Fiddler. This enables deployments where security or network policies require data to pass through a controlled intermediary (such as Amazon S3) before reaching Fiddler.
| Parameter | Type | Default | Description |
| --------------------------- | ------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `otlp_enabled` | `bool` | `True` | Set to `False` to disable direct OTLP export. `api_key` and `url` are not required when `False`. |
| `otlp_json_capture_enabled` | `bool` | `False` | When `True`, writes traces to local `.json` files in standard OTLP JSON format (`ExportTraceServiceRequest` envelope) compatible with the Fiddler S3 connector. |
| `otlp_json_output_dir` | `str` | `'fiddler_traces'` | Output directory for OTLP JSON files. Created automatically if it does not exist. Each span batch is written to a separate timestamped file. |
```python theme={null}
from fiddler_langgraph import LangGraphInstrumentor
from fiddler_otel import FiddlerClient
# No api_key or url needed — traces written to local files only
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID', # Required — used by S3 connector to route traces
otlp_enabled=False,
otlp_json_capture_enabled=True,
otlp_json_output_dir='./fiddler_traces',
)
instrumentor = LangGraphInstrumentor(client=client)
instrumentor.instrument()
# Traces are written to ./fiddler_traces/*.json — upload these files to S3
```
See [Offline and S3 Routing Mode](/integrations/agentic-ai/langgraph-sdk#offline-and-s3-routing-mode) in the integration guide for full details.
* **Documentation**
* Updated the [LangGraph SDK integration guide](/integrations/agentic-ai/langgraph-sdk) to use `LangGraphInstrumentor` throughout — all code examples, quick start, configuration, and troubleshooting sections now reflect the new import path.
* **Bug Fixes**
* **Fixed `set_conversation_id()` ignored after first LangGraph invocation**: When using `LangGraphInstrumentor`, calling `set_conversation_id()` between agent invocations had no effect — every span after the first run carried the conversation ID from the initial call. The root cause was a `@cached_property` on the internal `_CallbackHandler` that read the conversation ID contextvar once on first access and froze that value for the lifetime of the `_CallbackHandler` instance. The cached property has been replaced with a direct contextvar read on every span, matching the behavior of `FiddlerSpanProcessor`. `set_conversation_id()` now takes effect immediately for all subsequent invocations.
* **Added**
* **Decorator-based instrumentation**: New `@trace()` decorator for automatic function instrumentation.
* `@trace()` decorator for instrumenting any Python function with minimal boilerplate.
* `get_current_span()` to access the current span inside decorated functions.
* Support for span types: `span`, `generation`, `chain`, `tool`, with type-specific wrappers and helpers (e.g. `set_user_prompt()`, `set_tool_input()`).
* Automatic parent-child relationship from call stack.
* Decorator arguments for metadata: `model`, `system`, `user_id`, `version`.
* `capture_input` and `capture_output` arguments for controlling automatic argument/return value capture.
* Automatic async function detection — `@trace()` works with both sync and async functions.
* **Manual instrumentation**: Context managers and explicit control.
* `client.start_as_current_span(name, as_type=...)` for context-manager usage with automatic lifecycle and exception recording.
* `client.start_span(name, as_type=...)` for explicit control; caller must call `span.end()`.
* Support for all span types with both patterns.
* **Span wrapper classes**: Typed wrappers with semantic convention helpers.
* `FiddlerSpan` (base), `FiddlerGeneration`, `FiddlerChain`, `FiddlerTool` — returned by `start_as_current_span()`, `start_span()`, and `get_current_span()`.
* `FiddlerGeneration` helpers: `set_model()`, `set_system()`, `set_user_prompt()`, `set_completion()`, `set_usage()`, `set_messages()`, `set_output_messages()`, `set_tool_definitions()`.
* `FiddlerTool` helpers: `set_tool_name()`, `set_tool_input()`, `set_tool_output()`.
* Common helpers on all wrappers: `set_input()`, `set_output()`, `set_attribute()`, `set_agent_name()`, `set_conversation_id()`.
* **Global client accessor**: `get_client()` retrieves the active `FiddlerClient` singleton for use in decorators and utility functions.
* **Context isolation**: Isolation from other OpenTelemetry tracers.
* Each `FiddlerClient` uses its own isolated `Context`.
* `is_fiddler_span()` utility to verify span ownership.
* **Enhancements**
* **Automatic Span Flush on Exit**: `FiddlerClient` now registers an `atexit` handler to automatically flush and shut down the tracer provider when the process exits, reducing span loss from in-memory buffering.
* **Explicit Flush and Shutdown Methods**: New `force_flush(timeout_millis)` and `shutdown()` methods on `FiddlerClient` for explicit control over span export. `shutdown()` is idempotent and safe to call multiple times.
* **Asyncio Support**: New `aflush()` and `ashutdown()` async methods run flush and shutdown in a thread pool, avoiding event loop blocking in asyncio applications.
* **Context Manager Support**: `FiddlerClient` can now be used as a context manager (`with FiddlerClient(...) as client:`) to ensure automatic shutdown on exit.
* **Bug Fixes**
* **Fixed Pydantic Double-Encoding**: Corrected JSON serialization of Pydantic models to use `model_dump()` instead of `model_dump_json()`, preventing double-encoded JSON strings in span attributes.
* **Deduplicated Retriever Span Attributes**: Removed duplicate `TYPE` and `TOOL_NAME` attribute assignments in the `on_retriever_start` callback.
* **Replaced `print()` with `logging`**: All `print()` calls in `jsonl_capture.py` now use the standard `logging` module for proper log management.
* **Enhancements**
* **OpenTelemetry Version Upgrade**: Updated OpenTelemetry dependencies to version 1.39.1/0.60b1 for improved performance and compatibility:
* `opentelemetry-api`: now supports up to 1.39.1
* `opentelemetry-sdk`: now supports up to 1.39.1
* `opentelemetry-instrumentation`: now supports up to 0.60b1
* `opentelemetry-exporter-otlp-proto-http`: now supports up to 1.39.1
* **Enhancements**
* **Removed Hardcoded OpenTelemetry Limits**: Removed hardcoded default values for span limits and batch span processor configuration. The SDK now relies on OpenTelemetry SDK's built-in defaults, simplifying configuration and ensuring consistency with standard OpenTelemetry defaults.
* **Enhanced LangGraph Tracing with Full Message History**: Introduced comprehensive message lifecycle tracking with two new span attributes:
* `gen_ai.input.messages`: Captures the complete message history provided as input to the LLM, including system, user, assistant, and tool messages
* `gen_ai.output.messages`: Captures the output messages generated by the LLM, including tool calls and finish\_reason when available
* Both attributes are aligned with GenAI semantic conventions for standardized observability
* **Extracted Message History from Strands Span Events**: Added support for extracting `gen_ai.input.messages` and `gen_ai.output.messages` from Strands span events (emitted as events rather than attributes) and storing them as span-level attributes in ClickHouse for unified querying and analysis.
* **Breaking Changes**
* **Moved `add_session_attributes` to Tracing Module**: The `add_session_attributes()` method has been relocated from the Core module to the Tracing module to co-locate session management functions with related tracing utilities.
* **Old import**: `from fiddler_langgraph.core.attributes import add_session_attributes`
* **New import**: `from fiddler_langgraph.tracing.instrumentation import add_session_attributes`
* **Migration**: Update import statements in your code. No functional changes—all behavior remains identical.
* **Impact**: Existing code using the old import path will fail with ImportError
Initial release of Fiddler LangGraph SDK.
# Fiddler OTel SDK
Source: https://docs.fiddler.ai/changelog/otel-sdk
Fiddler's OTel SDK release history. Contains OTel SDK release highlights, SDK deprecation notices, and more.
[](https://pypi.org/project/fiddler-otel/)
## 1.3
### **1.3.0**
*June 2026*
#### New features
* **`FIDDLER_OTLP_DEBUG` debug flag** — Set `FIDDLER_OTLP_DEBUG=true` to log OTLP trace exports for troubleshooting. Failed exports are logged at `WARNING` (with bearer tokens scrubbed) and successful exports at `DEBUG`, and wire-level HTTP logging is elevated for the underlying HTTP libraries and the OpenTelemetry HTTP exporter.
#### Bug fixes
* **Thread-safe lazy tracer initialization** — `FiddlerClient` now guards lazy tracer initialization with double-checked locking. Concurrent first-span creation from multiple threads no longer registers the OTLP `BatchSpanProcessor` twice, which would previously have doubled trace ingest volume.
***
## 1.2
### **1.2.0**
*May 2026*
#### New features
* **Multimodal content normalization** — Large inline base64 content (images, PDFs) in span attributes can now be automatically uploaded to S3 and replaced with lightweight `fiddler-file://` URIs before OTLP export. This prevents trace loss at the 10 MB Kafka span limit.
Enable with `normalize_multimodal=True` on `FiddlerClient`:
```python theme={null}
from fiddler_otel import FiddlerClient
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID',
api_key='YOUR_API_KEY',
url='https://your-instance.fiddler.ai',
normalize_multimodal=True, # opt-in, default False
)
```
| Detail | Value |
| ------------------ | ------------------------------------------------- |
| Upload threshold | \~100 KB base64 (\~75 KB raw) |
| Supported content | `image_url` parts in content arrays |
| Upload endpoint | `POST /v3/files/upload` |
| Failure behavior | Warning logged, inline base64 kept (no data loss) |
| Requires | `otlp_enabled=True`, valid `api_key` and `url` |
| Server requirement | `ENABLE_MULTIMODAL_UPLOAD` enabled |
* **Multimodal content setters** — `set_user_prompt()` now accepts `str | list[dict]`, allowing users to pass OpenAI-format multimodal content arrays directly without manual JSON serialization.
```python theme={null}
generation.set_user_prompt([
{'type': 'text', 'text': 'Describe this image'},
{'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,...'}},
])
```
#### Dependencies
* Added `httpx>=0.24.0,<1.0` — used by `MediaUploader` for file uploads with thread-safe connection pooling.
***
## 1.1
### **1.1.1**
*April 2026*
#### Bug fixes
* **`add_session_attributes()` now accepts all OTel primitive value types** — Previously, `value` was restricted to `str`, forcing callers to convert numeric metadata to strings before attaching it to a session. Numeric values now flow through unchanged, so downstream charts and filters that expect numeric types (e.g. integer counts, floating-point scores) work as intended.
Supported value types: `str`, `bool`, `int`, `float`, and homogeneous sequences of these (per the OpenTelemetry attribute specification).
```python theme={null}
from fiddler_otel import add_session_attributes
add_session_attributes(key='user_id', value='user_12345') # str
add_session_attributes(key='request_count', value=42) # int
add_session_attributes(key='confidence', value=0.87) # float
add_session_attributes(key='is_premium', value=True) # bool
```
### **1.1.0**
*April 2026*
#### New features
* **Offline / S3 routing mode** — Traces can now be written to local `.json` files in standard OTLP JSON format and routed through an intermediate store (e.g. Amazon S3) instead of being sent directly to Fiddler. This enables deployment in environments where security or network policies require all data to pass through a controlled intermediary before reaching Fiddler.
New `FiddlerClient` constructor parameters:
| Parameter | Type | Default | Description |
| --------------------------- | ------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `otlp_enabled` | `bool` | `True` | Set to `False` to disable direct OTLP export to Fiddler. When `False`, `api_key` and `url` are not required. |
| `otlp_json_capture_enabled` | `bool` | `False` | When `True`, writes traces to local `.json` files in standard OTLP JSON format (`ExportTraceServiceRequest` envelope) compatible with the Fiddler S3 connector. |
| `otlp_json_output_dir` | `str` | `'fiddler_traces'` | Directory for OTLP JSON output files. Created automatically if absent. Each span batch is written to a separate timestamped file. |
Usage example:
```python theme={null}
from fiddler_otel import FiddlerClient
# No api_key or url needed in offline mode
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID', # Required — used by S3 connector to route traces
otlp_enabled=False,
otlp_json_capture_enabled=True,
otlp_json_output_dir='./fiddler_traces',
)
```
Upload the generated `.json` files from `otlp_json_output_dir` to S3. The Fiddler S3 connector ingests them directly with no reformatting required.
* **`application_id` as first positional parameter** — `FiddlerClient.__init__` now accepts `application_id` as the first (and only strictly required) argument. `api_key` and `url` default to `''` and are only validated when `otlp_enabled=True`.
#### Clarifications (no functional change)
* `console_tracer=True` has always been **additive** — it prints spans to stdout alongside the existing OTLP export. It does **not** suppress export to Fiddler. The documentation has been updated to reflect this clearly.
* `jsonl_capture_enabled=True` has always been **additive** — it writes spans to a local JSONL file alongside the existing OTLP export. It does **not** suppress export to Fiddler. Note: the JSONL format is Fiddler's custom format and is **not** compatible with the S3 connector; use `otlp_json_capture_enabled=True` for S3 ingestion.
#### Breaking changes
* **Parameter reordering in `FiddlerClient.__init__`**: `application_id` is now the first parameter, before `api_key` and `url`. Code using **keyword arguments** (the recommended pattern) is unaffected. Code passing arguments **positionally** will break — update to use keyword arguments.
***
## 1.0
### **1.0.0**
*March 18, 2026*
Initial release of the Fiddler OTel SDK as a standalone package. The core OpenTelemetry instrumentation functionality has been extracted from `fiddler-langgraph` into this independent package, which now serves as the foundation for all Fiddler framework integrations.
#### Core classes
* **`FiddlerClient`**: Main client that configures and manages the OTel tracer. Handles OTLP authentication, compression, span limits, sampling, and lifecycle (flush/shutdown). Registers an `atexit` handler for automatic span flushing.
* **`@trace` decorator**: Automatic function tracing with input/output capture. Supports sync and async functions, all four span types (`span`, `generation`, `chain`, `tool`), and decorator parameters for `model`, `system`, `user_id`, and `version`.
* **`get_current_span()`**: Retrieve the active Fiddler span inside a `@trace`-decorated function as a typed wrapper.
* **`get_client()`**: Retrieve the global `FiddlerClient` singleton.
#### Span wrappers
`FiddlerSpan` (base), `FiddlerGeneration`, `FiddlerChain`, `FiddlerTool` — typed wrappers with semantic convention helpers.
* `FiddlerGeneration`: `set_model()`, `set_system()`, `set_system_prompt()`, `set_user_prompt()`, `set_completion()`, `set_usage()`, `set_context()`, `set_messages()`, `set_output_messages()`, `set_tool_definitions()`
* `FiddlerTool`: `set_tool_name()`, `set_tool_input()`, `set_tool_output()`, `set_tool_definitions()`
* All wrappers: `set_input()`, `set_output()`, `set_attribute()`, `set_agent_name()`, `set_agent_id()`, `set_conversation_id()`, `record_exception()`, `end()`
#### Context and session
* **`set_conversation_id()`**: Set the conversation ID as a `ContextVar` that propagates to all spans in the current thread or async task.
* **`add_session_attributes(key, value)`**: Attach session-level metadata to all spans in the current thread or async coroutine. Emitted as `fiddler.session.user.{key}` and automatically propagated from parent to child spans.
* **`FiddlerSpanProcessor`**: OTel `SpanProcessor` that auto-propagates `gen_ai.agent.name`, `gen_ai.agent.id`, `gen_ai.conversation.id`, `session.id`, and `user.id` from parent to child spans.
* **Context isolation**: Each `FiddlerClient` uses its own isolated OTel `Context`, preventing interference with any existing global tracer in the application.
#### Configuration and debugging
* **JSONL local capture**: `jsonl_capture_enabled` and `jsonl_file_path` constructor parameters for local span capture. The `FIDDLER_JSONL_FILE` environment variable overrides the file path when `jsonl_file_path` is not set explicitly.
* **Console tracing**: `console_tracer=True` parameter to print spans to stdout during development.
* **Flush and shutdown**: `force_flush()`, `shutdown()`, `aflush()`, `ashutdown()`, and context manager support.
#### Constants
* **`FiddlerSpanAttributes`**: Constants for all Fiddler OTel span attribute keys.
* **`FiddlerResourceAttributes`**: Constants for Fiddler OTel resource attribute keys.
* **`SpanType`**: Constants for valid `fiddler.span.type` values.
## 0.1
### **0.1.1**
*March 17, 2026*
Pre-release.
# Product Releases
Source: https://docs.fiddler.ai/changelog/product-releases
Discover the latest updates to Fiddler's AI observability platform - new features for ML, LLM, GenAI, and agentic observability.
### Release 26.19
#### What's New and Improved
##### **Multi-Application Trace Routing**
This capability is currently in [private preview](/reference/feature-maturity-definitions#private-preview) and is not yet available to all customers. Contact your Fiddler Customer Success Manager to request access.
Send OpenTelemetry traces for many Fiddler applications through **one endpoint and one API key**.
Tag each span (or its resource) with an `application.id`, and Fiddler authorizes every span against the applications your API key may write to — storing the authorized ones under their application and returning the rest in the OTLP `partialSuccess` response.
A single shared gateway or collector can serve every application, with no separate pipeline or key per application.
See [Multi-Application Trace Routing](/integrations/agentic-ai/multi-application-trace-routing) for setup.
#### Deprecations and Removals
##### **Fiddler SDKs: Upcoming Python 3.10 Removal**
The Fiddler agentic SDKs — `fiddler-otel`, `fiddler-langgraph`, `fiddler-langchain`, `fiddler-strands`, and `fiddler-adk` — and the Fiddler Python client `fiddler-client` will drop support for Python 3.10 in a future release.
Python 3.10 reaches end-of-life in October 2026 (PEP 619).
These packages currently support Python 3.10 through 3.14; the release that removes Python 3.10 support will be announced ahead of time.
* *Action Recommended*: Plan to move environments running these SDKs to Python 3.11 or newer before the removal release.
### Release 26.18
#### What's New and Improved
##### **Claude Code Tool Input/Output Extraction**
Tool spans from [Claude Code](/integrations/agentic-ai/claude-code-integration) now show tool input arguments and output content when `OTEL_LOG_TOOL_CONTENT=1` is enabled. Fiddler extracts tool output and input from `tool.output` span events with per-tool dispatch:
* **Read** → output from `content`, input from `file_path`, `offset`, `limit`
* **Bash** → output from `output`, input from `bash_command`
* **Edit** → output from `diff`, input from `file_path`
* **Write** → no output (`content` and `file_path` are both inputs)
* **Unknown/MCP tools** → falls back to `content` as the output key
Non-string attribute values (e.g. `offset=10`) are preserved as native types in the JSON-serialized input.
##### **Fiddler Claude Code Plugin**
This capability is currently in [private preview](/reference/feature-maturity-definitions#private-preview) and is not yet available to all customers. Contact your Fiddler Customer Success Manager to request access.
The new [Fiddler Claude Code Plugin](/integrations/agentic-ai/fiddler-claude-code-plugin) captures per-turn OpenTelemetry traces from your Claude Code sessions — user prompt, LLM response, tool input/output, and token usage — and delivers them to Fiddler, so coding-agent activity lands alongside your other GenAI traces for exploration and scoring.
The plugin runs as Claude Code hooks and fails open, so a hook error never breaks or slows your session.
It requires Claude Code 2.1.120 or later.
Contact your Fiddler Customer Success Manager to get the plugin during private preview.
##### **Auto-Enrollment of Searchable Keys from Semantic Mappings**
Attributes mapped to the `input`, `output`, `tool_input`, or `tool_output` semantic concepts are now automatically enrolled for [full-text search](/observability/agentic/trace-explorer#searching-in-the-ui) in the Explorer — no separate configuration step required.
* *Applies to new data only*: existing spans are not re-indexed.
* *One-way enrollment*: removing a semantic mapping does not remove the attribute from the search index.
##### **LLM Gateway: New Models**
The LLM Gateway model catalog has been refreshed with the latest mainline Claude models on Azure AI (Microsoft Foundry).
Select them in **Settings > LLM Gateway** when you add or edit an Azure AI provider.
* **Azure AI** — `claude-fable-5`, `claude-opus-5`, `claude-sonnet-5`
### Release 26.17
#### What's New and Improved
##### **Guardrails for AgentGateway**
Explicit support for [implementing Fiddler guardrails in AgentGateway](/protection/agentgateway-guardrails),
redacting PII and secrets in real time before requests reach your LLM.
* **In-place masking** — unlike the Kong and LiteLLM integrations' block-only
behavior, AgentGateway's webhook protocol supports masking: detected PII and
secrets are redacted in the request or response, and the sanitized call
still proceeds instead of being rejected outright.
* *Scope*: PII and secrets, checked on both the request and response path.
* *Availability*: Requires AgentGateway version 1.4.0 or later.
##### **Track Errors in GenAI Applications with the New Status Metric**
Charts for GenAI applications can now monitor span errors with the
new **Status** metric, available in the chart builder alongside Traffic and Latency.
* **Chart error counts over time**: select the Status metric and choose which
span status values to count — `Error`, `Ok`, or `Unset` —
with one series plotted per selected value.
* **Consistent with Traffic**: status counts are computed over the same spans
as the Traffic metric,
so you can compare error volumes against total traffic to reason about error rates.
* **Built on OTel span status**: counts come directly from the OpenTelemetry
`StatusCode` recorded on your spans —
no evaluator, enrichment, or extra instrumentation required.
* **API support**: available programmatically via the charts API using
`metric_name: status` with the `status_values` parameter.
##### **LLM Gateway: New Models**
The LLM Gateway model catalog has been refreshed with the latest mainline models.
Select them in **Settings > LLM Gateway** when you add or edit a provider.
* **OpenAI** — `gpt-5.6`
* **Gemini** — `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash-lite`
* **Vertex AI** — `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash-lite`
##### **LLM Gateway: Model Deprecation Cues in the Model Pickers**
Model pickers now show when a model is scheduled for retirement or no longer available,
so you find out before a run fails rather than after.
* **Where**: **Settings > LLM Gateway** when you add or edit a provider, and the model picker in the evaluator form.
* **Retired models stay visible but cannot be selected**, so an evaluator already pointed at one keeps its selection instead of silently switching.
### Release 26.16
#### What's New and Improved
##### **Fiddler MCP Server**
The Fiddler MCP Server is currently in [public preview](/reference/feature-maturity-definitions#public-preview).
Fiddler now provides a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server,
so MCP-compatible AI assistants like Claude Code and OpenCode can explore your
applications, investigate traces, review evaluator results, and monitor metrics directly from your editor.
* **Set up from Settings > MCP**: copy a ready-to-use configuration for Claude Code, OpenCode, or any other OAuth 2.0-capable MCP client.
* **OAuth 2.0 authentication**: your client signs in through Fiddler in the browser and acts with your own account's permissions, with standard RBAC applied. No static keys in config files.
* **Tool coverage**: the server covers projects and apps, evaluators and rules, traces and sessions, charts and dashboards, alerts, metrics, LLM Gateway, evals datasets and experiments, attributes and annotations, and backfills.
* **Learn more**: see [Fiddler MCP Server](/integrations/agentic-ai/mcp-server) for setup instructions and details.
##### **OTel Log Ingestion for Claude Cowork**
This capability is currently in [private preview](/reference/feature-maturity-definitions#private-preview) and is not yet available to all customers. Contact your Fiddler Customer Success Manager to request access.
Fiddler now ingests Claude Cowork's OTel Log Records alongside existing trace data.
Contact your Fiddler Customer Success Manager to enable this feature for your deployment.
##### **Semantic Mappings Are Now the Default for Cross-Framework Trace Analytics**
Semantic mappings are now enabled by default for all organizations and are the primary mechanism for resolving OTel attribute keys to canonical concepts. Previously available as an opt-in feature, this removes the need to adopt Fiddler-specific naming conventions in your instrumentation.
* Raw attribute keys are stored exactly as sent by your application; Fiddler resolves them to canonical semantic concepts (like `input_tokens`, `model_name`, `input`) server-side using the semantic mappings table.
* 140+ pre-configured mappings ship out of the box, covering 13+ frameworks — OTel GenAI Semantic Conventions, OpenInference, Vercel AI SDK, Claude Code, LiteLLM, Langfuse, Google ADK, LiveKit, Genkit, MLflow, Traceloop/OpenLLMetry, and more.
* All downstream features — dashboards, alerts, custom metrics, evaluators, and the Explorer — query by semantic concept, so the same chart or alert works identically regardless of which framework produced the trace.
* **New Settings tab**: manage mappings from **Settings > Semantic Mappings** — view all current mappings, and add or delete custom mappings (Org Admin role required).
* **Custom metrics**: the `attribute()` FQL function now accepts a `semantic_name` argument as an alternative to a verbatim attribute `name` — for example, `average(attribute(semantic_name='input_tokens'))`. The `scope` and `type` arguments are no longer required; they default to `span` and `user`. Existing metric definitions continue to work unchanged. See [Custom Metrics for Agentic Applications](/observability/agentic/custom-metrics) for the full parameter reference and the list of supported semantic concepts.
* *Deprecation*: the `fiddler.span.system.*`, `fiddler.span.user.*`, and `fiddler.contents.*` prefix conventions are now deprecated. Existing data using these prefixes continues to resolve correctly via backward-compatible default mappings, so no migration is required for existing traces.
* *Benefit*: A single dashboard, alert, or custom metric works consistently across all supported frameworks — no per-framework instrumentation or client-side attribute renaming required.
* See [Semantic Mappings](/concepts/semantic-mappings) for the full concept reference.
##### **FQL: `in` and `not_in` for Multi-Value Matching**
FQL now has `in` and `not_in` functions, so matching a value against a list no longer requires a chain of `or` comparisons.
* **Available in ML segments, and in custom metrics for both ML models and GenAI applications.** For ML, write `in(country, 'US', 'CA', 'MX')` instead of `country == 'US' or country == 'CA' or country == 'MX'`. For GenAI expressions, use `in(attribute('country'), 'US', 'CA', 'MX')`.
* Values must be literals of a single type, and that type must match the tested expression. A null input returns null — see [Fiddler Query Language](/observability/platform/fiddler-query-language).
* *Benefit*: Cleaner, more readable filters — match a value against a list without chaining `or` comparisons.
* `greatest` and `least` now document that each accepts **one or more** aggregates — a documentation correction with no behavior change, since one-argument calls like `greatest(sum(col1))` were always valid. Both functions are now offered in the FQL editor's autocomplete.
##### **More Responsive Custom LLM-as-a-Judge Evaluators on Fiddler Centor Models**
Custom LLM-as-a-Judge evaluators running on select [Fiddler Centor Models](/glossary/centor-models) now complete significantly faster.
* *Scope*: Custom Judge evaluators backed by select Fiddler Centor Models
* *Benefit*: Evaluation latency reduced by 22–42% in internal benchmarks, with no change to scoring accuracy
##### **Guardrails for Kong AI Gateway**
Explicit support for [implementing Fiddler guardrails in Kong AI Gateway](/protection/kong-guardrails).
* *Availability*: The Kong AI Gateway adapter shipped in Release 26.14 and is now documented. Deployments on Release 26.14 or later already have it — no upgrade to 26.16 is needed.
##### **Annotations for Span-Level Human Evaluation**
Annotations is currently in [public preview](/reference/feature-maturity-definitions#public-preview).
Add human evaluation scores directly to individual spans in the Explorer. Annotations provides a human-in-the-loop mechanism for reviewing and scoring LLM application outputs alongside automated evaluators.
* **Score types**: Numeric, boolean, categorical, and free-text annotations, each with optional reasoning
* **In-context workflow**: Click **Annotate** on any span in the Explorer to add a new annotation, or manage existing ones directly from the **Annotations** section in the span detail panel
* **API access**: Create, read, update, and delete annotations programmatically via the `/v3/scores` API with `source=human`
##### **LLM Gateway: Refreshed Model Catalog**
* The LLM Gateway model catalog has been refreshed with current models from Anthropic, OpenAI, Gemini, and Vertex AI. Select them in **Settings > LLM Gateway** when you add or edit a provider.
* Models that have reached end-of-life are also removed from the catalog:
* **Anthropic** — `claude-opus-4-1-20250805`, `claude-opus-4-20250514`, `claude-sonnet-4-20250514`, `claude-3-5-haiku-20241022`
* **Databricks** — `databricks-meta-llama-3-1-405b-instruct`
#### What's Fixed
* **LLM Gateway: Models the provider no longer serves now fail immediately** — During ingestion, a span matching an evaluator rule is no longer retried when the error is due to model unavailability or incorrect credentials.
#### Deprecations and Removals
##### **Legacy LLM Ingestion Pipeline**
The legacy LLM ingestion pipeline — publishing prompts and responses as events for enrichment scoring — is deprecated in favor of OTel trace ingestion, and reaches end of life on **January 31, 2027**.
* *Action Required*: Migrate to the OTel ingestion pipeline before the end-of-life date, or contact your Fiddler Customer Success Manager for migration guidance.
##### **Legacy API Keys**
Legacy single-value API keys are deprecated in favor of named API keys, and reach end of life on **January 31, 2027**.
* *Action Required*: Migrate to named API keys under **Settings > Credentials** — see [Managing Credentials](/reference/administration/settings#credentials).
### Release 26.15
#### What's New and Improved
**Install Fiddler Agent Skills From the Docs Site**
Fiddler now publishes agent skills in the open agentskills.io format directly from the documentation site, so your AI coding agent can learn how to use Fiddler without you pasting docs into the chat.
* **Connect the docs MCP server** — start here: connect the documentation MCP server at `https://docs.fiddler.ai/mcp` and a connected agent can search the docs and read the skills without installing them.
* **One-line install** — run `npx skills add https://docs.fiddler.ai` to add the `fiddler` orientation skill and `fiddler-ml-onboarding` (end-to-end ML model onboarding and event publishing) to your supported agents.
* **New guide** — [Use Fiddler with AI Agents](/getting-started/use-fiddler-with-ai-agents) covers MCP setup, install, the manual fallback, and a tested-only compatibility matrix.
### Release 26.14
#### What's New and Improved
**Sensitive Information Guardrail: Curated PII Defaults and Score Normalization**
The Sensitive Information guardrail, powered by the [Centor Model for PII/PHI](/glossary/centor-models), now ships with a curated default PII entity set and per-entity score normalization for more consistent detection across entity types.
* **`PII` (default)** — 12 high-precision entities benchmarked across multiple datasets: person, address, email, phone number, credit card number, and more.
* **`PII_ALL` (new)** — All 28 supported PII entities for maximum coverage.
* **`PHI`** — 7 protected health information entities (unchanged).
* Per-entity score normalization ensures a single confidence threshold works uniformly across all entity types.
* If you need the full 28-entity set, use `entity_categories='PII_ALL'`.
**Explorer: Resource Attributes in Single-Trace View**
The single-trace (RCA) view now displays **resource attributes** for each span in a dedicated collapsible section. OTel resource-level attributes such as `service.name`, `deployment.environment`, and SDK info are shown inline, helping identify the origin of any span directly in the trace details view.
**Faster and More Consistent Pre-built LLM-as-a-Judge Evaluators**
All pre-built LLM-as-a-Judge evaluators now run faster and more consistently across all supported models, with no accuracy impact. Fiddler-hosted models see the largest improvement: 50–81% latency reduction depending on evaluator.
* *Scope*: Answer Relevance, Context Relevance, RAG Faithfulness, Conciseness, and Coherence evaluators
* *Benefit*: Evaluations complete faster with more predictable latency regardless of which LLM model is used
**LLM Gateway: OAuth 2.0 Authentication for Databricks**
You can now connect a **Databricks** provider in the LLM Gateway using **OAuth 2.0 (client-credentials / machine-to-machine)** authentication — a Databricks **service principal** instead of a static personal access token.
* Add a Databricks credential of type **OAuth Client Credentials** with your service principal's **Token URL**, **Client ID**, **Client Secret**, and **Scope**. Fiddler exchanges these for a short-lived bearer token and refreshes it automatically — no long-lived tokens to manage or rotate.
* *Availability*: Databricks only.
* *Benefit*: Use enterprise machine-to-machine auth for Databricks model serving instead of static, time-limited access tokens.
**Evaluator Downsampling for GenAI Applications**
This capability is currently in [private preview](/reference/feature-maturity-definitions#private-preview) and is not yet available to all customers. Contact your Fiddler Customer Success Manager to request access.
Evaluator rules can now score a representative sample of spans instead of every span, cutting evaluator cost and latency on high-volume GenAI applications. Set a `sampling_rate` greater than 0 and up to `1.0` (default `1.0` — no sampling) on any evaluator rule.
* Configure via the v3 evaluator-rules REST API, the evaluator rule UI, or MCP tools.
* A soft cap of 100 evaluator rules per application provides an early guardrail against rule sprawl.
* *Benefit*: Keep evaluator coverage on high-volume applications without paying to score every span.
#### What's Fixed
* **Charts: custom `token_count` configurations are preserved on edit** — Charts now retain their original metric parameters through a token-count round-trip, so custom token-count chart configurations are no longer silently dropped when charts are edited and saved.
* **Trace queries: out-of-memory failures on large GenAI applications are resolved** — Trace fetches now paginate attribute and evaluator lookups by span ID, preventing out-of-memory failures on large trace queries and stabilizing trace fetches under heavy load.
* **Ingestion: intermittent SSL connection errors are resolved** — Ingestion workers now dispose of their database engine after forking, correcting a connection-sharing defect that caused intermittent SSL errors and could destabilize ingestion.
* **ClickHouse: transient read failures are retried automatically** — A transient ClickHouse read error (`ATTEMPT_TO_READ_AFTER_EOF`) is now treated as retryable, so momentary read hiccups no longer surface as hard errors.
* **Custom Judge label validation now handles extra whitespace** — Custom Judge evaluators now strip leading and trailing whitespace from string outputs before validating them against configured choices, so values such as `"High "` match the `"High"` choice and downstream alert filters trigger reliably.
### Release 26.13
#### What's New and Improved
**Explorer: Span Events and Links in Single-Trace View**
The single-trace (RCA) view now displays **events** and **links** for each span in dedicated collapsible sections.
* **Events** — OTel span events (e.g., exceptions, log entries, `gen_ai.content.prompt` events) are shown with their timestamp, name, and attributes. Timestamps are formatted in your local timezone.
* **Links** — Links to related spans in the same or different trace are shown with Trace ID, Span ID, and optional trace state and attributes. Copy buttons allow quick extraction of IDs.
* Both sections are collapsed by default when empty, keeping the view clean for spans without events or links.
* *Benefit*: Access the full OTel span context — exceptions, cross-trace references, and structured event data — directly in the RCA view without needing to export traces to another tool.
**GenAI Traces TTL Can Now Be Configured via the Config Manager API**
This capability is currently in [private preview](/reference/feature-maturity-definitions#private-preview) and is not yet available to all customers. Contact your Fiddler Customer Success Manager to request access.
The retention window for GenAI trace, span, session, and evaluator data (introduced in 26.9) can now be configured at the organization level at runtime through the Config Manager APIs, without redeployment. Platform administrators can set the retention window to a minimum of 7 days. New data adopts the configured retention immediately.
**Fiddler Centor Model for Safety v2.7.23**
[Fiddler Centor Model for Safety](/observability/llm/enrichments#safety) has been updated with improved detection and calibrated scoring.
* **Stronger Jailbreak Detection**: Significantly improved recall on jailbreak and prompt-injection attempts
* **Fewer False Positives**: Reduced false flags on benign content such as technical jargon and roleplay
* **Calibrated Scoring**: Scores are calibrated so a single decision threshold of `0.5` currently applies across all eleven safety categories — no per-label threshold tuning required
* *Note*: No changes to the model interface or API/SDK identifiers (`ftl_prompt_safety`); existing integrations should adopt the new `0.5` decision threshold
#### What's Changed
**Custom Evaluator Output Description Field Removed**
The "Description" field on Custom Judge (LLM-as-a-Judge) output definitions has been removed from the UI. This field had no measurable impact on evaluator response. If you were using it for evaluation instructions, move those instructions into the prompt template instead.
**GenAI Monitoring Dashboards Under the v2 OTel Schema**
Attribute-based GenAI metrics — token counts and other numeric and categorical attributes — could return empty results on monitoring dashboards when the v2 OTel schema was enabled, because attribute lookups did not account for the stripped `fiddler.span.` name prefix. Attribute queries and filters now resolve correctly, restoring affected charts.
### Release 26.12
#### What's New and Improved
**Bulk Trace and Session Deletion for GenAI Applications**
You can now delete individual traces and sessions from GenAI applications via the REST API, enabling granular data cleanup without deleting the entire application.
* **Delete traces**: `DELETE /v3/traces` accepts an `application_id` and up to 1,000 `trace_ids` per request. All associated data within the specified traces are removed.
* **Delete sessions**: `DELETE /v3/sessions` accepts an `application_id` and up to 1,000 `session_ids` per request. All associated data within the specified sessions are removed.
* **Idempotent**: Requests succeed even if some or all of the specified IDs do not exist — no errors for already-deleted or non-existent data.
* **Authorization**: Deletion requires `DELETE` permission on the project that owns the application. Each request is scoped to a single application.
* **Rate limited**: Per-token rate limits apply to protect against accidental bulk data loss. See [REST API Rate Limiting](/sdk-api/rest-api#rate-limiting).
* *Use cases*: Designed for ad-hoc removal of test data or PII traces, rather than a programmatic cleanup workflow.
* *Scope*: Available for all GenAI Applications. Deletion is synchronous — data is invisible to subsequent queries immediately after the request returns.
**Explorer: Complete Span Attributes in Single-Trace View**
The single-trace (RCA) view in the Explorer now displays **all** attributes for **every** span type, grouped by semantic concept.
* Previously, span attributes were only shown for LLM spans, using a fixed set of curated fields, so many attributes — and entire span types such as tool and chain spans — never surfaced their details.
* Attributes are now resolved against the global semantic-mappings table and grouped under human-readable categories (Input, Output, Tool Input, Tool Output, and more), with any unmapped attributes collected under an **Unmapped Attributes** group. Empty categories are hidden.
* Custom-integration spans (for example, Claude Code) that previously showed empty System Input / User Input / Output now populate correctly.
* *Benefit*: Inspect the full set of attributes on any span — not just LLM spans — for faster, more complete root-cause investigation.
**Agentic Alert Backtesting in the Alert Chart Preview**
When you create a GenAI (agentic) alert rule, the Chart Preview now backtests your thresholds against historical data so you can validate an alert before saving it.
* The preview plots your selected metric over a chosen time range with the **critical** and **warning** thresholds overlaid as mark-lines, so you can see exactly where past data would have breached.
* The preview updates as you adjust the metric, thresholds, and comparison operator in the rule form.
* *Benefit*: Tune alert thresholds with confidence by seeing how they would have behaved on real historical data, reducing noisy or missed alerts.
**Date-Time Range Picker for Alert Backtesting**
The Chart Preview header now includes an interactive date-time range picker, replacing the previous fixed "Last 30 days" view.
* Choose from preset ranges or a custom date range to control the backtest window.
* Defaults to the last 30 days; the chart x-axis follows the selected range.
* *Benefit*: Backtest alert thresholds over the exact time period you care about — a recent incident, a seasonal window, or a longer historical baseline.
**Breach Summary in the Alert Chart Preview**
A breach summary now appears below the Chart Preview, quantifying how often your thresholds would have triggered over the selected range.
* Shows the count of **critical** and **warning** breaches out of the total number of data points in the range.
* A data point that breaches both thresholds is counted once, as critical.
* *Benefit*: Get an at-a-glance sense of how sensitive an alert configuration is before saving it.
**Trace Explorer is Now "Explorer"**
We renamed the Trace Explorer to the Explorer — a simpler name that better reflects the view's purpose.
* Page names and routes no longer include the word "trace."
* Existing bookmarks redirect automatically to the new Explorer URLs.
* *Benefit*: A clearer, shorter name for the view, with no disruption to your saved links.
**Add Spans to an Evaluation Dataset from the Explorer**
You can now select spans directly in the Explorer and add them to an evaluation dataset, turning real production traffic into test data without leaving the page.
* Select spans with the new per-row checkboxes, then choose **Add to dataset** in the toolbar.
* A guided dialog walks you through choosing the target dataset and mapping span fields — Input, Output, Tools & Retrieval, and Attributes — to dataset columns by drag-and-drop, with a final review step before you confirm.
* *Benefit*: Build and grow evaluation datasets from the traces you are already investigating, so your test cases reflect real user traffic.
**Secret Detection in Prompts and Responses**
Fiddler can now detect leaked credentials — API keys, tokens, and other secrets — in LLM prompts and responses, both in real time and in offline scans.
* **Real-time guardrail**: Fiddler Guardrails block or redact secrets before a request reaches the model.
* **Offline evaluation**: The new [`FTLSecretDetection`](/sdk-api/evals/ftl-secret-detection) evaluator scans datasets for credentials, returning one score per detected secret with its type label.
* Detection covers 42 known credential formats and also catches custom or unknown secrets that don't match a known format, with sub-millisecond latency.
* *Benefit*: Catch credential leakage in agent and LLM traffic before it is stored or sent downstream. See the [Secret Detection tutorial](/developers/tutorials/guardrails/guardrails-secrets) to get started.
**Kong AI Gateway Integration**
You can now capture LLM observability data through the [Kong AI Gateway](/integrations/agentic-ai/kong-integration) (v3.13+) using Kong's OpenTelemetry plugin.
* Point your application at Kong instead of the provider, and Kong exports traces — prompts, responses, token usage, and latency — directly to Fiddler over OTLP.
* Supports OpenAI, Anthropic, Cohere, Azure OpenAI, Google Gemini, and other providers via Kong's `ai-proxy` plugin.
* *Benefit*: Get full LLM observability with zero application instrumentation — no SDK to add to your code.
**Executive Dashboard**
This capability is currently in [private preview](/reference/feature-maturity-definitions#private-preview) and is not yet available to all customers. Contact your Fiddler Customer Success Manager to request access.
A new cross-project Executive Dashboard gives organization leadership a single, at-a-glance overview of GenAI and ML health across every application.
* **Top-line KPIs** — Total Apps, Active Apps, Total Traces, Errors, and Sessions over a selectable 7-day, 14-day, or 30-day window.
* **Evaluator Scores** — Average Quality alongside Answer Relevance, Context Relevance, Faithfulness, Prompt Safety, and PII Detection, aggregated across all applications.
* **Traffic and Token Usage** trends, plus an **Application Health Overview** table (traces, average latency, tokens, and average quality per app) with drill-down links into each application.
* *Benefit*: Give leadership a cross-project view of GenAI adoption, quality, and cost in one place — without stitching together per-application dashboards.
#### Improvements
**Score Distribution in Experiment Results**
Each score column in the Experiment Details results grid now shows a mini distribution chart in its header — a histogram for numeric evaluators, a stacked bar for categorical ones — with a `Mean · Pass %` summary below it. The pass rate is computed from each evaluator's configured thresholds, so it stays consistent with the color-coded scores in the cells. *Benefit*: See the shape of your results at a glance, without scrolling the grid.
**Evaluator Rule Limit**
Each application now supports up to 100 evaluator rules by default, keeping evaluation and monitoring responsive as rule counts grow. Creating a rule beyond the limit returns a `422` error; delete unused rules to make room. On self-hosted deployments, an administrator can adjust the limit. See [Evaluator Rules](/evaluate-and-test/evaluator-rules#rule-limit).
### Release 26.11
#### What's New and Improved
**Trace Explorer Filtering Improvements**
* Introduced a new filtering sidebar/persistent drawer to the Trace Explorer UI with more robust filtering options.
* Search for any column to filter from the search input, with options to sort and filter the results.
* Filter settings persist in local storage and as URL params, making it easier to save and share views.
**Long-Running Jobs Alert on Jobs In Progress**
The Jobs In Progress tab now surfaces jobs that have been queued or running for more than 2 hours, helping you spot stalled ingestion and processing work sooner.
* A warning banner appears when one or more jobs have been queued or running for over 2 hours.
* Click **Show long-running jobs** in the banner to filter the grid to just those jobs; an info banner indicates the filter is active and can be cleared with one click.
* The Duration column is now filterable by elapsed running time (in hours) directly from the column filter menu (for example, `Duration > 2`).
* The grid defaults to sorting by Duration so the longest-running jobs appear at the top.
* *Benefit*: Detect stuck or slow jobs at a glance and drill into them without manually scanning the list.
#### Client Versions
| Component | Version |
| --------------------- | ------- |
| Fiddler Python Client | 3.12.0 |
| Fiddler Evals SDK | 0.3.0 |
| Fiddler LangChain SDK | 1.1.2 |
| Fiddler LangGraph SDK | 1.5.2 |
| Fiddler Strands SDK | 0.6.0 |
| Fiddler OTel SDK | 1.2.0 |
The Fiddler Python client 3.12.0 migrates to **Pydantic v2** (`pydantic>=2.0`) and adds **Python 3.13 and 3.14** support. Environments pinned to Pydantic v1 must upgrade. Refer to the [Python client changelog](/changelog/python-sdk) for version-specific details.
#### Deprecations and Removals
**`Model.publish()` is deprecated (Python client 3.12.0)**
The implicit-dispatch `Model.publish()` method is deprecated in favor of two explicit methods; calling it now emits a `DeprecationWarning`.
* **Replacement**: Use `Model.publish_stream(events=...)` for streaming/live events, or `Model.publish_batch(source=...)` for batch files (Parquet/CSV) and pandas DataFrames.
* *Action Required*: Update calls to `model.publish(...)`. Pass `update=True` to either method to update previously published events.
* See the [Python Client SDK changelog](/changelog/python-sdk) and [Publishing production data](/developers/client-library-reference/publishing-production-data).
### Release 26.10
#### What's New and Improved
**Fiddler Centor Models — Product Rebrand**
Fiddler's purpose-built LLM evaluators are now called **Fiddler Centor Models**. This release updates all documentation and UI labels to reflect the new name.
* **Fiddler Trust Models** → **Fiddler Centor Models** (product family name)
* **Fast Safety** → **Centor Safety**
* **Fast Faithfulness** → **Centor Faithfulness**
* **Fast PII** → **Centor PII**
* API field names and SDK identifiers (`ftl_prompt_safety`, `ftl_response_faithfulness`, etc.) are **unchanged** — no code changes are required.
* Alongside this rename, Fiddler has launched the [Fiddler Evals TCO calculator](https://www.fiddler.ai/evals-tco-calculator), which lets you compare the cost of Centor Models to third-party LLM evaluators.
**RCA Column Filtering for Predictive/LLM Monitoring Charts**
You can now filter by any column in the RCA chart for Predictive and LLM Monitoring charts.
* Click the filter icon in the upper-right corner of the chart, or open the column header menu on any column.
* Specify up to 10 filter conditions; data is filtered server-side.
* Column filters combine with existing saved segments or ad-hoc segments; all conditions are joined with AND.
* Must be enabled with feature flag: `enableRCAColumnFilteringUI`
* *Scope*: Applies to Predictive and LLM Monitoring charts. Not available for GenAI Application charts.
* *Benefit*: Narrow the RCA view to a specific data slice without leaving the chart, and layer column filters on top of saved or ad-hoc segments for targeted root-cause investigation.
**Project Admin Access for Org Admins**
This feature is currently in [private preview](/reference/feature-maturity-definitions#private-preview) and not yet available to all customers. Contact your Customer Success Manager to enable.
When enabled, Org Admins automatically receive Project Admin access on every project in their organization, without needing to be explicitly assigned to each one.
* Org Admins get Project Admin permissions on all projects, models, alerts, dashboards, and charts in their organization.
* Org Admins can no longer be assigned a project role while the feature is enabled.
* Any existing project role assignments for Org Admins are no longer shown in project member listings; those Org Admins automatically have Project Admin access instead.
* Per-project role assignments for non-admin users are unaffected.
* Org Admins can still be added to teams; their effective access on a team's projects is always Project Admin (the higher permission wins).
* If an Org Admin is later demoted to Org Member, they lose implicit Project Admin access immediately and revert to whatever direct or team-mediated project roles (if any) were assigned to them previously.
* Must be enabled with feature flag: `enableOrgAdminImplicitProjectAdmin`
* *Benefit*: Eliminates per-project role-assignment overhead for organizations where Org Admins are expected to have full platform access.
**Enhanced API Rate Limiting**
The REST API now exposes rate limit information through standard HTTP response headers (`X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`, `Retry-After`) on every request, making it straightforward for clients to implement graceful retry handling. Per-token rate limits use separate budgets for read and write operations to protect service availability and fairness. See the [REST API Rate Limiting](/sdk-api/rest-api#rate-limiting) documentation for current limits and retry guidance.
**Trace Explorer Enhancements**
Trace Explorer gains several improvements in 26.10.
* **Full-page view**: Open any trace in a dedicated full-page view in addition to the side drawer, giving you more space to inspect complex trace trees.
* **Search on historical spans**: Full-text search over prompt and response content is now available on spans ingested before 26.9. Existing applications gain retroactive search coverage automatically — no action required.
* **Improved filter dropdowns**: The span name, span type, and agent name filter dropdowns now load options from the server with search support. Boolean evaluator fields now support `IN` and `NOT IN` operators.
* **Trace drawer layout**: The trace drawer defaults to List view with a 70/30 split (span details 70%, trace tree 30%) and adapts to narrow screens.
* *Benefit*: Investigate individual spans across an entire application with broader search coverage, more precise filters, and a more usable trace view.
**GenAI Application Dashboard: Inline Add Chart**
You can now add charts to a GenAI Application dashboard directly from edit mode, without navigating away.
* In edit mode, a **+** tile appears in the dashboard grid. Click it to choose between adding a saved chart or creating a new one.
* **Saved Charts**: Opens a searchable drawer listing existing GenAI charts for the current project. Select one or more to add them to the dashboard layout.
* **New Chart**: Opens the chart creation flow with the current project pre-selected. On save, you are returned to the dashboard with the new chart added to the layout automatically.
* Your draft layout persists across page refreshes and round-trips to the chart creation page, so unsaved edits are not lost.
* *Benefit*: Build and iterate on GenAI Application dashboards without losing your in-progress layout.
#### Fixes and Security Updates
* **Breadcrumbs Fixed for Charts**: When you navigate to a chart from a dashboard, the breadcrumbs now route back to the originating dashboard. *Benefit*: Preserves navigation context so you can return directly to the dashboard you came from.
**Release 26.9**
#### What's New and Improved
**Custom Histogram Bins for Numerical Columns**
Numerical columns now support custom bin boundaries for histograms and drift metrics (PSI, JSD), enabling domain-specific or quantile-based binning instead of the default 10 uniform bins.
* Set custom bins at model creation or update via `model.schema["column"].bins = [...]` and `model.update()`, or through the UI schema editor
* Bins must be strictly increasing, span \[min, max], with at most 16 boundary values (15 bins)
* When omitted, 10 uniform bins are auto-generated (default behavior unchanged)
* Updating bins triggers re-computation of historical aggregates; models with more than 10M events per environment are not eligible for bin updates
* See [Customizing Your Model Schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema#modifying-the-histogram-bins)
**GenAI Traces TTL**
You can now bound the storage growth of GenAI trace, span, session, and evaluator data in long-running deployments by applying a retention window to raw GenAI data.
* **Anchor**: Retention is measured from Fiddler-side ingestion time, so backfilled or late-arriving spans are retained for the full window after ingestion
* **Availability**: Retention applies to GenAI data ingested into newly-onboarded deployments. To enable retention on an existing deployment, contact your Fiddler Customer Success Manager
**FQL Enhancements: `min()`/`max()` Aggregate Functions and `null` Keyword**
Two additions to Fiddler Query Language (FQL) expand the expressiveness of custom metric definitions for both ML and agentic applications.
* **`min(x)` and `max(x)` aggregate functions**: Compute the minimum or maximum value of a numeric column or expression across all rows in a time window. Use them to track feature or attribute ranges (`max(col) - min(col)`), detect extreme values, or build ratio-based metrics (`min(col) / max(col)`)
* For ML: accepts column references (e.g., `min(Prediction)`)
* For agentic/GenAI: accepts `attribute()` calls (e.g., `max(attribute('response_time_ms', type='user', scope='span'))`)
`min(x)` / `max(x)` aggregate a single expression across rows. To compare multiple already-computed aggregates, use `least(...)` / `greatest(...)` instead.
* **`null` keyword**: The `null` literal can now be used directly as a return value in FQL expressions — for example, in `if()` to propagate missing data: `if(is_null(col), null, col * 2)`. Use `is_null()` and `is_not_null()` to filter by null values
Available for both ML and GenAI/agentic custom metrics. See [Fiddler Query Language](/observability/platform/fiddler-query-language) for the full reference.
**Session-Level Attributes in GenAI Charts**
GenAI Application charts now support **Session** as a query scope alongside **Span**, enabling Attribute metrics over custom session-level attributes published via the Fiddler SDK in addition to per-span attributes.
* Pick **Session** from the **Query Scope** dropdown when configuring an Attribute metric — the attribute picker then lists session-scoped attributes you have published for the application
* *Scope*: Available for the Attribute metric only. Other built-in metrics (Traffic, Latency, Token Count, Evaluator) remain Span-only in the chart Query Scope dropdown
**Executive Dashboard**
Executive Dashboard is currently in private preview, not yet available to all customers.
* **Improved Error States**: We now give more helpful information if the user viewing the dashboard has no access to projects or if the projects they can access have no onboarded models, instead of displaying a generic error message as we were previously doing.
**LLM Gateway: AWS Bedrock, Azure OpenAI, and Azure AI Provider Support**
The LLM Gateway now supports AWS Bedrock, Azure OpenAI, and Azure AI as providers, enabling teams to route evaluator and LLM-as-a-Judge requests through AWS and Azure-hosted LLM deployments.
Configure each provider in **Settings > LLM Gateway** by selecting the provider and configuring models and authentication credentials.
Bedrock provides access to foundation models from leading AI providers:
* **Anthropic** (`bedrock/anthropic.*`) — Claude models for advanced reasoning and long-context tasks
* **Amazon** (`bedrock/amazon.nova-*`) — Nova family for multimodal and text generation
* **Meta** (`bedrock/meta.llama*`) — Open-weights Llama models across multiple sizes
* **Mistral AI** (`bedrock/mistral.*`) — High-performance models optimized for efficiency
Both default and custom model IDs (e.g., fine-tuned or newly released models) are supported.
**Authentication methods**
* **API Key** — Use Fiddler's managed AWS credentials
* **AWS Access Key** — Provide your own AWS access key ID, secret access key, and region
**Optional VPC endpoint support**
Configure a custom API Base URL to route requests through an AWS VPC endpoint (e.g., `https://vpce-xxx.bedrock-runtime.us-east-1.vpce.amazonaws.com`) instead of the public Bedrock endpoint.
Access OpenAI models hosted on Azure OpenAI Service:
* GPT-4 and GPT-4 Turbo for advanced reasoning and long-context tasks
* GPT-3.5 Turbo for fast, cost-effective completions
* GPT-4o for multimodal capabilities
* Custom fine-tuned models
Customers specify their Azure deployment name (which may differ from the base model name).
**Authentication methods**
* **API Key** — Use your Azure deployment's API key for simple authentication
* **Microsoft Entra ID** (formerly Azure AD) — OAuth 2.0 client credentials flow with tenant ID, client ID, and client secret for enterprise SSO integration
**Custom endpoint support**
Configure a custom API Base URL to target specific Azure regions or resource endpoints, e.g., `https://.openai.azure.com/`.
Access models from the Azure AI Model Catalog:
* **Anthropic** — Claude models (Opus, Sonnet, Haiku)
* **Meta** — Llama models (8B to 70B parameters)
* **Mistral AI** — Mistral and Mixtral models
* **Cohere** — Command and Embed models
* Other third-party foundation models available in the Azure AI catalog
Customers specify their Azure AI deployment name.
**Authentication methods**
* **API Key** — Use your Azure deployment's API key for simple authentication
* **Microsoft Entra ID** (formerly Azure AD) — OAuth 2.0 client credentials flow with tenant ID, client ID, and client secret for enterprise SSO integration
**Custom endpoint support**
Configure a custom API Base URL to target specific Azure regions or resource endpoints, e.g., `https://.services.ai.azure.com/`.
**Named Access Keys and Configurable Limits**
You can now create multiple access keys with descriptive names, making it easier to manage credentials across different integrations and environments. Keys are managed under **Settings > Credentials**, in the Access Keys section.
* **Named access keys**: Assign a descriptive name to each key at creation so you can quickly identify which integration or environment uses it.
* **Configurable limit**: Each user can create up to 5 keys by default. Contact Fiddler Support to request a higher limit.
A key's full value is shown only once, at creation. Copy and store it securely before closing the dialog — Fiddler cannot retrieve it later.
Existing access keys continue to work. We recommend creating any new keys with names going forward; the option to create unnamed keys will be removed in a future release.
See [Managing Credentials](/reference/administration/settings#credentials) for setup steps and security guidance.
**Trace Explorer**
Trace Explorer is a new spans-first exploration surface for GenAI applications, providing a paginated, filterable, searchable view of every span ingested into your application. Access it via the **Trace Explorer** tab on any GenAI Application Details page.
Trace Explorer is currently in public preview.
* **Server-side filterable DataGrid**: Browse spans across a 7-day default window with columns for type, name, input, output, timestamp, duration, and status (`Ok` / `Error` / `Unset`); open the trace drawer from the Actions column to inspect the full trace tree
* **Full-text search with scope**: Search prompt and response content with a scope selector (All / Input / Output), backed by a new dedicated content index for fast substring lookups
* **Structured filters**: Filter by span fields, IDs, token counts, user-defined span attributes, and evaluator outputs, with type-aware operators including negation
* **Dynamic columns**: Evaluator outputs and user-defined span attributes appear as columns automatically based on your application's schema
* **Sorting**: Sort by start time (default) or duration in the UI
***
**Release 26.8**
#### What's New and Improved
**Dashboard and Chart PDF/PNG Export**
All dashboard surfaces — Executive Dashboard, Symphony dashboards, GenAI Application dashboards, and chart detail pages (Monitoring, GenAI Monitoring, Feature Analytics, and Embedding Visualization) — now include an **Export** button that captures the current view as a PDF or PNG file.
* **PDF export**: Renders the full dashboard at a consistent page width, scales content to fit, and produces a multi-page PDF when the content is taller than a single page
* **PNG export**: Captures the visible dashboard as a single high-resolution image
* **Smart exclusions**: UI chrome such as control panels, date pickers, and fullscreen toggles are excluded from the export so the output contains only chart content
* **Force-load**: Before capturing, the exporter pre-fetches all chart data and waits for charts to finish rendering, so exports are never blank or partially loaded
* *Scope*: Available on all dashboard surfaces when the `enable_dashboard_export_ui` feature flag is enabled
**Chart Legend Bulk Selection and Isolate**
Charts with more than three legend items now include a context-aware bulk selection button and support double-click to isolate a single series. This applies to GenAI charts and ML charts.
* **Dynamic bulk action label**: The button reads "Hide All" when all series are visible, "Show All" when all are hidden, and "Invert Selection" when the chart has a mix of visible and hidden series
* **Double-click to isolate**: Double-clicking a legend item hides every other series and shows only the clicked one, making it easy to focus on a single metric
* *Benefit*: Quickly toggle visibility of many series at once or drill into a single series without clicking each legend item individually
* *Scope*: Applies to both Monitoring and GenAI Application charts when more than three legend items are present
**Chart Legend Selection Filters RCA View for GenAI Charts**
The RCA table for GenAI charts will now be filtered by whatever is selected in the chart legend, making the filtering experience faster and more intuitive.
* **Works with Filters Panel**: RCA chart will be filtered by the intersection of filter selections between the legend and the filters panel in the sidebar
**Agentic Custom Metrics**
You can now define custom aggregate metrics over OpenTelemetry span attributes using Fiddler Query Language (FQL). Agentic Custom Metrics are scoped to your organization and can be optionally narrowed to specific projects, making them ideal for tracking cross-cutting concerns across agentic workflows.
* **FQL-based definitions**: Write expressions like `average(attribute('gen_ai.usage.input_tokens', type='system', scope='span'))` to define metrics over any span attribute — system or user-defined
* **Flexible scoping**: Create organization-wide global metrics or scope them to specific projects for team-level visibility
* **Full lifecycle management**: Create, view, and delete custom metrics directly from the Fiddler UI.
* **Use cases**: Track cost per span, compute custom latency percentiles, measure token usage ratios, or aggregate any numeric span attribute over time
* *Benefit*: Define the exact measurements that matter to your agentic applications without waiting for built-in metric support
* *Scope*: Available for all GenAI Applications
**Fiddler Fast Trust Safety Model v2.6.0**
[Fiddler Fast Trust Safety](/observability/llm/enrichments#safety) model has been updated with improved detection capabilities.
* **Improved Jailbreaking Detection**: Enhanced accuracy for identifying jailbreaking attempts in prompts
* **Improved Roleplaying Detection**: Enhanced accuracy for detecting content where the model is instructed to adopt a specific persona or character
* *Note*: No changes to the model interface — works as a drop-in replacement for the previous version
**LLM Gateway: Fiddler-Hosted Ministral 3 8B Model**
The LLM Gateway now offers Mistral's Ministral 3 8B Instruct as a Fiddler-hosted model for evaluators and LLM-as-a-Judge workflows. This feature is currently in public preview.
* **Model ID**: `fiddler/ministral3-8b`
* **Use case**: Complex or nuanced evaluations that benefit from stronger reasoning capabilities
* **Trade-off**: Slower than `fiddler/llama3.1-8b` but better suited for evaluations requiring deeper analysis
* *Benefit*: Run sophisticated evaluators without configuring external provider credentials or incurring per-token charges
**Evaluator Editing**
Evaluators can now be edited via the Evaluator Library. You can now modify and refine evaluators to meet your monitoring needs.
**Gini Coefficient Custom Metric (FQL)**
The `gini()` function is now available in Fiddler Query Language (FQL) for defining custom metrics on ML models. It computes the Gini coefficient derived from the Lorenz curve, measuring how well a model's predicted scores rank actual values.
* **Syntax**: `gini(actual=Target, predicted=Score)` — both `actual` and `predicted` are required keyword arguments of type `Number`
* **All ML task types**: Works for binary classification, multiclass classification, regression, and ranking models
* **Normalized Gini**: Compute the normalized Gini coefficient (scaled to 1 for a perfect model) with `gini(actual=Target, predicted=Score) / gini(actual=Target, predicted=Target)`
* **AUC shortcut**: For binary classification, the normalized Gini can also be derived as `2 * auroc() - 1`
* **Categorical targets**: For binary classification models with categorical target columns, use an `if` expression to convert to numeric (e.g. `gini(actual=if(churn == 'yes', 1, 0), predicted=score)`)
* *Benefit*: Compute Gini-based model performance metrics — widely used in insurance and credit risk modeling — directly in custom metric definitions, charts, and alerts
* *Scope*: Available for ML custom metrics only (not for GenAI or agentic applications). See [Fiddler Query Language](/observability/platform/fiddler-query-language) and [Custom Metrics](/observability/platform/custom-metrics) for full documentation
***
**Release 26.7**
#### What's New and Improved
**LLM Gateway: Vertex AI Provider Support**
The LLM Gateway now supports Google Vertex AI as a provider, enabling teams to route evaluator and LLM-as-a-Judge requests through GCP using service account credentials. Vertex AI provides access to a wide range of models from the Model Garden, organized by family:
* **Gemini**: `vertex_ai/gemini-*`
* **Anthropic (Claude)**: `vertex_ai/claude-*`
* **Meta/Llama**: `vertex_ai/meta/{MODEL}`
* **Mistral**: `vertex_ai/mistral-*`
* **DeepSeek**: `vertex_ai/deepseek-ai/{MODEL}`
* **Qwen**: `vertex_ai/qwen/*`
* **AI21 (Jamba)**: `vertex_ai/jamba-*`
* **OpenAI (GPT-OSS)**: `vertex_ai/openai/gpt-oss-*`
* **ZAI (GLM)**: `vertex_ai/zai-org/{MODEL}`
* **OpenAI-compatible**: `vertex_ai/openai/`
* **Non-OpenAI-compatible**: `vertex_ai/`
Configure Vertex AI access in **Settings > LLM Gateway** by providing a GCP service account JSON key, project ID, and location.
**Evaluator Library**
A new Evaluator Library provides a centralized place to create, view, duplicate, and manage evaluators independently of rules. Create evaluators directly from the library and select from them when setting up new rules.
**Multi-Target Event Updates for LLM Models**
LLM and NOT\_SET models with multiple target columns now correctly preserve all target columns during event updates. Previously, only the first target column was retained and additional targets were silently dropped.
* *Benefit*: You can update multiple target columns (e.g., `comment` and `feedback`) in a single PATCH request without data loss
* *Scope*: Affects only `LLM` and `NOT_SET` model task types, which are the only tasks that support multiple targets. Classification, regression, and ranking models (single target) are unaffected
**Evaluation Dataset Management**
Evaluation datasets can now be created, populated, and deleted directly from the UI. You can upload items via CSV with guided column mapping, including auto-mapping based on the existing dataset schema when adding to datasets that already contain data.
* *Benefit*: You can manage evaluation datasets entirely from the UI without writing code; smart auto-mapping reduces friction for repeated uploads
* *Note*: CSV uploads are limited to 1,000 rows per upload
#### Fixes and Security Updates
* **LLM Gateway**: Provider delete and update operations now return `409 Conflict` instead of a generic `400` when credentials are referenced by evaluators
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay current
* **Chart zoom and legend state preserved across re-renders**: Zoom position and legend selection no longer reset when hovering over chart elements or when the chart re-renders. Previously, user-interactive chart state was discarded on every option update, causing charts to "jitter" and lose filter settings
***
**Release 26.6**
#### What's New and Improved
**Executive Dashboard (Preview)**
A new Executive Dashboard provides organization-wide visibility into model health and alerting across all projects in a single view. This feature is now available as *private preview*; if you would like access to this feature, please contact your Customer Success Manager.
* **KPI Summary Cards**: At-a-glance metrics including total models, critical and warning alert counts, and monitoring coverage percentage
* *Benefit*: You can quickly assess overall AI system health without navigating into individual projects
* **Alert Breakdown by Type and Severity**: Alert summary cards grouped by type (drift, performance, data integrity) showing affected models, top triggered metrics, and time since last trigger
* *Benefit*: Identify which alert categories need attention and drill down to specific alerts with one click
* **Cross-Project Model Health Table**: A unified table listing all models across projects with severity status indicators, traffic volume, last event timestamp, and model owner
* *Benefit*: Compare model health across the entire organization and quickly spot models that need investigation
* **Configurable Time Range**: Selectable time windows (7D, 14D, 30D, 90D) to control the reporting period for all dashboard metrics
* *Benefit*: Adjust the analysis window to match your reporting cadence or investigate recent vs. long-term trends
* **Model Fleet Health Donut Chart**: Visual distribution of healthy, warning, critical, and unmonitored models across the organization
* *Benefit*: Instantly understand the proportion of your model fleet in each health state
**LiteLLM Proxy OpenTelemetry Integration**
Fiddler now automatically ingests OpenTelemetry traces emitted by LiteLLM proxy, with no SDK or code changes required in your application.
* **Zero-configuration ingestion**: Point `OTEL_EXPORTER_OTLP_ENDPOINT` at your Fiddler instance and set your application ID — traces flow in automatically
* **Purpose-built span mapper**: Handles LiteLLM proxy's span format (`acompletion` operation name, JSON message attributes, infrastructure spans) and maps them to Fiddler's semantic convention
* **Full conversation capture**: Extracts system prompt, last user turn, and assistant response from LiteLLM's JSON message attributes (`gen_ai.input.messages`, `gen_ai.output.messages`)
* **Cost and proxy metadata**: LiteLLM cost fields (`gen_ai.cost.*`) and proxy metadata (`metadata.*`) are preserved as user-visible span attributes for auditing and cost attribution
See the [LiteLLM Integration guide](/integrations/agentic-ai/litellm-integration) for setup instructions.
**GenAI Dashboard Inline Editing**
The GenAI Application dashboard now supports inline editing, allowing you to rearrange, resize, and delete chart tiles directly within the application view without navigating to a separate configuration page.
* **Drag-and-Drop Rearrangement**: Reorder dashboard charts by dragging and dropping tiles into new positions
* *Benefit*: Quickly customize your dashboard layout to prioritize the metrics that matter most
* **Inline Resize**: Resize chart tiles directly on the dashboard using drag handles
* *Benefit*: Adjust chart sizes to give more space to complex visualizations without leaving the page
* **Swap Behavior**: When dragging a tile onto another, the tiles swap positions rather than overlapping
* *Benefit*: Predictable layout behavior that preserves all your charts during rearrangement
**Sortable Numeric Columns in RCA Events Table**
Numeric columns (integers and floats) in the RCA Raw Events tab can now be sorted in ascending or descending order. Sorting is performed on the currently displayed data (up to 1,000 events), making it easier to identify outliers and top contributors within the loaded set.
* *Benefit*: You can quickly find the highest or lowest values for evaluator scores, custom metrics, and other numeric columns without manual scanning.
**Clickable URL Hyperlinks in Data Tables**
URL strings in RCA data table cells are now automatically detected and rendered as clickable hyperlinks that open in a new browser tab.
* *Benefit*: You can click directly from an RCA row to open source documents, external links, or trace URLs without copying and pasting.
**Test Alert Notifications**
You can now send test notifications for alert rules directly from the UI, verifying that your notification channels (email, webhook) are correctly configured before alerts fire in production.
* **Send Test Alert Action**: A new "Send Test Alert" option is available in the alert rules 3-dot menu
* *Benefit*: Validate notification delivery without waiting for a real alert to trigger
* **Backend API**: New `POST /v3/alert-rules/{id}/test-notification` endpoint supports test notification delivery with org-scoping and input validation
* *Benefit*: Programmatic access to test notifications for automation and CI/CD workflows
#### Fixes and Security Updates
* **Alert Email Input**: Email addresses typed into the alert notification recipient field are now automatically captured when the field loses focus, preventing silent data loss when the user clicks away without pressing Enter. Email validation logic has been consolidated into a shared `webapp-common` helper for consistent behavior across all alert configuration surfaces
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
***
**Release 26.5**
#### What's New and Improved
*No user-facing features in this release.*
#### Fixes and Security Updates
* **Charting**: Fixed an issue where the Metric Query Name displayed as empty when backend data was missing; a default name is now shown instead
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
***
**Release 26.4**
#### What's New and Improved
**Fiddler LangGraph SDK 1.4.0: Decorator and Manual Instrumentation**
The Fiddler LangGraph SDK (1.4.0) introduces decorator-based and manual instrumentation modes alongside the existing `LangGraphInstrumentor` auto-instrumentation, giving you fine-grained control over trace structure and metadata for custom application logic and non-LangGraph components.
Key additions include the `@trace()` decorator, context managers for explicit span control, typed span wrappers with semantic helpers, and a global `get_client()` singleton.
For the full changelog, see the [LangGraph SDK 1.4.0 release notes](/changelog/langgraph-sdk). For implementation guides and code examples, see the [LangGraph SDK integration guide](/integrations/agentic-ai/langgraph-sdk) and the [SDK API reference](/sdk-api/langgraph/fiddler-client).
**GenAI Application Page Redesign**
The GenAI Applications page now offers a grid-based card view alongside the existing list view, giving you a more visual and scannable overview of your applications at a glance.
* **Grid-Based Application View**: A new card-based grid layout is available as an alternative to the existing list view
* *Benefit*: Faster visual scanning and easier identification of applications, especially in environments with many GenAI applications
* **Project-Level Administration**: Projects are now exposed as a manageable entity within the admin console
* *Benefit*: Administrators can manage project-level RBAC permissions and perform project deletion directly from the UI
* *Note*: This enables customers to manage access control and lifecycle for projects without requiring backend intervention
**GenAI Chart and Alert Controls Rework**
Chart controls and alert controls across GenAI experiences have been redesigned for a more unified and intuitive monitoring workflow.
* **Unified Chart Controls**: Reworked chart controls provide a consistent experience across all GenAI monitoring views
* *Benefit*: Streamlined chart configuration with fewer clicks and a more intuitive layout
* **Reworked Alert Controls**: Alert configuration within GenAI experiences has been updated to align with the new chart controls
* *Benefit*: Consistent interaction patterns between charting and alerting workflows
* **Automatic Chart Migration**: Existing GenAI charts are automatically migrated to the new format during the upgrade
**Breaking Change** — This is a breaking change to chart and alert configurations. The automatic migration supports all valid configurations. In the rare event that a migration fails, the affected chart should be recreated manually.
**Model TTL Management**
You can now configure Time-to-Live (TTL) policies for ClickHouse event data at the organization, project, or model level, enabling fine-grained control over data retention.
* **Hierarchical TTL Configuration**: TTL values can be set at organization, project, or model scope by platform administrators using the config manager APIs
* *How it works*: More specific levels take precedence — model-level TTL overrides project-level, which overrides organization-level
* *Unit*: Days (minimum 7 days)
* **Lazy Pruning**: Data pruning occurs on a background schedule, at intervals of at least 4 hours
* *Note*: Dashboards and historical charts are unaffected; only raw event queries are limited to data within the TTL window
* **Opt-In by Default**: Existing deployments are unaffected unless custom TTL values are explicitly configured
* *Note*: The deployment-wide default is controlled by the `EVENTS_TABLE_TTL_DAYS` environment variable, which defaults to no TTL
Pruning is irreversible. Once data is pruned, it cannot be recovered.
**OTel Collector Performance Improvements**
Infrastructure improvements to the OpenTelemetry Collector enhance throughput and reliability for high-volume trace ingestion.
* **Size-Based Batching**: Sending queues now use byte-based sizing instead of count-based
* *Benefit*: More predictable batching behavior for payloads of varying sizes, especially large spans
* **Percentage-Based Memory Limiter**: The collector's memory limiter is now percentage-based, adapting to container memory limits automatically
* *Benefit*: Better suited to containerized deployments compared to the previous fixed-value approach
* **Configurable Kafka Producer**: Kafka producer max request size is now configurable with optional compression
* *Benefit*: Supports larger trace and event payloads without hitting message size limits
#### Client Versions
Refer to the [LangGraph SDK changelog](/changelog/langgraph-sdk) for version-specific details.
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
***
**Release 26.3**
#### What's New and Improved
**RAG Health Metrics: Answer Relevance 2.0, Context Relevance, and RAG Faithfulness**
RAG Health Metrics is a purpose-built diagnostic framework for Retrieval-Augmented Generation applications. The three evaluators work together to pinpoint exactly where RAG pipelines fail — whether in retrieval, generation, or query understanding — transforming debugging from manual trial-and-error into targeted root cause analysis.
* **Answer Relevance 2.0** (Enhanced): Improved ordinal scoring system replaces binary scoring
* *Scoring*: High (1.0), Medium (0.5), Low (0.0) with detailed reasoning
* *Benefit*: Granular assessment of how well responses address user queries
* *Availability*: Agentic Monitoring, Experiments, LLM Observability
* **Context Relevance** (New): Measures whether retrieved documents are relevant to the query
* *Scoring*: High (1.0), Medium (0.5), Low (0.0) with detailed reasoning
* *Benefit*: Isolate retrieval problems from generation problems in your RAG pipeline
* *Availability*: Agentic Monitoring and Experiments (not available in LLM Observability)
* **RAG Faithfulness** (Repackaged): LLM-as-a-Judge faithfulness evaluator with standardized inputs and outputs, now part of the RAG Health Metrics triad
* *Scoring*: Binary — Yes (1.0) / No (0.0) with detailed reasoning
* *Inputs*: `user_query`, `rag_response`, `retrieved_documents`
* *Outputs*: `label`, `value`, `reasoning`
* *Benefit*: Combined with Answer Relevance and Context Relevance for comprehensive RAG pipeline diagnostics
* *Availability*: Agentic Monitoring, Experiments, LLM Observability
* *Note*: FTL Faithfulness (`ftl_response_faithfulness`) continues unchanged for guardrails and LLM Observability use cases
* *Note*: RAG Faithfulness and FTL Faithfulness are separate evaluators with different architectures
**Diagnostic Framework Benefits:**
Use the three evaluators together to diagnose RAG pipeline issues:
| What the metrics tell you | Why it's happening | Next Step |
| --------------------------------- | ------------------------------------- | ------------------------------------------------ |
| High relevance + Low faithfulness | Hallucinations despite being on-topic | Check if retrieval provided sufficient grounding |
| High faithfulness + Low relevance | Grounded but didn't answer the query | Check if retrieval provided relevant information |
| Low Context Relevance | Retrieval pulling wrong documents | Fix retrieval mechanism |
RAG Health Metrics works alongside Fiddler's existing 80+ LLM metrics (toxicity, PII, coherence, and more) — providing targeted RAG diagnostics that complement your existing observability stack.
**CustomJudge in Evals SDK**
The `CustomJudge` evaluator class is now available in the Fiddler Evals SDK (0.3.0), enabling custom LLM-as-a-Judge evaluators in Experiments workflows.
* **Prompt Template Style**: Define custom evaluators using `prompt_template` with Jinja `{{ placeholder }}` syntax and `output_fields` for structured evaluation results
* *Benefit*: Build domain-specific evaluation criteria for any use case — brand voice compliance, bias detection, topic classification, and more
* *Note*: This is the same `CustomJudge` class available in Agentic Monitoring, now also accessible through the Evals SDK for Experiments workflows
**Improved Reasoning Quality for LLM-as-a-Judge Evaluators**
The quality of the `reasoning` field has been improved for all LLM-as-a-Judge evaluators when using the Fiddler-hosted Llama model.
* **Better Reasoning Output**: More intelligent, consistent reasoning that better aligns with predicted labels
* *Scope*: All pre-built LLM-as-a-Judge evaluators in Agentic Monitoring, and custom LLM-as-a-Judge evaluators using Prompt Template-style prompt specs in Traditional Monitoring
* *Benefit*: More detailed and accurate reasoning explanations for evaluation results
* *Note*: Evaluation speed may be slightly slower in some cases due to higher quality reasoning output; tokens per second remains constant
**Fiddler LangGraph SDK Enhancements**
The Fiddler LangGraph SDK (1.3.1) includes infrastructure upgrades and new observability capabilities.
* **OpenTelemetry Upgrade**: Updated to OpenTelemetry 1.39.1 / 0.60b1 (api, sdk, instrumentation, otlp exporter)
* *Note*: Removed hardcoded OpenTelemetry span/batch limits; SDK now uses OpenTelemetry defaults
* **Full LLM Message History**: New span attributes `gen_ai.input.messages` and `gen_ai.output.messages` capture complete LLM message history
* *Benefit*: Better debugging and observability of LLM interactions within agentic workflows
**Fiddler Strands Pipeline Support**
Message history from Strands span events is now extracted and stored as span attributes for querying.
* *Benefit*: Enables observability and analysis of Strands-based agentic pipelines within the Fiddler platform
#### Client Versions
| Component | Version |
| --------------------- | ------- |
| Fiddler Python Client | 3.11 |
| Fiddler Evals SDK | 0.3.0 |
| Fiddler LangGraph SDK | 1.3.1 |
| Fiddler Strands SDK | 0.4.0 |
Refer to the [Evals SDK changelog](/changelog/evals-sdk) and [LangGraph SDK changelog](/changelog/langgraph-sdk) for version-specific details.
#### Deprecations and Removals
**DSPy-style Prompt Specifications Removed from Documentation**
DSPy-style Prompt Specifications are not supported in Agentic Monitoring or Experiments. Example notebooks have been updated to use the `CustomJudge` class with Prompt Template style exclusively.
* **Replacement**: Use `CustomJudge` with `prompt_template` (Jinja syntax) and `output_fields` for structured evaluation results
* *Note*: DSPy-style prompt specs remain available in Traditional Monitoring but are no longer the recommended approach
* *Action Required*: If using custom LLM-as-a-Judge evaluators, adopt `CustomJudge` with Prompt Template syntax
**Fixes and Security Updates**
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
***
**Release 26.2**
#### What's New and Improved
**Enhanced Sentiment Evaluator Performance**
The Sentiment evaluator has been optimized to leverage GPU acceleration when available, significantly improving evaluation speed for all input sizes.
* **GPU Acceleration**: Sentiment evaluator now automatically detects and utilizes GPU resources when available
* *Benefit*: Dramatically faster evaluation performance - under 80ms on the scoring endpoint for all input sizes
* *Note*: Falls back to CPU processing if GPU is unavailable, maintaining backward compatibility
* **Performance Optimization**: Replaced model weights with GPU-compatible version
* *Note*: Input is truncated at 512 tokens as before
**GenAI Application Deletion**
You can now delete GenAI applications directly from the GenAI Applications List page, providing better application lifecycle management.
* *Benefit*: Streamlined cleanup of test and deprecated applications
#### Deprecations and Removals
**Answer Relevance Evaluator**
The Answer Relevance evaluator has been removed from the list of available evaluators as we prepare to release an improved version.
* **Removed from Available Evaluators**: Answer Relevance is no longer available for new evaluation rules
* *Note*: Existing Answer Relevance rules will continue to function normally
* **Replacement Timeline**: Answer Relevance v2 will be introduced in Fiddler v26.3
* *Benefit*: The updated evaluator will provide improved accuracy and performance
**Fixes and Security Updates**
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
**Release 26.1**
#### What's New and Improved
**Performance Optimization for Metrics**
Significant improvements to the metric fetching engine and metadata caching to enhance query performance and reduce database load.
* **Optimized Metric Fetching**: Metric fetching engine now retrieves sparse data from the database more efficiently
* *Benefit*: Significantly reduced data transfer and processing overhead for metric queries
* **Smart In-Memory Processing**: Missing values are efficiently filled in-memory only when necessary
* *Note*: For example, when preserving zero-traffic bins
* *Benefit*: Improved query performance without sacrificing data completeness
* **LRU Metadata Caching**: Implemented Least Recently Used (LRU) caching for frequently accessed metadata
* *Scope*: Caches Projects, Models, Baselines, Segments, and Applications metadata
* *Benefit*: Further reduces database load and improves response times for common queries
**Span Latency Monitoring**
New charting capabilities for monitoring span latency across your agentic and LLM applications.
* **Span Latency Charts**: Added new charts specifically designed for span latency visualization
* *Benefit*: Better visibility into performance characteristics of individual operations
* **Granular Latency Filtering**: Filter latencies by different time granularities
* *Options*: Microseconds, milliseconds, or seconds
* *Benefit*: Analyze performance at the appropriate scale for your use case
**Charts Preview in Agentic Alerts**
Enhanced alert configuration with visual chart previews to help you set accurate thresholds.
* **Visual Threshold Tuning**: Charts preview shows data points that fall under Critical and Warning thresholds
* *Benefit*: Dynamically fine-tune alert thresholds based on observed trends
* *Benefit*: Reduce false positives by visualizing actual data distribution before setting alerts
**Fixes and Security Updates**
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
**Release 25.22**
### What's New and Improved
**Enhanced Agentic Observability Dashboards**
Improved out-of-the-box monitoring experience for LLM and agentic applications with expanded default dashboards and automated safety evaluation.
* **Default Safety Evaluator**: New applications automatically include FTL safety evaluator rule
* *Benefit*: Immediate safety monitoring for all LLM spans and outputs without manual configuration
* **Expanded Dashboard Charts**: Default dashboards now include 4 charts in addition to traffic monitoring and traffic metrics
* *Safety Analysis*: Monitor safety evaluator results for responsible AI deployment
* *Token Consumption*: Track LLM token usage for cost optimization
* **Accurate Token Counting**: SDK-based token counting replaces previous estimation method
* *Benefit*: Precise cost tracking and optimization insights for LLM operations
* **Production Monitoring Parity**: Metric cards gaining feature parity with monitoring charts
* *New Support*: Token counts and custom attributes now available in metric cards
* *Benefit*: Consistent monitoring capabilities across different visualization types
**Extended Alert Capabilities for Agentic Applications**
Alert system expanded beyond traffic monitoring to support comprehensive LLM application observability.
* **Token Count Alerts**: Set thresholds for LLM token consumption
* *Benefit*: Proactive cost management and budget protection for API usage
* **Attribute-Based Alerts**: Create alerts on numerical and categorical custom attributes
* *Benefit*: Monitor business-specific metrics and application-specific dimensions
**Programmatic Model Schema Updates**
Add columns to production models after initial onboarding without downtime or data re-ingestion. Supports adding inputs, outputs, targets, and metadata columns with all data types.
*Action Required*: Requires Python Client SDK 3.11 or higher
**Important Notes**:
* New columns will have null values for historical events
* Future events must include data for newly added columns
**Event Deletion and Pipeline Improvements**
Critical fixes to data pipeline operations and rate limit handling for improved reliability.
* **Event Deletion Bug Fixes**: Resolved issues with event deletion operations in the data pipeline
* *Benefit*: Reliable data cleanup and management for production systems
* **Rate Limit Handling**: Added HTTP 429 responses for rate limit scenarios
* *Benefit*: Clear feedback when API rate limits are exceeded, enabling proper retry logic
### Client Version
This release works best with Python Client SDK **3.11** or higher.
**Release 25.21**
### What's New and Improved
**Harrier Alerts MVP: Enhanced Traffic Monitoring**
Introducing the first release of Harrier Alerts, enabling you to set up granular traffic monitoring for your ML and LLM applications with configurable alert thresholds.
* **1-Hour Traffic Alerts**: Monitor traffic patterns with hourly granularity to catch issues quickly
* *Benefit*: Detect traffic anomalies and potential problems within an hour of occurrence
* **1-Day Traffic Alerts**: Track daily traffic patterns for broader trend analysis
* *Benefit*: Identify long-term traffic patterns and plan capacity accordingly
* **Configurable Thresholds**: Set custom alert thresholds based on your application needs
* *Note*: Available for both hourly and daily monitoring periods
**User-Defined Attributes for Enhanced Observability**
Add custom attributes to your agentic and LLM application traces for deeper insights into your specific use cases and business metrics.
* **Span-Level Attributes**: Add granular metadata to individual operations
* *Format*: `fiddler.span.user.{key}` for operation-specific attributes
* **Flexible Metadata**: Capture any custom data relevant to your monitoring needs
* *Benefit*: Filter, group, and analyze traces using your own business dimensions
* *Action Required*: SDK update required to use this feature
**Rule-Based Evaluators**
Create custom evaluation rules for your LLM applications with flexible, condition-based logic.
* **Custom Evaluation Logic**: Define specific rules and conditions for evaluating LLM outputs
* **Flexible Conditions**: Set up complex evaluation criteria based on your use case
* *Benefit*: Tailor evaluation to your specific quality and safety requirements
**Custom LLM as a Judge (LLMaaJ)**
Build custom LLM-based evaluators using your preferred models and evaluation criteria.
* **Flexible Model Selection**: Use any LLM as an evaluator for your applications
* **Custom Prompts**: Define evaluation criteria with your own prompting strategies
* *Benefit*: Create domain-specific evaluators that understand your unique requirements
**Pre-Built LLMaaJ Improvements**
Enhanced Fiddler's pre-built LLM evaluators with improved accuracy and updated evaluation criteria.
* **Updated Evaluation Logic**: Refined prompts and evaluation criteria for better accuracy
* **Improved Response Quality**: More consistent and reliable evaluation results
* *Note*: Input and output formats have been updated for improved performance
* *Compatibility*: These evaluators were deactivated in 25.20 to prepare for these improvements
**Streamlined GenAI Application Onboarding**
Redesigned the GenAI application onboarding flow for a simpler, more intuitive setup experience.
* **Simplified Steps**: Reduced complexity in creating new GenAI applications
* **Improved Guidance**: Better in-app instructions and tooltips
* *Benefit*: Get your agentic and LLM applications monitored faster with less configuration
**Performance Optimization: Event Deletion**
Improved memory efficiency when deleting events by time range.
* **Reduced Memory Usage**: Delete operations now process data in chunks to minimize memory consumption
* **Better Scalability**: Handle large-scale deletions without resource constraints
* *Performance Impact*: Significantly lower memory footprint for bulk delete operations
### Deprecations
**Enrichment Features: Toxicity and Custom LLM Classifier**
The following enrichment features are being deprecated in favor of the improved LLMaaJ evaluators:
* **Deprecated Features**:
* Toxicity enrichment
* Custom LLM Classifier enrichment
* **Replacement**: Use the new Custom LLMaaJ or Pre-Built LLMaaJ evaluators
* Enhanced evaluation capabilities with LLM-based judges
* More flexible and accurate evaluation
* **Migration Path**: Configure equivalent evaluators using the new LLMaaJ framework
* *Benefit*: Modern evaluation infrastructure with better accuracy and flexibility
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### Client Version
Client version updates are required for User-Defined Attributes functionality. Refer to the [Python SDK changelog](/changelog/python-sdk) for version-specific details.
#### Documentation Updates
* **OpenTelemetry Integration**: New comprehensive documentation for OpenTelemetry integration
* Quick Start guide for custom agent frameworks
* Advanced patterns and production configurations
* [OpenTelemetry Quick Start →](/developers/quick-starts/opentelemetry-quick-start)
* **User-Defined Attributes**: Documentation for adding custom attributes to traces
* Session and span-level attribute patterns
* Best practices for custom metadata
**Release 25.20**
### What's New and Improved
**Enhanced Fiddler Fast Trust Safety Model**
[Fiddler Fast Trust Safety](/observability/llm/enrichments#safety) model has been updated to improve accuracy and expand classification capabilities based on customer feedback.
* **Improved Cryptocurrency Handling**: Reduced false positives when analyzing content that references cryptocurrency and related financial terms
* *Benefit*: More accurate safety assessments for fintech and Web3 applications without unnecessary flagging of legitimate cryptocurrency discussions
* **New Roleplaying Detection**: Added `roleplaying` label to identify content where the model is instructed to adopt a specific persona or character
* *Impact*: Enhanced ability to detect potential prompt injection attempts and inappropriate persona adoption
* *Note*: Users with existing FTL Safety enrichments that did not specify label subsets via the `classifiers` configuration will automatically see the new `roleplaying` label on newly ingested data
### Deprecations
**Custom LLM Classifier**
The Custom LLM Classifier feature is now deprecated and will be removed in a future release. Users should migrate to the more powerful and flexible LLM-as-a-Judge with Prompt Spec feature.
* **Replacement**: [LLM-as-a-Judge with Prompt Specs](/observability/llm/llm-evaluation-prompt-specs) provides enhanced capabilities for custom evaluations with greater control and flexibility
* **Migration Path**: Refer to the LLM-as-a-Judge documentation for implementation guidance and examples
* *Benefit*: LLM-as-a-Judge offers more sophisticated evaluation workflows, better prompt engineering capabilities, and improved consistency for custom classification tasks
### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### What's New and Improved
**Enhanced Session Management**
Fiddler 25.19 introduces improved session management that keeps users authenticated during active usage while maintaining security through standardized timeout policies.
* **Extended Active Sessions**: Users remain authenticated while actively using Fiddler, eliminating the previous 2-hour hard timeout that logged out active users
* *Benefit*: Uninterrupted workflow for active users while maintaining robust security controls
* **Intelligent Idle Detection**: Sessions automatically expire after 2 hours of inactivity
* *Default Configuration*: 2-hour idle timeout, 24-hour maximum session lifetime, 1-hour access token lifetime
* **Multi-Tab Session Synchronization**: Session state is synchronized across multiple browser tabs, providing a consistent experience
* *Benefit*: Seamless transitions between browser tabs without unexpected logouts
* **Proactive Token Refresh**: Access tokens are automatically refreshed in the background when actively using the application
**Jobs Page Optimization**
The Jobs page now provides a more focused view of job history while improving system performance through intelligent data retention policies.
* **Streamlined Job Display**: The Succeeded/Failed tab shows only the jobs that require attention or review
* Failed streaming jobs remain visible for 30 days
* All batch ingestion jobs (successful and failed) remain visible for 30 days
* Successful streaming jobs are removed immediately upon completion
* *Benefit*: Cleaner job history makes it easier to identify jobs that need attention
* **Performance Improvements**: Chunk-level data purging reduces internal metadata table sizes
* *Benefit*: Faster page load times, especially for high-volume environments
* **Smart Data Retention**: The system maintains a 30-day retention period for jobs that may need to be replayed or investigated while removing unnecessary successful streaming job entries
### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### What's New and Improved
* **Login Redirection**
* Users are now automatically redirected to their originally requested URL after signing in. This enhancement improves the user experience by eliminating the need to manually navigate back to the intended page after authentication.
* Customizable Identity Provider Group Prefix
* Organizations can now configure a custom prefix for identity provider group mapping instead of the default `fiddler_` prefix. This enhancement allows Fiddler to work with existing organizational naming conventions for Active Directory, LDAP, and SSO groups.
Previously, Fiddler required groups to follow the exact naming pattern:
* `fiddler_ORG_ADMIN`
* `fiddler_ORG_MEMBER`
* `fiddler_ `
With this update, administrators can specify a custom prefix that matches their organization's group naming policy. Refer to the [Mapping LDAP Groups & Users to Fiddler Teams](/reference/access-control/mapping-ad-groups-to-fiddler-teams) guide for more details.
#### What's New and Improved
**New Fiddler AuthN Management Console**
We're excited to introduce the new Fiddler AuthN management console, built on a robust authentication framework. This upgrade enhances Fiddler's authentication and user management capabilities with improved security and administrative control.
**Key Features:**
* **Dedicated Management Console**: Access authentication administration at `https://authn-{your-instance}.cloud.fiddler.ai`
* **Enhanced SSO Integration**: Improved support for Okta, Microsoft Entra ID, Google, and Ping Identity with standardized configuration
* **Identity Provider Group Sync**: Automatic mapping of external groups to Fiddler teams with customizable group prefixes
* **Role-Based Administration**: Granular admin roles including "Org Owner" and "Org User Manager" for delegated management
* **Mixed Authentication Support**: Simultaneous SSO and email-based authentication methods
This enhancement delivers enterprise-grade authentication management while maintaining backward compatibility with existing configurations.
*Benefit*: Provides improved security, streamlined administration, and enhanced user experience through modern authentication infrastructure.
[Explore Authentication Documentation →](/reference/access-control)
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### What's New and Improved
**Fast Personally Identifiable Information (PII) Guardrails**
We've added Fast PII detection to Fiddler Guardrails, joining Fast Safety and Fast Faithfulness to provide comprehensive real-time protection for LLM applications.
* **Comprehensive PII Detection**: Automatically detects and flags PII leakage across 25+ categories including personal identifiers, financial data, government IDs, and digital identifiers
* **Detailed Output**: Returns detected PII spans with labels, confidence scores, and character offsets for precise identification and redaction
* **Enterprise Ready**: Optimized for low latency and high-scale deployment with consistent sub-second response times
* **Compliance Support**: Helps meet privacy regulations like GDPR, CCPA, and HIPAA by preventing PII exposure
The Fast PII Guardrails integrate seamlessly via REST API and can be used independently or combined with existing guardrails for layered protection.
For technical implementation details, see the [Guardrails documentation](/getting-started/guardrails).
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
* **Alerts UI Consistency**: Fixed various display inconsistencies in the Alerts interface to improve user experience and visual consistency across the platform.
#### What's New and Improved
* **Concurrent Enrichment Processing**
* Introduced parallel processing for independent enrichment operations, significantly improving performance when multiple enrichments are applied to your LLM data
* *Performance Impact:* Reduces overall enrichment processing time by up to 70% when using multiple enrichments (e.g., combining PII detection, toxicity analysis, and sentiment scoring)
* *Benefit:* Faster data processing enables near real-time monitoring for high-volume LLM applications, reducing the delay between data ingestion and actionable insights
* *Note:* This feature is currently in controlled rollout. Contact your customer success manager if you'd like early access
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### What's New and Improved
* **Custom LLM Enrichment & Evaluations Enhancements**
* *Improved Determinism:* Temperature parameter now defaults to 0 for classification use cases, ensuring more consistent and repeatable results
* *Enhanced Flexibility:* The Evaluations endpoint now accepts a temperature parameter, matching the configuration options available in Enrichment
* *Benefit:* Teams can achieve more predictable classification outputs while maintaining the flexibility to adjust temperature when needed for specific use cases
* **Fiddler Fast Trust Faithfulness v2.4.1**
* *Performance Boost:* Up to 150ms faster processing on longer input texts
* *Benefit:* Reduced latency for faithfulness checks on documents and extended conversations, enabling more responsive LLM monitoring at scale
* *Impact:* Particularly beneficial for applications processing lengthy contexts or high-volume document analysis workflows
#### What's New and Improved
* **Custom Webhook Support for Alert Notifications**
* Extended webhook integration capabilities beyond Slack and Microsoft Teams to support any webhook provider
* *Key Features:*
* [Create custom webhook](/reference/administration/settings) configurations for any third-party service or internal system
* Configure custom webhooks with alert rules for flexible notification routing
* Maintain existing Slack and Microsoft Teams integrations alongside custom webhooks
* *Benefit:* Organizations can now integrate Fiddler alerts with their preferred communication platforms and incident management systems
* **Enhanced Data Ingestion Performance with ClickHouse Optimization**
* *Performance Improvements include:*
* End-to-end ingestion latency reduced by up to 10x for faster data processing
* Label update operations now complete significantly faster
* Event deletion performance dramatically improved
* Enhanced ClickHouse storage efficiency and query performance
* *Benefit:* Teams can now process and analyze data in near real-time, enabling faster decision-making and more responsive monitoring of production models
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### What's New and Improved
* **Updated Fiddler Fast Trust Faithfulness Model**
* *Classification Improvements:*
* Improved accuracy on Q\&A and simple knowledge-retrieval tasks
* Enhanced accuracy on Q\&A with longer contexts
* Improved accuracy on "off-label" tasks like JSON-to-Text and Dialogue/Chat exchanges
* *Performance Boost:*
* 15-20% faster processing on longer contexts
* **Enhanced Fiddler Fast Trust Safety Model**
* Increased Safety model context window from 4,000 to 800,000 tokens
* *Benefit:* Enables comprehensive safety analysis on much larger documents and conversations without truncation
* **Multi-threading for Embedding Enrichment**
* Implemented parallel processing for embedding generation
* *Performance Impact:* At least a 5x improvement in processing speed for embedding enrichments
* *Scalability:* Can achieve even greater performance with additional threads and resources
* *Benefit:* Significantly reduces processing time for high-volume LLM monitoring pipelines
* **Microsoft Teams Webhook Integration**
* Added native support for Microsoft Teams webhook notifications alongside existing Slack integration
* *Benefit:* Teams can now receive alert notifications directly in their Microsoft Teams channels
* *How to use:* Configure Microsoft Teams webhooks in the Webhook Integrations tab of the [Settings](/reference/administration/settings) page
* *Impact:* Streamlines communication workflows for organizations using Microsoft Teams as their primary collaboration platform
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
**Documentation Updates**
* **New Glossary Feature:** We've expanded our Product Concepts guide with a comprehensive [glossary](/glossary). Each term now has its own dedicated page containing:
* Detailed explanations
* Implementation guidance
* Related resources and references
This enhancement makes it easier to understand key product terminology and concepts while providing deeper technical context when needed. We'll continue to add new terms to the glossary incrementally over time.
#### What's New and Improved
* **Custom LLM Enrichment**: Leverage [Llama3.1 8B](https://huggingface.co/meta-llama/Llama-3.1-8B) to categorize input data using your own prompts and custom categories
* *Benefit*: Enables flexible classification tasks tailored to your specific business needs, going beyond pre-defined enrichment types
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches and version updates to application frameworks to stay up-to-date with the latest improvements.
#### Documentation Updates
* **Removed Deployment Guide**: The Deployment Guide is no longer relevant to Fiddler SaaS and managed SaaS offerings.
#### Improvements
* **Baseline Name Length**: Increased the maximum allowed characters from 30 to 256. This change enables more descriptive baseline names for complex projects with multiple models and datasets.
* *Action Required*: None. Existing baselines remain unchanged.
* **Enhanced Job Error Messages**: Error messages during metrics aggregation now specifically identify which step in the data ingestion process failed, helping you troubleshoot pipeline issues faster.
* *Benefit*: Reduces debugging time by pinpointing in which step jobs are failing.
#### Fixes and Security Updates
* **Security Updates**: Applied routine security patches to container images and application frameworks to address recent vulnerabilities.
* **Homepage Cache Timestamp**: Fixed an issue where cached dashboard data would display incorrect "Last Updated" timestamps, leading to confusion about data freshness.
#### Documentation Updates
* **Streamlined Structure**: Reorganized documentation with improved navigation paths between related topics.
* *New Section*: Consolidated technical guides and API references into the "Technical Reference" section.
#### What's New and Improved
#### Now Available in Public Preview
* **UI-based Model Onboarding with Draft Mode**: Iteratively refine your model schema before publishing. Validate sample data, collaborate with your team, and deploy with confidence. This streamlines the model deployment process for faster time-to-value. See our new [Model Guides](/observability/model-ui) for details and best practices.
#### Improvements
* **Enhanced Charts Framework**: Implemented significant improvements to our charting system, delivering more consistent rendering and reliable performance across all dashboards and visualizations.
* **Fast Faithfulness Trust Model Enhancements**: Improved classification accuracy overall by 23% and reduced the Q\&A benchmark error rate by 36%.
#### Fixes and Security Updates
* **Performance Optimization**: Updated application framework components to improve validation speed during model onboarding and data publishing processes.
* **Enhanced Error Handling**: Redesigned validation messages during Baseline creation to provide more actionable and detailed troubleshooting guidance.
* **Concurrency Improvements**: Optimized metrics calculation when model edits trigger recalculations, reducing processing time while preventing disruption to production data pipelines.
#### Documentation Updates
* **Streamlined Structure**: Merged the former UI Guide into the Product Guide for a more intuitive navigation experience.
* **Expanded Content**: Added comprehensive data publishing guides with practical examples and best practices for various data types and formats.
* **New UI Onboarding Guide**: Published detailed documentation for the new UI-based model onboarding feature, including step-by-step instructions and best practices.
#### What's New and Improved
**New Priority Queue for Streaming Data**
We've added a dedicated queue for processing streaming inference data. This improvement gives streaming events priority handling compared to batch processing jobs.
**Improvements**
This release enhances the clarity of error messages when configuring different baseline types for a model.
**How it helps you**
* Faster processing times for streaming data
* Reduced latency for real-time monitoring applications
* No delays from large batch upload operations
**How It Works**
Streaming events now flow through a separate, high-priority processing lane—similar to an HOV lane on a highway—bypassing any congestion from batch operations. This feature works automatically with your existing implementation. No configuration changes required.
#### What's New and Improved
We have added Token Count as a new addition to Fiddler's out-of-the-box enrichments. Token count visibility is a key factor for monitoring and optimizing LLM applications. This enrichment is particularly useful for cost analysis, as it tracks API usage and helps teams understand the financial implications of their LLM usage. It also aids in identifying performance issues related to token limits and supports system health monitoring by detecting unusual patterns or truncated responses. Teams can use token metrics to optimize prompts for efficiency and quality while understanding usage patterns across their application. Combined with existing quality metrics, token counts offer a more complete view of LLM system performance and help teams make data-driven decisions about their prompt engineering and resource allocation.
#### Client Version
Python client version is updated to [3.8](/changelog/python-sdk) and includes new support for updating additional parameters of Alert rules, including `warning_threshold`, `critical_threshold`, and `evaluation_delay`.
#### What's New and Improved
**Introducing Fiddler Guardrails!**
Fiddler AI has introduced Fiddler Guardrails, a new feature that extends the Fiddler Trust Service and is designed to enhance the safety and security of Large Language Model (LLM) applications. This tool proactively detects and mitigates risks such as hallucinations and prompt injection attacks, ensuring more reliable and trustworthy AI operations. Organizations can confidently deploy LLM applications with improved oversight and protection by integrating Fiddler Guardrails. You can see the full announcement [here](/changelog) for a comprehensive overview of this feature.
**Improvements**
This release includes updates focused on improving system performance, stability, and scalability. These improvements ensure a smoother user experience and provide a more robust platform for future developments.
**Guardrails Endpoint Change**
This release updates the following Guardrails REST API endpoints. The [Guardrails guide ](/getting-started/guardrails)provides detailed usage information.
| Guardrail Service | Previous Endpoint | Current Endpoint |
| ---------------------------- | --------------------------- | ------------------------- |
| Fast Safety Guardrails | ftl\_prompt\_safety | ftl-safety |
| Fast Faithfulness Guardrails | ftl\_response\_faithfulness | ftl-response-faithfulness |
#### What's New and Improved
Introducing **UI Model Onboarding**, a powerful new capability that enables teams to onboard models directly through the Fiddler user interface. This streamlined approach to model integration enhances the platform's accessibility while maintaining robust monitoring capabilities.
Here's what this means for you:
* Easy to use: We designed the Model Onboarding UI to be user-friendly, making it simple and intuitive to onboard your models.
* More accessible: This new feature makes it easy to onboard your models even if you're not a Python expert.
What can you do with it?
* Onboard models for different tasks: Currently, it supports three model task types: Not Set, Regression, and Binary/Multiclass Classification.
* Upload your data: Upload your sample data, and Fiddler will automatically try to understand its structure, saving you time from entering details manually.
* Review and edit: You can quickly review and edit the data structure (schema) inferred by Fiddler.
* Define targets: Specify the target variable that your model is designed to predict.
* Provide model details: Give your model a name and other important information.
This feature is currently in public preview which you can learn more about [here](/reference/feature-maturity-definitions#public-preview). We appreciate your feedback as we work to enhance the UI Model Onboarding experience.
#### What's New and Improved
We're excited to introduce three updates in this release that will significantly improve workflow efficiency. These updates include enhancements to alert notifications, schema management, and model onboarding workflows.
* **Pause Alert Notifications**
* Users can now pause and resume notifications for specific alert rules without interrupting their evaluation. This feature helps reduce alert fatigue and prioritize critical alerts.
* Highlights:
* New bell icon for toggling notification status directly from the alert rule table.
* Notifications can be paused or resumed with clear visual feedback and confirmation messages.
* Benefits:
* Reduced alert fatigue.
* Continued evaluation of paused alerts without disruption.
* **Model Schema Editing (Private Preview)**
* Say goodbye to re-onboarding models for schema updates! This feature allows users to edit numerical and categorical columns and add metadata columns directly within the platform.
* Highlights:
* Adjust numerical ranges and add new categories to categorical columns.
* Add metadata columns for enhanced schema flexibility.
* Automatic recalculation of metrics and aggregates after edits.
* Benefits:
* Faster schema updates with no re-onboarding required.
* Keeps metrics and alerts in sync with the latest schema changes.
* **UI-Based Model Onboarding (Private Preview)**
* Simplify model onboarding with our new interactive UI. Add models without relying on Python APIs, making onboarding faster and more accessible to all team members.
* Highlights:
* Supports key task types: Not Set, Regression, and Binary Classification.
* Automatic schema inference and validation with error detection.
* Benefits:
* Streamlines onboarding for non-technical users.
* Reduces errors with built-in validation checks.
**Private Preview**
* The Model Schema Editing and UI-Based Model Onboarding features are available for private preview. For more details on how to participate, please refer to the private preview [guide](/reference/feature-maturity-definitions#private-preview).
#### What's New and Improved
* **Native Integration with AWS SageMaker AI**
* Fiddler is now natively supported within the newly launched Amazon SageMaker partner AI ecosystem. This integration enables enterprises to validate, monitor, analyze, and improve their ML models in production, all within their existing private and secure Amazon SageMaker AI environment. Read the official announcement [here](https://www.fiddler.ai/blog/fiddler-delivers-native-enterprise-grade-ai-observability-to-amazon-sagemaker-ai-customers). Note Fiddler Python client version [3.7+](/changelog/python-sdk) is required for this feature.
* **Download Dataset Code in UI**
* Now you can download your baseline and non-production datasets faster than ever with just a click! Building on the popular feature we introduced for production data in the Root Cause Analysis Events table, we've added ready-to-use Python code snippets right in the interface. Simply copy and paste these snippets to jumpstart your data analysis in your notebooks.
* **Python Client Highlights**
* The latest release of Fiddler's Python client brings two powerful new convenience features to streamline your workflow:
* Our new Project.get\_or\_create() class function simplifies project creation in notebooks. This feature prevents name conflict errors during project creation when your notebook runs multiple times, saving you time and reducing the need for additional exception handling.
* We've also added model.remove\_column(), a simpler way to remove columns during model onboarding. This function replaces the multi-step process previously required, making model configuration faster and more intuitive.
* To enhance reliability, we've implemented a configurable HTTP retry mechanism that you can fine-tune to match your network environment.
* For more details, please refer to the [Python Client Release notes](/changelog/python-sdk).
#### Discontinued
* **The SQL Analyze Page Discontinued**
* The legacy SQL Analyze page has been removed as of 24.18. The new Analyze experience within monitoring charts Root Cause Analysis now enables data table generation using Fiddler Query Language (FQL) and supports the creation of analytical charts such as confusion matrices, feature distribution charts, and more.
#### Client Version
Client version [3.7+](/changelog/python-sdk) is required for the updates and features mentioned in this release.
#### What's New and Improved
* **Feature Analytics in Root Cause Analysis (Public Preview)**
* The root cause analysis experience within monitoring charts now allows users to view feature distribution, feature correlation, and correlation matrix.
#### Discontinued
* **SQL Methods in the Python Client Discontinued**
* From Client 3.6 and onwards, `get_slice` and `download_slice` are discontinued. In their stead, use the new `download_data` method to download production and non-production data from your Fiddler models. If you have any questions or need any assistance migrating scripts using the deprecated methods, please contact your Fiddler customer success manager.
* Use of or support for **Python 3.8 is discontinued** by Fiddler. Note Python 3.8 has been designated [End of Life](https://devguide.python.org/versions/) as of October 7, 2024.
#### What's New and Improved
* **New Chart Type: Correlation Matrix (Public Preview)**
* The Correlation Matrix chart enables users to visualize relationships between up to eight columns in a heatmap, making it easy to spot significant patterns. By clicking on any cell representing the relationship between two features, users can open a Feature Correlation chart for that pair, offering more detailed insights into the correlation score.
* **Events Table in Root Cause Analysis (Public Preview)**
* The root cause analysis experience within monitoring charts now allows users to perform deeper investigations by viewing and downloading up to 1,000 raw events, providing valuable insights for understanding and addressing potential issues.
#### What's New and Improved
* **New Chart Type: Metric Card (Public Preview)**
* We’re excited to introduce the Metric Card chart type, which allows users to display up to four key numerical values in a clear and concise card format. This new visualization enhances data presentation by enabling quick insights into critical metrics, making it easier for decision-makers to spot trends or performance indicators at a glance.
* **New Chart Type: Feature Correlation (Public Preview)**
* The Feature Correlation chart, part of Feature Analytics charts, enables users to analyze and visualize the relationships between different features within their models. By offering a clear view of correlations, this tool supports more informed model diagnostics and refinement.
#### What's New and Improved
* This release focused on system performance, stability, and security enhancements. These improvements ensure a smoother user experience and provide a more robust platform for future developments.
#### What's New and Improved
* NEW Standalone Feature Distribution Chart (Public Preview)
* Create feature distribution charts for numerical and categorical data types that can be added to dashboards.
* Embedding Visualization UX Improvements
* User interface and usability improvements to the UMAP embedding visualization chart.
* Additional database performance improvements.
#### Deprecated and Decommissioned
* Fairness was decommissioned in v24.8, and the documentation has now been removed.
* Surfacing Error Messages for Failed Jobs
* Error messages for failed jobs are now visible directly on the UI job status page, simplifying the process of diagnosing and resolving issues.
* User Selected Default Dashboards
* Any dashboard within a project can now be assigned as the default dashboard for a model, with all insights leading directly to the assigned default dashboard.
* Custom Feature Impact Feature Release Notes
* Introducing Custom Feature Impact: Upload custom feature impact scores for your models, leveraging domain-specific knowledge or external data without requiring the corresponding model artifact.
* Easy data upload via API endpoint with required parameters: Model UUID, Feature Names, and Impact Scores.
* View updated feature impact scores in:
* Model details page
* Charts page
* Explain page
* Flexible update options: Update existing feature impact data by uploading new data for the same model and Seamless integration with existing model artifacts.
* Flexible Model Deployment
* The `python-38` base image is no longer supported.
#### Client Version
Client version 3.3+ is required for the updates and features mentioned in this release.
#### What's New and Improved
* Performance Analytics (Preview) Embedded in Monitoring Charts
* Visualize performance analytics charts as part of the root cause analysis flow for Binary Classification, Multiclass Classification, and Regression models, spanning from confusion matrices, precision recall charts, prediction scatterplots and more.
#### Client Version
Client version 3.3+ is required for the updates and features mentioned in this release.
#### What's New and Improved
* Support for applied segments in monitoring charts
* Create and apply segments dynamically in monitoring charts for exploratory analysis without requiring them to be saved to the model.
* User-Defined Feature Impact
* The User-Defined Feature Impact enables you to upload custom feature impact for models. This feature addresses several issues reported by our customers, including model artifact size, onboarding complexity, and the need for custom feature impact.
* Key highlights
* New method: UploadFeatureImpact
* Improved Fiddler UI to display uploaded feature impact
#### What's New and Improved
* Enhanced access controls
* Control access with precision: Manage user access to resources with Role-Based Access Control (RBAC), ensuring the right users have the right permissions.
* Simplify user management: Assign roles to users and teams to streamline access control and enhance collaboration. \*≠ Protect sensitive resources: Restrict access to sensitive resources, such as models and project settings, with granular permissions.
* Work efficiently: Focus on your work without worrying about unauthorized access or data breaches.
#### Release of Fiddler Platform Version 24.8
* **Performance Analytics Charts (Public Preview)**
* Visualize charts to aid in analyzing model performance for Binary Classification, Multiclass Classification, and Regression models.
* Leverage applied segments in Performance Analytics charts to explore problematic cohorts of data.
#### What's New and Improved
TBD
#### Release of Fiddler Platform Version 24.6
* Performance improvements
* Improved the performance of various modules / APIs.
* Improved observability which can help monitor health and performance of the operations.
#### Client Version
* Client version 3.1.2+ is required for the updates and features mentioned in this release.
#### Release of Fiddler Platform Version 24.5
* Support for model versions for streamlined model management
#### What's New and Improved
* **Model Versions**
* Efficiently manage related models by creating structured versions, facilitating tasks like retraining and comparison analyses.
* Users can maintain model lineage, efficiently manage updates, flexibly modify schemas, and adjust parameters.
* **Airgapped Enrichments (alpha)**
* For privacy sensitive use cases, all data getting enriched stays within customer premises.
* **New Deployment Base Images**
* We have added new deployment base images to support model versioning.
**Client Version**
Client version 3.1.0+ is required for the updates and features mentioned in this release.
#### Release of Fiddler Platform Version 24.4
* UMAP UI changes
* SSO integration changes
* New concept: **Environments**
* Fundamental changes to product concepts
#### What's New and Improved
* **UMAP UI**
* Vertical scrolling instead of horizontal scrolling for data cards
* "View More" option to open data cards in maximized modal
* Ability to toggle between data cards in the maximized modal
* **SSO integration changes**
* Fiddler now integrates with Azure AD SSO, allowing you to leverage existing user roles for access control within Fiddler. This eliminates the need for manual user creation and simplifies user management within your organization.
* **Environments**
* Each Model now has two environments (Pre-Production and Production) used to house data in different ways.
* A Model's Pre-Production environment is used to house non-time series data (Datasets).
* A Model's Production environment is used to house time series data.
* **Product concept changes**
* Datasets are no longer stored at the Project level. Instead, they're stored at the Model level under the Pre-Production Environment.
* The Model Details page has been updated with a new design.
**Client Version**
Client version 3.0+ is required for the updates and features mentioned in this release.
#### Client 3.x Release
We are launching Client 3.x, this is revamped client 2.x as we move to more object-oriented based methods. This means, any pipeline setup in client 2.x would eventually be required to upgrade to the new methods. **Client 2.x will sunset approximately 6 months post this release.**
#### Deprecations and Removals
* All IDs will be UUIDs instead of strings.
* Dataset deletion is not allowed anymore.
For API level changes and updates please check [client history](/changelog/python-sdk)
# Python Client SDK
Source: https://docs.fiddler.ai/changelog/python-sdk
Fiddler's Python client history reference. Contains Python client release highlights, client deprecation notices, and more.
[](https://pypi.org/project/fiddler-client/)
## 3.14
* **Bug Fixes**
* **`delete_events()` now rejects mixing selectors**: Passing both a time range (`start_time`/`end_time`) and `event_ids` in the same `Model.delete_events()` call now raises an error instead of silently applying one. The two selection modes are mutually exclusive.
* **`Authorization` header redacted in logs**: The HTTP client no longer writes the `Authorization` header value to log output, preventing API tokens from leaking into logs.
* **Changes**
* **500 MB file-size limit on multipart upload**: `multipart_upload` now enforces a 500 MB per-file limit, failing fast with a clear error rather than attempting an oversized upload.
## 3.13
* **New Features**
* Added `Model.delete_events()` to delete published production events for a model. Wraps the [Delete Events REST API](/sdk-api/rest-api/events/delete-events) (`DELETE /v3/events`); deletion runs asynchronously and returns a `Job`.
* **Select events** by a time range or an explicit list of event IDs (the two are mutually exclusive):
* `start_time` / `end_time` — must be provided together. They accept a timezone-aware `datetime` or an ISO 8601 string with an explicit offset (e.g., `'2025-11-17T01:00:00Z'`); non-UTC values are normalized to UTC, and naive datetimes are rejected.
* `event_ids` — up to 10,000 IDs per call.
* `update_metrics` (default `False`) — whether the server recomputes metrics after deletion. Applies to time-range deletions only.
* **Safety guards**: a call with no filter is rejected (to avoid deleting all events for a model), both time bounds are required together, and providing more than 10,000 `event_ids` is rejected.
```python theme={null}
job = model.delete_events(
start_time='2025-11-17T01:00:00.000Z',
end_time='2025-11-17T02:00:00.000Z',
update_metrics=True,
)
job.wait()
# Or delete specific events by ID
job = model.delete_events(event_ids=['pred_001', 'pred_002'])
job.wait()
```
## 3.12
* **New Features**
* `Model.update()` now supports updating column schema properties (bins, min, max, categories) on existing models
* Set custom histogram bins: `model.schema["column"].bins = [...]; model.update()`
* Update value ranges: `model.schema["column"].min = value; model.update()`
* Update categories: `model.schema["column"].categories = [...]; model.update()`
* Added explicit `Model.publish_stream()` and `Model.publish_batch()` methods for publishing inference data
* `publish_stream(events=...)` publishes streaming/live events (a list of event dicts)
* `publish_batch(source=...)` publishes a batch from a Parquet/CSV file path or a pandas DataFrame
* Both accept `update=True` to update previously published events
* **Deprecations**
* `Model.publish()` is deprecated and now emits a `DeprecationWarning`. Its implicit dispatch is replaced by the explicit `publish_stream()` (streaming events) and `publish_batch()` (batch files and DataFrames) methods. See [Publishing production data](/developers/client-library-reference/publishing-production-data).
* **Dependencies & Python Support**
* **Migrated to Pydantic v2** (`pydantic>=2.0`, previously `pydantic>=1.10`). Environments pinned to Pydantic v1 must upgrade to v2.
* Added support for **Python 3.13 and 3.14** (enabled by the Pydantic v2 migration).
* Bumped `pandas>=2.1.0` (previously `>=1.2.5`) and `pyarrow>=17.0.0` (previously `>=15.0.0`); `pip` is no longer a runtime dependency.
## 3.11
* **New Features**
* Added `Model.add_column()` method to dynamically add columns to existing model schemas after initial onboarding. Supports adding input features, outputs, targets, and metadata columns.
* **Parameters**:
* `column` (Column): The column definition to add to the model
* `column_type` (str): Type of column - one of 'inputs', 'outputs', 'targets', or 'metadata' (default)
* **Important**: New columns have null values for historical events; future events must include the new column
* **Validation**: Prevents duplicate column names (raises ValueError)
* **New Features**
* Added a helper function `create_columns_from_df` to create a `ModelSchema` from a pandas `Dataframe`.
* **New Features**
* Added ability to specify `auto_threshold_params` while creating/updating alert rules.
* **Modifications**
* Updated minimum required version of pyarrow from 'pyarrow>=7.0.0' to 'pyarrow>=15.0.0' to address potential security issues from CVE-2025-30065.
* **New Features**
* Added the ability to create and use Microsoft Teams Webhooks.
* **Bug Fixes**
* Fixed pydantic validation error for `Webhook` object.
* **Modifications**
* Improved log message for timeout failure on how timeout can be increased.
* **Bug Fixes**
* Added missing support for specifying `category` while creating Alert Rule.
* **New Features**
* Added a new `update` method to the `AlertRule` object, allowing updates to the following fields: `warning_threshold`, `critical_threshold`, and `evaluation_delay`.
* Added ability to create alert rules with Fiddler-determined automatic thresholds.
* **Modifications**
* **Project Deletion Uses v3 API**:
* `project.delete()` now utilizes the v3 API for deleting projects.
* The method signature remains unchanged.
* **Modifications**
* **Connection Timeout Settings**: You can now configure network timeout settings when [initializing](/developers/python-client-guides/installation-and-setup) the Python client. The new timeout parameter in init() accepts:
* A single number (in seconds) to set the connection timeout
* A tuple of two numbers (in seconds) to set both connection and read timeouts separately
Release highlights:
* **Robustness via retrying**: this release introduces a persistent HTTP request retrying strategy to enhance fault tolerance in view of transient network problems and retryable HTTP request errors. You can take control of the maximum duration for which an HTTP request is retried by setting the environment variable `FIDDLER_CLIENT_RETRY_MAX_DURATION_SECONDS`.
* **AWS SageMaker authentication support**: to enable that, install version 2.236.0+ of the [AWS Python SageMaker SDK](https://pypi.org/project/sagemaker/). Then, before calling `init()`, set the environment variable `AWS_PARTNER_APP_AUTH` to `true` and set `AWS_PARTNER_APP_ARN`/`AWS_PARTNER_APP_URL` to meaningful values.
* **Logging improvements**: messages are now emitted to `stderr` instead of `stdout`. Only if the calling context does not configure a root logger this library will actively declare a handler for its own log messages (this automation can be disabled by setting `auto_attach_log_handler=False` during `init()`).
Compatibility changes:
* Pydantic 2.x is now supported (and compatibility with Pydantic 1.x has been retained).
* Support for Python 3.8 has been dropped.
API surface additions:
* Introduced `Project.get_or_create()` to reduce code required for instantiating a project.
* Introduced `model.remove_column()` to allow for removing a column from a model object.
Fixes:
* A transient error during a job status update does not prematurely terminate waiting for a job anymore.
* GET requests do not contain the `Content-Type` header anymore.
* **Removed**
* The `get_slice` and `download_slice` methods are removed. Please use `download_data` to retrieve some data.
* The `get_mutual_info` method is removed.
* The `SqlSliceQueryDataSource` option is removed from explain, feature impact and importance. Please use the `DatasetDataSource` instead or the UI.
* **New Features**
* New `download_data` method, to download a slice of data given an environment, time range and segment. Resulted file can be downloaded either as a CSV or a Parquet file.
* **Removed**
* The `get_fairness` method is removed. Please use charts and custom metrics to track / compute fairness metrics on your model.
* **Modifications**
* Fixed the error while setting notification config for alert rule.
* **Modifications**
* Added validation while adding notifications to alert rules.
* Upgraded dependencies to resolve known vulnerabilities - deepdiff, mypy, pytest, pytest-mock, python-decouple, types-requests and types-simplejson.
* **New Features**
* Introduced `upload_feature_impact()` method to upload or update feature impact manually.
* **New Features**
* Introduced evaluation delay in Alerts Rule.
* Optional `evaluation_delay` parameter added to `AlertRule.__init__` method.
* It is used to introduce a delay in the evaluation of the alert.
* **Modifications**
* Fix windows file permission error bug with publish method.
* **Modifications**
* Adds support to get schema of Column object by `fdl.Column`
* **Modifications**
* Updated `pydantic` and `typing-extensions` dependencies to support Python 3.12.
* **New Features**
* Introduced the native support for model versions.
* Optional `version` parameter added to `Model`, `Model.from_data`, `Model.from_name` methods.
* New `duplicate()` method to seamlessly create new version from existing model.
* Optional `name` parameter added to `Model.list` to offer the ability to list all the versions of a model.
* **New Features**
* Allowed usage of `group_by()` to form the grouped data for ranking models.
* **Modifications**
* Return Job in ModelDeployment update.
* **New Features**
* Added `Webhook.from_name()`
* **Modifications**
* Import path fix for packtools.
* **Modifications**
* Fix pydantic issue with typing-extensions versions > 4.5.0
* **New Features**
* General
* Moving all functions of client to an Object oriented approach
* Methods return resource object or a deserialized object wherever possible.
* Support to search model, project, dataset, baselines by their names using `from_name()` method.
* List methods will return iterator which handles pagination internally.
* Data
* Concept of environments was introduced.
* Ability to download slice data to a parquet file.
* Publish dataframe as stream instead of batch.
* New methods for baselines.
* Multiple datasets can be added to a single model. Ability to choose which dataset to use for computing feature impact / importance, surrogate generation etc.
* Model can be added without dataset.
* Ability to generate schema for a model.
* Model delete is async and returns job details.
* Added cached properties for `model`: `datasets`, `model_deployment`.
* Alerts
* New methods for alerts: `enable_notification`, `disable_notification`, `set_notification_config` and `get_notification_config`.
* Explainability
* New methods in explainability: `precompute_feature_impact`, `precompute_feature_importance`, `get_precomputed_feature_importance`, `get_precomputed_feature_impact`, `precompute_predictions`.
* Decoupled model artifact / surrogate upload and feature impact / importance pre-computation.
* **Modifications**
* All IDs will be UUIDs instead of strings
* Dataset delete is not allowed anymore
# Fiddler Strands Agent SDK
Source: https://docs.fiddler.ai/changelog/strands-sdk
Fiddler's Strands Agent SDK release history. Contains Strands Agent SDK release highlights, SDK deprecation notices, and more.
[](https://pypi.org/project/fiddler-strands/)
## 0.6
* **New Features**
* **`clear_llm_context()` function**: New convenience function to explicitly remove LLM context from a `Model` instance, preventing faithfulness evaluation on subsequent non-RAG spans. In multi-step agent workflows, context set after a RAG retrieval step previously leaked into later non-RAG LLM calls (tool planning, routing, etc.), causing unintended faithfulness evaluation. `set_llm_context(model, None)` also now clears context.
* **Python 3.14 support**: Added Python 3.14 to the CI test matrix and PyPI classifiers. All runtime dependencies have compatible wheels.
* **Fiddler attributes bridged to Strands `trace_attributes`**: Custom span and session attributes set via `set_span_attributes()` and `set_session_attributes()` are now bridged to Strands' native `trace_attributes` mechanism, enabling propagation through the Strands telemetry pipeline without custom hook plumbing for every span type.
* **Changes**
* **`strands-agents` floor bumped from `>=1.10.0` to `>=1.19.0`**: Required to align with the Strands telemetry features the SDK now consumes. Users on Strands `1.10.x`–`1.18.x` must upgrade to at least `strands-agents 1.19.0`.
* **Extended `strands-agents` upper bound to `<=1.33.0`**: Tested compatibility extended to support newer Strands releases.
* Bumps `fiddler-otel` dependency to `1.1.1` (gains `add_session_attributes()` support for all OTel primitive value types — `str`, `bool`, `int`, `float`, and homogeneous sequences — previously restricted to `str`).
* **Bug Fixes**
* **Defensive guard for `trace_attributes` access**: Added a defensive check before accessing `trace_attributes` to prevent attribute errors when bridging Fiddler attributes to Strands telemetry on agents that have not yet initialized the field.
## 0.5
* **Initial PyPI Release**
* First version-tagged build of `fiddler-strands` published to PyPI from the consolidated `fiddler-sdk` monorepo (`pip install fiddler-strands`).
* Includes all functionality from the 0.4.0 development build (auto-instrumentation, conversation tracking, session and span attributes, LLM context management, and tool-definition tracing). Version 0.4.0 was never published to PyPI.
## 0.4
* **Enhancements**
* **Tool Definition Tracing**: Added `gen_ai.tool.definitions` span attribute to all child spans, providing visibility into tool schemas within agent traces.
* **Enhancements**
* **API Export Fix**: Added `get_llm_context` to module exports — previously available but not importable from the top-level package.
* **Documentation**: Added comprehensive docstrings to all critical public APIs to support automatic documentation generation.
* **Examples**: Fixed missing example dependencies and Fiddler configuration setup.
* **Enhancements**
* **LLM Context Management**: New `set_llm_context()` and `get_llm_context()` functions for setting additional context for LLM processing within Strands agent workflows.
* **System Prompt Telemetry**: System prompts are now included as part of telemetry events for backend processing.
* **Initial Release**
* Core `StrandsAgentInstrumentor` with `instrument()` and `uninstrument()` for automatic OpenTelemetry instrumentation of Strands agents.
* **Conversation Tracking**: `set_conversation_id()` and `get_conversation_id()` for multi-turn conversation tracking.
* **Session Attributes**: `set_session_attributes()` and `get_session_attributes()` for adding custom metadata to trace sessions.
* **Span Attributes**: `set_span_attributes()` and `get_span_attributes()` for adding custom metadata to individual spans.
* Available as `pip install fiddler-strands`.
# RAG Health Diagnostics
Source: https://docs.fiddler.ai/concepts/rag-health-diagnostics
Understand how RAG Health Metrics diagnose Retrieval-Augmented Generation pipeline failures using Answer Relevance, Context Relevance, and RAG Faithfulness evaluators.
## The RAG Debugging Challenge
Retrieval-Augmented Generation (RAG) applications fail in specific ways that generic metrics cannot diagnose. When a RAG system produces a poor response, the failure could originate in any of three stages:
1. **Retrieval** — The system retrieved irrelevant or insufficient documents
2. **Generation** — The LLM generated content not grounded in the retrieved documents
3. **Query understanding** — The response doesn't address what the user actually asked
Without targeted diagnostics, debugging RAG failures is manual trial-and-error — inspecting retrieved documents, re-running queries, and guessing where the pipeline broke. RAG Health Metrics transforms this into targeted root cause analysis.
## The RAG Health Metrics Triad
RAG Health Metrics is a purpose-built diagnostic framework consisting of three evaluators that work together to pinpoint exactly where RAG pipelines fail:
| Evaluator | What It Measures | Scoring | Inputs |
| ------------------------ | ---------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------- |
| **Answer Relevance 2.0** | Does the response address the user's query? | High (1.0), Medium (0.5), Low (0.0) | `user_query`, `rag_response` (+ optional `retrieved_documents`) |
| **Context Relevance** | Are the retrieved documents relevant to the query? | High (1.0), Medium (0.5), Low (0.0) | `user_query`, `retrieved_documents` |
| **RAG Faithfulness** | Is the response grounded in the retrieved documents? | Yes (1.0) / No (0.0) | `user_query`, `rag_response`, `retrieved_documents` |
All three evaluators return detailed `reasoning` explaining the score, enabling you to understand not just *what* failed but *why*.
### Availability
| Evaluator | Agentic Monitoring | Experiments | LLM Observability |
| -------------------- | ------------------ | ----------- | ----------------- |
| Answer Relevance 2.0 | Yes | Yes | Yes |
| Context Relevance | Yes | Yes | **No** |
| RAG Faithfulness | Yes | Yes | Yes |
**Context Relevance** is available in Agentic Monitoring and Experiments only. It is not available in LLM Observability.
## Diagnostic Workflow
Use the three evaluators together to diagnose specific failure modes in your RAG pipeline:
| What the metrics tell you | Why it's happening | Next Step |
| --------------------------------- | ------------------------------------- | ------------------------------------------------ |
| High relevance + Low faithfulness | Hallucinations despite being on-topic | Check if retrieval provided sufficient grounding |
| High faithfulness + Low relevance | Grounded but didn't answer the query | Check if retrieval provided relevant information |
| Low Context Relevance | Retrieval pulling wrong documents | Fix retrieval mechanism |
### Scenario 1: Hallucination (High Relevance, Low Faithfulness)
The response addresses the user's question (high Answer Relevance) but includes claims not supported by the retrieved documents (low RAG Faithfulness). This indicates the LLM is generating plausible-sounding content rather than grounding its response in the provided context.
**Root cause:** Generation layer — the LLM is not sufficiently constrained by the retrieved documents.
**Actions to investigate:**
* Review the system prompt — does it instruct the LLM to only use provided context?
* Check if the retrieved documents contain enough detail to answer the question
* Consider adding explicit grounding instructions to the prompt
### Scenario 2: Off-Topic Response (Low Relevance, High Faithfulness)
The response accurately reflects the retrieved documents (high RAG Faithfulness) but doesn't answer what the user asked (low Answer Relevance). The LLM faithfully summarized the wrong information.
**Root cause:** Retrieval layer — the retrieved documents are relevant enough to generate a response, but don't contain the information needed to answer the specific query.
**Actions to investigate:**
* Review the retrieval query — is it capturing the user's intent?
* Check if the knowledge base contains the needed information
* Consider improving query expansion or embedding quality
### Scenario 3: Bad Retrieval (Low Context Relevance)
The retrieved documents are not relevant to the user's query at all (low Context Relevance). Downstream evaluators may score unpredictably because the entire pipeline is working with the wrong source material.
**Root cause:** Retrieval mechanism — the vector search, keyword matching, or hybrid retrieval is pulling wrong documents.
**Actions to investigate:**
* Review the embedding model — does it capture semantic similarity for your domain?
* Check chunking strategy — are documents split at appropriate boundaries?
* Examine the retrieval query formulation — is the user's intent preserved?
* Consider re-indexing with a domain-adapted embedding model
## Using RAG Health Metrics
### In Experiments (Evals SDK)
Evaluate RAG pipelines systematically against test datasets:
```python theme={null}
from fiddler_evals import evaluate
from fiddler_evals.evaluators import AnswerRelevance, ContextRelevance, RAGFaithfulness
evaluators = [
AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential"), # Is the response relevant? (High/Medium/Low)
ContextRelevance(model="openai/gpt-4o", credential="your-llm-credential"), # Are retrieved docs relevant? (High/Medium/Low)
RAGFaithfulness(model="openai/gpt-4o", credential="your-llm-credential"), # Is the response grounded? (Yes/No)
]
results = evaluate(
dataset=dataset,
task=my_rag_task,
evaluators=evaluators,
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["user_query"],
"rag_response": "rag_response",
"retrieved_documents": lambda x: x["inputs"]["retrieved_documents"],
}
)
```
### In Agentic Monitoring
Configure Evaluator Rules to continuously evaluate production RAG spans. Navigate to your application's **Evaluator Rules** tab and add rules for Answer Relevance, Context Relevance, and RAG Faithfulness. Map the evaluator inputs to your span attributes.
See [Evaluator Rules](/evaluate-and-test/evaluator-rules) for step-by-step configuration instructions.
## RAG Faithfulness vs Faithfulness (Centor Model)
Fiddler provides two separate faithfulness evaluators with different architectures:
| Feature | RAG Faithfulness | Faithfulness (Centor Model) |
| ---------------------------- | --------------------------------------------------- | ------------------------------------------------------- |
| **Type** | LLM-as-a-Judge | Centor Model for Faithfulness |
| **Class** | `RAGFaithfulness` | `FTLResponseFaithfulness` |
| **Inputs** | `user_query`, `rag_response`, `retrieved_documents` | `context`, `response` |
| **Outputs** | `label` (yes/no), `value` (1/0), `reasoning` | `faithful_prob` (0.0–1.0) |
| **Best for** | RAG pipeline diagnostics, comprehensive evaluation | Guardrails, real-time monitoring, low-latency use cases |
| **Part of RAG Health triad** | Yes | No |
Choose RAG Faithfulness when you need detailed reasoning and diagnostic context as part of the RAG Health Metrics triad. Choose Faithfulness (Centor Model) when you need sub-100ms latency for production guardrails.
## Complementary Metrics
RAG Health Metrics works alongside Fiddler's existing 80+ LLM metrics. RAG systems fail in specific ways that generic metrics cannot diagnose — RAG Health Metrics provides the targeted diagnostics, while metrics like toxicity, PII detection, coherence, and sentiment provide complementary quality and safety coverage.
## Next Steps
* [RAG Health Metrics Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial) — End-to-end hands-on guide
* [Experiments Getting Started](/getting-started/experiments#rag-system-evaluation) — RAG evaluation use case
* [Evaluator Rules](/evaluate-and-test/evaluator-rules) — Configure production monitoring
# Semantic Mappings
Source: https://docs.fiddler.ai/concepts/semantic-mappings
How Fiddler maps raw OTel attribute keys to canonical semantic concepts for cross-framework analytics, alerts, and dashboards
## Overview
Fiddler ingests OpenTelemetry traces from a wide range of AI frameworks and SDKs — OTel [GenAI Semantic Conventions](https://github.com/open-telemetry/semantic-conventions-genai), OpenInference (LlamaIndex, LangChain), Vercel AI SDK, Claude Code, LiteLLM, Langfuse, and many more. Each framework uses its own attribute key names for the same underlying concept. For example, the number of input tokens might be sent as `gen_ai.usage.input_tokens` by one framework, `llm.token_count.prompt` by another, and `ai.usage.promptTokens` by a third.
Server-side semantic mappings solve this problem by associating each raw attribute key with a canonical **semantic concept** — a framework-agnostic name like `input_tokens`, `model_name`, or `input`. This enables Fiddler's dashboards, alerts, custom metrics, and evaluators to work consistently across all frameworks without requiring per-framework instrumentation (client-side).
Semantic mappings are enabled by default for all organizations and are the recommended way to work with span attributes. Legacy prefix conventions are deprecated — see [Span and Resource Attributes](/integrations/agentic-ai/attributes#deprecated-prefix-conventions) for migration guidance. Existing data using legacy prefixes continues to resolve correctly via backward-compatible default mappings.
## How It Works
When your application sends OpenTelemetry traces to Fiddler, the raw attribute keys are stored exactly as sent — Fiddler does not transform or rename them. In the backend, Fiddler enables you to map each attribute key to a semantic concept, producing a canonical name alongside the original key.
All downstream features — monitoring charts, alert rules, custom metrics, evaluator input configuration, and the Explorer — use the semantic concept for queries. This means a dashboard showing "average input tokens over time" works identically whether your application uses the OTel GenAI convention (`gen_ai.usage.input_tokens`), the OpenInference convention (`llm.token_count.prompt`), or any other supported framework's naming.
Custom attributes that don't map to any semantic concept are still stored and queryable by their original key name. Semantic mapping only applies to attributes that Fiddler recognizes as carrying a known concept.
## Semantic Concepts
Fiddler defines over 30 semantic concepts, organized into the following categories.
### Token Usage
| Concept | Description |
| ----------------------------- | ---------------------------------------------- |
| `input_tokens` | Number of tokens in the input/prompt |
| `output_tokens` | Number of tokens in the output/completion |
| `total_tokens` | Total token count (input + output) |
| `cache_read_input_tokens` | Cached input tokens read (Claude, GPT caching) |
| `cache_creation_input_tokens` | Cached input tokens created |
| `reasoning_tokens` | Reasoning tokens (o1/o3 models) |
### Cost (Estimated)
| Concept | Description |
| ------------- | ---------------------- |
| `total_cost` | Total LLM usage cost |
| `input_cost` | Input/prompt cost |
| `output_cost` | Output/completion cost |
### Model and Provider
| Concept | Description |
| --------------- | --------------------------------------------------------- |
| `model_name` | LLM model identifier (e.g., `gpt-4.1`, `claude-sonnet-4`) |
| `provider_name` | LLM provider identifier (e.g., `openai`, `anthropic`) |
### Agent and Tool
| Concept | Description |
| ------------------- | -------------------------------- |
| `agent_name` | Agent name |
| `agent_id` | Agent identifier |
| `agent_description` | Agent description |
| `tool_name` | Tool/function name |
| `tool_id` | Tool call identifier |
| `tool_type` | Tool type or category |
| `tool_definitions` | Tool/function schema definitions |
### Session and User
| Concept | Description |
| ------------ | ---------------------------------- |
| `session_id` | Conversation or session identifier |
| `user_id` | End-user identifier |
### Content
| Concept | Description |
| --------------------- | ---------------------------------- |
| `input` | LLM prompt or user input content |
| `output` | LLM completion or response content |
| `system_instructions` | System prompt or instructions |
| `retrieval_context` | RAG retrieval context |
| `tool_input` | Tool call arguments |
| `tool_output` | Tool call results |
Mapping an attribute key to `input`, `output`, `tool_input`, or `tool_output` also makes it searchable in the [Explorer](/observability/agentic/trace-explorer#searching-in-the-ui). No additional configuration is required.
### Performance
| Concept | Description |
| --------- | ------------------- |
| `latency` | Span duration |
| `ttft` | Time to first token |
### Span Identity
| Concept | Description |
| ----------- | --------------------------------------------------------- |
| `span_name` | Span name |
| `span_type` | Span type or kind (e.g., `llm`, `tool`, `agent`, `chain`) |
### Metadata
| Concept | Description |
| --------------- | ----------------------- |
| `received_time` | Ingestion timestamp |
| `request_id` | LLM request identifier |
| `response_id` | LLM response identifier |
| `finish_reason` | LLM stop/finish reason |
## Default Mappings
Fiddler ships with **140+ pre-configured mappings** covering 13+ AI frameworks and SDKs. If your application uses any of the following frameworks, traces are automatically mapped to semantic concepts with no configuration required:
* **OTel GenAI Semantic Conventions** — the standard OpenTelemetry conventions for generative AI (`gen_ai.usage.*`, `gen_ai.request.*`, `gen_ai.response.*`)
* **OpenInference** — used by LlamaIndex, LangChain, and other frameworks (`llm.token_count.*`, `input.value`, `output.value`)
* **Vercel AI SDK** — (`ai.usage.*`, `ai.prompt`, `ai.response.*`)
* **Claude Code** — (`input_tokens`, `output_tokens`, `user_prompt`, `model`)
* **LiteLLM** — (`gen_ai.cost.*`)
* **Langfuse SDK** — (`langfuse.session.id`, `langfuse.user.id`)
* **Google Agent Development Kit (ADK, part of Gemini Enterprise Agent Platform)** — (`gcp.vertex.agent.*`)
* **LiveKit** — (`lk.input_text`, `lk.response.text`)
* **Genkit** — (`genkit:input`, `genkit:output`)
* **MLflow** — (`mlflow.spanInputs`, `mlflow.spanOutputs`)
* **Traceloop / OpenLLMetry** — (`traceloop.entity.input`, `traceloop.entity.output`)
Many semantic concepts are mapped from multiple frameworks. For example, the `input` concept is mapped from over a dozen different attribute keys across all supported frameworks, so Fiddler can display user prompts regardless of which SDK generated the trace.
## When to Add Custom Mappings
The default mappings cover most common frameworks. You may need to add custom mappings when:
* **Using a framework not in the default list** — if your application uses an AI framework or SDK that Fiddler doesn't ship mappings for, you can add mappings for that framework's attribute keys.
* **Custom instrumentation** — if your application uses proprietary or internal attribute key names that differ from standard OTel conventions.
* **Internal naming conventions** — if your organization has adopted non-standard attribute key naming that you want Fiddler to recognize.
For example, if your application sends `my_framework.prompt_tokens` for input token counts, you can create a mapping from `my_framework.prompt_tokens` to the `input_tokens` semantic concept. After the mapping propagates, all charts, alerts, and metrics that use `input_tokens` will include data from this key.
## Span Types
### Why span type mapping is needed
Each AI framework labels span operations differently. OTel GenAI uses `gen_ai.operation.name` with values like `chat` and `completion`. OpenInference uses `openinference.span.kind` with values like `LLM` and `RETRIEVER`. Vercel AI SDK uses `ai.operationId` with values like `ai.generateText`. Langfuse uses `langfuse.observation.type` with values like `generation`. Without normalization, filtering for "all LLM calls" or "all tool executions" in Fiddler would require knowing every framework's convention.
Fiddler resolves this by automatically normalizing raw span type values into a set of canonical types. When a span is ingested, Fiddler checks its attributes for known span type keys (determined by the semantic mapping for the `span_type` concept), extracts the raw value, lowercases it, and maps it to a canonical type.
### Canonical span types
Fiddler recognizes 10 canonical span types:
| Type | Description |
| ----------- | ------------------------------------------------------- |
| `llm` | LLM inference calls (chat completions, text generation) |
| `tool` | Tool or function executions |
| `agent` | Agent invocations |
| `chain` | Chain or pipeline steps |
| `embedding` | Embedding operations |
| `retriever` | Retrieval operations (RAG) |
| `reranker` | Reranking operations |
| `guardrail` | Guardrail checks |
| `evaluator` | Evaluator runs |
| `span` | Generic span (default for unrecognized values) |
### How span type is inferred
Fiddler checks the following attribute keys to find the span type value:
| Attribute Key | Framework |
| --------------------------- | ------------------------------------- |
| `span_type` | Generic / direct |
| `span.type` | Claude Code |
| `fiddler.span.type` | Fiddler OTel SDK |
| `openinference.span.kind` | OpenInference (LlamaIndex, LangChain) |
| `langfuse.observation.type` | Langfuse SDK |
| `gen_ai.operation.name` | OTel GenAI, LiteLLM, Strands |
| `ai.operationId` | Vercel AI SDK |
| `genkit:metadata:subtype` | Genkit |
Once a raw value is found, it is lowercased and mapped to a canonical type using the table below. If the raw value is not recognized, the span type defaults to `span`.
### Recognized raw values
The following table lists all recognized raw values and the canonical span type they map to. Raw values are case-insensitive (lowercased before lookup).
#### Identity values
These are already canonical and map directly:
| Raw Value | Span Type |
| ----------- | ----------- |
| `llm` | `llm` |
| `tool` | `tool` |
| `agent` | `agent` |
| `chain` | `chain` |
| `embedding` | `embedding` |
| `retriever` | `retriever` |
| `reranker` | `reranker` |
| `guardrail` | `guardrail` |
| `evaluator` | `evaluator` |
| `span` | `span` |
#### OpenInference
| Raw Value | Span Type |
| --------- | --------- |
| `UNKNOWN` | `span` |
| `PROMPT` | `span` |
All other uppercase OpenInference values (`LLM`, `TOOL`, `AGENT`, `CHAIN`, `EMBEDDING`, `RETRIEVER`, `RERANKER`, `GUARDRAIL`) are lowercased and match the identity values above.
#### Claude Code
| Raw Value | Span Type |
| ---------------------- | --------- |
| `interaction` | `agent` |
| `llm_request` | `llm` |
| `tool.blocked_on_user` | `chain` |
| `tool.execution` | `chain` |
#### Langfuse
| Raw Value | Span Type |
| ------------ | --------- |
| `generation` | `llm` |
| `event` | `span` |
#### OTel GenAI / LiteLLM / Strands
| Raw Value | Span Type |
| -------------------------- | ----------- |
| `chat` | `llm` |
| `completion` | `llm` |
| `acompletion` | `llm` |
| `text_completion` | `llm` |
| `atext_completion` | `llm` |
| `responses` | `llm` |
| `aresponses` | `llm` |
| `_aresponses_websocket` | `llm` |
| `anthropic_messages` | `llm` |
| `generate_content` | `llm` |
| `agenerate_content` | `llm` |
| `generate_content_stream` | `llm` |
| `agenerate_content_stream` | `llm` |
| `generate` | `llm` |
| `execute_tool` | `tool` |
| `invoke_agent` | `agent` |
| `create_agent` | `agent` |
| `embeddings` | `embedding` |
| `aembedding` | `embedding` |
#### Genkit
| Raw Value | Span Type |
| ------------------ | ----------- |
| `model` | `llm` |
| `background-model` | `llm` |
| `embedder` | `embedding` |
| `tool.v2` | `tool` |
#### Vercel AI SDK
| Raw Value | Span Type |
| ------------------------------ | ----------- |
| `ai.generateText` | `llm` |
| `ai.generateText.doGenerate` | `llm` |
| `ai.streamText` | `llm` |
| `ai.streamText.doStream` | `llm` |
| `ai.generateObject` | `llm` |
| `ai.generateObject.doGenerate` | `llm` |
| `ai.streamObject` | `llm` |
| `ai.streamObject.doStream` | `llm` |
| `ai.embed` | `embedding` |
| `ai.embed.doEmbed` | `embedding` |
| `ai.embedMany` | `embedding` |
| `ai.embedMany.doEmbed` | `embedding` |
| `ai.toolCall` | `tool` |
### Adding new span type mappings
If your framework uses span type values that Fiddler doesn't recognize (appearing as `span` in the Explorer), contact your Fiddler deployment administrator to add the mapping. New mappings take effect within approximately 5 minutes and apply to all subsequently ingested spans.
## Managing Mappings via the UI
Semantic mappings are managed from the **Settings > Semantic Mappings** tab in the Fiddler UI.
The tab displays all current mappings in a table with two columns:
* **Attribute Name** — the raw OTel attribute key (e.g., `gen_ai.usage.input_tokens`)
* **Semantic Name** — the canonical concept it maps to (e.g., `input_tokens`)
To add a new mapping:
1. Click the **Add Mapping** button
2. Enter the raw attribute key in the **Attribute Name** field
3. Select the semantic concept from the **Semantic Name** dropdown
4. Click **Add**
To delete a mapping, use the delete action on the mapping row.
Only **Org Admins** can create or delete mappings. All users can view the current mappings.
## Propagation
After creating or deleting a mapping, changes take effect within approximately **5 minutes**. Only new spans ingested after the mapping propagates are affected — existing data is not reprocessed.
## Constraints and Limits
| Constraint | Value |
| ---------------------------------- | ------------------------------------ |
| Maximum total mappings | 1,000 |
| Maximum mappings per request | 100 |
| Maximum attribute key length | 256 characters |
| Write rate limit | 10 requests/minute, 100 requests/day |
| Propagation delay | \~5 minutes |
| Required role for write operations | Org Admin |
## Unmapped Attributes
Attributes that don't map to any semantic concept are still fully functional. They are stored with their original key name, appear in the Explorer's span detail view, and are queryable in FQL custom metrics via the `attribute()` function. The only difference is that they won't appear in built-in dashboards or metrics that rely on semantic concepts (such as the token usage chart or the model breakdown view). If you need an unmapped attribute to participate in these features, add a semantic mapping for it.
## Deleting or Changing a Mapping
When you delete a mapping or remap an attribute key to a different semantic concept, the change only affects new spans ingested after the mapping propagates (\~5 minutes). Existing data retains the semantic name that was assigned at ingestion time and is not reprocessed. This means:
* **Deleting a mapping** — new spans with that attribute key will no longer have a semantic name assigned. The attribute is still stored and queryable by its original key, but it stops appearing in concept-based dashboards and metrics.
* **Remapping to a different concept** — new spans will use the new semantic concept. Older spans will continue to use the previous concept. During the transition period, both old and new data are independently queryable.
# Model Onboarding
Source: https://docs.fiddler.ai/developers/client-library-reference/model-onboarding
* [Create a Project and Model](/developers/python-client-guides/model-onboarding/create-a-project-and-model): Explore our guide to creating a project and onboarding a model for observation. Learn how projects organize models and define a ModelSpec and Model Task.
* [Customizing Your Model Schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema): Delve into our guide to customize your Model Schema with Fiddler. Learn how to adjust a column’s value range, possible values, and data type to match your model.
* [Updating Model Schema](/developers/python-client-guides/model-onboarding/updating-model-schema): Learn how to modify your model's schema after initial creation by adding new columns using the Python client's add\_column() method. Add features, metadata, and tracking columns to existing models.
* [Task Types](/developers/python-client-guides/model-onboarding/task-types): Explore our guide to selecting a model task type when onboarding your ML models and LLM applications.
* [Custom Missing Values](/developers/python-client-guides/model-onboarding/specifying-custom-missing-value-representations): Learn how you can customize a model column to assign values to be treated as missing or null data in order to handle a value or token that is inserted in place of null.
# Publishing Production Data
Source: https://docs.fiddler.ai/developers/client-library-reference/publishing-production-data
Navigate our client guide to publishing production data. Learn how to provide event data to Fiddler, update it, and retrieve it efficiently.
### Publish Inference Events to Fiddler
After you onboard an ML model or LLM application as a Fiddler Model, you can publish inference events for analysis, performance monitoring, and reporting. There are two types of inference data:
* Pre-production data: Static datasets such as training or testing data that serve as Baseline references for comparison
* Production data: Time series data from live model inferences that Fiddler monitors against your baselines
#### Integration Methods
Fiddler offers two ways to publish inference data:
**Python Client Library**
Use the Python client for Python environments. Publish production and pre-production inference data with `Model.publish_stream()` for streaming events or `Model.publish_batch()` for batch uploads.
For more details, see the [Python client SDK](/sdk-api/python-client/connection) documentation.
**REST API**
Use the [REST API](/sdk-api/rest-api/alert-rules) for language-agnostic integration across any platform. Both production and pre-production inference data use a common interface.
### Publish Pre-Production Data
Publish pre-production data to Fiddler as a single dataset. You can add multiple baseline datasets to a model to create customized references for different metrics and alert rules.
Fiddler accepts pre-production data in these formats:
* Pandas DataFrame
* Parquet file
* CSV file
For detailed instructions, see the [Creating a Baseline](/developers/python-client-guides/publishing-production-data/creating-a-baseline-dataset) guide.
> **Note**:
>
> Pre-production datasets are immutable after publication. You can't update them or delete individual rows.
#### Upload a Static Pre-Production Baseline
```python theme={null}
dataset_file_path = 'path_to_your_data.parquet'
dataset_name = 'a_unique_identifying_name'
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
job = model.publish_batch(
source=dataset_file_path,
environment=fdl.EnvType.PRE_PRODUCTION,
dataset_name=dataset_name,
)
# publish_batch() is asynchronous. Use the publish job's wait() method
# if synchronous behavior is desired.
# job.wait()
```
### Publish Production Data
Fiddler provides several methods for publishing and editing production inference data:
* Batch publishing (`Model.publish_batch()`): Send data in batches using pandas DataFrames, Parquet files, or CSV files
* Stream publishing (`Model.publish_stream()`): Send individual events or small batches in near real-time
* Update publishing: Modify previously published data
* Delete publishing: Remove published data when needed
Fiddler accepts production data in these formats:
* Pandas DataFrames
* Parquet files
* CSV files
* List of Python dictionaries (limited to stream and updates)
A list of dictionaries is an additional data format on top of the three common to pre-production and production data.
Choose the method that best fits your use case by reviewing the publishing guides below:
* [Publish batch events](/developers/python-client-guides/publishing-production-data/publishing-batches-of-events)
* [Stream live events](/developers/python-client-guides/publishing-production-data/streaming-live-events)
* [Update published events](/developers/python-client-guides/publishing-production-data/updating-events)
* [Delete events](/developers/python-client-guides/publishing-production-data/deleting-events)
* [Publish ranking events](/developers/python-client-guides/publishing-production-data/ranking-events)
* [Event deduplication](/developers/python-client-guides/publishing-production-data/event-deduplication)
#### Key Considerations
Here are some considerations to keep in mind as you onboard models and begin publishing production data to Fiddler.
**Inference Event Unique Identifier**
Fiddler requires a unique identifier on each event published should you later need to update ground truth labels and/or metadata columns.
* Define the unique identifier column name when onboarding a model: `Model.event_id_col`
* A unique index on the event id column is not enforced
* When [event deduplication](/developers/python-client-guides/publishing-production-data/event-deduplication) is not enabled, duplicate values are accepted and events sharing the same event id value will all be used in calculating metrics
**Event Deduplication**
When event deduplication is enabled, Fiddler uses the `event_id_col` value to detect and drop duplicate events across multiple layers — within a single batch, against recently stored events, and across concurrent publish calls. This prevents duplicate events from inflating aggregated metrics such as drift scores, event counts, and model performance indicators.
* Event deduplication is not enabled by default — contact Fiddler support to enable it
* Events without an event ID (null or missing) are never deduplicated
* Deduplication applies to insert operations only — update operations are not deduplicated
For full details on how each layer works, see [Event Deduplication](/developers/python-client-guides/publishing-production-data/event-deduplication).
**Inference Event Timestamp**
Fiddler requires a timestamp for each inference event which is used as the event occurrence timestamp in time-series monitoring charts and alert rule evaluation.
* Define the timestamp column name when onboarding a model: `Model.event_ts_col`
* If not defined, Fiddler will use the time of publication as the event occurrence timestamp
* Timestamps are stored and rendered in UTC
* Timestamps with timezone are accepted but will be converted to UTC
* Fiddler supports basic pandas timestamp formats by inferring from the data
### Data Retention Policy
Fiddler retains production inference event data for 90 days. Contact your Fiddler customer success representative if you need a different retention period.
#### Raw Event Data
* Retained for 90 days from publication date
* Automatically deleted after 90 days
* Policy applies globally
#### Pre-Calculated Metrics
* Standard metrics derived from raw data are retained indefinitely
* Dashboards and charts continue to display historical trends after raw data expires
* **Coming soon:** the same long-term retention is being extended to individual saved GenAI charts (attribute breakdowns, custom metrics, and other chart types) in Mission Control (formerly Executive Dashboard) and elsewhere in the product. This is still in development, is not yet available to any customer, and has no committed release date.
#### Runtime features
* Custom metrics require raw event data and aren't available for data older than 90 days
# Monitoring Agentic Content Generation
Source: https://docs.fiddler.ai/developers/cookbooks/agentic-content-generation
Monitor agentic content generation for quality, safety, and brand compliance using built-in evaluators and custom LLM-as-a-Judge scoring.
Ensure quality, safety, and brand compliance in content generation agents using a combination of Fiddler's built-in evaluators for baseline quality and custom `CustomJudge` evaluators for domain-specific governance.
**Use this cookbook when:** You have content generation agents (writing reports, customer communications, marketing copy) and need automated quality gates to replace manual review of every draft.
**Time to complete**: \~20 minutes
```mermaid theme={null}
graph LR
A["Generated\nContent"] --> B["Built-In Evaluators"]
A --> C["Custom Judges"]
subgraph "Baseline Quality"
B --> D["Relevance"]
B --> E["Coherence"]
B --> F["Conciseness"]
end
subgraph "Domain Governance"
C --> G["Brand Voice"]
C --> H["Compliance"]
end
D --> I{"Quality\nGate"}
E --> I
F --> I
G --> I
H --> I
I -->|Pass| J["APPROVED"]
I -->|Fail| K["REJECTED"]
I -->|Marginal| L["REVIEW"]
style J fill:#6f9,stroke:#333
style K fill:#f96,stroke:#333
style L fill:#ffd,stroke:#333
```
**Prerequisites**
* Fiddler account with API access
* LLM credential configured in **Settings > LLM Gateway**
* `pip install fiddler-evals pandas`
***
## The Content Generation Challenge
Enterprise content generation agents produce volume that exceeds human review capacity. Without automated quality gates, teams face:
* **Reviewer fatigue** — manually reviewing hundreds of drafts per day
* **Inconsistent quality** — different reviewers apply different standards
* **Brand drift** — subtle changes in tone or style go undetected
The solution: combine Fiddler's built-in evaluators (quality, safety) with custom LLM-as-a-Judge evaluators (brand voice, compliance) for automated governance.
## Recommended Evaluators
### Built-In Evaluators (Baseline Quality)
| Evaluator | What It Measures | Value |
| -------------------- | ---------------------------------------------- | --------------------- |
| **Answer Relevance** | Does the output address the input instruction? | Instruction adherence |
| **Coherence** | Logical flow and clarity | Narrative quality |
| **Conciseness** | Brevity without losing meaning | Message clarity |
| **Sentiment** | Positive, negative, or neutral tone | Brand alignment |
| **Prompt Safety** | 11 safety dimensions (toxicity, bias, etc.) | Risk mitigation |
### Custom Evaluators (Domain-Specific Governance)
| Evaluator | What It Measures | Value |
| --------------------- | ----------------------------------------- | ------------------------------ |
| **Brand Voice Match** | Adherence to company style guide | Automated brand governance |
| **Bias Detection** | Potential bias across multiple dimensions | Compliance and risk mitigation |
***
Replace `URL`, `TOKEN`, and credential names with your Fiddler account details. Find your credentials in **Settings > Access Tokens** and **Settings > LLM Gateway**.
```python theme={null}
import pandas as pd
from fiddler_evals import init
from fiddler_evals.evaluators import (
AnswerRelevance,
Coherence,
Conciseness,
CustomJudge,
)
URL = 'https://your-org.fiddler.ai'
TOKEN = 'your-access-token'
LLM_CREDENTIAL_NAME = 'your-llm-credential'
LLM_MODEL_NAME = 'openai/gpt-4o'
init(url=URL, token=TOKEN)
# Built-in evaluators for baseline quality
relevance = AnswerRelevance(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME)
coherence = Coherence(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME)
conciseness = Conciseness(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME)
```
Use `CustomJudge` to evaluate content against your company's style guide:
```python theme={null}
brand_voice_judge = CustomJudge(
prompt_template="""
Determine whether the provided content adheres to the provided
brand guidelines.
Content: {{ content }}
Brand Guidelines: {{ brand_guidelines }}
""",
output_fields={
'voice_match_score': {
'type': 'string',
'choices': ['Perfect Match', 'Minor Deviations', 'Off-Brand'],
},
'reasoning': {'type': 'string'},
},
model=LLM_MODEL_NAME,
credential=LLM_CREDENTIAL_NAME,
)
```
See [Building Custom Judge Evaluators](/developers/cookbooks/custom-judge-evaluators) for a deep-dive into `prompt_template`, `output_fields`, and iterative prompt improvement.
```python theme={null}
# Example: your brand guidelines
brand_guidelines = (
"Use professional, approachable tone. "
"Address customers as 'you'. "
"Avoid jargon, slang, and exclamation marks. "
"Keep sentences under 25 words."
)
# Sample content from your agent
generated_content = [
{
'instruction': 'Write a welcome email for new customers',
'content': 'Welcome to our platform. We are glad you chose us. '
'Your account is ready and you can start exploring features '
'right away.',
},
{
'instruction': 'Write a welcome email for new customers',
'content': 'OMG WELCOME!!! You are going to LOVE this!! '
'Our platform is literally the BEST thing ever!!!',
},
{
'instruction': 'Explain the refund process',
'content': 'To request a refund, navigate to your order history, '
'select the item, and click Request Refund. Processing takes '
'3-5 business days.',
},
]
# Evaluate each piece of content
for item in generated_content:
# Built-in evaluators
rel_score = relevance.score(
user_query=item['instruction'],
rag_response=item['content'],
)
coh_score = coherence.score(prompt=item['instruction'], response=item['content'])
con_score = conciseness.score(response=item['content'])
# Custom brand voice judge
brand_scores = brand_voice_judge.score(inputs={
'content': item['content'],
'brand_guidelines': brand_guidelines,
})
brand_dict = {s.name: s for s in brand_scores}
print(f"\nInstruction: {item['instruction'][:50]}...")
print(f" Relevance: {rel_score.label} ({rel_score.value})")
print(f" Coherence: {coh_score.label}")
print(f" Conciseness: {con_score.label}")
print(f" Brand Voice: {brand_dict['voice_match_score'].label}")
print(f" Reason: {brand_dict['reasoning'].label}")
```
**Expected output:**
```
Instruction: Write a welcome email for new customers...
Relevance: high (1.0)
Coherence: high
Conciseness: high
Brand Voice: Perfect Match
Reason: Professional tone, addresses customer directly, no jargon or
exclamation marks, sentences are concise.
Instruction: Write a welcome email for new customers...
Relevance: medium (0.5)
Coherence: low
Conciseness: low
Brand Voice: Off-Brand
Reason: Uses all-caps, multiple exclamation marks, slang ("OMG",
"literally"), and informal tone — violates all brand guidelines.
Instruction: Explain the refund process...
Relevance: high (1.0)
Coherence: high
Conciseness: high
Brand Voice: Perfect Match
Reason: Clear, professional instructions with appropriate tone and
sentence length.
```
Combine evaluator scores into an automated quality gate that flags content for human review:
```python theme={null}
def quality_gate(instruction, content, brand_guidelines):
"""Automated quality gate for content generation agents.
Returns 'APPROVED', 'REVIEW', or 'REJECTED' with reasons.
"""
issues = []
# Check relevance
rel = relevance.score(user_query=instruction, rag_response=content)
if rel.value < 0.5:
issues.append(f'Low relevance ({rel.label})')
# Check coherence
coh = coherence.score(prompt=instruction, response=content)
if coh.value < 0.5:
issues.append(f'Low coherence ({coh.label})')
# Check brand voice
brand = brand_voice_judge.score(inputs={
'content': content,
'brand_guidelines': brand_guidelines,
})
brand_dict = {s.name: s for s in brand}
voice = brand_dict['voice_match_score'].label
if voice == 'Off-Brand':
issues.append(f'Off-brand content')
elif voice == 'Minor Deviations':
issues.append(f'Minor brand deviations')
if not issues:
return 'APPROVED', []
elif any('Off-Brand' in i or 'Low' in i for i in issues):
return 'REJECTED', issues
else:
return 'REVIEW', issues
# Run the quality gate
for item in generated_content:
status, issues = quality_gate(
item['instruction'], item['content'], brand_guidelines,
)
print(f"{status}: {item['content'][:60]}...")
if issues:
print(f" Issues: {', '.join(issues)}")
```
**Expected output:**
```
APPROVED: Welcome to our platform. We are glad you chose us. Your ...
REJECTED: OMG WELCOME!!! You are going to LOVE this!! Our platform...
Issues: Low coherence (low), Off-brand content
APPROVED: To request a refund, navigate to your order history, sel...
```
***
## Production Monitoring
To deploy these evaluators in production:
1. **Evaluator Rules**: Configure built-in evaluators (Answer Relevance, Coherence, Conciseness) as Evaluator Rules in your Agentic Monitoring application. See [Evaluator Rules](/evaluate-and-test/evaluator-rules).
2. **Custom Judges in Experiments**: Run the Brand Voice Match judge as a recurring experiment against sampled production outputs to track brand compliance over time.
3. **Alerting**: Set up alerts on evaluator score degradation to catch systemic quality drift after model updates or prompt changes.
***
## Next Steps
* [Building Custom Judge Evaluators](/developers/cookbooks/custom-judge-evaluators) — Deep-dive into `CustomJudge` capabilities
* [Evaluator Rules](/evaluate-and-test/evaluator-rules) — Deploy evaluators in production
* [Evals SDK Integration](/integrations/agentic-ai/evals-sdk) — Integration patterns for agentic workflows
***
**Related**: [Evaluator Rules](/evaluate-and-test/evaluator-rules) — Configure evaluators for production monitoring
# Agentic Document Extraction
Source: https://docs.fiddler.ai/developers/cookbooks/agentic-document-extraction
Build observable, measurable document extraction pipelines using Fiddler's agentic tracing, custom evaluators, and experiments.
Build reliable, measurable document extraction pipelines using Fiddler's agentic observability (tracing), custom evaluators, and experiments to catch hallucinated fields, schema drift, and silent accuracy degradation. While this cookbook uses invoice extraction as a running example, the patterns apply to any pipeline that extracts structured data from documents — medical records, legal filings, research papers, support tickets, or any other source.
**Use this cookbook when:** You have an LLM-based agent extracting structured data from any document type — invoices, medical records, legal contracts, research papers, support tickets, or other sources — and need observability, automated quality evaluation, and production monitoring.
**Time to complete**: \~30 minutes
```mermaid theme={null}
graph LR
A["Source\nDocument"] --> B["Parse"]
B --> C["Extract\n(LLM)"]
C --> D["Validate"]
subgraph "Fiddler Observability"
E["Tracing\n(OpenTelemetry)"]
F["Evaluators\n(Custom + Built-In)"]
G["Experiments\n(Benchmarking)"]
end
D --> E
D --> F
F --> G
G --> H{"Quality\nGate"}
H -->|Pass| I["Production"]
H -->|Fail| J["Review"]
style I fill:#6f9,stroke:#333
style J fill:#f96,stroke:#333
```
**Prerequisites**
* Fiddler account with API access
* LLM credential configured in **Settings > LLM Gateway**
* `pip install fiddler-evals pandas`
***
## Understanding the Problem
Document extraction — pulling structured data from unstructured or semi-structured documents — is one of the most common enterprise AI applications. Whether the source is an invoice, a medical record, a legal filing, or a support ticket, the core challenge is the same. When an LLM-based agent handles this work, it introduces new failure modes that traditional rule-based parsers never had: hallucinated field values, inconsistent schemas, and silent accuracy degradation after model updates.
A document extraction agent typically follows a multi-step workflow:
1. **Parse** — normalize raw document text into a consistent format
2. **Extract** — use an LLM to pull structured fields (vendor name, invoice number, line items, totals) into JSON
3. **Validate** — check that the output is complete, correctly formatted, and mathematically consistent
Each step can fail in different ways, and the failures compound. A parsing step that drops a line item produces an extraction that looks correct but is missing data. An LLM that hallucinates a subtotal produces a validation that passes the schema check but fails the math check. Without observability at every step, these issues are invisible until a downstream system — or a customer — catches them.
**Common failure modes in document extraction:**
* **Schema drift:** The model starts omitting fields it previously extracted reliably
* **Numeric hallucination:** Dollar amounts or quantities that don't match the source document
* **Date format inconsistency:** Dates returned in varying formats despite explicit instructions
* **Math errors:** Extracted totals that don't equal subtotal + tax
* **Silent degradation:** Accuracy drops gradually after a model update, with no hard errors to trigger alerts
***
## How Fiddler Helps: Three Layers of Observability
Fiddler provides three complementary capabilities for document extraction pipelines:
### 1. Agentic Observability (Tracing)
By instrumenting your extraction pipeline with OpenTelemetry, every step — parse, extract, validate — appears as a span in Fiddler's trace view. This gives you:
* **Span hierarchy:** A root `chain` span with child `tool` and `llm` spans, showing exactly how the agent orchestrates each step
* **LLM telemetry:** Model name, token usage (input/output/total), and the full prompt and response captured on the extraction span
* **Error attribution:** When an extraction fails, you can see which step failed and why, including error type and message
* **Custom attributes:** Tag spans with metadata like `document_type`, `invoice_id`, or any business-relevant dimension for filtering in dashboards
**Learn more:** [OpenTelemetry Quick Start](/developers/quick-starts/opentelemetry-quick-start)
**Other integration options:** While this cookbook uses OpenTelemetry for maximum flexibility, Fiddler also provides dedicated SDKs with auto-instrumentation for popular agentic frameworks — including [LangGraph](/integrations/agentic-ai/langgraph-sdk), [Strands](/integrations/agentic-ai/strands-sdk), [LangChain](/integrations/agentic-ai/langchain-sdk), and [LiteLLM](/integrations/agentic-ai/litellm-integration). These SDKs require minimal code changes (often a single `instrument()` call) and produce the same traces in Fiddler. For custom Python agents without a framework, the [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) (`fiddler-otel`) provides a `@trace` decorator for lightweight instrumentation. See the [full integration guide](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) to choose the right option for your stack.
### 2. Fiddler Experiments (Offline Evaluation)
Experiments let you systematically measure extraction quality against a ground-truth dataset. You define a dataset of test cases (source documents + expected extractions), run your pipeline against them, and score the results with custom evaluators. This gives you:
* **Repeatable benchmarks:** Compare extraction accuracy across model versions, prompt changes, or schema updates
* **Per-test-case drill-down:** See exactly which fields mismatched and why, for every test case
* **Side-by-side comparison:** Run the same dataset against different model versions and compare field accuracy in a single view
**Learn more:** [Experiments](/getting-started/experiments)
### 3. Production Monitoring Signals
In production, you compute aggregate signals over rolling time windows and set alerts on threshold breaches. These signals act as early warning systems for extraction quality degradation.
**Learn more:** [Agentic Monitoring](/getting-started/agentic-monitoring)
***
## Definition: Built-In Evaluators (Fiddler Evals SDK)
The Fiddler Evals SDK (`fiddler-evals`) provides pre-built evaluators that deliver immediate, generalized assessments of LLM performance. For document extraction, they establish a baseline for output quality without requiring ground-truth data. Built-in evaluators include `Coherence`, `Conciseness`, `AnswerRelevance`, `Sentiment`, and more — each available as a Python class you instantiate and pass to the `evaluate()` function.
**Learn more:** [Evals SDK Reference](/sdk-api/evals/evaluate)
## Definition: Custom Evaluators
Custom evaluators allow you to encode domain-specific quality standards directly into the evaluation process. For document extraction, this means comparing extracted fields against known correct values, checking schema completeness, and validating mathematical consistency. The Fiddler Evals SDK provides three approaches for building custom evaluators:
* **`CustomJudge`** — An LLM-as-a-Judge evaluator that uses a Jinja prompt template and structured output fields. Ideal for nuanced, qualitative assessments like per-field accuracy or math consistency checks.
* **`EvalFn`** — Wraps any Python function as an evaluator. Best for deterministic checks like schema completeness or exact-match comparisons.
* **Subclass `Evaluator`** — Extend the base `Evaluator` class for full control over scoring logic, input handling, and multi-score returns.
**Learn more:** [CustomJudge Evaluators](/sdk-api/evals/custom-judge)
***
## Recommended Evaluators for Document Extraction
| Evaluator | What does it measure? | What value does it provide? |
| ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| **Coherence** | Assesses the logical flow and clarity of the LLM's extraction output. | **Output Quality:** Catches garbled or malformed extraction responses before they reach downstream systems. |
| **Conciseness** | Evaluates whether the extraction output is focused and free of extraneous commentary. | **Schema Discipline:** Ensures the model returns structured data, not explanations or caveats mixed into the output. |
| **"Field Accuracy" Custom Evaluator** (See Deep Dive below) | Compares each extracted scalar field (vendor name, invoice number, date, subtotal, tax, total) against a ground-truth value. | **Granular Quality Control:** Pinpoints exactly which fields the model extracts reliably vs. which require prompt tuning or model changes. |
| **"Schema Completeness" Custom Evaluator** (See Deep Dive below) | Measures the fraction of required fields that are present and non-null in the extraction output. | **Completeness Assurance:** Catches schema drift — when a model starts silently omitting fields it previously extracted correctly. |
| **"Per-Field Accuracy" `CustomJudge`** (See Deep Dive below) | Uses a `CustomJudge` evaluator to assess extraction accuracy for each specific field against the source text. | **Automated Review:** Provides human-like accuracy assessment at scale without requiring manual comparison of every extraction. |
| **"Math Consistency" Custom Evaluator** | Checks whether extracted numeric fields are internally consistent (e.g., total == subtotal + tax). | **Numeric Integrity:** Catches hallucinated dollar amounts that pass schema validation but fail basic arithmetic. |
## Recommended Production Monitoring Signals
Beyond per-document evaluators, you should track aggregate signals over rolling time windows to detect systemic issues.
| Signal | What does it measure? | Alert threshold (suggested) |
| --------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------- |
| **Success Rate** | Fraction of extractions completing without exception. | Alert when \< 95%. Catches API errors, timeouts, and crashes. |
| **Validation Failure Rate** | Fraction of extractions with at least one validation error (missing fields, bad date format, math mismatch). | Alert when > 20%. Catches silent quality degradation. |
| **Field Completeness** | Average fraction of required fields present and non-null across all extractions. | Alert when \< 90%. Catches schema drift after model updates. |
| **Math Accuracy** | Fraction of extractions where `total == subtotal + tax` within a small tolerance. | Alert when \< 90%. Catches numeric hallucination trends. |
In production, these signals would be computed over rolling windows (e.g., hourly or daily) and configured as Fiddler alerts. A sudden drop in math accuracy after a model update, for example, would trigger an alert before the bad data propagates to downstream accounting systems.
***
## Deep Dive: Tracing an Extraction Pipeline with OpenTelemetry
Fiddler's agentic observability uses OpenTelemetry (OTEL) to capture the full execution of your extraction pipeline as a structured trace. Each trace consists of spans with a parent-child hierarchy that mirrors your agent's logic.
A typical extraction trace has the following structure:
```
extraction_pipeline (chain)
├── parse_document (tool)
├── extract_fields (llm)
└── validate_output (tool)
```
* **`extraction_pipeline`** — the root span, typed as `chain`. Carries agent-level metadata: `gen_ai.agent.name`, `gen_ai.agent.id`, and custom attributes like `document_type` and `invoice_id`.
* **`parse_document`** — a `tool` span. Records the raw input and cleaned output of the normalization step.
* **`extract_fields`** — an `llm` span. This is where the richest telemetry lives: `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.input.messages`, and `gen_ai.output.messages`.
* **`validate_output`** — a `tool` span. Records the validation result: which checks passed, which failed, and the specific error messages.
When an extraction produces incorrect data, the span hierarchy lets you pinpoint the root cause:
* **Parse failed?** The `parse_document` span shows the raw input was garbled or the normalization dropped content.
* **LLM hallucinated?** The `extract_fields` span shows the exact prompt and response, plus token usage that may indicate the model was truncating output.
* **Validation caught it?** The `validate_output` span shows which checks failed, so you know whether the issue is a missing field, a bad date, or a math error.
Without this trace structure, you only see "extraction failed" — with it, you see *why*.
When any step throws an exception, the root span captures `fiddler.error.message` and `fiddler.error.type`, making it filterable in Fiddler dashboards. You can quickly find all traces where the LLM returned unparseable JSON, or where the OpenAI API timed out, without searching through logs.
**Learn more:** [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration)
***
## Deep Dive: Custom Evaluators for Extraction Quality
While built-in evaluators like `Coherence` catch general output quality issues, document extraction requires domain-specific evaluators that understand your schema and can compare against ground truth. The Fiddler Evals SDK provides multiple ways to build these: subclassing `Evaluator` for complex scoring logic, wrapping functions with `EvalFn` for simple checks, and using `CustomJudge` for LLM-based assessment.
This evaluator compares each extracted scalar field against a known correct value. It handles numeric fields with a tolerance (to account for rounding differences) and string fields with case-insensitive comparison.
**What it checks:** `vendor_name`, `invoice_number`, `date`, `subtotal`, `tax`, `total`
**Scoring:** Returns the fraction of fields that match (0.0–1.0). A score of `0.83` means 5 out of 6 fields matched.
```python theme={null}
from fiddler_evals import Evaluator, Score
class FieldAccuracyEvaluator(Evaluator):
NUMERIC_FIELDS = ['subtotal', 'tax', 'total']
def score(self, extracted: dict, expected: dict) -> Score:
matches = 0
total_fields = len(expected)
for field in expected.keys():
ext_val = extracted.get(field)
exp_val = expected.get(field)
if ext_val is None or exp_val is None:
continue
if field in self.NUMERIC_FIELDS:
if abs(float(ext_val) - float(exp_val)) < 0.01:
matches += 1
else:
if str(ext_val).strip().lower() == str(exp_val).strip().lower():
matches += 1
accuracy = matches / total_fields
return Score(
name="field_accuracy",
evaluator_name=self.name,
value=accuracy,
reasoning=f"{matches}/{total_fields} fields matched",
)
```
**Why it matters:** Aggregate field accuracy tells you how reliable your pipeline is. But the per-field breakdown tells you *where* to focus improvement. If `date` is the field that most often mismatches, you know to adjust the date formatting instructions in your prompt — not rebuild the entire pipeline.
This evaluator checks what fraction of required fields are present and non-null in the extraction output, independent of whether the values are correct. Wrapping a simple Python function with `EvalFn` is the most concise way to build deterministic evaluators.
**What it checks:** All required schema fields (`vendor_name`, `invoice_number`, `date`, `line_items`, `subtotal`, `tax`, `total`)
**Scoring:** Returns the fraction present (0.0–1.0). A score of `0.86` means 6 out of 7 required fields were populated.
```python theme={null}
from fiddler_evals.evaluators import EvalFn
REQUIRED_FIELDS = ['vendor_name', 'invoice_number', 'date',
'line_items', 'subtotal', 'tax', 'total']
def schema_completeness(extracted: dict) -> float:
present = sum(1 for f in REQUIRED_FIELDS
if extracted.get(f) is not None)
return present / len(REQUIRED_FIELDS)
schema_completeness_evaluator = EvalFn(
schema_completeness, score_name="schema_completeness"
)
```
**Why it matters:** Schema completeness is the earliest signal of extraction degradation. A model may still extract *some* fields correctly while silently dropping others. Tracking completeness separately from accuracy lets you distinguish between "the model is wrong" and "the model isn't even trying."
For cases where you want a more nuanced assessment — or where ground-truth data is unavailable — you can use a `CustomJudge` to evaluate extraction quality directly from the source text. `CustomJudge` uses a Jinja prompt template with `{{ placeholder }}` syntax and structured `output_fields` to define what the LLM judge should return.
```python theme={null}
from fiddler_evals.evaluators import CustomJudge
field_accuracy_judge = CustomJudge(
prompt_template="""
Evaluate the accuracy of extracted fields from the source document.
Compare each extracted field against the original text and determine
whether it was extracted correctly.
Source Document:
{{ source_text }}
Extracted Data:
{{ extracted_fields }}
Required Fields:
{{ required_fields }}
""",
output_fields={
"vendor_name_correct": {
"type": "boolean",
"description": "Was the vendor name extracted correctly?",
},
"invoice_number_correct": {
"type": "boolean",
"description": "Was the invoice number extracted correctly?",
},
"date_correct": {
"type": "boolean",
"description": "Was the date extracted correctly?",
},
"amounts_correct": {
"type": "boolean",
"description": "Were the dollar amounts extracted correctly?",
},
"overall_accuracy": {
"type": "string",
"choices": ["All Correct", "Partially Correct", "Mostly Incorrect"],
},
"reasoning": {
"type": "string",
"description": "Explain which fields matched or mismatched and why.",
},
},
model="openai/gpt-4o",
credential="your-openai-credential",
)
# Score a single extraction
scores = field_accuracy_judge.score(inputs={
"source_text": raw_document_text,
"extracted_fields": json.dumps(extracted_data),
"required_fields": "vendor_name, invoice_number, date, subtotal, tax, total",
})
# Access individual scores
scores_dict = {s.name: s for s in scores}
print(scores_dict["overall_accuracy"].label) # e.g. "All Correct"
print(scores_dict["reasoning"].label) # detailed explanation
```
* The judge takes the original source document and the extracted fields as inputs — no ground truth required.
* It assesses each field independently, producing both per-field booleans and an overall accuracy classification.
* By flagging "Mostly Incorrect" extractions, you can route them for human review or reprocessing.
* This approach scales to production volumes where maintaining ground-truth datasets is impractical.
This `CustomJudge` evaluates whether the extracted numeric fields are internally consistent with the source document.
```python theme={null}
from fiddler_evals.evaluators import CustomJudge
math_consistency_judge = CustomJudge(
prompt_template="""
Verify the mathematical consistency of extracted invoice fields.
Check that:
(1) the total equals subtotal plus tax,
(2) line item prices multiplied by quantities sum to the subtotal,
(3) all numeric values match the source document.
Source Document:
{{ source_text }}
Extracted Data:
{{ extracted_fields }}
""",
output_fields={
"total_matches_sum": {
"type": "boolean",
"description": "Does total equal subtotal + tax?",
},
"line_items_sum_correct": {
"type": "boolean",
"description": "Do line item totals sum to the subtotal?",
},
"values_match_source": {
"type": "boolean",
"description": "Do all numeric values match the source document?",
},
"math_consistency": {
"type": "string",
"choices": ["Fully Consistent", "Minor Discrepancy", "Major Error"],
},
"reasoning": {
"type": "string",
"description": "Explain any discrepancies found.",
},
},
model="openai/gpt-4o",
credential="your-openai-credential",
)
```
* This judge catches a category of errors that schema validation alone cannot: values that are present and correctly formatted but numerically wrong.
* A "Major Error" flag on `math_consistency` indicates the model hallucinated dollar amounts — a critical failure for any financial document extraction system.
* Tracking the `line_items_sum_correct` field over time reveals whether the model struggles more with line-item aggregation than with reading individual totals.
***
## Deep Dive: Using Fiddler Experiments for Extraction Benchmarking
Fiddler Experiments provide a structured way to benchmark your extraction pipeline against a ground-truth dataset. This is especially valuable when you need to:
* **Compare model versions:** Does your current model extract dates more accurately than a smaller or newer variant?
* **Test prompt changes:** Does adding "Return 0 for tax if not applicable" reduce errors on tax-exempt invoices?
* **Validate schema changes:** After adding a new required field, does the existing accuracy hold?
The Fiddler Evals SDK (`fiddler-evals`) provides the `evaluate()` function as the main entry point for running experiments. The workflow is:
1. **Create a Dataset** with source documents as inputs and ground-truth extractions as expected outputs
2. **Define a Task** — a Python function that runs your extraction pipeline on each input
3. **Attach Evaluators** — built-in evaluators, `CustomJudge` instances, `EvalFn`-wrapped functions, or `Evaluator` subclasses
4. **Run with `evaluate()`** and review per-test-case scores in the Fiddler UI
Replace `url`, `token`, and credential names with your Fiddler account details. Find your credentials in **Settings > Access Tokens** and **Settings > LLM Gateway**.
```python theme={null}
import json
from fiddler_evals import init, Project, Application, Dataset, evaluate
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
from fiddler_evals.evaluators import Coherence, Conciseness
# 1. Initialize connection
init(url="https://your-org.fiddler.ai", token="your-token")
# 2. Set up project hierarchy: Project > Application > Dataset
project = Project.get_or_create(name="document_extraction")
app = Application.get_or_create(
name="invoice_extractor", project_id=project.id
)
dataset = Dataset.create(
name="invoice_test_set", application_id=app.id
)
# 3. Add test cases with ground-truth expected outputs
dataset.insert([
NewDatasetItem(
inputs={
"source_text": "Invoice #12345\nFrom: Acme Corp\nDate: 2025-01-15\n"
"Widget A 2 x $50.00 = $100.00\nSubtotal: $100.00\n"
"Tax: $8.00\nTotal: $108.00",
},
expected_outputs={
"vendor_name": "Acme Corp",
"invoice_number": "12345",
"date": "2025-01-15",
"subtotal": 100.00,
"tax": 8.00,
"total": 108.00,
},
metadata={"document_type": "invoice"},
),
# ... additional test cases
])
# 4. Define the extraction task
def extraction_task(inputs: dict, extras: dict, metadata: dict) -> dict:
# Replace with your actual extraction logic
result = run_extraction_pipeline(inputs["source_text"])
return {
"extracted_fields": result, # raw dict for custom evaluators
"source_text": inputs["source_text"],
"response": str(result), # string for built-in evaluators
}
# 5. Run the experiment with multiple evaluators
results = evaluate(
dataset=dataset,
task=extraction_task,
evaluators=[
FieldAccuracyEvaluator(), # Evaluator subclass
schema_completeness_evaluator, # EvalFn
field_accuracy_judge, # CustomJudge
math_consistency_judge, # CustomJudge
Coherence(model="openai/gpt-4o", credential="your-cred"),
Conciseness(model="openai/gpt-4o", credential="your-cred"),
],
name_prefix="invoice_extraction_v1",
# Maps dataset columns and task outputs to evaluator score() arguments
score_fn_kwargs_mapping={
"extracted": "extracted_fields", # for FieldAccuracyEvaluator (from task output)
"expected": lambda x: x["expected_outputs"], # for FieldAccuracyEvaluator (from dataset item)
"source_text": "source_text", # for CustomJudge evaluators
"extracted_fields": "extracted_fields", # for CustomJudge evaluators
"response": "response", # for Coherence & Conciseness
},
max_workers=4,
)
# 6. Analyze results
for result in results.results:
print(f"\nDataset item {result.experiment_item.dataset_item_id}:")
for score in result.scores:
print(f" {score.name}: {score.value} — {score.reasoning}")
```
The experiment results show you not just aggregate scores, but individual test cases where your pipeline struggled. If 7 out of 8 invoices score 100% field accuracy but one scores 67%, you can drill into that specific case to see what went wrong — perhaps it was an invoice with an unusual date format or a tax-exempt order.
The real power of Experiments is iterative improvement. Because `evaluate()` returns structured results and each experiment is tracked in the Fiddler UI, you can run A/B comparisons:
1. Run a baseline experiment with your current prompt and model
2. Identify the weakest test cases (lowest field accuracy or schema completeness)
3. Adjust your prompt, model, or preprocessing to address those failures
4. Run a new experiment against the same dataset and compare side-by-side with the baseline
5. Repeat until accuracy meets your threshold
```python theme={null}
# Compare two model versions against the same dataset
results_gpt4o = evaluate(
dataset=dataset, task=extraction_task_gpt4o,
evaluators=evaluators, name_prefix="gpt4o_baseline",
)
results_gpt4o_mini = evaluate(
dataset=dataset, task=extraction_task_gpt4o_mini,
evaluators=evaluators, name_prefix="gpt4o_mini_comparison",
)
# View side-by-side in the Fiddler UI
print(f"GPT-4o: {results_gpt4o.experiment.get_app_url()}")
print(f"GPT-4o-mini: {results_gpt4o_mini.experiment.get_app_url()}")
```
This workflow turns prompt engineering from guesswork into a measurable process.
**Learn more:** [Experiments](/getting-started/experiments)
***
## How These Evaluators Can Help
### 1. Catching Silent Quality Degradation
Document extraction pipelines rarely fail loudly. More often, they degrade gradually — a model update causes date formatting to shift, or a prompt change inadvertently reduces field completeness. By tracking Field Accuracy and Schema Completeness over time, you catch these regressions before they reach downstream systems. A 5% drop in math accuracy after a model update is invisible in error logs but immediately visible in Fiddler dashboards.
### 2. Reducing Manual Review Burden
Without automated evaluation, every extracted document needs human review to verify accuracy. With Fiddler evaluators acting as an automated quality gate, your team reviews only the extractions flagged for low accuracy or missing fields. If 95% of extractions pass validation, your reviewers focus on the 5% that need attention — not the full volume.
### 3. Enabling Confident Model and Prompt Changes
Changing a model or prompt in a document extraction pipeline is risky without a way to measure the impact. Fiddler Experiments give you a controlled environment to test changes against a known dataset before deploying to production. You can prove that a prompt change improved date accuracy from 87% to 98% before it touches a single real document.
### 4. Building Trust with Downstream Consumers
Finance teams, compliance officers, and ERP systems that consume extracted data need to trust its accuracy. Fiddler's monitoring signals — success rate, field completeness, math accuracy — provide auditable evidence that your extraction pipeline is performing within acceptable bounds. When a downstream consumer questions a data point, you can trace it back to the specific span that produced it.
### 5. Detecting Document-Type-Specific Weaknesses
By tagging traces with custom attributes like `document_type` (invoice, receipt, contract) or `vendor_name`, you can segment your monitoring signals and discover that your pipeline handles standard invoices at 98% accuracy but struggles with receipts (85%) or international invoices with VAT (78%). This guides where to invest in prompt engineering or training data.
***
## Next Steps
* [Building Custom Judge Evaluators](/developers/cookbooks/custom-judge-evaluators) — Deep-dive into `CustomJudge` capabilities
* [Running RAG Experiments at Scale](/developers/cookbooks/rag-experiments-at-scale) — Structured experiments with Datasets and golden label validation
* [Monitoring Agentic Content Generation](/developers/cookbooks/agentic-content-generation) — Quality and brand compliance for content agents
* [Evaluator Rules](/evaluate-and-test/evaluator-rules) — Deploy evaluators in production
* [Evals SDK Integration](/integrations/agentic-ai/evals-sdk) — Integration patterns for agentic workflows
***
**Related**: [OpenTelemetry Quick Start](/developers/quick-starts/opentelemetry-quick-start) | [Experiments Quick Start](/developers/quick-starts/experiments-quick-start) | [CustomJudge Evaluators](/developers/cookbooks/custom-judge-evaluators)
# AI Security
Source: https://docs.fiddler.ai/developers/cookbooks/ai-security
Monitor for AI security concerns using built-in evaluators and custom LLM-as-a-Judge scoring.
## Overview
As GenAI applications handle increasingly sensitive data and interact directly with users, security becomes a pressing concern.
A single security breach, whether it's a successful jailbreak attempt, leaked PII, or harmful content reaching users, can undermine trust and expose organizations to significant risk.
This cookbook provides a framework for using Fiddler's evaluators and guardrails to protect your AI applications from threats, including jailbreak attempts, harmful content generation, PII leakage, and policy violations.
***
## Understanding AI Security Risks
AI security encompasses multiple threat vectors:
**Prompt-based attacks:**
* **Jailbreaking:** Attempts to bypass safety restrictions and make the model behave in unintended ways
* **Prompt injection:** Malicious instructions embedded in user inputs to manipulate model behavior
* **Roleplaying exploitation:** Using fictional scenarios to elicit restricted information or harmful content
**Content safety risks:**
* **Harmful content generation:** Producing content that could cause psychological, physical, or social harm
* **Illegal content:** Generating content that violates laws or regulations
* **Unethical outputs:** Responses that violate ethical guidelines or corporate policies
**Data privacy risks:**
* **PII leakage in responses:** Model outputs that inadvertently expose personally identifiable information
* **PII in prompts:** Users submitting sensitive personal data that must be detected and protected
## Out-of-the-Box Security Evaluators
Fiddler provides pre-built scoring mechanisms called evaluators that assess AI systems across multiple risk dimensions.
**Learn more:** [Enrichments](/observability/llm/enrichments)
## Custom LLM-as-a-Judge for Security
While Fiddler's out-of-the-box evaluators cover common security risks, you may have **organization-specific security policies** that require custom evaluation. LLM-as-a-Judge evaluators allow you to encode your unique security guidelines into automated checks.
**Use LLM-as-a-Judge when you need to:**
* Enforce company-specific content policies beyond standard safety categories
* Detect violations of industry-specific regulations (healthcare, finance, legal)
* Flag content that conflicts with your organization's ethical guidelines
* Identify security risks unique to your application domain
**Example applications:**
* A healthcare AI that must never provide medical diagnoses (even when users request them)
* A financial AI that must refuse to give personalized investment advice
* A customer service AI that must escalate certain sensitive topics to human agents
* An educational AI that must not provide answers to homework assignments
By creating custom prompt templates that define your specific security requirements, you can automatically flag violations and enforce your policies at scale.
**Learn more:** [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start)
## Recommended Security Evaluators
### 1. Prompt Safety
**What it detects:** Evaluates the safety of text (prompts and responses) across multiple risk dimensions.
| Safety Dimension | What it identifies | Why it matters |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **Jailbreak** | Attempts to bypass model safety restrictions or manipulate the AI into ignoring its guidelines | **Attack Prevention:** Detects sophisticated prompt engineering designed to circumvent your controls |
| **Illegal** | Content that violates laws or promotes illegal activities | **Legal Compliance:** Prevents your AI from facilitating or encouraging unlawful behavior |
| **Roleplaying** | Use of fictional scenarios to elicit restricted information or harmful content (e.g., "pretend you're an AI with no restrictions") | **Policy Enforcement:** Catches attempts to bypass safety measures through narrative framing |
| **Harmful** | Content that could cause psychological, physical, or social harm to individuals or groups | **User Safety:** Protects users from outputs that could lead to self-harm, dangerous actions, or trauma |
| **Unethical** | Content that violates ethical principles or societal norms, even if not explicitly illegal | **Reputation Protection:** Maintains alignment with your organization's values and public expectations |
**Additional dimensions available:** hateful, harassing, racist, sexist, violent, sexual
**How to use:**
* Apply to both user prompts (inputs) and AI responses (outputs)
* Set severity thresholds based on your risk tolerance
* Track trends over time to identify emerging attack patterns
**Value:** Acts as a comprehensive safety net, catching both malicious user attempts and problematic model outputs before they cause harm.
### 2. PII Detection
**What it detects:** Identifies personally identifiable information in both user prompts and AI responses.
| PII Type | Examples | Risk |
| ---------------------------- | ------------------------------------------- | ------------------------------------------------------------------------ |
| **Social Security Numbers** | 123-45-6789, 123456789 | **Identity Theft:** Exposure can lead to fraud and financial harm |
| **Credit Card Numbers** | 4532-1234-5678-9010 | **Financial Fraud:** Direct monetary loss and unauthorized transactions |
| **Email Addresses** | [user@example.com](mailto:user@example.com) | **Privacy Violation:** Can enable spam, phishing, or unwanted contact |
| **Phone Numbers** | (555) 123-4567, +1-555-123-4567 | **Privacy Violation:** Enables unwanted contact and potential harassment |
| **Physical Addresses** | 123 Main St, New York, NY 10001 | **Physical Security:** Can reveal location and enable stalking or theft |
| **IP Addresses** | 192.168.1.1, 2001:0db8:85a3::8a2e:0370:7334 | **Privacy & Security:** Can reveal location and enable tracking |
| **Driver's License Numbers** | D1234567 | **Identity Theft:** Can be used for fraud and impersonation |
| **Passport Numbers** | 123456789 | **Identity Theft:** International fraud and identity compromise |
| **Medical Record Numbers** | MRN-789456 | **HIPAA Violation:** Protected health information exposure |
| **Bank Account Numbers** | 123456789012 | **Financial Fraud:** Unauthorized access to funds |
| **National ID Numbers** | Varies by country | **Identity Theft:** Government ID compromise |
| **Tax IDs / EINs** | 12-3456789 | **Business Fraud:** Corporate identity theft |
| **Dates of Birth** | 01/15/1990 | **Identity Verification:** Combined with other data, enables fraud |
| **Names** | John Smith, Jane Doe | **Privacy Context:** When combined with other PII, increases risk |
**Where to apply:**
**Input Detection (User Prompts):**
* **Purpose:** Identify when users are submitting sensitive personal information
* **Actions:**
* Warn users not to share PII
* Redact PII before processing
* Log incidents for security review
* **Example:** User asks "Can you analyze my credit report? My SSN is 123-45-6789..."
**Output Detection (AI Responses):**
* **Purpose:** Catch when the model inadvertently includes PII in responses
* **Actions:**
* Block response from reaching the user
* Regenerate without PII
* Flag for investigation (How did the model access this PII?)
* **Example:** Model trained on customer service logs accidentally includes someone's phone number in a response
**Value:** Prevents privacy violations, ensures compliance with GDPR/CCPA/HIPAA, and protects users from identity theft.
***
## Guardrails vs. Post-Production Observability
Fiddler supports two complementary approaches to AI security: **real-time guardrails** and **post-production observability**. Understanding when to use each is critical for building secure AI systems.
### Real-Time Guardrails
**What they are:** Security checks that evaluate and potentially block AI inputs or outputs **before they are processed or delivered to users**.
**How they work:**
1. User submits a prompt → Guardrail evaluates for safety/PII
2. If violation detected → Request is blocked or modified
3. If safe → Request proceeds to model
4. Model generates response → Guardrail evaluates output
5. If violation detected → Response is blocked or regenerated
6. If safe → Response delivered to user
**When to use real-time guardrails:**
| Scenario | Why Guardrails are Essential | Example |
| ---------------------------- | -------------------------------------------------- | ------------------------------------------------------------ |
| **User-facing applications** | Cannot allow harmful content to reach users | Customer support chatbot must block offensive responses |
| **High-stakes domains** | Single failure has severe consequences | Healthcare AI must never provide medical diagnoses |
| **Compliance requirements** | Regulations mandate prevention, not just detection | Financial AI must block PII from being logged or transmitted |
| **Jailbreak prevention** | Must stop malicious prompts before processing | Detect and reject prompt injection attacks |
| **PII protection** | Cannot allow sensitive data to be exposed | Prevent credit card numbers from appearing in responses |
**Tradeoffs:**
* **Latency:** Adds processing time to each request (typically 100-500ms)
* **False positives:** May occasionally block legitimate requests
* **Cost:** Requires additional compute for real-time evaluation
**Learn more:** [Guardrails Quick Start](/developers/quick-starts/guardrails-quick-start)
### Post-Production Observability
**What it is:** Continuous monitoring and analysis of AI behavior **after** requests have been processed, using historical data to identify patterns, trends, and emerging threats.
**How it works:**
1. AI processes requests normally (no blocking)
2. All prompts and responses are logged to Fiddler
3. Security evaluators run asynchronously on logged data
4. Dashboards show trends, patterns, and anomalies
5. Alerts trigger when thresholds are exceeded
6. Teams investigate and respond to issues
**When to use post-production observability:**
| Scenario | Why Observability is Valuable | Example |
| --------------------------------- | -------------------------------------------------- | ------------------------------------------------------------------- |
| **Threat intelligence** | Identify attack patterns and emerging risks | Notice spike in jailbreak attempts targeting a specific weakness |
| **Model behavior analysis** | Understand how the model responds to edge cases | Discover the model occasionally generates PII in rare scenarios |
| **Compliance auditing** | Maintain historical records for regulatory review | Demonstrate you monitor and address security issues over time |
| **Performance optimization** | Improve guardrails based on real-world data | Reduce false positive rate by analyzing blocked legitimate requests |
| **Trend monitoring** | Track security metrics over time | See if harmful content attempts increase after a news event |
| **A/B testing security measures** | Compare security performance across model versions | Evaluate if new prompt reduces jailbreak success rate |
**Tradeoffs:**
* **No prevention:** Issues are detected after they occur
* **Requires follow-up:** Teams must act on insights
* **Best for learning:** Ideal for understanding threats and improving defenses
**Learn more:** [Enrichments](/observability/llm/enrichments)
### Using Both Approaches Together
**The most secure AI systems combine real-time guardrails with post-production observability:**
**Real-time guardrails provide:**
* Immediate protection for users
* Prevention of high-severity incidents
* Compliance with "must prevent" requirements
**Post-production observability provides:**
* Insights to improve guardrails
* Detection of sophisticated attacks that evade guardrails
* Trend analysis for proactive security
**Example workflow:**
to block high-confidence threats (Jailbreak score > 0.9, PII detected)
to log all requests and responses and alert on issues
for medium-severity flags (Jailbreak score 0.5-0.9)
in flagged content
based on findings (tighten thresholds, add custom rules)
as new threats emerge
***
## How These Evaluators Can Help
### 1. Prevent Security Incidents Before They Occur
Real-time guardrails act as a security perimeter, blocking malicious inputs and harmful outputs before they reach users. This prevents:
* Reputational damage from AI generating offensive content
* Legal liability from privacy violations
* User harm from dangerous or misleading information
### 2. Detect and Respond to Emerging Threats
Post-production observability helps you identify:
* **New attack vectors:** Novel jailbreak techniques not caught by existing rules
* **Systematic weaknesses:** Topics or phrasings where the model consistently fails safety checks
* **Coordinated attacks:** Patterns suggesting organized attempts to compromise your AI
### 3. Maintain Compliance and Auditability
For regulated industries, security monitoring provides:
* **Audit trails** demonstrating proactive security measures
* **Compliance evidence** for GDPR, CCPA, HIPAA, and other regulations
* **Incident documentation** showing how you detected and responded to threats
* **Risk assessment data** to support security reviews and certifications
### 4. Build Trust with Users
Transparent security practices signal to users that you take their safety and privacy seriously:
* Publish security metrics and response times
* Communicate how you protect user data
* Demonstrate continuous improvement in safety measures
### 5. Optimize Security vs. User Experience
By analyzing false positives in observability dashboards, you can:
* Tune guardrails to reduce unnecessary blocking
* Identify legitimate use cases that trigger safety flags
* Balance security rigor with user experience
***
## Get Started
Ready to secure your AI applications? Here's how to begin:
* Enable **Prompt Safety** for comprehensive threat detection
* Enable **PII Detection** for privacy protection
* Review evaluation results in Fiddler dashboards
* Identify your highest-priority security requirements
* Configure guardrails with appropriate thresholds
* Test thoroughly before production deployment
* Set up dashboards for key security metrics
* Configure alerts for anomalies
* Schedule regular security reviews
* Analyze patterns in flagged content
* Refine guardrail thresholds based on false positive/negative rates
* Add custom LLM-as-a-Judge evaluators for organization-specific policies
**For step-by-step tutorials:**
* **Guardrails setup:** [Guardrails Quick Start](/developers/quick-starts/guardrails-quick-start)
* **Enrichments & observability:** [Enrichments](/observability/llm/enrichments)
* **Custom LLM-as-a-Judge:** [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start)
**Security is not a one-time configuration, it's an ongoing practice.** By combining real-time guardrails with continuous observability, you can protect users, maintain compliance, and build AI systems worthy of trust.
# Tracking Bias and Accuracy Across Cohorts
Source: https://docs.fiddler.ai/developers/cookbooks/bias-and-accuracy
Track GenAI accuracy and bias across protected cohorts using Fiddler out-of-the-box evaluators, custom LLM-as-a-Judge scoring, and segment-based fairness analysis.
## Overview
GenAI applications span a wide variety of tasks, from summarization and information extraction to Q\&A systems and autonomous agents. Each task type presents unique challenges for accuracy measurement and bias detection. This cookbook provides a framework for selecting the right Fiddler evaluators (both out-of-the-box and custom LLM-as-a-Judge) to ensure your models deliver reliable, fair, and high-quality outputs across different use cases.
## Understanding Bias in GenAI
**Bias** in GenAI primarily refers to unfair treatment of people by an LLM based on protected characteristics such as:
* Nationality
* Gender
* Age
* Socioeconomic status
* Sexual orientation
* Race/ethnicity
This is the most critical concern when monitoring for bias in production systems. A model exhibits bias when it performs differently for users or content associated with different demographic groups.
Beyond these social considerations, models can also exhibit **inconsistent behavior across different input types**—such as performing differently on technical documents vs. narrative content. While less critical than bias related to protected characteristics, tracking these patterns can still improve overall model quality.
### Measuring Bias with Fiddler
**A great way to detect bias is to compare accuracy metrics across protected cohorts.** For example:
* Does your Q\&A system achieve lower Answer Relevance scores for questions about certain cultural topics?
* Does your summarization model produce less faithful summaries for content written by authors from specific demographic backgrounds?
* Does your chatbot provide less helpful responses when discussing topics related to certain protected characteristics?
Fiddler provides two complementary approaches for bias detection:
1. **LLM-as-a-Judge for individual assessment:** Custom evaluators that assess each request for potential bias indicators
2. **Segment analysis for comparative measurement:** Compare accuracy metrics across protected cohorts to identify disparities
Both approaches are detailed in the Deep Dive sections below.
## Out-of-the-Box Evaluators
These pre-built metrics provide immediate, generalized assessments of LLM performance. They offer a quick, low-effort way to establish a baseline for quality and compliance.
**Learn more:** [Enrichments](/observability/llm/enrichments)
## Custom LLM-as-a-Judge Evaluators
These evaluators allow you to inject specific business rules and task-specific quality standards directly into the evaluation process. By using a structured prompt template to define the criteria, the LLM acts as an automated, scalable subject matter expert, turning nuanced judgments into objective, trackable data.
**Learn more:** [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start)
## Recommended Evaluators by GenAI Task
### Summarization
**Use Case Description:** Summarization models condense long-form content (documents, articles, meeting transcripts, research papers) into concise summaries while preserving key information and maintaining factual accuracy.
**Accuracy Evaluators:**
| Evaluator | What does it measure? | What value does it provide? |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| **Faithfulness** (optimized for summarization) | Assesses whether the summary accurately represents the source material without introducing hallucinations or distortions. | **Accuracy Baseline:** Ensures the summary doesn't fabricate or misrepresent information from the original text. |
| **Conciseness** | Evaluates whether the summary is appropriately brief while retaining essential information. | **Efficiency:** Confirms the model is truly summarizing, not just excerpting or paraphrasing at length. |
**Bias Detection Strategy:**
* Tag summaries with metadata about source content (author demographics, topic categories, document type)
* Compare Faithfulness and Conciseness scores across segments
* You can also monitor token count differences between summaries of similar-length documents from different segments
* Example: Do summaries of technical papers by women researchers average 20% fewer tokens than those by men, suggesting less thorough coverage?
### Information Extraction
**Use Case Description:** Information extraction models parse unstructured or semi-structured text (invoices, receipts, contracts, emails, forms) to identify and extract specific data fields like names, dates, amounts, addresses, or custom entities relevant to your business.
**Accuracy Evaluators:**
| Evaluator | What does it measure? | What value does it provide? |
| ------------------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Per-Field Accuracy LLM-as-a-Judge** | Evaluates extraction accuracy for each specific field (e.g., "name," "date," "amount"). | **Granular Quality Control:** Pinpoints which fields the model extracts reliably vs. which require improvement. |
| **Overall Accuracy LLM-as-a-Judge** | Assesses whether all required information was extracted correctly in aggregate. | **Completeness Check:** Ensures no critical data is missed during extraction. |
**Bias Detection Strategy:**
* Tag extractions with metadata about the source (document format, language, content domain, demographic context)
* Compare per-field accuracy rates across segments
* Example: Does the model correctly extract names from resumes with non-Western names at the same rate as Western names?
### Question & Answer (RAG Systems)
**Use Case Description:** RAG (Retrieval-Augmented Generation) systems answer user questions by first retrieving relevant information from a knowledge base, then generating responses grounded in that context. These systems power customer support chatbots, internal knowledge assistants, and documentation search tools.
**Note:** Evaluation strategies differ for **closed-ended questions** (verifiable facts) vs. **open-ended questions** (requiring interpretation).
**Accuracy Evaluators:**
| Evaluator | What does it measure? | What value does it provide? |
| --------------------- | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Answer Relevance** | Assesses whether the response accurately addresses the user's question. | **Intent Matching:** Especially critical for open-ended questions—confirms the response addresses the *spirit* of the question, not just surface-level keywords. |
| **RAG Faithfulness** | Evaluates whether the answer is grounded in the retrieved context without hallucination. | **Factual Accuracy:** For closed-ended questions, ensures the model said the "right thing" based on retrieved documents. For open-ended questions, confirms claims are supported by context. |
| **Context Relevance** | Measures whether the retrieved documents are actually relevant to the question. | **Retrieval Quality:** Identifies when the retrieval subsystem is returning poor-quality or off-topic context. |
| **Conciseness** | Evaluates whether answers are appropriately brief. | **User Experience:** Prevents over-explaining simple questions or burying key information in verbose responses. |
**Closed-Ended vs. Open-Ended Questions:**
* **Closed-Ended:** "What is the capital of France?" → Focus on **RAG Faithfulness** (Did it say "Paris"?)
* **Open-Ended:** "How can I improve team productivity?" → Focus on **Answer Relevance** (Does it address the intent?) and **RAG Faithfulness** (Are suggestions grounded in retrieved best practices?)
**Bias Detection Strategy:**
* Tag questions with metadata about topic category, user demographics (if available), or question complexity
* Compare Answer Relevance and RAG Faithfulness scores across segments
* Monitor **response token count** and **sentiment** across different question types
* Example: Do questions about minority health issues receive lower Answer Relevance scores or shorter responses than general health questions?
### Chatbots
**Use Case Description:** Conversational AI agents engage in multi-turn dialogues with users for customer support, HR assistance, IT helpdesk, sales qualification, or general-purpose assistance. Unlike simple Q\&A, chatbots maintain conversation context and handle follow-up questions.
**Accuracy Evaluators:**
| Evaluator | What does it measure? | What value does it provide? |
| ---------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| **Answer Relevance** | Assesses whether responses stay on-topic throughout the conversation. | **Conversational Coherence:** Prevents the chatbot from drifting off-topic or ignoring user intent. |
| **RAG Faithfulness** | Ensures responses are grounded in retrieved knowledge (if using RAG). | **Trust Building:** Users trust chatbots that cite or reference real information rather than making things up. |
| **Sentiment Analysis** | Tracks the emotional tone of chatbot responses. | **Tone Management:** Ensures the bot maintains an appropriate, helpful tone even when users are frustrated. |
**Note:** Apply the same **Context Relevance** and **Conciseness** evaluators recommended for Q\&A systems if your chatbot uses retrieval.
**Bias Detection Strategy:**
* Tag conversations with user demographics, conversation topic, or user satisfaction ratings
* Compare Answer Relevance, RAG Faithfulness, and Sentiment scores across segments
* Monitor **average response length** and **response time** across different user groups
* Example: Do users from certain demographic groups receive responses with consistently more negative sentiment or fewer tokens?
### Classification
**Use Case Description:** Classification models categorize text inputs into predefined classes. Common applications include sentiment analysis (positive/negative/neutral), topic categorization (finance/sports/politics), intent detection (purchase/support/information), content moderation, and spam filtering.
**Accuracy Measurement Strategy:**
For GenAI classification tasks, traditional ML performance metrics such as precision, recall, and F1-score are highly effective.
Since the task can be treated the same way as a predictive ML task, these metrics provide quantitative benchmarks for application performance.
**Bias Detection Strategy:**
* Tag classifications with metadata about input characteristics (writing style, topic, demographic context)
* Compare precision, recall, and F1 scores across segments in Fiddler
* Example: Does sentiment classification achieve 85% accuracy for product reviews written by younger users but only 70% for older users?
### Autonomous Agents
**Use Case Description:** Autonomous agents are AI systems that independently plan, execute multi-step workflows, call external tools or APIs, and make decisions to accomplish complex goals. Examples include AI assistants that can book travel, research assistants that gather and synthesize information, or automation agents that handle business processes.
**Accuracy Evaluators:**
| Evaluator | What does it measure? | What value does it provide? |
| ------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
| **Tool Call Accuracy LLM-as-a-Judge** | Evaluates whether the agent selected and invoked the correct tool with proper parameters. | **Execution Reliability:** Prevents the agent from calling the wrong APIs or passing invalid arguments. |
| **Context Relevance** | Assesses whether the agent retrieved or used appropriate information before taking action. | **Decision Quality:** Ensures the agent's actions are informed by relevant context. |
**Bias Detection Strategy:**
* Tag agent interactions with task type, user demographics, or workflow complexity
* Compare Context Relevance and Tool Call Accuracy across segments
* Monitor **number of tool calls** and **task completion time** across different user groups
* Example: Does the agent require more steps to complete identical tasks for certain user populations?
### Code Generation
**Use Case Description:** Code generation models translate natural language descriptions into working code. Applications range from developer productivity tools (generating boilerplate, writing tests, explaining code) to low-code/no-code platforms where non-developers can create automation scripts or data processing pipelines.
**Accuracy Evaluators:**
| Evaluator | What does it measure? | What value does it provide? |
| ------------------------- | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| **Custom LLM-as-a-Judge** | Evaluates code quality, correctness, security, and adherence to coding standards. | **Code Review Automation:** Acts as a first-pass reviewer to catch obvious errors, security vulnerabilities, or style violations. |
**Bias Detection Strategy:**
* Tag code generation requests with programming language, user experience level, or problem domain
* Compare correctness and security scores across segments
* Monitor **generated code token count** for similar requests across different segments
* Example: Does the model generate less secure code for web development tasks compared to data science tasks?
### Content Generation
**Use Case Description:** Content generation models create original written content tailored to specific audiences and purposes. Applications include marketing copy, blog posts, social media content, email campaigns, product descriptions, internal communications, and creative writing assistance.
**Accuracy Evaluators:**
| Evaluator | What does it measure? | What value does it provide? |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **Answer Relevance** | Assesses whether the content addresses the original prompt or brief. | **Instruction Adherence:** Ensures the model delivers what was requested, not something tangentially related. |
| **Custom LLM-as-a-Judge: Audience Alignment** | Evaluates whether the content is appropriate for the target audience (tone, complexity, terminology). | **Targeted Communication:** Confirms the content speaks to the intended reader (e.g., executives vs. technical users vs. general public). |
| **Coherence** | Assesses logical flow and narrative quality. | **Readability:** Ensures content is easy to follow and professionally written. |
| **Sentiment Analysis** | Tracks the emotional tone of generated content. | **Brand Safety:** Prevents overly negative or inappropriate messaging from reaching audiences. |
**Bias Detection Strategy:**
* Tag content with target audience demographics, topic category, or content type
* Compare Answer Relevance, Coherence, and audience alignment scores across segments
* Monitor **content token count** and **sentiment scores** for similar prompts targeting different audiences
* Example: Does content generated for female audiences consistently receive lower Coherence scores or different sentiment patterns than content for male audiences?
## Deep Dive: Detecting Bias with LLM-as-a-Judge
LLM-as-a-Judge evaluators can be designed to assess individual requests for potential bias indicators. This approach is most effective when you have **specific bias criteria** you want to detect in each prompt or response.
### When to Use LLM-as-a-Judge for Bias Detection
Use this approach when you need to:
* **Flag potentially biased content** in individual responses (e.g., stereotypical language, exclusionary phrasing)
* **Assess prompt safety** for bias-related concerns (e.g., requests that might elicit biased responses)
* **Evaluate fairness** of individual outputs (e.g., does a job description use inclusive language?)
* **Score bias indicators** that can later be aggregated across segments
### Limitations of LLM-as-a-Judge for Bias Detection
**Important:** LLM-as-a-Judge evaluators process one request at a time and **cannot compare performance across demographic groups**. They can identify potentially problematic content, but they cannot tell you if your model performs differently for different user populations.
For that, you need **segment analysis** (see next section).
## Deep Dive: Detecting Bias Through Segment Analysis
Segment analysis is the most powerful method for detecting **performance disparities** across protected cohorts. By comparing accuracy metrics between demographic groups, you can identify systemic bias in your GenAI systems.
### How Segment Analysis Works
Since LLM-as-a-Judge evaluators process requests individually and cannot compare across prompts, Fiddler enables bias detection through a four-step process:
#### Step 1: Apply Accuracy Evaluators to All Requests
Use the appropriate out-of-the-box and custom LLM-as-a-Judge evaluators for your use case (as outlined above) to score every request for accuracy and quality.
**Examples:**
* **Summarization:** Faithfulness, Conciseness
* **Q\&A:** Answer Relevance, RAG Faithfulness
* **Content Generation:** Answer Relevance, Coherence, Audience Alignment
* **Information Extraction:** Per-Field Accuracy, Overall Accuracy
#### Step 2: Tag Requests with Relevant Metadata
Enrich your data with tags that enable meaningful comparisons. The specific tags depend on your use case:
**User-level tags:**
* Demographics (age group, gender, location, language)
* User segment (enterprise vs. SMB, premium vs. free tier)
* Experience level (new user vs. power user)
**Content-level tags:**
* Topic category (e.g., "women's health" vs. "men's health" vs. "general health")
* Content source (author demographics, publication type)
* Language or dialect
* Domain (technical vs. general, formal vs. informal)
**Interaction-level tags:**
* Channel (web, mobile, API)
* Time of day
* Session length
#### Step 3: Compare Metrics Across Segments in Fiddler
Use Fiddler's dashboards to calculate and compare:
**Accuracy metrics:**
* Average Answer Relevance, Faithfulness, Coherence scores by segment
* Per-field extraction accuracy rates by segment
* Classification precision/recall/F1 by segment
**Behavioral metrics:**
* Token count distributions (are responses equally detailed?)
* Response time (do certain queries take longer?)
* Retrieval quality (Context Relevance by segment)
* Sentiment patterns (are responses equally positive/neutral?)
#### Step 4: Investigate Disparities
When you identify statistically significant differences:
**Quantify the gap:**
* What's the magnitude of the disparity? (e.g., 15% difference in Faithfulness scores)
* How many users are affected?
* Is the gap consistent over time or growing?
**Root cause analysis:**
* Is the bias in the training data?
* Is it in the prompt or system instructions?
* Is it in the retrieval system (for RAG applications)?
* Is it in the evaluation criteria themselves?
**Remediation:**
* Adjust prompts to be more explicit about fairness requirements
* Augment training data to balance representation
* Retrain or fine-tune models
* Filter or reweight retrieval results
* Update evaluation criteria to catch edge cases
### Example: Detecting Gender Bias in a Healthcare Q\&A System
**Scenario:** You want to ensure your healthcare Q\&A system provides equally helpful answers regardless of the topic's demographic context.
**Implementation:**
**Step 1: Apply Accuracy Evaluators**
* Enable **Answer Relevance** evaluator for all questions
* Enable **RAG Faithfulness** evaluator for all questions
**Step 2: Tag Questions by Topic** Create a tagging system for health topics:
* `topic: "women's health"` (menstruation, pregnancy, menopause, etc.)
* `topic: "men's health"` (prostate health, testosterone, etc.)
* `topic: "general health"` (nutrition, exercise, sleep, etc.)
* `topic: "pediatric care"` (child development, vaccinations, etc.)
**Step 3: Compare Metrics in Fiddler**
Create segments for each topic category and compare.
Example:
| Segment | Avg Answer Relevance | Avg RAG Faithfulness | Avg Response Length (tokens) | Sample Size |
| -------------- | -------------------- | -------------------- | ---------------------------- | ----------- |
| Women's health | 0.78 | 0.82 | 145 | 1,247 |
| Men's health | 0.91 | 0.89 | 203 | 1,156 |
| General health | 0.88 | 0.87 | 195 | 3,892 |
| Pediatric care | 0.85 | 0.86 | 188 | 891 |
**Step 4: Investigate & Remediate**
**Finding:** Women's health questions receive:
* 14% lower Answer Relevance scores (0.78 vs. 0.91)
* 8% lower RAG Faithfulness scores (0.82 vs. 0.89)
* 29% shorter responses (145 vs. 203 tokens)
**Root Cause Investigation:**
1. Examine retrieved documents for women's health queries → Discovery: Knowledge base has 40% fewer articles on women's health topics
2. Review sample low-scoring responses → Discovery: Model often retrieves general health content instead of women's-health-specific sources
3. Check Context Relevance scores → Discovery: Retrieved documents are less relevant for women's health (0.72 vs. 0.85 for men's health)
**Remediation Actions:**
1. **Immediate:** Adjust retrieval parameters to prioritize topic-specific matches
2. **Short-term:** Expand knowledge base with high-quality women's health content
3. **Medium-term:** Fine-tune retrieval model on balanced health topic dataset
4. **Ongoing:** Monitor disparity metrics weekly to ensure improvement
**Example Result After Remediation:**
*Numbers below are illustrative.*
| Segment | Avg Answer Relevance | Avg RAG Faithfulness | Avg Response Length (tokens) |
| -------------- | -------------------- | -------------------- | ---------------------------- |
| Women's health | 0.86 ↑ | 0.87 ↑ | 192 ↑ |
| Men's health | 0.91 | 0.89 | 203 |
Gap reduced from 14% to 5% for Answer Relevance.
### Advanced Segment Analysis Techniques
**1. Intersectional Analysis**
You can also examine intersections in protected groups rather than just single demographic attributes:
* How does the system perform for questions about "women's health" in Spanish vs. English?
* Do younger users get different quality responses than older users for the same topics?
**2. Temporal Monitoring**
Track metrics over time to catch drift:
* Did a model update introduce new disparities?
* Are gaps widening or narrowing?
* Do disparities appear at certain times of day or during high-traffic periods?
**3. Cohort Comparison**
Compare multiple segments simultaneously:
```text theme={null}
Segment Analysis: Answer Relevance by Topic
- Women's health: 0.78
- Men's health: 0.91
- LGBTQ+ health: 0.74 ← Additional disparity discovered
- General health: 0.88
```
**4. Statistical Significance Testing**
Ensure observed differences aren't due to random variation:
* Use adequate sample sizes for each segment
* Calculate confidence intervals
* Apply appropriate statistical tests (t-tests, ANOVA, etc.)
## How These Evaluators Can Help
### 1. Establishing Accuracy Baselines
By combining out-of-the-box metrics (like Faithfulness, Answer Relevance) with custom LLM-as-a-Judge evaluators, you create a comprehensive accuracy profile for your GenAI application. This lets you:
* Set quality thresholds (e.g., "95% of summaries must achieve Faithfulness > 0.85")
* Compare model versions objectively
* Track accuracy degradation over time
### 2. Detecting Bias at Scale
Manual review of thousands of responses for bias is impractical. By combining LLM-as-a-Judge flagging with segment analysis, you can:
* **Flag problematic individual outputs** for immediate remediation
* **Quantify fairness gaps** across demographic groups with statistical rigor
* **Identify systemic disparities** that would be invisible in aggregate metrics
* **Build audit trails** demonstrating proactive bias monitoring for compliance
### 3. Continuous Monitoring & Improvement
By tracking these metrics in Fiddler over time, you can:
* Identify systemic drift in accuracy or bias after model updates
* Catch when prompt changes inadvertently introduce new biases
* Build feedback loops that route low-quality outputs for retraining data
* Measure the effectiveness of bias mitigation efforts
### 4. Compliance & Governance
For regulated industries, bias detection isn't just about fairness—it's about compliance. Automated bias tracking provides an audit trail showing you actively monitor and mitigate unfair treatment, which can be critical for regulatory reviews and demonstrating responsible AI practices.
## Related Resources
* [Evaluator Rules](/evaluate-and-test/evaluator-rules)
* [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start)
# Overview
Source: https://docs.fiddler.ai/developers/cookbooks/cookbooks
Use-case oriented guides for solving real-world AI evaluation and monitoring problems with Fiddler.
Cookbooks are use-case oriented guides that demonstrate end-to-end workflows for solving real problems with Fiddler. Unlike quick starts (which introduce product features) or tutorials (which deep-dive into specific capabilities), cookbooks are organized by scenario — they show you how to combine multiple Fiddler features to achieve a practical goal.
**Prerequisites for all cookbooks**
* Fiddler account with API access
* Python 3.10+
* Fiddler Evals SDK: `pip install fiddler-evals`
* LLM credential configured in **Settings > LLM Gateway**
## RAG Evaluation & Monitoring
| Cookbook | Use Case | Key Features |
| ----------------------------------------------------------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------------- |
| [RAG Evaluation Fundamentals](/developers/cookbooks/rag-evaluation-fundamentals) | "I have a RAG app and want to evaluate its quality" | RAG Faithfulness, Answer Relevance, direct `.score()` API |
| [Running RAG Experiments at Scale](/developers/cookbooks/rag-experiments-at-scale) | "I want to compare RAG pipeline configurations" | Datasets, Experiments, `evaluate()`, golden label validation |
| [Detecting Hallucinations in RAG](/developers/cookbooks/hallucination-detection-pipeline) | "I want to monitor my RAG app for hallucinations" | RAG Health triad, Evaluator Rules, LLM Observability enrichments |
## Custom Evaluators
| Cookbook | Use Case | Key Features |
| --------------------------------------------------------------------------------- | -------------------------------------------- | ----------------------------------------------------------------------- |
| [Building Custom Judge Evaluators](/developers/cookbooks/custom-judge-evaluators) | "I need domain-specific evaluation criteria" | `CustomJudge`, prompt templates, `output_fields`, iterative improvement |
## Agentic AI
| Cookbook | Use Case | Key Features |
| ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------- |
| [Monitoring Agentic Content Generation](/developers/cookbooks/agentic-content-generation) | "I want to ensure quality and brand compliance in content generation agents" | Built-in evaluators + custom Brand Voice Match judge |
***
## Related Resources
* [Experiments Getting Started](/getting-started/experiments) — Product overview
* [Experiments Quick Start](/developers/quick-starts/experiments-quick-start) — SDK setup and first experiment
* [RAG Health Diagnostics](/concepts/rag-health-diagnostics) — Conceptual guide to the diagnostic triad
* [Evals SDK Advanced Guide](/developers/tutorials/experiments/evals-sdk-advanced) — Advanced patterns
# Building Custom Judge Evaluators
Source: https://docs.fiddler.ai/developers/cookbooks/custom-judge-evaluators
Build domain-specific LLM-as-a-Judge evaluators using CustomJudge with prompt templates, structured output fields, and iterative prompt improvement.
Create domain-specific evaluators using `CustomJudge` to encode business rules, quality criteria, or classification tasks that built-in evaluators don't cover.
**Use this cookbook when:** You need evaluation criteria specific to your domain, such as topic classification, brand voice matching, compliance checking, or custom quality rubrics.
**Time to complete**: \~20 minutes
```mermaid theme={null}
graph LR
A["Define Prompt\nTemplate"] --> B["Set Output\nFields"]
B --> C["Run\nEvaluation"]
C --> D["Check\nAccuracy"]
D -->|Misclassifications| E["Refine Prompt\n& Add Constraints"]
E --> C
D -->|Accurate| F["Deploy"]
style E fill:#ffd,stroke:#333
style F fill:#6f9,stroke:#333
```
**Prerequisites**
* Fiddler account with API access
* LLM credential configured in **Settings > LLM Gateway**
* `pip install fiddler-evals pandas`
***
Replace `URL`, `TOKEN`, and credential names with your Fiddler account details. Find your credentials in **Settings > Access Tokens** and **Settings > LLM Gateway**.
```python theme={null}
import pandas as pd
from fiddler_evals import init
from fiddler_evals.evaluators import CustomJudge
URL = 'https://your-org.fiddler.ai'
TOKEN = 'your-access-token'
LLM_CREDENTIAL_NAME = 'your-llm-credential'
LLM_MODEL_NAME = 'openai/gpt-4o'
init(url=URL, token=TOKEN)
```
This example classifies news summaries into topics — **Sci/Tech**, **Sports**, **Business**, or **World**:
```python theme={null}
df = pd.DataFrame(
[
{
'text': 'Google announces new AI chip designed to accelerate '
'machine learning workloads.',
'ground_truth': 'Sci/Tech',
},
{
'text': 'The Lakers defeated the Celtics 112-108 in overtime, '
'with LeBron James scoring 35 points.',
'ground_truth': 'Sports',
},
{
'text': 'Federal Reserve raises interest rates by 0.25% citing '
'persistent inflation concerns.',
'ground_truth': 'Business',
},
{
'text': 'United Nations Security Council votes to impose new '
'sanctions on North Korea.',
'ground_truth': 'World',
},
{
'text': 'Microsoft acquires gaming company Activision Blizzard '
'for $69 billion.',
'ground_truth': 'Sci/Tech',
},
]
)
```
Define your evaluation criteria using a `prompt_template` with `{{ placeholder }}` markers and `output_fields` that define the structured response:
```python theme={null}
simple_judge = CustomJudge(
prompt_template="""
Determine the topic of the given news summary.
Pick one of: Sports, World, Sci/Tech, Business.
News Summary:
{{ news_summary }}
""",
output_fields={
'topic': {'type': 'string'},
'reasoning': {'type': 'string'},
},
model=LLM_MODEL_NAME,
credential=LLM_CREDENTIAL_NAME,
)
```
**How It Works**
* **`prompt_template`**: Your evaluation prompt with `{{ placeholder }}` markers (Jinja syntax). Placeholders are filled from the `inputs` dict passed to `.score()`.
* **`output_fields`**: Schema defining the expected outputs. Each field specifies a `type` (`string`, `boolean`, `integer`, `number`) and optional `choices` or `description`.
```python theme={null}
results = []
for _, row in df.iterrows():
scores = simple_judge.score(inputs={'news_summary': row['text']})
scores_dict = {s.name: s for s in scores}
results.append(
{
'ground_truth': row['ground_truth'],
'predicted': scores_dict['topic'].label,
'reasoning': scores_dict['reasoning'].label,
}
)
results_df = pd.DataFrame(results)
accuracy = (results_df['ground_truth'] == results_df['predicted']).mean()
print(f'Accuracy: {accuracy:.0%}')
# Show misclassified
misclassified = results_df[results_df['ground_truth'] != results_df['predicted']]
if len(misclassified) > 0:
print(f'\nMisclassified ({len(misclassified)}):')
for _, row in misclassified.iterrows():
print(f' Expected: {row["ground_truth"]}, Predicted: {row["predicted"]}')
```
**Expected output:**
```
Accuracy: 80%
Misclassified (1):
Expected: Sci/Tech, Predicted: Business
```
The simple prompt often confuses tech company acquisitions (like the Microsoft-Activision deal) with Business news. The next step shows how to fix this with clearer topic guidelines.
Add clearer topic guidelines and constrain outputs with `choices`:
```python theme={null}
improved_judge = CustomJudge(
prompt_template="""
Determine the topic of the given news summary.
Use topic 'Sci/Tech' if the news summary is about a company or
business in the tech industry, or if the news summary is about
a scientific discovery or research, including health and medicine.
Use topic 'Sports' if the news summary is about a sports event
or athlete.
Use topic 'Business' if the news summary is about a company or
industry outside of science, technology, or sports.
Use topic 'World' if the news summary is about a global event
or issue.
News Summary:
{{ news_summary }}
""",
output_fields={
'topic': {
'type': 'string',
'choices': ['Sci/Tech', 'Sports', 'Business', 'World'],
},
'reasoning': {'type': 'string'},
},
model=LLM_MODEL_NAME,
credential=LLM_CREDENTIAL_NAME,
)
```
Key improvements:
* **Explicit guidelines** for each topic eliminate ambiguity
* **`choices`** constrains the LLM output to valid categories only
**Compare Results**
```python theme={null}
improved_results = []
for _, row in df.iterrows():
scores = improved_judge.score(inputs={'news_summary': row['text']})
scores_dict = {s.name: s for s in scores}
improved_results.append(
{
'ground_truth': row['ground_truth'],
'predicted': scores_dict['topic'].label,
}
)
improved_df = pd.DataFrame(improved_results)
original_accuracy = (results_df['ground_truth'] == results_df['predicted']).mean()
improved_accuracy = (improved_df['ground_truth'] == improved_df['predicted']).mean()
print(f'Simple prompt: {original_accuracy:.0%}')
print(f'Improved prompt: {improved_accuracy:.0%}')
```
**Expected output:**
```
Simple prompt: 80%
Improved prompt: 100%
```
***
## Output Field Types
`CustomJudge` supports four output field types:
| Type | Description | Example Use |
| --------- | ---------------------------------------------- | --------------------------------------- |
| `string` | Free-form text or categorical (with `choices`) | Topic classification, reasoning |
| `boolean` | True/False | Compliance checks, binary quality gates |
| `integer` | Whole numbers | 1-5 rating scales |
| `number` | Floating-point | 0.0-1.0 confidence scores |
### Using `choices` for Categorical Output
```python theme={null}
output_fields={
'sentiment': {
'type': 'string',
'choices': ['positive', 'negative', 'neutral'],
},
}
```
### Using `description` to Guide the LLM
```python theme={null}
output_fields={
'quality_score': {
'type': 'integer',
'description': 'Overall quality rating from 1 (poor) to 5 (excellent)',
},
}
```
***
## Real-World Examples
### Brand Voice Match
Evaluate whether generated content adheres to brand guidelines:
```python theme={null}
brand_judge = CustomJudge(
prompt_template="""
Determine whether the provided content adheres to the provided
brand guidelines.
Content: {{ content }}
Brand Guidelines: {{ brand_guidelines }}
""",
output_fields={
'voice_match_score': {
'type': 'string',
'choices': ['Perfect Match', 'Minor Deviations', 'Off-Brand'],
},
'reasoning': {'type': 'string'},
},
model=LLM_MODEL_NAME,
credential=LLM_CREDENTIAL_NAME,
)
scores = brand_judge.score(inputs={
'content': 'Hey! Check out our AMAZING new product!!!',
'brand_guidelines': 'Use professional tone. Avoid exclamation marks '
'and all-caps. Address customers formally.',
})
```
**Expected output:**
```
voice_match_score: Off-Brand
reasoning: The content uses informal language ("Hey!"), multiple exclamation
marks, and all-caps ("AMAZING"), all of which violate the brand guidelines.
```
### Compliance Checking
Verify responses meet regulatory requirements:
```python theme={null}
compliance_judge = CustomJudge(
prompt_template="""
Review the following financial advice response for regulatory
compliance.
Customer Question: {{ question }}
Advisor Response: {{ response }}
Check for: unauthorized guarantees, missing disclaimers,
inappropriate risk characterization.
""",
output_fields={
'compliant': {
'type': 'boolean',
'description': 'Does the response meet regulatory standards?',
},
'issues_found': {
'type': 'string',
'description': 'List any compliance issues identified',
},
},
model=LLM_MODEL_NAME,
credential=LLM_CREDENTIAL_NAME,
)
```
***
## Next Steps
* [Running RAG Experiments at Scale](/developers/cookbooks/rag-experiments-at-scale) — Use CustomJudge evaluators in structured experiments
* [Monitoring Agentic Content Generation](/developers/cookbooks/agentic-content-generation) — Combine built-in evaluators with custom Brand Voice judges
* [Evaluator Rules](/evaluate-and-test/evaluator-rules) — Deploy custom evaluators in production monitoring
***
**Source notebook**: [Fiddler Cookbook: Custom Judge Evaluators](https://github.com/fiddler-labs/fiddler-examples/blob/main/cookbooks/Fiddler_Cookbook_Custom_Judge_Evaluators.ipynb)
# Detecting Hallucinations in RAG
Source: https://docs.fiddler.ai/developers/cookbooks/hallucination-detection-pipeline
Build a complete hallucination detection pipeline combining Evals SDK evaluation with LLM Observability enrichments for continuous RAG monitoring.
Build a hallucination detection pipeline that combines pre-deployment evaluation with the Evals SDK and continuous production monitoring through LLM Observability enrichments and Evaluator Rules.
**Use this cookbook when:** You want to monitor your RAG application for hallucinations across both testing and production environments.
**Time to complete**: \~25 minutes
```mermaid theme={null}
graph LR
subgraph "Layer 1: Pre-Deployment"
A["Design Test Cases"] --> B["Run Diagnostic\nEvaluation"]
B --> C["Classify\nFailure Modes"]
end
subgraph "Layer 2: Production"
D["Evaluator Rules\n(Agentic)"]
E["LLM Obs\nEnrichments"]
end
C --> D
C --> E
D --> F["Alerts &\nInvestigation"]
E --> F
style C fill:#ffd,stroke:#333
style F fill:#f96,stroke:#333
```
**Prerequisites**
* Fiddler account with API access
* LLM credential configured in **Settings > LLM Gateway**
* `pip install fiddler-evals fiddler-client pandas`
***
## The Two-Layer Approach
Hallucination detection works best as a two-layer pipeline:
| Layer | Tool | Purpose |
| ------------------ | ----------------------------------- | --------------------------------------------------------- |
| **Pre-deployment** | Evals SDK | Test against known scenarios, validate with golden labels |
| **Production** | LLM Observability + Evaluator Rules | Continuous monitoring of live traffic |
***
## Layer 1: Pre-Deployment Evaluation
Use the RAG Health Metrics triad to distinguish hallucinations from other failure modes:
Replace `URL`, `TOKEN`, and credential names with your Fiddler account details. Find your credentials in **Settings > Access Tokens** and **Settings > LLM Gateway**.
```python theme={null}
import pandas as pd
from fiddler_evals import init, evaluate, Project, Application, Dataset
from fiddler_evals.evaluators import (
AnswerRelevance,
ContextRelevance,
RAGFaithfulness,
)
URL = 'https://your-org.fiddler.ai'
TOKEN = 'your-access-token'
LLM_CREDENTIAL_NAME = 'your-llm-credential'
LLM_MODEL_NAME = 'openai/gpt-4o'
init(url=URL, token=TOKEN)
project = Project.get_or_create(name='hallucination_detection')
app = Application.get_or_create(
name='rag-hallucination-test',
project_id=project.id,
)
dataset = Dataset.get_or_create(
name='hallucination-scenarios',
application_id=app.id,
)
```
Design test cases that specifically probe for hallucination patterns:
```python theme={null}
hallucination_scenarios = pd.DataFrame(
[
{
'scenario': 'Grounded response',
'user_query': 'What is the return policy?',
'retrieved_documents': [
'Returns accepted within 30 days with receipt.',
],
'rag_response': 'You can return items within 30 days '
'if you have a receipt.',
},
{
'scenario': 'Fabricated details',
'user_query': 'What is the return policy?',
'retrieved_documents': [
'Returns accepted within 30 days with receipt.',
],
'rag_response': 'You can return items within 60 days. '
'No receipt needed. We also offer free shipping on returns.',
},
{
'scenario': 'Insufficient context',
'user_query': 'What are the shipping costs?',
'retrieved_documents': [
'We ship to all 50 US states.',
],
'rag_response': 'Standard shipping is $5.99 and express '
'shipping is $12.99.',
},
]
)
dataset.insert_from_pandas(
df=hallucination_scenarios,
input_columns=['user_query', 'retrieved_documents', 'rag_response'],
metadata_columns=['scenario'],
)
```
```python theme={null}
def passthrough_task(inputs, extras, metadata):
return {
'rag_response': inputs['rag_response'],
'retrieved_documents': inputs['retrieved_documents'],
}
result = evaluate(
dataset=dataset,
task=passthrough_task,
evaluators=[
RAGFaithfulness(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME),
AnswerRelevance(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME),
ContextRelevance(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME),
],
score_fn_kwargs_mapping={
'user_query': lambda x: x['inputs']['user_query'],
'retrieved_documents': 'retrieved_documents',
'rag_response': 'rag_response',
},
)
```
Use the diagnostic workflow to classify failures:
```python theme={null}
for r in result.results:
scores = {s.evaluator_name: s for s in r.scores}
scenario = r.dataset_item.metadata.get('scenario', 'unknown')
faithfulness = scores.get('rag_faithfulness')
relevance = scores.get('answer_relevance')
context = scores.get('context_relevance')
# Classify the failure mode
if faithfulness and faithfulness.value == 0:
diagnosis = 'HALLUCINATION'
elif context and context.value < 0.5:
diagnosis = 'BAD RETRIEVAL'
elif relevance and relevance.value < 0.5:
diagnosis = 'OFF-TOPIC'
else:
diagnosis = 'HEALTHY'
print(f'{scenario}: {diagnosis}')
if faithfulness:
print(f' Faithfulness: {faithfulness.label} — {faithfulness.reasoning}')
```
**Expected output:**
```
Grounded response: HEALTHY
Faithfulness: yes — The response accurately reflects the return policy
stated in the retrieved document.
Fabricated details: HALLUCINATION
Faithfulness: no — The response claims a 60-day return window and no
receipt requirement, but the source document states 30 days with receipt.
Insufficient context: HALLUCINATION
Faithfulness: no — The response provides specific prices ($5.99, $12.99)
that are not supported by the retrieved document.
```
**Reading the diagnosis:** The triad distinguishes *why* a response failed:
* **HALLUCINATION** = Faithfulness fails (response fabricates information)
* **BAD RETRIEVAL** = Context Relevance fails (wrong documents retrieved)
* **OFF-TOPIC** = Answer Relevance fails (response doesn't address the question)
***
## Layer 2: Production Monitoring
For applications using Agentic Monitoring, configure Evaluator Rules to continuously evaluate production spans:
1. Navigate to your application's **Evaluator Rules** tab
2. Add a rule for **RAG Faithfulness**
3. Map evaluator inputs to your span attributes:
* `user_query` → your query span attribute
* `rag_response` → your response span attribute
* `retrieved_documents` → your context span attribute
4. Set alert thresholds (e.g., alert when faithfulness drops below 80%)
See [Evaluator Rules](/evaluate-and-test/evaluator-rules) for step-by-step instructions.
For applications using LLM Observability, configure enrichments during model onboarding to monitor for hallucinations:
```python theme={null}
import fiddler as fdl
fiddler_enrichments = [
# Faithfulness (Centor Model) for low-latency hallucination detection
fdl.Enrichment(
name='Faithfulness',
enrichment='ftl_response_faithfulness',
columns=['source_docs', 'response'],
config={
'context_field': 'source_docs',
'response_field': 'response',
'threshold': 0.5,
},
),
# Safety enrichments
fdl.Enrichment(
name='Safety',
enrichment='ftl_prompt_safety',
columns=['question', 'response'],
),
# Embeddings for drift detection
fdl.TextEmbedding(
name='Prompt TextEmbedding',
source_column='question',
column='Enrichment Prompt Embedding',
),
]
```
LLM Observability uses **Faithfulness (Centor Model)** (`ftl_response_faithfulness`), powered by a proprietary Fiddler Centor Model for low-latency scoring. This is a **different evaluator** from the RAG Faithfulness used in Layer 1 — it has different inputs (`context`, `response`) and outputs probability scores (`faithful_prob` 0.0–1.0) rather than binary labels. For detailed diagnostic reasoning, use **RAG Faithfulness** via Evaluator Rules or the Evals SDK.
***
## Combining Both Layers
The most effective hallucination detection pipeline uses both layers:
| Stage | What to Do | Tool |
| ----------------- | ------------------------------------------ | -------------------------------------- |
| **Development** | Test against known hallucination scenarios | Evals SDK + RAG Faithfulness |
| **Pre-release** | Run experiments comparing pipeline changes | Evals SDK + full diagnostic triad |
| **Production** | Continuous monitoring with alerting | Evaluator Rules or LLM Obs enrichments |
| **Investigation** | Deep-dive into flagged events | Evals SDK `.score()` on specific cases |
***
## Next Steps
* [RAG Health Diagnostics](/concepts/rag-health-diagnostics) — Conceptual guide to failure mode diagnosis
* [RAG Evaluation Fundamentals](/developers/cookbooks/rag-evaluation-fundamentals) — Direct evaluation with `.score()` API
* [Evaluator Rules](/evaluate-and-test/evaluator-rules) — Configure production monitoring rules
***
**Source notebooks**:
* [Fiddler Cookbook: RAG Evaluation Fundamentals](https://github.com/fiddler-labs/fiddler-examples/blob/main/cookbooks/Fiddler_Cookbook_RAG_Evaluation_Fundamentals.ipynb)
* [Fiddler Quickstart: LLM Chatbot](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Chatbot.ipynb)
# RAG Evaluation Fundamentals
Source: https://docs.fiddler.ai/developers/cookbooks/rag-evaluation-fundamentals
Evaluate RAG application quality using Fiddler's built-in evaluators with direct scoring for rapid iteration on retrieval and generation quality.
Evaluate your RAG application's retrieval and generation quality using Fiddler's built-in evaluators. This cookbook demonstrates the direct `.score()` API for rapid iteration on test cases before scaling to full experiments.
**Use this cookbook when:** You have a RAG application and want to quickly assess whether responses are faithful to retrieved documents and relevant to user queries.
**Time to complete**: \~15 minutes
```mermaid theme={null}
graph LR
A["Define Test Cases"] --> B["Score with Evaluators"]
B --> C{"Faithfulness?"}
B --> D{"Relevance?"}
C -->|Yes| E["Grounded"]
C -->|No| F["Hallucination"]
D -->|High| G["On-topic"]
D -->|Low| H["Off-topic"]
style F fill:#f96,stroke:#333
style H fill:#f96,stroke:#333
style E fill:#6f9,stroke:#333
style G fill:#6f9,stroke:#333
```
**Prerequisites**
* Fiddler account with API access
* LLM credential configured in **Settings > LLM Gateway**
* `pip install fiddler-evals pandas`
***
Replace `URL`, `TOKEN`, and credential names with your Fiddler account details. Find your credentials in **Settings > Access Tokens** and **Settings > LLM Gateway**.
```python theme={null}
import pandas as pd
from fiddler_evals import init
from fiddler_evals.evaluators import RAGFaithfulness, AnswerRelevance
URL = 'https://your-org.fiddler.ai'
TOKEN = 'your-access-token'
LLM_CREDENTIAL_NAME = 'your-llm-credential' # From Settings > LLM Gateway
LLM_MODEL_NAME = 'openai/gpt-4o' # Or your preferred model
init(url=URL, token=TOKEN)
# Initialize evaluators
faithfulness = RAGFaithfulness(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME)
relevance = AnswerRelevance(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME)
```
Define representative test cases that cover both successful and failing RAG scenarios:
```python theme={null}
test_cases = pd.DataFrame(
[
{
'scenario': 'Perfect Match',
'user_query': 'What is the capital of France?',
'retrieved_documents': ['Paris is the capital of France.'],
'rag_response': 'The capital of France is Paris.',
},
{
'scenario': 'Hallucination',
'user_query': 'What are the office hours?',
'retrieved_documents': ['We are closed on weekends.'],
'rag_response': 'We are open 9 AM to 5 PM every day.',
},
{
'scenario': 'Irrelevant Answer',
'user_query': 'How do I reset my password?',
'retrieved_documents': ['To reset, click "Forgot Password".'],
'rag_response': 'Our system is very secure and uses 256-bit encryption.',
},
]
)
```
Use the `.score()` method to evaluate each test case directly. Each evaluator returns a `Score` object with `value`, `label`, and `reasoning`:
```python theme={null}
def evaluate_row(row):
f_score = faithfulness.score(
user_query=row['user_query'],
rag_response=row['rag_response'],
retrieved_documents=row['retrieved_documents'],
)
r_score = relevance.score(
user_query=row['user_query'],
rag_response=row['rag_response'],
)
return pd.Series(
{
'Faithfulness': f_score.label,
'Relevance': r_score.label,
'Status': 'HEALTHY'
if f_score.label == 'yes' and r_score.value >= 0.5
else 'ISSUE DETECTED',
}
)
results = test_cases.join(test_cases.apply(evaluate_row, axis=1))
```
```python theme={null}
results[['scenario', 'Faithfulness', 'Relevance', 'Status']]
```
**Expected output:**
| scenario | Faithfulness | Relevance | Status |
| ----------------- | ------------ | --------- | -------------- |
| Perfect Match | yes | high | HEALTHY |
| Hallucination | no | high | ISSUE DETECTED |
| Irrelevant Answer | yes | low | ISSUE DETECTED |
The hallucination case scores high on relevance (it addresses the question) but fails faithfulness (the response fabricates hours not in the context). The irrelevant answer is faithful to the context but doesn't actually answer the user's question.
***
## Understanding the Evaluators
### [RAG Faithfulness](/sdk-api/evals/rag-faithfulness)
RAG Faithfulness checks whether the response is grounded in the retrieved documents.
* **Inputs**: `user_query`, `rag_response`, `retrieved_documents`
* **Scoring**: Binary — Yes (1.0) / No (0.0)
* **Use for**: Detecting hallucinations where the LLM generates plausible but unsupported claims
### [Answer Relevance](/sdk-api/evals/answer-relevance)
Answer Relevance measures how well the response addresses the user's query.
* **Inputs**: `user_query`, `rag_response` (+ optional `retrieved_documents`)
* **Scoring**: Ordinal — High (1.0), Medium (0.5), Low (0.0)
* **Use for**: Detecting off-topic responses where the LLM answers a different question
**RAG Faithfulness vs. Faithfulness (Centor Model):** This cookbook uses `RAGFaithfulness`, an LLM-as-a-Judge evaluator. Fiddler also offers `FTLResponseFaithfulness`, powered by a proprietary Fiddler Centor Model with different inputs (`context`, `response`) and probability-based scoring (`faithful_prob` 0.0–1.0). These are separate evaluators — see the [Evaluators Glossary](/glossary/enrichment) for details.
***
## Next Steps
* [Running RAG Experiments at Scale](/developers/cookbooks/rag-experiments-at-scale) — Use Datasets and Experiments to evaluate systematically across larger test sets
* [Detecting Hallucinations in RAG](/developers/cookbooks/hallucination-detection-pipeline) — Set up continuous hallucination monitoring in production
* [RAG Health Diagnostics](/concepts/rag-health-diagnostics) — Conceptual guide to the diagnostic triad
***
**Source notebook**: [Fiddler Cookbook: RAG Evaluation Fundamentals](https://github.com/fiddler-labs/fiddler-examples/blob/main/cookbooks/Fiddler_Cookbook_RAG_Evaluation_Fundamentals.ipynb)
# Running RAG Experiments at Scale
Source: https://docs.fiddler.ai/developers/cookbooks/rag-experiments-at-scale
Run structured RAG experiments with Datasets, golden label validation, and side-by-side comparison of pipeline configurations.
Move beyond ad-hoc evaluation to structured experiments that track results, validate against golden labels, and enable side-by-side comparison of RAG pipeline configurations.
**Use this cookbook when:** You want to compare different retrieval strategies, LLM models, or prompt configurations across a standardized test set.
**Time to complete**: \~25 minutes
```mermaid theme={null}
graph TD
A["Project"] --> B["Application"]
B --> C["Dataset"]
C --> D["Experiment v1"]
C --> E["Experiment v2"]
D --> F["Compare Results"]
E --> F
subgraph Evaluators
G["Context Relevance"]
H["RAG Faithfulness"]
I["Answer Relevance"]
end
D -.-> Evaluators
E -.-> Evaluators
```
**Prerequisites**
* Fiddler account with API access
* LLM credential configured in **Settings > LLM Gateway**
* `pip install fiddler-evals pandas`
* Familiarity with [RAG Evaluation Fundamentals](/developers/cookbooks/rag-evaluation-fundamentals) recommended
***
Experiments are organized as: **Project > Application > Dataset > Experiment**
Replace `URL`, `TOKEN`, and credential names with your Fiddler account details. Find your credentials in **Settings > Access Tokens** and **Settings > LLM Gateway**.
```python theme={null}
import pandas as pd
from fiddler_evals import Application, Dataset, Project, evaluate, init
from fiddler_evals.evaluators import AnswerRelevance, ContextRelevance, RAGFaithfulness
URL = 'https://your-org.fiddler.ai'
TOKEN = 'your-access-token'
LLM_CREDENTIAL_NAME = 'your-llm-credential'
LLM_MODEL_NAME = 'openai/gpt-4o'
init(url=URL, token=TOKEN)
project = Project.get_or_create(name='rag_experiments')
application = Application.get_or_create(
name='rag-pipeline-comparison',
project_id=project.id,
)
dataset = Dataset.get_or_create(
name='rag-test-cases',
application_id=application.id,
)
```
Include `expected_quality` labels so you can validate whether evaluators correctly identify good and bad responses:
```python theme={null}
rag_data = pd.DataFrame(
[
{
'scenario': 'Perfect Match',
'expected_quality': 'good',
'user_query': 'What is the capital of France?',
'retrieved_documents': [
'Paris is the capital and largest city of France.',
'France is located in Western Europe.',
],
'rag_response': 'The capital of France is Paris.',
},
{
'scenario': 'Irrelevant Context',
'expected_quality': 'bad',
'user_query': 'How do I reset my password?',
'retrieved_documents': [
'To make pasta, boil water and add salt.',
'Italian cuisine features many pasta dishes.',
],
'rag_response': 'To reset your password, go to the login page '
'and click Forgot Password.',
},
{
'scenario': 'Hallucination',
'expected_quality': 'bad',
'user_query': 'What are the business hours?',
'retrieved_documents': [
'Our office is located at 123 Main Street.',
'We are closed on federal holidays.',
],
'rag_response': 'Our business hours are Monday through Friday, '
'9 AM to 5 PM.',
},
{
'scenario': 'Irrelevant Answer',
'expected_quality': 'bad',
'user_query': 'What is your return policy?',
'retrieved_documents': [
'Returns are accepted within 30 days of purchase.',
'Items must be unused and in original packaging.',
],
'rag_response': 'We offer free shipping on orders over $50. '
'Delivery takes 3-5 business days.',
},
]
)
```
```python theme={null}
if not list(dataset.get_items()):
dataset.insert_from_pandas(
df=rag_data,
input_columns=['user_query', 'retrieved_documents', 'rag_response'],
expected_output_columns=['expected_quality'],
metadata_columns=['scenario'],
)
print(f'Inserted {len(rag_data)} test cases')
else:
print('Dataset already has items, skipping insert')
```
**Expected output:**
```
Inserted 4 test cases
```
The idempotency check (`if not list(dataset.get_items())`) prevents duplicate inserts if you re-run the notebook. Remove this check if you want to refresh the dataset.
Define a task function that returns the RAG response, then run the experiment with all three RAG Health evaluators:
```python theme={null}
def rag_task(inputs: dict, extras: dict, metadata: dict) -> dict:
"""Return pre-recorded RAG response.
Replace with your actual RAG pipeline in production.
"""
return {'rag_response': inputs['rag_response']}
evaluators = [
ContextRelevance(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME),
RAGFaithfulness(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME),
AnswerRelevance(model=LLM_MODEL_NAME, credential=LLM_CREDENTIAL_NAME),
]
result = evaluate(
dataset=dataset,
task=rag_task,
evaluators=evaluators,
score_fn_kwargs_mapping={
'user_query': lambda x: x['inputs']['user_query'],
'retrieved_documents': lambda x: x['inputs']['retrieved_documents'],
'rag_response': 'rag_response',
},
)
print(f'Experiment: {result.experiment.name}')
print(f'Evaluated {len(result.results)} test cases')
```
**Understanding `score_fn_kwargs_mapping`:** This dict maps evaluator parameter names to data sources. Use a **lambda** to extract values from dataset inputs (`x['inputs']['...']`), or a **string** to reference a key from the task function's return dict.
**Expected output:**
```
Experiment: rag-test-cases-2026-02-07-001
Evaluated 4 test cases
```
Check whether the evaluators correctly identified quality issues by comparing their scores against your expected labels:
```python theme={null}
from fiddler_evals.pydantic_models.experiment import ExperimentItemResult
from fiddler_evals.pydantic_models.score import Score
validation_results = []
correct = 0
for r in result.results:
expected = r.dataset_item.expected_outputs.get('expected_quality')
has_problem = any(s.value < 0.5 for s in r.scores)
predicted = 'bad' if has_problem else 'good'
if expected == predicted:
correct += 1
validation_results.append(
ExperimentItemResult(
experiment_item=r.experiment_item,
dataset_item=r.dataset_item,
scores=[
Score(
name='predicted_quality',
evaluator_name='OverallQuality',
value=1.0 if predicted == 'good' else 0.0,
label=predicted,
reasoning=f'Expected: {expected}',
)
],
)
)
result.experiment.add_results(validation_results)
print(
f'Evaluator Accuracy: {correct}/{len(result.results)} '
f'({100 * correct / len(result.results):.0f}%)'
)
```
**Expected output:**
```
Evaluator Accuracy: 4/4 (100%)
```
```python theme={null}
print(f'View in Fiddler: {URL}/evals/experiments/{result.experiment.id}')
# Build results DataFrame
rows = []
for r, v in zip(result.results, validation_results):
row = {
'scenario': r.dataset_item.metadata.get('scenario'),
'expected': r.dataset_item.expected_outputs.get('expected_quality'),
'predicted': v.scores[0].label,
}
row.update({s.evaluator_name: s.value for s in r.scores})
rows.append(row)
pd.DataFrame(rows)
```
**Expected output:**
| scenario | expected | predicted | ContextRelevance | RAGFaithfulness | AnswerRelevance |
| ------------------ | -------- | --------- | ---------------- | --------------- | --------------- |
| Perfect Match | good | good | 1.0 | 1.0 | 1.0 |
| Irrelevant Context | bad | bad | 0.0 | 0.0 | 0.5 |
| Hallucination | bad | bad | 0.5 | 0.0 | 1.0 |
| Irrelevant Answer | bad | bad | 1.0 | 0.0 | 0.0 |
***
## Comparing Pipeline Configurations
To compare different RAG configurations, run multiple experiments against the same dataset:
Replace `rag_pipeline_v1` and `rag_pipeline_v2` with your actual RAG pipeline functions. Each function must accept `(inputs, extras, metadata)` and return a dict containing `rag_response`.
```python theme={null}
# Experiment 1: Default retrieval
result_v1 = evaluate(
dataset=dataset,
task=rag_pipeline_v1,
evaluators=evaluators,
score_fn_kwargs_mapping={
'user_query': lambda x: x['inputs']['user_query'],
'retrieved_documents': lambda x: x['inputs']['retrieved_documents'],
'rag_response': 'rag_response',
},
)
# Experiment 2: Improved retrieval with re-ranking
result_v2 = evaluate(
dataset=dataset,
task=rag_pipeline_v2,
evaluators=evaluators,
score_fn_kwargs_mapping={
'user_query': lambda x: x['inputs']['user_query'],
'retrieved_documents': lambda x: x['inputs']['retrieved_documents'],
'rag_response': 'rag_response',
},
)
# Compare results side-by-side in the Fiddler UI
print(f'V1: {URL}/evals/experiments/{result_v1.experiment.id}')
print(f'V2: {URL}/evals/experiments/{result_v2.experiment.id}')
```
Both experiments appear in the Fiddler UI under the same Application, enabling side-by-side comparison of scores across all test cases.
***
## Next Steps
* [RAG Evaluation Fundamentals](/developers/cookbooks/rag-evaluation-fundamentals) — Direct `.score()` API for quick iteration
* [Building Custom Judge Evaluators](/developers/cookbooks/custom-judge-evaluators) — Add domain-specific evaluation criteria
* [RAG Health Metrics Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial) — Step-by-step tutorial
***
**Source notebook**: [Fiddler Cookbook: RAG Experiments at Scale](https://github.com/fiddler-labs/fiddler-examples/blob/main/cookbooks/Fiddler_Cookbook_RAG_Experiments_at_Scale.ipynb)
# Overview
Source: https://docs.fiddler.ai/developers/index
Practical guides, tutorials, and reference documentation for building with Fiddler.
## ⚡ Quick Starts
Get up and running in minutes with step-by-step quick start guides:
* [**Get Started in \<10 Minutes**](/developers/quick-starts/get-started-in-less-than-10-minutes) - Fastest way to integrate Fiddler
* **Agentic Monitoring Quick Starts** - Monitor AI agents and multi-step workflows
* [OpenTelemetry Quick Start](/developers/quick-starts/opentelemetry-quick-start) (`fiddler-otel`) or
* [LangChain SDK Quick Start](/developers/quick-starts/langchain-sdk-quick-start) (`fiddler-langchain`) or
* [LangGraph SDK Quick Start](/developers/quick-starts/langgraph-sdk-quick-start) (`fiddler-langgraph`) or
* [Strands Agent SDK Quick Start](/developers/quick-starts/strands-agent-quick-start) (`fiddler-strands`)
* [**Experiments Quick Start**](/developers/quick-starts/experiments-quick-start) - Evaluate LLM outputs with custom metrics
* [**Guardrails Quick Start**](/developers/quick-starts/guardrails-quick-start) - Add safety guardrails to your AI applications
## 📚 Tutorials
In-depth, hands-on tutorials organized by product area:
### Experiments
Learn how to evaluate and test your LLM applications:
* [RAG Health Metrics Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial)
* [Evals SDK Advanced Guide](/developers/tutorials/experiments/evals-sdk-advanced)
* [Advanced Prompt Specs](/developers/tutorials/llm-monitoring/prompt-specs-advanced)
### Agentic & LLM Monitoring
Monitor production LLM applications and AI agents:
* [LangGraph SDK Quick Start](/developers/quick-starts/langgraph-sdk-quick-start)
* [LangGraph SDK Advanced](/developers/tutorials/llm-monitoring/langgraph-sdk-advanced)
* [Simple LLM Monitoring](/developers/quick-starts/simple-llm-monitoring)
### Guardrails
Implement safety controls for your AI applications:
* [Faithfulness Guardrails](/developers/tutorials/guardrails/guardrails-faithfulness)
* [Safety Guardrails](/developers/tutorials/guardrails/guardrails-safety)
* [PII Detection Guardrails](/developers/tutorials/guardrails/guardrails-pii)
### ML Monitoring
Monitor traditional ML models in production:
* [ML Monitoring Quick Start](/developers/quick-starts/simple-ml-monitoring)
* [NLP Model Monitoring](/developers/tutorials/ml-monitoring/simple-nlp-monitoring-quick-start)
* [Class Imbalance Handling](/developers/tutorials/ml-monitoring/class-imbalance-monitoring-example)
* [Model Versions](/developers/tutorials/ml-monitoring/ml-monitoring-model-versions)
* [Ranking Models](/developers/tutorials/ml-monitoring/ranking-model)
* [Regression Models](/developers/tutorials/ml-monitoring/ml-monitoring-regression)
* [Feature Impact Analysis](/developers/tutorials/ml-monitoring/user-defined-feature-impact)
* [Computer Vision Monitoring](/developers/tutorials/ml-monitoring/cv-monitoring)
## 🍳 Cookbooks
Use-case oriented guides that demonstrate end-to-end workflows for solving real problems:
* [**RAG Evaluation Fundamentals**](/developers/cookbooks/rag-evaluation-fundamentals) — Evaluate RAG quality with built-in evaluators
* [**Running RAG Experiments at Scale**](/developers/cookbooks/rag-experiments-at-scale) — Compare pipeline configurations systematically
* [**Building Custom Judge Evaluators**](/developers/cookbooks/custom-judge-evaluators) — Create domain-specific evaluation criteria
* [**Detecting Hallucinations in RAG**](/developers/cookbooks/hallucination-detection-pipeline) — Monitor for hallucinations in production
* [**Monitoring Agentic Content Generation**](/developers/cookbooks/agentic-content-generation) — Quality and brand compliance for content agents
## 📖 Client Library Reference
Comprehensive reference documentation for Fiddler's Python client:
### Getting Started
* [Installation and Setup](/developers/python-client-guides/installation-and-setup)
* [Naming Convention Guidelines](/developers/python-client-guides/naming-convention-guidelines)
* [Alerts with Fiddler Client](/developers/python-client-guides/alerts-with-fiddler-client)
### Model Onboarding
* [Create a Project and Model](/developers/python-client-guides/model-onboarding/create-a-project-and-model)
* [Customizing Your Model Schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema)
* [Task Types](/developers/python-client-guides/model-onboarding/task-types)
* [Custom Missing Values](/developers/python-client-guides/model-onboarding/specifying-custom-missing-value-representations)
### Publishing Production Data
* [Creating a Baseline Dataset](/developers/python-client-guides/publishing-production-data/creating-a-baseline-dataset)
* [Publishing Batches of Events](/developers/python-client-guides/publishing-production-data/publishing-batches-of-events)
* [Streaming Live Events](/developers/python-client-guides/publishing-production-data/streaming-live-events)
* [Updating Events](/developers/python-client-guides/publishing-production-data/updating-events)
* [Deleting Events](/developers/python-client-guides/publishing-production-data/deleting-events)
* [Ranking Events](/developers/python-client-guides/publishing-production-data/ranking-events)
For supported task setup and schema guidance, continue with the
[*Model Onboarding guide*](/developers/client-library-reference/model-onboarding)
and
[*Updating Model Schema*](/developers/python-client-guides/model-onboarding/updating-model-schema).
***
## Related Documentation
* [**SDK & API Reference**](/sdk-api/index) - Complete API documentation
* [**Integrations**](/integrations) - Connect Fiddler with your ML stack
* [**Documentation**](/) - Product guides and platform documentation
# Alerts with Fiddler Client
Source: https://docs.fiddler.ai/developers/python-client-guides/alerts-with-fiddler-client
Discover our guide to alerts with Fiddler Client. Learn to set up alert rules to add, delete, and list all alerts, including triggered alerts.
The Fiddler API client provides programmatic control for alert management alongside the Fiddler UI, enabling these key workflows:
* Create alert rules
* Remove alert rules
* Retrieve all configured alert rules
* Access triggered alert history
> 📘 Note: For UI-based alert configuration, refer to the [alert setup guide](/observability/platform/alerts-platform).
## Creating Alert Rules
The Fiddler client can be used to create a variety of alert rule types, including **Data Drift**, **Performance**, **Data Integrity**, **Service Metrics**, and **Custom Metrics**.
The Fiddler client supports multiple alert rule types, including:
* Data Drift
* Performance
* Data Integrity
* Service Metrics
* Custom Metrics
### Understanding Alert Thresholds
When configuring thresholds:
* **Absolute thresholds** (`CompareTo.RAW_VALUE`): For percentage-based metrics like `null_violation_percentage`, express values as percentages (e.g., 10 for 10%).
* **Relative thresholds** (`CompareTo.TIME_PERIOD`): Express values as decimal fractions regardless of metric type (e.g., 0.1 for 10%)
## Example Implementations
### Example 1: Data Integrity Alert with Static Threshold
This example creates an alert that monitors for missing values in the `age` column. It triggers notifications when null values exceed 5% (warning) or 10% (critical) of daily values.
```python theme={null}
MODEL_ID = '299c7b40-b87c-4dad-bb94-251dbcd3cbdf'
alert_rule = AlertRule(
name='Bank Churn Missing Values Static Percent',
model_id=MODEL_ID,
metric_id='null_violation_percentage',
priority=Priority.HIGH,
compare_to=CompareTo.RAW_VALUE,
condition=AlertCondition.GREATER,
bin_size=BinSize.DAY,
critical_threshold=10,
warning_threshold=5,
columns=['age'],
).create()
notifications = alert_rule.set_notification_config(
emails=['your.email@example.com', 'alerts.group@example.com'],
)
```
### Example 2: Data Integrity Alert with Historical Comparison
This example creates an alert that compares today's missing values against yesterday's values. It triggers when null values increase by 5% (warning) or 10% (critical) compared to the previous day.
```python theme={null}
MODEL_ID = '299c7b40-b87c-4dad-bb94-251dbcd3cbdf'
alert_rule = AlertRule(
name='Bank Churn Missing Values Rolling Historical Percent',
model_id=MODEL_ID,
metric_id='null_violation_percentage',
priority=Priority.HIGH,
compare_to=CompareTo.TIME_PERIOD,
compare_bin_delta=1,
condition=AlertCondition.GREATER,
bin_size=BinSize.DAY,
critical_threshold=0.1,
warning_threshold=0.05,
columns=['age'],
).create()
notifications = alert_rule.set_notification_config(
emails=['your.email@example.com', 'alerts.group@example.com'],
)
```
### Example 3: Performance Alert with Time-Based Comparison
This example creates a performance alert that monitors for precision changes. It compares today's precision with yesterday's values and triggers when precision decreases by 5% (warning) or 10% (critical).
```python theme={null}
MODEL_ID = '4531bfd9-2ca2-4a7b-bb5a-136c8da09ca2'
alert_rule = AlertRule(
name='Bank Churn Precision Relative',
model_id=MODEL_ID,
metric_id='precision',
priority=Priority.HIGH,
compare_to=CompareTo.TIME_PERIOD,
compare_bin_delta=1,
condition=AlertCondition.LESSER,
bin_size=BinSize.DAY,
critical_threshold=0.1,
warning_threshold=0.05,
).create()
notifications = alert_rule.set_notification_config(
emails=['your.email@example.com', 'alerts.group@example.com'],
)
```
> 🚧 Please note, the possible values for compare\_bin\_delta vs bin\_size are:
| Bin Size | Allowed Compare bin delta |
| ------------- | ------------------------------------- |
| BinSize.Hour | \[1, 24, 24 \* 7, 24 \* 30, 24 \* 90] |
| BinSize.Day | \[1, 7, 30, 90] |
| BinSize.Week | \[1] |
| BinSize.Month | \[1] |
### Retrieving Alert Rules
The AlertRule.list() method retrieves alert rules that match your specified criteria. This method returns a Python iterator of matching rules.
```python theme={null}
MODEL_ID = '4531bfd9-2ca2-4a7b-bb5a-136c8da09ca2'
alert_rules = AlertRule.list(
model_id=MODEL_ID, # Optional parameter
metric_id='jsd', # Optional parameter
columns=['age'], # Optional parameter
ordering=[
'critical_threshold'
], # Add **-** prefix for descending sort ['-critical_threshold']
)
```
> **Tip**: All filter parameters are optional. Omit them to retrieve all alert rules.
### Removing Alert Rules
To delete an alert rule, call the delete() method on an AlertRule object. You can retrieve the rule object using either:
* The list() method with filters
* The get() method with the rule's unique identifier
```python theme={null}
# Delete from a list of alerts
MODEL_ID = '4531bfd9-2ca2-4a7b-bb5a-136c8da09ca2'
alert_rules = AlertRule.list(model_id=MODEL_ID)
for alert_rule in alert_rules:
if alert_rule.name == 'Bank Churn Precision Relative':
alert_rule.delete()
break
# Delete using the alert rule's unique identifier
ALERT_RULE_ID = '6da9c3c0-a9fa-4ab6-8b64-8d07b0736e77'
rule = AlertRule.get(id_=ALERT_RULE_ID)
rule.delete()
```
> **Tip**: Find the alert rule ID in the Alert Rule tab of your Fiddler Alerts page.
### Accessing Triggered Alerts
Use the AlertRecord.list() method to retrieve alerts triggered by a specific rule:
```python theme={null}
from datetime import datetime
triggered_alerts = AlertRecord.list(
alert_rule_id=ALERT_RULE_ID, # Required: ID of the alert rule
start_time=datetime(2024, 9, 1), # Optional: Start of time range
end_time=datetime(2024, 9, 24), # Optional: End of time range
ordering=['alert_time_bucket'], # Optional: Sort order
# Use '-alert_time_bucket' for descending
)
```
### Configuring Notifications
After creating an alert rule, configure how notifications are sent when the rule triggers. Fiddler supports these notification channels:
* Email
* PagerDuty
* Slack webhooks
* Custom webhooks
You can specify multiple notification types, and each type can have multiple recipients.
```python theme={null}
ALERT_RULE_ID = '72e8835b-cde2-4dd2-a435-a35d4b51196b'
rule = AlertRule.get(id_=ALERT_RULE_ID)
rule.set_notification_config(
emails=['your.email@example.com', 'alerts.group@example.com'],
webhooks=['8b403d99-530a-4c5a-a519-89688d65ddc1'], # Webhook UUID
pagerduty_services=[
'pagerduty_service_1',
'pagerduty_service_2',
], # PagerDuty service names
pagerduty_severity='critical', # Only applies to PagerDuty
)
```
> **Important**: PagerDuty, Slack webhooks, and custom webhooks must be pre-configured by your Fiddler administrator.
# Installation and Setup
Source: https://docs.fiddler.ai/developers/python-client-guides/installation-and-setup
Explore our installation guide to set up Fiddler’s Python SDK client. Learn how to connect, install, import, authorize, and set log levels in your environment.
Fiddler offers a **Python SDK client** that lets you connect directly to your Fiddler environment from a Jupyter Notebook or automated pipeline.
***
### Install the Fiddler Client
The client is available for download from PyPI via pip:
```bash theme={null}
pip install -q fiddler-client
```
Fiddler follows the Python Software Foundation's support schedule for Python versions. See the [Python version support policy](/reference/python-support-policy) for details.
### Import the Fiddler Client
Once you've installed the client, you can import the `fiddler` package into any Python script:
```python theme={null}
import fiddler as fdl
```
***
### Authorize the Client
To use the Fiddler client, you will need **authorization details** that contain
* The [URL](#find-your-url) you are connecting to
* A [personal access token](#find-your-authorization-token) for your user
#### Find Your URL
The URL should point to **where Fiddler has been deployed** for your organization.
On-premise customers will use the URL specified by their IT operations team. If you are using Fiddler's managed cloud service, you will have been provided with a unique URL, which will be in one of the forms shown below:
```html theme={null}
# Managed SaaS
https://.fiddler.ai
# Managed SaaS Peering
https://.cloud.fiddler.ai
```
#### Find Your Authorization Token
To find your authorization token, navigate to the **Settings** page, click the **Credentials** tab, and then use the **Create Key** button (if there is not already an authorization token for your user).
#### Connect the Client to Fiddler
Once you've located the URL of your Fiddler environment and your authorization token, you can connect the Fiddler client to your environment.
```python theme={null}
import fiddler as fdl
URL = 'https://app.fiddler.ai'
AUTH_TOKEN = ''
# Connect to the Fiddler client
# This call will also validate the client vs server version compatibility.
fdl.init(url=URL, token=AUTH_TOKEN)
print(f'Client version: {fdl.__version__}')
print(f'Server version: {fdl.conn.server_version}')
```
### Set Log Level
Set the log level for the desired verbosity.
```python theme={null}
import fiddler as fdl
import logging
# Create and configure the root logger prior to calling init()
logging.basicConfig(
level=logging.WARN,
format="%(asctime)s.%(msecs)03d [%(name)s] %(levelname)s: %(message)s",
datefmt="%y%m%d-%H:%M:%S",
)
fdl.init(url=URL, token=AUTH_TOKEN)
```
> 📘 Info
>
> For detailed documentation on the Fiddler Python Client SDK’s many features, check out the [Python Client SDK reference](/sdk-api/python-client/connection) section.
# Create a Project and Model
Source: https://docs.fiddler.ai/developers/python-client-guides/model-onboarding/create-a-project-and-model
Explore our guide to creating a project and onboarding a model for observation. Learn how projects organize models and define a ModelSpec and Model Task.
### What is a Project?
A project helps organize models under observation and serves as the authorization unit to manage access to your models. To onboard a model to Fiddler, you need to have a project to associate it with. Once Fiddler's Python client is connected to your environment, you can either create a new project or use an existing one to onboard your model.
#### Create a Project
Using the Python client, you can create a project by calling the Project object's create function after setting the desired project name.
```python theme={null}
# Please be mindful that the Project name must be unique and should not have spaces or special characters.
PROJECT_NAME = 'quickstart_example'
## Create the Project
project = fdl.Project(name=PROJECT_NAME)
project.create()
print(f'New project created with id = {project.id}')
```
You should now see the newly created project on the Projects page in the Fiddler UI.
#### List All Projects
Using an existing project, you may list all the projects that you are authorized to view.
```python theme={null}
for project in fdl.Project.list():
print(f'Project: {project.id} - {project.name}')
## This will print ALL project IDs and Names that you have access to.
```
### Onboarding a Model
To onboard a model you need to define a **ModelSpec** and optionally a **Model Task**. If you do not specify a model task during Model creation it can be set later or left unset.
#### Define the ModelSpec
A **ModelSpec object** defines what role each column of your inference data serves in your model.
Fiddler supports five column roles:
1. Inputs (features),
2. Outputs (predictions),
3. Targets (ground truth labels),
4. Metadata (additional information passed along with the inference)
5. Custom features (additional information that Fiddler should generate like embeddings or enrichments)
```python theme={null}
model_spec = fdl.ModelSpec(
inputs=['CreditScore', 'Geography', 'Gender', 'Age', 'Tenure', 'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember', 'EstimatedSalary'],
outputs=['probability_churned'],
targets=['Churned'],
decisions=[],
metadata=[],
custom_features=[],
)
```
#### Define the Model Task
Fiddler supports a variety of model tasks. Create a `ModelTask` object and an additional `ModelTaskParams` object to specify the ordering of labels.
```python theme={null}
model_task = fdl.ModelTask.BINARY_CLASSIFICATION
task_params = fdl.ModelTaskParams(target_class_order=['no', 'yes'])
```
#### Infer the Model Schema
Onboard the model schema to Fiddler by passing in:
1. the data sample dataframe, called `sample_df` below
2. the `ModelSpec` object
3. the `ModelTask` and `ModelTaskParams` objects
4. the event/inference ID column and event/inference timestamp columns
```python theme={null}
MODEL_NAME = 'my_model'
model = fdl.Model.from_data(
name=MODEL_NAME,
project_id=fdl.Project.from_name(PROJECT_NAME).id,
source=sample_df,
spec=model_spec,
task=model_task,
task_params=task_params,
event_id_col=id_column,
event_ts_col=timestamp_column
)
```
Depending on the input size this step might take a moment to complete. It is not a local operation, but requires uploading the sample dataframe to the Fiddler HTTP API.
#### Review and Edit the Schema
Schema inference is just a helping hand. The resulting schema needs human review and potentially some edits, as documented in the section titled [Customizing your Model Schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema).
#### Define Model Columns Using Pandas DataFrame Column Types
When defining the columns of a model, use the data types from a Pandas DataFrame. Note that no type inference is performed; it is the responsibility of the user to ensure that the DataFrame's column types are correct. The mapping of column types is as follows:
| Pandas | Fiddler (fdl.DataType) |
| -------------------- | ---------------------- |
| int8 | INTEGER |
| int16 | INTEGER |
| int32 | INTEGER |
| int64 | INTEGER |
| uint8 | INTEGER |
| uint16 | INTEGER |
| uint32 | INTEGER |
| uint64 | INTEGER |
| float16 | FLOAT |
| float32 | FLOAT |
| float64 | FLOAT |
| complex64 | STRING |
| complex128 | STRING |
| string | STRING |
| object | STRING |
| bool | BOOLEAN |
| boolean | BOOLEAN |
| datetime64\[ns] | TIMESTAMP |
| datetime64\[ns, UTC] | TIMESTAMP |
| timedelta64\[ns] | STRING |
| period\[freq] | STRING |
| category | CATEGORY |
| interval | STRING |
```python theme={null}
MODEL_NAME = 'my_model'
model_schema = fdl.utils.column_generator.create_columns_from_df(sample_df)
model = fdl.Model(
name=MODEL_NAME,
project_id=project.id,
schema=model_schema,
spec=model_spec,
task=model_task,
task_params=task_params)
```
The resulting model's schema should be reviewed by the user. Any necessary edits should be made according to the guidelines outlined in the preceding section.
#### Onboard the Model
After making sure the schema looks good, the model can be onboarded with the following API call:
```python theme={null}
model.create()
```
# Customizing Your Model Schema
Source: https://docs.fiddler.ai/developers/python-client-guides/model-onboarding/customizing-your-model-schema
Delve into our guide to customize your Model Schema with Fiddler. Learn how to adjust a column’s value range, possible values, and data type to match your model.
There can be occasions when the `fdl.ModelSchema` object generated by `fdl.Model.from_data` infers a column's data type **differently than the type intended** by the model developer. In these cases you can modify the ModelSchema columns as needed prior to creating the model in Fiddler.
Let's walk through an example of how to do this.
***
Suppose you've loaded in a dataset as a pandas DataFrame.
```python theme={null}
import pandas as pd
df = pd.read_csv('example_dataset.csv')
```
Below is an example of what is displayed upon inspection.
***
You then create a `fdl.Model` object by inferring the column schema details from this DataFrame.
```python theme={null}
model = fdl.Model.from_data(
name='my_model',
project_id=PROJECT_ID,
source=df,
)
```
Below is an example of what is displayed upon inspection of `model.schema`.
```json theme={null}
```
Upon inspection you may notice that **a few things are off**:
1. The [value range](#modifying-the-value-range) of `output_column` is set to `[0.01, 0.99]`, when it should really be `[0.0, 1.0]`.
2. There are no [possible values](#modifying-the-possible-values) set for `feature_3`.
3. The [data type](#modifying-the-data-type) of `feature_3` is set to `DataType.STRING`, when it should really be `DataType.CATEGORY`.
4. The [histogram bins](#modifying-the-histogram-bins) for a numerical column may need to be adjusted for better distribution analysis (e.g., quantile-based instead of uniform).
What's the downside of not making sure that ranges and categories are reviewed and properly set? A production traffic event that encodes a number outside of the specified value range or a category value that is not in the set of valid category values will further down the line be flagged as a so-called Data Integrity violation. Depending on the alerting config, this may result in an alert. It's also worth noting however that an event which has a violation in its columns is still processed, and metrics that can be generated are still generated.
The below examples demonstrate how to address each of the issues noted:
#### Modifying the Value Range
Let's say we want to modify the range of `output_column` in the above `fdl.Model` object to be `[0.0, 1.0]`.
You can do this by setting the `min` and `max` of the `output_column` column.
```python theme={null}
model.schema['output_column'].min = 0.0
model.schema['output_column'].max = 1.0
```
#### Modifying the Histogram Bins
By default, Fiddler auto-generates 10 uniform bins for numerical columns based on the column’s min and max values. You can customize these bins to better represent your data distribution — for example, using quantile-based bins or domain-specific ranges.
```python theme={null}
model.schema['output_column'].bins = [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]
```
Custom bins must:
* Be strictly monotonically increasing
* Start at the column’s `min` value
* End at the column’s `max` value
* Have at least 2 and at most 16 boundary values (1 to 15 bins)
Custom bins affect how feature distributions and drift metrics (PSI, JSD) are computed. Bins are used for histogram bucketing in data drift calculations.
#### Modifying the Possible Values
Let’s say we want to modify the possible values of `feature_3` to be `[‘Yes’, ‘No’]`.
You can do this by setting the `categories` of the `feature_3` column.
```python theme={null}
model.schema['feature_3'].categories = ['Yes', 'No']
```
#### Modifying the Data Type
Let's say we want to modify the data type of `feature_3` to be `DataType.CATEGORY`.
You can do this by setting the `data_type` of the `feature_3` column.
```python theme={null}
model.schema['feature_3'].data_type = fdl.DataType.CATEGORY
```
**Note**: when converting a column to a CATEGORY, you must also set the list of unique possible values:
```python theme={null}
model.schema['feature_3'].categories = ['Yes', 'No']
```
**Note**: if converting a column from numeric (integer or float) to a category, you must also remove the min/max numeric range values and bins that were automatically calculated from the sample data.
```python theme={null}
model.schema['output_column'].min = None
model.schema['output_column'].max = None
model.schema['output_column'].bins = None
```
A complete example might look like this:
```python theme={null}
sample_data_df = pd.read_csv('some.csv')
# configure your ModelSpec here
# model_spec = ...
# configure your ModelTask here, or use NOT_SET
# model_task = ...
# Infer your model's schema from a sample of data
ml_model = fdl.Model.from_data(
source=sample_data_df,
name=MODEL_NAME,
version=VERSION_NAME,
project_id=project.id,
spec=model_spec,
task=model_task
)
# Make any adjustments to the inferred ModelSchema BEFORE creating the model
ml_model.schema['feature_3'].data_type = fdl.DataType.CATEGORY
ml_model.schema['feature_3'].categories = ['0', '1']
# The original datatype was inferred as an integer, but we preferred a category.
# Clear out the min, max, and bins values derived from the sample data as they do not apply to categories
ml_model.schema['feature_3'].min = None
ml_model.schema['feature_3'].max = None
ml_model.schema['feature_3'].bins = None
# Now create the model with the inferred schema and your overrides
ml_model.create()
```
# Custom Missing Values
Source: https://docs.fiddler.ai/developers/python-client-guides/model-onboarding/specifying-custom-missing-value-representations
Learn how you can customize a model column to assign values to be treated as missing or null data in order to handle a value or token that is inserted in place of null.
Your data may contain missing values represented in nonstandard ways instead of `null` or `NaN`. For example, an upstream system might use "-1.0" or "-999" in a Float column to indicate missing data. Fiddler lets you specify custom missing value representations for each column when defining your model schema.
## Customize Missing Data Values in Your Schema
To specify which values should be treated as nulls when publishing data to Fiddler:
You can modify your ModelSchema object just before onboarding your model to include details about which values should be replaced with nulls when publishing data to Fiddler.
```python theme={null}
# Assume an instantiated Fiddler Model:
# model = Model.from_data(...)
# Modify your ModelSchema object before calling model.create()
model.schema['my_column'].replace_with_nulls = [
'-1.0',
'-999'
]
```
This configuration tells Fiddler to automatically consider these values as `null` when processing your data and generating data integrity metrics.
For more information, see our in-depth [guide](/developers/python-client-guides/model-onboarding/customizing-your-model-schema) on customizing your model schema before creating your Fiddler model.
# Task Types
Source: https://docs.fiddler.ai/developers/python-client-guides/model-onboarding/task-types
Explore our guide to selecting a model task type when onboarding your ML models and LLM applications.
Fiddler supports **six** model tasks. These include:
* Binary Classification
* Multi-class Classification
* Regression
* Ranking
* LLM
* Not set
**Binary classification** is the task of classifying the elements of an outcome set into two groups (each called class) on the basis of a classification rule.
Onboarding a Binary classification task in Fiddler requires the following:
* A single output column of type float (range 0-1) which represents the soft output of the model. This column has to be defined.
* A single target column that represents the true outcome. This column has to be defined.
* A list of input features has to be defined.
Typical binary classification problems include:
* Determining whether a customer will churn or not. Here the outcome set has two outcomes: The customer will churn or the customer will not. Further, the outcome can only belong to either of the two classes.
* Determining whether a patient has a disease or not. Here the outcome set has two outcomes: the patient has the disease or does not.
**Multiclass classification** is the task of classifying the elements of an outcome set into three or more groups (each called class) on the basis of a classification rule.
Onboarding a Multiclass classification task in Fiddler requires the following:
* Multiple output columns (one per class) of type float (range 0-1) which represent the soft outputs of the model. Those columns have to be defined.
* A single target column that represents the true outcome. This column has to be defined.
* A list of input features has to be defined.
Typical multiclass classification problems include:
* Determining whether an image is a cat, a dog, or a bird. Here the outcome set has more than two outcomes. Further, the image can only be determined to be one of the three outcomes and it's thus a multiclass classification problem.
**Regression** is the task of predicting a continuous numeric quantity.
Onboarding a Regression task in Fiddler requires the following:
* A single numeric output column that represents the output of the model. This column has to be defined.
* A single numeric target column that represents the true outcome. This column has to be defined.
* A list of input features has to be defined.
Typical regression problems include:
* Determining the average home price based on a given set of housing-related features such as its square footage, number of beds and baths, its location, etc.
* Determining the income of an individual based on features such as age, work location, job sector, etc.
**Ranking** is the task of constructing a rank-ordered list of items given a particular query that seeks some information.
Onboarding a Ranking task in Fiddler requires the following:
* A single numeric output column that represents the output of the model. This column has to be defined.
* A single target column that represents the true outcome. This column has to be defined.
* A list of input features has to be defined.
* A `target_class_order` that ranks the target's possible values from least to most relevant. This is **required** for ranking models — Fiddler no longer infers it, and onboarding fails without it (even when the target is a float).
`target_class_order` tells Fiddler how to interpret your target column by relevance, from least to most relevant. The target is the ground truth — what actually happened — not the model's predicted score. For example, a search-and-book use case might record the outcomes `No click`, `Click`, and `Booked`; you would pass `task_params=fdl.ModelTaskParams(target_class_order=['No click', 'Click', 'Booked'])` so Fiddler knows `Booked` is the most relevant result. Labels can be numeric or string — only their order matters. Fiddler uses this ordering to compute ranking performance metrics out of the box.
Typical ranking problems include:
* Ranking documents in information retrieval systems.
* Ranking relevancy of advertisements based on user search queries.
**LLM** is the task dedicated for Large Language Models, a type of transformer model. It represents a deep learning model that can process human languages.
Onboarding an LLM task in Fiddler doesn't require any specific format with regards to the targets/outputs/inputs definition. Those can be defined or not, with any type and no minimum or maximum column has to be defined. However, in that setting, Fiddler doesn't offer XAI functionalities or performance metrics.
Typical LLM problems include:
* Chatbots and Virtual assistants that can answer questions.
* Content creation like articles, blog posts, etc.
**Not set** is the task to choose if a use-case is not covered by the previous tasks or the use-case doesn't need performance metrics. In this setting, the user doesn't have to specify the required data as discussed above to onboard their model.
Onboarding a `NOT_SET` task in Fiddler doesn't require any specific format with regards to the targets/outputs/inputs definition Those can be defined or not, with any type and no minimum or maximum column has to be defined. However, in that setting, Fiddler doesn't offer XAI functionalities or performance metrics.
# Updating Model Schema
Source: https://docs.fiddler.ai/developers/python-client-guides/model-onboarding/updating-model-schema
Learn how to modify your model's schema after initial creation by adding new columns using the Python client's add\_column() method. Add features, metadata, and tracking columns to existing models.
Learn how to modify your model's schema after initial creation by adding new columns.
## Overview
Sometimes you need to add new columns to an existing model in production. Common scenarios include:
* Adding new features that weren't in the original training data
* Including additional metadata for monitoring purposes
* Extending the model with derived features
* Adding tracking columns for business metrics
Fiddler allows you to update your model schema programmatically using the Python client's `add_column()` method.
**Availability:** This feature requires Fiddler Python Client SDK version 3.11 or later.
For detailed API reference, see [Model.add\_column()](/sdk-api/python-client/model#add_column).
## Prerequisites
* An existing model in Fiddler
* Python client installed and initialized (version 3.11+)
* Appropriate permissions to modify the model
## Adding a Column
Use the `add_column()` method on your model instance to add a new column:
### Basic Example
```python theme={null}
import fiddler as fdl
from fiddler import Column, DataType
# Fetch existing model
model = fdl.Model.from_name(
name="fraud_detector",
project_id="YOUR_PROJECT_ID"
)
# Define new column (bins are optional — auto-generated from min/max if omitted)
new_column = Column(
name="transaction_amount",
data_type=DataType.FLOAT,
min=0.0,
max=100000.0,
bins=[0, 100, 500, 1000, 5000, 10000, 50000, 100000]
)
# Add to model schema
model.add_column(column=new_column, column_type='metadata')
```
## Column Types
The `column_type` parameter specifies where the column will be used in your model. Available types:
* **`'inputs'`**: Model input features used for predictions
* **`'outputs'`**: Model prediction outputs (probabilities, scores, etc.)
* **`'targets'`**: Ground truth labels for evaluation
* **`'metadata'`**: Tracking/monitoring data (default)
## Data Type Examples
Fiddler supports the following data types for model columns:
* **Integer** (`DataType.INTEGER`): Whole numbers (e.g., age, count)
* **Float** (`DataType.FLOAT`): Decimal numbers (e.g., price, score, probability)
* **Category** (`DataType.CATEGORY`): Categorical values from a predefined set
* **String** (`DataType.STRING`): Text data
* **Boolean** (`DataType.BOOLEAN`): True/false values
* **Vector** (`DataType.VECTOR`): Multi-dimensional numerical arrays (embeddings)
* **Timestamp** (`DataType.TIMESTAMP`): Date and time values
### Numeric Column (Integer)
```python theme={null}
from fiddler import Column, DataType
age_col = Column(
name="customer_age",
data_type=DataType.INTEGER,
min=18,
max=100
)
model.add_column(column=age_col, column_type='metadata')
```
### Numeric Column (Float)
```python theme={null}
score_col = Column(
name="risk_score",
data_type=DataType.FLOAT,
min=0.0,
max=1.0
)
model.add_column(column=score_col, column_type='outputs')
```
### Categorical Column
```python theme={null}
category_col = Column(
name="product_category",
data_type=DataType.CATEGORY,
categories=["Electronics", "Clothing", "Food", "Books"]
)
model.add_column(column=category_col, column_type='inputs')
```
### String Column
```python theme={null}
text_col = Column(
name="customer_feedback",
data_type=DataType.STRING
)
model.add_column(column=text_col, column_type='metadata')
```
### Boolean Column
```python theme={null}
bool_col = Column(
name="is_premium_customer",
data_type=DataType.BOOLEAN
)
model.add_column(column=bool_col, column_type='metadata')
```
### Vector Column (Embeddings)
```python theme={null}
embedding_col = Column(
name="text_embedding",
data_type=DataType.VECTOR,
n_dimensions=768
)
model.add_column(column=embedding_col, column_type='inputs')
```
### Timestamp Column
```python theme={null}
timestamp_col = Column(
name="transaction_time",
data_type=DataType.TIMESTAMP
)
model.add_column(column=timestamp_col, column_type='metadata')
```
## Important Considerations
### Historical Data
Adding a column doesn't automatically populate historical data. The new column will have `null` values for all past events. Only newly published events will contain values for the added column.
Additionally, the baseline dataset won't have data for this new column. If you need to compute drift metrics for the new column, upload a new baseline dataset that includes the column data:
```python theme={null}
import pandas as pd
# Prepare baseline data with the new column included
baseline_df = pd.DataFrame({
"feature1": [...],
"feature2": [...],
"region": ["US", "EU", "APAC", ...] # New column added to schema
})
# Upload new baseline dataset
baseline_publish_job = model.publish_batch(
source=baseline_df,
environment=fdl.EnvType.PRE_PRODUCTION,
dataset_name='baseline_with_new_column',
)
print(f'Baseline upload initiated with Job ID = {baseline_publish_job.id}')
```
### Schema Validation
The column definition must pass Fiddler's validation rules:
* Column names must be unique within the model
* Data types must be valid
* Numeric columns should specify min/max ranges
* Numeric columns may optionally specify custom bins (must span \[min, max], be strictly increasing, at most 16 boundary values)
* Categorical columns should specify categories
* Vector columns must specify dimensions
### Publishing Data
After adding a column, remember to include it when publishing new events:
```python theme={null}
import fiddler as fdl
import pandas as pd
# Add new column to model
model.add_column(
column=Column(name="region", data_type=DataType.STRING),
column_type='metadata'
)
# Publish events including the new column
events_df = pd.DataFrame({
"timestamp": [...],
"feature1": [...],
"feature2": [...],
"region": ["US", "EU", "APAC", ...] # New column
})
model.publish_batch(source=events_df, environment=fdl.EnvType.PRODUCTION)
```
## Common Use Cases
### Adding Multiple Columns
```python theme={null}
# Define multiple columns
columns_to_add = [
Column(name="customer_segment", data_type=DataType.INTEGER, min=1, max=5),
Column(name="region", data_type=DataType.STRING),
Column(name="is_returning", data_type=DataType.BOOLEAN)
]
# Add each column
for col in columns_to_add:
model.add_column(column=col, column_type='metadata')
print(f"Added column: {col.name}")
```
### Adding a Feature Column
```python theme={null}
# Add a new feature that wasn't in original training data
new_feature = Column(
name="days_since_last_purchase",
data_type=DataType.INTEGER,
min=0,
max=365
)
model.add_column(column=new_feature, column_type='inputs')
```
## Error Handling
### Duplicate Column Names
```python theme={null}
try:
model.add_column(
column=Column(name="existing_column", data_type=DataType.STRING),
column_type='metadata'
)
except ValueError as e:
print(f"Error: {e}")
# Output: Column 'existing_column' already exists in model schema
```
## Complete Example
Here's a complete workflow for adding columns to an existing model:
```python theme={null}
import fiddler as fdl
from fiddler import Column, DataType
import pandas as pd
# Initialize Fiddler client
fdl.init(
url="https://your-instance.fiddler.ai",
token="your-api-token"
)
# Get existing model
model = fdl.Model.from_name(
name="credit_risk_model",
project_id="my-project-id"
)
print(f"Current columns: {[col.name for col in model.schema.columns]}")
# Add new metadata columns
new_columns = [
Column(
name="customer_tier",
data_type=DataType.CATEGORY,
categories=["Bronze", "Silver", "Gold", "Platinum"]
),
Column(
name="account_age_days",
data_type=DataType.INTEGER,
min=0,
max=10000
),
Column(
name="transaction_history_summary",
data_type=DataType.STRING
)
]
# Add each column
for col in new_columns:
try:
model.add_column(column=col, column_type='metadata')
print(f"✓ Added: {col.name}")
except ValueError as e:
print(f"✗ Failed to add {col.name}: {e}")
# Verify columns were added
print(f"Updated columns: {[col.name for col in model.schema.columns]}")
# Publish new events with the added columns
new_events = pd.DataFrame({
# Existing columns
"timestamp": ["2024-01-01T12:00:00Z"],
"credit_score": [720],
"annual_income": [75000],
# Newly added columns
"customer_tier": ["Gold"],
"account_age_days": [365],
"transaction_history_summary": ["Regular activity, no delinquencies"]
})
job = model.publish_batch(source=new_events, environment=fdl.EnvType.PRODUCTION)
print(f"Published events with new columns. Job ID: {job.id}")
```
## Related Documentation
* [Create a Project and Model](/developers/python-client-guides/model-onboarding/create-a-project-and-model)
* [Customizing Your Model Schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema)
* [Publishing Production Data](/developers/client-library-reference/publishing-production-data)
* [Model Task Types](/developers/python-client-guides/model-onboarding/task-types)
## Frequently Asked Questions (FAQ)
**Q: Can I modify an existing column?**
A: `add_column()` is only for adding new columns. To modify an existing column's properties (like min, max, bins, or categories), update the property on the schema and call `model.update()`:
```python theme={null}
model = fdl.Model.get(id_="abc-123")
model.schema["credit_score"].bins = [350, 500, 650, 850]
model.update()
```
> **Note:** Bin boundaries must span the column's existing \[min, max] range. If you are also changing the range, update min and max in the same call.
See [Customizing Your Model Schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema) for details.
**Q: What happens to existing alerts and monitors?**
A: Existing alerts and monitors continue to work. However, you may want to create new monitors for the added columns.
**Q: Can I add multiple columns at once?**
A: You need to call `add_column()` separately for each column. The method updates the model after each addition.
# Naming Convention Guidelines
Source: https://docs.fiddler.ai/developers/python-client-guides/naming-convention-guidelines
Learn Fiddler's naming requirements: start with lowercase letters, use only a-z, 0-9, and underscores afterwards. See examples and best practices.
## Core Fiddler Asset Naming Requirements
The following naming convention applies specifically to these core Fiddler platform assets: Projects, Models, Baselines, and Webhooks. Other elements in the platform may have different naming rules.
### Required Format
* Names must start with a lowercase letter (a-z)
* After the first character, names may contain:
* Lowercase letters (a-z)
* Numbers (0-9)
* Underscores (\_)
* Names cannot contain uppercase letters, spaces, hyphens, or special characters
* Each asset type has specific length requirements:
| Fiddler Asset | Minimum Name Length | Maximum Name Length |
| ------------- | ------------------- | ------------------- |
| Baseline | 3 | 256 |
| Model | 2 | 30 |
| Project | 2 | 32 |
| Webhook | 2 | 127 |
### Examples
#### Valid Names
* `model_1`
* `test_project_2023`
* `prod_env`
* `my_baseline_123`
#### Invalid Names
* `Model1` (starts with uppercase)
* `1_model` (starts with a number)
* `test-project` (contains hyphen)
* `my feature` (contains space)
* `project#1` (contains special character)
* `m` (too short for any asset type)
* `_test` (starts with underscore)
* `extremely_long_name_that_exceeds_the_character_limit_for_models_and_projects` (exceeds length limits for Models, Projects, and Webhooks)
### Best Practices
While following the required format, we recommend these additional best practices:
* Use descriptive names that indicate the purpose or content
* Use underscores to separate words for better readability
* Keep names reasonably short while maintaining clarity
* Be consistent with naming patterns across similar items
* Consider using prefixes to group related assets (e.g., `prod_`, `dev_`, `test_`)
### Note on Other Platform Elements
This naming convention applies specifically to core Fiddler assets (Projects, Models, Baselines, and Webhooks). Other elements in the Fiddler platform may have different naming requirements or restrictions. Consult the specific documentation for those elements if needed.
# Creating a Baseline Dataset
Source: https://docs.fiddler.ai/developers/python-client-guides/publishing-production-data/creating-a-baseline-dataset
Learn to create a baseline dataset and detect data drift in production. Explore baseline types in Fiddler and start building one that fits your model best.
Fiddler requires a baseline to detect data drift in production data. Baselines serve as a point of reference for Fiddler to understand what data distributions the model expects to see for all of its inputs and outputs. In other words, a baseline is a **representative sample** of the data you expect to see in production. It represents the ideal data that your model works best on. For this reason, in most cases, **a baseline dataset should be sampled from your model’s training set.**
A few things to remember when designing baselines:
> * It’s important to include **enough data** to ensure you have a representative sample of the expected data distributions. If using the training data for your baseline, you don't necessarily need all the training data, but rather enough to properly denote the minimums, maximums, and each distinct categorical value to be encountered.
> * You may want to consider **including extreme values (min/max)** of each column in your training set so you can adequately monitor range violations in production data. However, if you choose not to, you can manually specify these ranges before onboarding your model (see [customizing your dataset schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema)).
## Types of Baselines
There are four (4) types of baselines in Fiddler:
1. **Static Pre-production** - A static pre-production baseline is created by publishing a dataset to Fiddler. Typically, these datasets are training datasets or subsets thereof. Training data is an excellent baseline as it represents the data the model should expect to receive in production. You may also choose to use testing datasets or samples of production inference data as well.
2. **Static Production** - Using time ranges, previously ingested inference logs can be defined to define a static production baseline. For example, you can define a baseline as a point of reference from every event or inference ingested from March 2024. It is worth noting that static production baselines are not immutable. For example, if new events are published into the time range of the baseline, the distributions defined by the baseline will be recalculated.
3. **Rolling Production** - A rolling production baseline provides a dynamic reference point that shifts with time, like a sliding window that always reflects a fixed period prior to the present moment. For example, with a one-month rolling baseline, today's data is compared with data from exactly one month ago, and this comparison window automatically shifts forward each day. Unlike static baselines, which remain fixed, rolling baselines maintain a consistent time distance from your current data, making them ideal for analyzing recurring patterns in your production environment.
4. **Default Static** - Fiddler always creates this type of baseline for you for every model. This ensures Fiddler can deliver drift calculations even if no other baseline type was defined. This default baseline is named *default\_static\_baseline* and gets created at the time of initial model schema creation. It is defined as a static production baseline using the time range starting 12 months before the Fiddler model object creation date and ending 3 months after that date. It is worth noting that default static production baselines are not immutable like static production baselines. For example, if new events are published into the time range of the baseline, the distributions defined by the baseline will be recalculated, and this may have some impact on previously calculated drift values used in Alerts and Charts.
An example of how to create each type of baseline can be found below.
### Static Pre-production Baseline
A baseline gets created when you upload your pre-production dataset as shown in the example below. Note that `dataset_name` is used to specify the baseline name and the environment is set to PRE\_PRODUCTION to indicate this is not monitored, time-series data.
```python theme={null}
baseline_publish_job = model.publish_batch(
source=sample_data_df,
environment=fdl.EnvType.PRE_PRODUCTION,
dataset_name=STATIC_BASELINE_NAME,
)
print(
f'Initiated pre-production dataset upload with Job ID = {baseline_publish_job.id}'
)
```
### Static Production Baseline
```python theme={null}
static_prod_baseline = fdl.Baseline(
name='static_prod_1',
model_id=model.id,
environment=fdl.EnvType.PRODUCTION,
type_=fdl.BaselineType.STATIC,
start_time=(datetime.now() - timedelta(days=0.5)).timestamp(),
end_time=(datetime.now() - timedelta(days=0.25)).timestamp(),
)
static_prod_baseline.create()
print(f'Static production baseline created with id - {static_prod_baseline.id}')
```
### Rolling Production Baseline
```python theme={null}
rolling_prod_baseline = fdl.Baseline(
name='rolling_prod_1',
model_id=model.id,
environment=fdl.EnvType.PRODUCTION,
type_=fdl.BaselineType.ROLLING,
window_bin_size=fdl.WindowBinSize.WEEK,
offset_delta=4,
)
rolling_prod_baseline.create()
print(f'Rolling production baseline created with id - {rolling_prod_baseline.id}')
```
#### Default Static Baseline
(*Note: while this example doesn't illustrate explicitly creating a default baseline, it illustrates when the default static baseline gets created.*)
```python theme={null}
model = fdl.Model.from_data(
name=MODEL_NAME,
project_id=project.id,
source=sample_data_df,
spec=model_spec,
task=model_task,
task_params=task_params,
event_id_col=id_column,
event_ts_col=timestamp_column,
)
model.create()
# Default static production baseline created automatically at this point
# with a time range between 12 months prior to and 3 months after today's date
```
### List A Model's Baselines
```python theme={null}
for baseline in fdl.Baseline.list(model_id=model.id):
print(f'Baseline: {baseline.id} - {baseline.name}')
```
# Deleting Events
Source: https://docs.fiddler.ai/developers/python-client-guides/publishing-production-data/deleting-events
Dive into our guide on deleting existing events from your data whether due to regulatory compliance, custom data retention policies, or publishing mistakes.
Fiddler enables you to delete previously published production inference data by unique identifier or by event timestamp, using either the Python client's `Model.delete_events()` method or the [Delete Events REST API](/sdk-api/rest-api/events/delete-events).
Deleting events with the Python client requires `fiddler-client` 3.13.0 or later. On earlier client versions, use the REST API directly. See the [Python SDK changelog](/changelog/python-sdk) for release details.
Common reasons for deletion include:
* Complying with data privacy regulations ([GDPR](https://commission.europa.eu/law/law-topic/data-protection/legal-framework-eu-data-protection_en), [CCPA](https://www.oag.ca.gov/privacy/ccpa), etc.)
* Removing erroneously published data
* Correcting data inaccuracies
## Deletion Methods
Fiddler supports two filtering methods for deletion (only one method can be used per request):
### Delete by Event ID
Delete specific inference events by providing their unique identifiers. The event ID corresponds to the `event_id_col` specified during [Model](/sdk-api/python-client/model) onboarding.
Event IDs are designed to be unique. If duplicate `event_id` values have been published, deleting by event ID will remove all events matching those IDs from the raw events table. Aggregated metrics are not recalculated for event ID-based deletions — requests that combine `event_ids` with `update_metrics=True` are rejected. See [event deduplication](/developers/python-client-guides/publishing-production-data/event-deduplication) for avoiding duplicate events. You can delete up to 10,000 events per request. If you need to delete more events in one request, please contact Fiddler support.
### Delete by Timestamp Range
Delete events that fall within a designated time range by specifying start and end timestamps. The filtering is based on the timestamp column (`event_ts_col`) defined during [Model](/sdk-api/python-client/model) onboarding. Deletions are processed in batches (default batch size: 100K events) to ensure efficiency and resilience.
Both timestamps must be aligned to hour boundaries (for example, `2025-11-17T01:00:00Z`), and the range applies as `start_time` ≤ t \< `end_time`.
Ensure that the `max_memory_usage` setting in ClickHouse is configured to handle the number of events specified by the batch size (controlled by the `EVENT_DELETION_BATCH_SIZE` environment variable). You can delete up to 100 million events per day using this endpoint. If you require a higher daily deletion limit, please contact Fiddler support.
## Impact on Monitoring Metrics
When deleting inference data with the time range filter, you can control how the deletion affects your monitoring metrics using the `update_metrics` parameter:
* Preserve existing aggregated metrics (default): `update_metrics=False`
* Recalculate metrics to reflect the data removal: `update_metrics=True`
`update_metrics` applies only to time range deletions. Event ID-based deletions do not recalculate aggregated metrics, and requests that set `update_metrics=True` together with `event_ids` are rejected.
## Important Considerations
* Deletions are permanent and cannot be undone
* Pre-production (baseline) datasets do not support row-level deletion
* Large-scale deletions may temporarily impact data ingestion performance.
* By default, the event deletion API allows up to 30 requests per day.
* If the rate limit is exceeded, the API will return a `429 Too Many Requests` response, along with rate limiting headers:
* `X-RateLimit-Limit` – The maximum number of allowed requests in the current window.
* `X-RateLimit-Remaining` – The number of remaining requests in the current window.
* `Retry-After` – The number of seconds to wait before retrying, or the HTTP date when the rate limit resets.
* `X-RateLimit-Reset` – The UTC epoch time (seconds) when the current rate limiting window resets.
## Deleting Events with the Python Client
Use `Model.delete_events()` to delete events directly from the Python client (`fiddler-client` 3.13.0+). Deletion runs asynchronously on the server: the method returns a [Job](/sdk-api/python-client/job) you can wait on or poll for completion.
### Example: Deleting Events by Timestamp Range
```python theme={null}
import fiddler as fdl
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
# Delete one hour of events. Timestamps must be timezone-aware (an ISO 8601
# string with an explicit offset, or a timezone-aware datetime) and aligned
# to hour boundaries. The range applies as start_time <= t < end_time.
job = model.delete_events(
start_time='2025-11-17T01:00:00Z',
end_time='2025-11-17T02:00:00Z',
update_metrics=True, # Recalculate monitoring metrics after deletion
)
job.wait()
```
### Example: Deleting Events by Event ID
```python theme={null}
import fiddler as fdl
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
# Delete specific events by ID (up to 10,000 IDs per request).
job = model.delete_events(event_ids=['pred_001', 'pred_002'])
job.wait()
```
Validation to be aware of:
* Provide either a time range or a list of event IDs in a single request — not both.
* `start_time` and `end_time` must be provided together. Naive datetimes (without timezone information) are rejected; non-UTC values are normalized to UTC.
* A call with no filter is rejected, to avoid accidentally deleting all events for a model.
## Deleting Events with the REST API
The [Delete Events REST API](/sdk-api/rest-api/events/delete-events) (`DELETE /v3/events`) accepts the following parameters:
| Parameter | Type | Default | Description |
| --------------- | --------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| model\_id | UUID | - | Unique identifier for the model from which production events are deleted. |
| time\_range | Optional\[dict] | - | Dictionary with `start_time` (inclusive) and `end_time` (exclusive) in UTC, both aligned to the hour (format: `'YYYY-MM-DD HH:00:00'`). Specify timestamps so that the entire range covers complete hours—do not use minutes or seconds. The range filter applies as `start_time` ≤ t \< `end_time`. |
| event\_ids | Optional\[list] | - | List of event\_ids to be deleted. |
| update\_metrics | Optional\[bool] | False | Determines if the monitoring metrics are recalculated to reflect the changes. Supported for time range deletions only. |
### Example: Deleting Inference Events by Timestamp Range
```python theme={null}
import requests
headers = {'Content-Type': 'application/json', 'Authorization': f'Bearer {token}'}
data = {'model_id': model.id,
'time_range':{
'start_time':'2024-09-27 17:00:00',
'end_time':'2024-09-27 18:00:00',
},
'update_metrics':True,
}
response = requests.delete(
url=f'{url}/v3/events',
headers=headers,
json=data,
)
```
### Example: Deleting Inference Events by Event IDs
```python theme={null}
import requests
headers = {'Content-Type': 'application/json', 'Authorization': f'Bearer {token}'}
data = {
'model_id': model.id,
'event_ids': ['event_id1', 'event_id2'],
}
response = requests.delete(
url=f'{url}/v3/events',
headers=headers,
json=data,
)
```
> 📘 Please delete events with caution when `update_metrics=True`. We recommend not deleting events while there is an ongoing publish or update operation within the same data range.
## Related Resources
* [Delete Events REST API reference](/sdk-api/rest-api/events/delete-events)
* [Model SDK reference](/sdk-api/python-client/model)
* [Event deduplication](/developers/python-client-guides/publishing-production-data/event-deduplication)
* [Python SDK changelog](/changelog/python-sdk)
# Event Deduplication
Source: https://docs.fiddler.ai/developers/python-client-guides/publishing-production-data/event-deduplication
Learn how Fiddler prevents duplicate inference events from inflating your monitoring metrics.
When enabled, Fiddler's event deduplication feature ensures that the same inference event is stored and counted only once within the deduplication windows, even when events are published multiple times due to retries, overlapping batch uploads, or client-side errors.
Event deduplication is not enabled by default. To enable it for your deployment, contact Fiddler support.
## Prerequisites
Event deduplication requires that a unique identifier column is defined on your model:
* Set `event_id_col` to the name of the column containing unique event identifiers when onboarding your model using `Model.from_data()` or `Model()`.
* Events that do not have a value in this column (null or missing) are always accepted and stored without deduplication.
```python theme={null}
model = fdl.Model.from_data(
name='your_model_name',
project_id=fdl.Project.from_name('your_project_name').id,
source=sample_df,
spec=model_spec,
task=model_task,
event_id_col='your_event_id_column', # required for deduplication
event_ts_col='your_timestamp_column',
)
```
## How Deduplication Works
Fiddler applies multiple independent layers of deduplication as events move through the ingestion pipeline. Each layer targets a different source of duplication.
| Layer | What it prevents | Window |
| ------------------------------- | ----------------------------------------------------------- | ------------------- |
| Request retry protection | Same HTTP request processed twice due to client retry | 30 minutes |
| Within-batch deduplication | Duplicate event IDs in the same publish call | Entire batch |
| Against recently stored events | Event IDs that already exist in recent ingestion history | 1 hour (default) |
| Across concurrent publish calls | Same event ID arriving in parallel chunks | 5 minutes (default) |
| Storage-level idempotency | Same internal batch resubmitted due to infrastructure retry | 7 days |
Request retry protection and storage-level idempotency are always active. The remaining layers are part of event deduplication.
### Request Retry Protection
When the Fiddler client SDK retries a publish request due to a network timeout or transient error, Fiddler detects the retry and returns the original result without reprocessing the events. This prevents duplicate job submissions and duplicate event ingestion caused by client-side retry logic.
* **Window:** 30 minutes. A retried request received within 30 minutes of the original is recognized and deduplicated.
* **Always active:** This layer requires no configuration and is always enabled.
### Within-Batch Deduplication
If the same event ID appears more than once within a single publish call (for example, duplicate rows in a DataFrame or CSV), Fiddler keeps the first occurrence and drops subsequent duplicates before the events are stored.
* **Window:** The entire publish batch.
### Against Recently Stored Events
Before storing incoming events, Fiddler checks whether their event IDs already exist in recently ingested data. If a match is found, the incoming event is dropped.
* **Window:** 1 hour (default). Event IDs are checked against events ingested within the past hour.
### Across Concurrent Publish Calls
Large batch uploads are internally split into chunks and processed in parallel. If the same event ID appears across multiple chunks — for example, when overlapping files are uploaded concurrently — Fiddler uses a short-lived shared cache to detect and drop the duplicates.
* **Window:** 5 minutes (default). Event IDs seen within the past 5 minutes are held in cache and used to deduplicate overlapping chunks.
### Storage-Level Idempotency
If an internal processing failure causes the same batch of events to be retried and resubmitted to storage, Fiddler's storage layer silently ignores the duplicate insert using a per-chunk idempotency token.
* **Always active:** This layer requires no configuration.
* It protects against infrastructure-level retries and does not deduplicate individual events by event ID.
> **Note:**
>
> If all events in a publish call are identified as duplicates, the call succeeds with no new events stored — it is not treated as an error.
## Without Deduplication
When event deduplication is not enabled, duplicate events published via retries or overlapping batches are accepted and stored. This may affect the accuracy of aggregated metrics such as event counts, drift scores, and model performance indicators shown in monitoring dashboards, charts, and alerts.
To enable deduplication for your deployment, contact Fiddler support.
## Limitations
* Deduplication applies to **insert operations only**. Update operations (used to add ground truth labels or metadata) are never deduplicated.
* Events with a null or missing event ID are never deduplicated. They are always accepted and stored, and receive a system-generated identifier.
* Enabling deduplication does not retroactively correct metrics affected by previously published duplicates.
## Related Resources
* [Publishing Batches of Events](/developers/python-client-guides/publishing-production-data/publishing-batches-of-events)
* [Streaming Live Events](/developers/python-client-guides/publishing-production-data/streaming-live-events)
* [Deleting Events](/developers/python-client-guides/publishing-production-data/deleting-events)
* [Publishing Production Data Reference](/developers/client-library-reference/publishing-production-data)
# Publishing Batches of Events
Source: https://docs.fiddler.ai/developers/python-client-guides/publishing-production-data/publishing-batches-of-events
Dive into our guide on publishing batches of events. Learn how Fiddler supports multiple source formats when publishing batches of events.
Fiddler provides multiple options for publishing batches of production data, allowing you to choose the format and method that best suits your needs.
### Supported Data Formats
* pandas DataFrame
* CSV file (`.csv`),
* Parquet file (`.parquet`)
### Supported Data Locations
* In memory - pandas DataFrame
* Local disk - CSV, parquet
> **Note:**
>
> Fiddler's Python client offers the ability to integrate with Cloud data stores such as AWS S3. Refer to our [Integrations Guides](/integrations) for examples.
### Batch Publishing Examples
Publish a batch of inference events using a parquet file, CSV file, or DataFrame using `Model.publish_batch()`. This method executes asynchronously and returns a Job object. The job can be used to:
* Track by ID in the UI on the Jobs page
* Poll for status until completion
* Use the `wait()` method for synchronous behavior
* Log the job ID for reference
#### Parquet File
```python theme={null}
import fiddler as fdl
# Instantiate the Model object for your model
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
publish_job = model.publish_batch(source='my_events_batch.parquet')
# publish_batch() is asynchronous. Use the publish job's wait() method
# if synchronous behavior is desired.
publish_job.wait()
```
#### CSV File
```python theme={null}
import fiddler as fdl
# Instantiate the Model object for your model
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
publish_job = model.publish_batch(source='my_events_batch.csv')
```
#### Pandas DataFrame
```python theme={null}
import pandas as pd
import fiddler as fdl
# Instantiate the Model object for your model
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
my_events_df = pd.read_csv('my_events_batch.csv')
publish_job = model.publish_batch(source=my_events_df)
```
*Please allow a few minutes for events to populate the related charts.* *Total processing time is a function of both width and count of the inference events.*
# Ranking Events
Source: https://docs.fiddler.ai/developers/python-client-guides/publishing-production-data/ranking-events
Explore our guide to publishing production data. Learn how to publish and update ranking events in a grouped format with a detailed example.
### Publish Inference Events for Ranking Models
A ranking machine learning model orders items based on their relevance to a specific query or context. Unlike classification or regression models that predict absolute values, ranking models focus on relative ordering between items.
#### Grouping Format for Ranking Model Inference Events
Before publishing ranking model events to Fiddler, format them in a **grouped format**. This means items returned for the same query ID (specified by the group\_by parameter in TaskParams) appear in a single row with values stored as lists.
Example of correctly grouped format:
| srch\_id | price\_usd | review\_score | ... | prediction | target |
| -------- | -------------------------- | ------------------ | ------ | ------------------------ | ---------- |
| 101 | \[134.77,180.74,159.80] | \[5.0,2.5,4.5] | \[...] | \[1.97, 0.84,-0.69] | \[1,0,0] |
| ... | ... | | ... | ... | ... |
| 112 | \[26.00,51.00,205.11,73.2] | \[3.0,4.5,2.0,1.0] | \[...] | \[10.75,8.41,-0.23,-3.2] | \[0,1,0,0] |
In this example, `srch_id` is the `group_by` column, and all other columns contain lists corresponding to each group.
#### How to Convert a Flat CSV File Into Grouped Format
If your data is in a flat CSV file (one item per row), use Fiddler's utility function to convert it to the required grouped format:
```python theme={null}
from fiddler.utils.helpers import group_by
import pandas as pd
# Read the flat CSV file
df = pd.read_csv('path/to/ranking_events.csv')
# Convert to grouped format
df_grouped = group_by(df=df, group_by_col='srch_id')
```
The `group_by_col` argument should match the column specified in the `group_by` parameter of your [Model](/sdk-api/python-client/connection) object.
### Publish Grouped Ranking Events
After preparing your grouped data, publish it to Fiddler:
```python theme={null}
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
# df_grouped DataFrame from above example
publish_job = model.publish_batch(source=df_grouped)
# publish_batch() is asynchronous. Use the publish job's wait() method
# if synchronous behavior is desired.
publish_job.wait()
```
### Update Ranking Events
You can update existing ranking events while maintaining the same grouped format.
#### Prepare the Updating DataFrame
To update the ground truth labels (called 'target values' in Fiddler), create a dataframe with the required updates while keeping the grouped format.
Call `Model.publish_batch()` with update set to true:
```python theme={null}
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
# Publish updates to existing events
update_job = model.publish_batch(source=modified_df_grouped, update=True)
print(f'Update job: {update_job.id}')
```
Fiddler will update only the columns included in your modified dataframe while preserving all other data. Event update behavior is described in more detail in the [Event Update](/developers/python-client-guides/publishing-production-data/updating-events) guide.
# Streaming Live Events
Source: https://docs.fiddler.ai/developers/python-client-guides/publishing-production-data/streaming-live-events
Learn how to stream your ML model's inference event data using the Fiddler Python client.
You can stream production data to Fiddler as an alternative to batch publishing. Streaming offers lower latency, which is beneficial for high-velocity or near real-time models.
### When to Use Streaming vs. Batch Publishing
* Use `Model.publish_stream()` when low latency is a priority and you're working with individual events or small batches
* Use `Model.publish_batch()` for large datasets or when you need to track longer-running processes with Job objects
#### Stream Individual Inference Events
To stream a single inference event:
```python theme={null}
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
# A single event must still be passed as a list.
model.publish_stream(events=[
{
'customer_id': 1234,
'timestamp': 1710428785,
'CreditScore': 650,
'Geography': 'France',
'Gender': 'Female',
'Age': 45,
'Tenure': 2,
'Balance': 10000.0,
'NumOfProducts': 1,
'HasCrCard': 'Yes',
'isActiveMember': 'Yes',
'EstimatedSalary': 120000,
'probability_churned': 0.105,
'churn': 1
}
])
```
#### Stream Small Batches of Events
For better efficiency, you can stream multiple events at once:
```python theme={null}
# For multiple events, where `my_events` is a list of Python dictionaries
model.publish_stream(events=my_events)
```
> 🚧 Note
>
> Convert a pandas DataFrame to a list of event dictionaries using the to\_dict function.
```python theme={null}
my_events = my_df.to_dict(orient='records')
```
For batches larger than 5,000 events, prefer [`Model.publish_batch()`](/developers/python-client-guides/publishing-production-data/publishing-batches-of-events) over streaming.
# Updating Events
Source: https://docs.fiddler.ai/developers/python-client-guides/publishing-production-data/updating-events
Dive into our guide on updating inference events. Learn how to update your ground truth labels and metadata using the Fiddler Python client.
Fiddler allows you to update specific fields in previously published events. While your model feature values can't be updated, you can update:
* Target column values (ground truth labels)
* Metadata columns
Input, Output, and Custom Feature column types cannot be updated once published. If values for these columns are included with your update (`update=True`), they will be ignored.
### Updating Ground Truth Labels
Updating ground truth labels is the most common use case for post-publish updates.
You can update events when:
* Actual values become available for events initially published without labels
* You discover that initially uploaded labels are incorrect
Things to keep in mind about ground truth labels (target values):
* Use null values when initially publishing inferences that don't yet have labels
* Fiddler automatically keeps aggregated performance metrics current as labels are updated
* Labels can be updated multiple times if necessary
### Updating Metadata Columns
Fiddler supports updating **metadata** columns with new values. This is particularly useful for supporting alternate labels that can be used with Custom Metrics to calculate alternative performance metrics.
Things to keep in mind regarding metadata updates:
* Updated values are visible in Feature Analytics and [Root Cause Analysis ](/observability/analytics#root-cause-analysis)views
* Pre-calculated aggregated metrics will not reflect the updated values
* Custom Metrics, used in charts and alerts, will always use the current values since they're calculated at runtime
* Updating metadata columns requires additional processing time, so only send updates when necessary
### Update Rate Limits
To protect ingestion and downstream metric computation from unbounded load, the update path (`update=True`, `PATCH /v3/events`) is rate-limited per API token. These are the default limits—your administrator can tune them per environment, so your deployment may be configured differently. If you need higher throughput for a large recurring batch of label or metadata updates, contact Fiddler support to discuss raising the limits for your environment.
**Request and volume limits**
* Up to 100 update requests per hour and 1,000 update requests per day.
* Up to 1,000,000 events per hour and 10,000,000 events per day.
* Batch/file updates (`Model.publish_batch()`) are additionally capped at 100,000 events per request. Stream updates (`Model.publish_stream()`) are chunked by the client into requests of 1,000 events each, so streaming's effective ceiling is governed by the request-rate limit (about 100,000 events/hour) rather than the higher event-volume limit above.
If a single request exceeds the 100,000-events-per-request cap, the API returns `413 Request Entity Too Large`—this is not retryable as-is; split the payload into smaller requests. If a request-rate or event-volume window limit is exceeded instead, the API returns `429 Too Many Requests` along with rate-limiting headers:
* `X-RateLimit-Limit` – The maximum number of allowed requests/events in the current window.
* `X-RateLimit-Remaining` – The number remaining in the current window.
* `Retry-After` – The number of seconds to wait before retrying.
* `X-RateLimit-Reset` – The UTC epoch time (seconds) when the current rate-limiting window resets.
### Label Update Examples
#### Stream Label Updates
As with inference publishing, label updates can be sent as streams or batches.
**Stream Update Data Formats**
* List of Python dictionaries
**Stream Update One or More Events**
Integrate Fiddler directly into your production system to publish label updates as they occur. The event ID, the column chosen for Model.event\_id\_col, is required.
```python theme={null}
import fiddler as fdl
# Instantiate the Model object for your model
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
event_update = [
{
'label_value': 1,
'event_id': 'A1',
},
...
]
# Streaming update returns the list of event ID(s) updated.
event_id = model.publish_stream(
events=event_update,
environment=fdl.EnvType.PRODUCTION,
update=True,
)
```
**Stream Update Label and Metadata**
Metadata column(s) can also be updated.
```python theme={null}
import fiddler as fdl
# Instantiate the Model object for your model
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
event_update = [
{
'metadata_col': 'yes',
'label_value': 1,
'event_id': 'A1',
},
]
event_id = model.publish_stream(
events=event_update,
environment=fdl.EnvType.PRODUCTION,
update=True,
)
```
#### Batch Update Data Formats
* pandas DataFrame
* CSV file (`.csv`),
* Parquet file (`.parquet`)
**Batch Update Events**
Publish larger sets of label updates using batch publishing. The event ID, the column chosen for Model.event\_id\_col, is required.
```python theme={null}
import pandas as pd
import fiddler as fdl
project = fdl.Project.from_name(name='your_project_name')
model = fdl.Model.from_name(name='your_model_name', project_id=project.id)
label_batch = pd.read_parquet('path_to_your_updated_labels/data.parquet')
update_job = model.publish_batch(
source=label_batch,
environment=fdl.EnvType.PRODUCTION,
update=True,
)
```
Refer to `Model.publish_stream()` and `Model.publish_batch()` documentation for more details on different sources and parameters.
> 📘 **There are a few points to be aware of:**
>
> * Performance metrics (available in monitoring charts and alert rules) will be computed as ground truth labels are inserted and recomputed when later updated.
> * For example, if the ground truth values are originally missing from events in a given time range, there will be **no performance metrics available** for that time range. Once the events are updated, performance metrics will be computed and will populate the monitoring charts.
> * Events that do not originally have ground truth labels should be **uploaded with empty values**—not dummy values. If dummy values are used, you will have improper performance metrics, and once the new values come in, the old, incorrect values will still be present.
> * Metrics based on **Metadata** columns won't reflect updates.
> * In order to update existing events, you will need access to the event IDs used at the time of upload.
> * Updating the event timestamp, `Model.event_ts_col`, is not supported.
# Experiments Quick Start
Source: https://docs.fiddler.ai/developers/quick-starts/experiments-quick-start
Quick start guide for running experiments on LLM applications using the Fiddler Evals SDK
Systematically evaluate your LLM applications, RAG systems, and AI agents using the Fiddler Evals SDK with built-in evaluators and custom metrics.
**Time to complete**: \~20 minutes
## What You'll Learn
* Initialize the Fiddler Evals SDK and organize your experiments
* Create experiment datasets with test cases
* Use built-in evaluators (faithfulness, relevance, coherence, etc.)
* Create custom evaluators for domain-specific requirements
* Run experiments and analyze results
***
## Prerequisites
* **Fiddler Account**: Active account with API access
* **Python 3.10+**
* **Fiddler Evals SDK**: `pip install fiddler-evals`
* **Access Token**: From [Settings > Credentials](/reference/administration/settings#credentials)
***
## Quick Start
### Step 1: Connect to Fiddler
```python theme={null}
from fiddler_evals import init
# Initialize connection
init(
url='https://your-org.fiddler.ai',
token='your-access-token'
)
```
### Step 2: Create Project and Application
```python theme={null}
from fiddler_evals import Project, Application, Dataset
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
# Create organizational structure
project = Project.get_or_create(name='my_eval_project')
application = Application.get_or_create(
name='my_llm_app',
project_id=project.id
)
# Create experiment dataset
dataset = Dataset.create(
name='experiment_dataset',
application_id=application.id,
description='Test cases for LLM experiments'
)
```
### Step 3: Add Test Cases
```python theme={null}
# Define test cases
test_cases = [
NewDatasetItem(
inputs={"question": "What is the capital of France?"},
expected_outputs={"answer": "Paris is the capital of France"},
metadata={"type": "Factual", "category": "Geography"}
),
NewDatasetItem(
inputs={"question": "Explain photosynthesis"},
expected_outputs={"answer": "Photosynthesis is the process by which plants convert sunlight into energy"},
metadata={"type": "Explanation", "category": "Science"}
),
]
# Insert test cases into dataset
dataset.insert(test_cases)
print(f"✅ Added {len(test_cases)} test cases")
```
### Step 4: Define Your LLM Task
```python theme={null}
def my_llm_task(inputs, extras, metadata):
"""Your LLM application logic."""
question = inputs.get("question", "")
# Call your LLM here (example uses placeholder)
# In production, call OpenAI, Anthropic, or your LLM
answer = f"Mock response to: {question}"
return {"answer": answer}
```
### Step 5: Run Experiment with Built-In Evaluators
```python theme={null}
from fiddler_evals import evaluate
from fiddler_evals.evaluators import (
AnswerRelevance,
Conciseness,
FTLPromptSafety
)
MODEL = "openai/gpt-4o"
CREDENTIAL = "your-llm-credential" # From Settings > LLM Gateway
# Run evaluation
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[
AnswerRelevance(model=MODEL, credential=CREDENTIAL),
Conciseness(model=MODEL, credential=CREDENTIAL),
FTLPromptSafety() # Centor Models run locally, no model= needed
],
name_prefix="my_experiment",
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["question"],
"rag_response": "answer",
"response": "answer",
"text": "answer",
}
)
print(f"✅ Evaluated {len(results.results)} test cases")
```
### Step 6: Analyze Results
```python theme={null}
# Access results programmatically
for item_result in results.results:
print(f"\nTest Case: {item_result.dataset_item_id}")
print(f"Task Output: {item_result.task_output}")
# View scores from each evaluator
for score in item_result.scores:
print(f" {score.name}: {score.value} - {score.reasoning}")
# View results in Fiddler UI
print(f"\n🔗 View results: https://your-org.fiddler.ai")
```
***
## Built-In Evaluators
The Fiddler Evals SDK includes 13 pre-built evaluators:
### Quality & Accuracy
* **AnswerRelevance** - Measures response relevance to the question (High / Medium / Low)
* **Coherence** - Evaluates logical flow and consistency
* **Conciseness** - Checks for unnecessary verbosity
### Safety & Trust
* **FTLPromptSafety** - Detects prompt injection, jailbreaks, and unsafe prompts
* **FTLResponseFaithfulness** - Evaluate faithfulness of LLM responses (powered by the Centor Model for Faithfulness)
### RAG Health Metrics
* **AnswerRelevance** - Measures how well responses address user queries (High / Medium / Low)
* **ContextRelevance** - Evaluates whether retrieved documents are relevant to the query (High / Medium / Low)
* **RAGFaithfulness** - Checks if response is grounded in retrieved documents (Yes / No)
Use these three evaluators together as a diagnostic triad to pinpoint whether RAG pipeline issues originate in retrieval, generation, or query understanding.
### Example: RAG Experiment
```python theme={null}
from fiddler_evals.evaluators import AnswerRelevance, ContextRelevance, RAGFaithfulness
# Add context to test cases
rag_test_cases = [
NewDatasetItem(
inputs={
"user_query": "What is the capital of France?",
"retrieved_documents": "Paris is the capital and largest city of France."
},
expected_outputs={"rag_response": "Paris"}
),
]
dataset.insert(rag_test_cases)
# Evaluate with RAG Health Metrics evaluators
rag_results = evaluate(
dataset=dataset,
task=my_rag_task, # Your RAG system
evaluators=[
AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential"),
ContextRelevance(model="openai/gpt-4o", credential="your-llm-credential"),
RAGFaithfulness(model="openai/gpt-4o", credential="your-llm-credential")
],
score_fn_kwargs_mapping={
"rag_response": "rag_response",
"retrieved_documents": lambda x: x["inputs"]["retrieved_documents"],
"user_query": lambda x: x["inputs"]["user_query"]
}
)
```
***
## Custom Evaluators
Create domain-specific evaluators for your use case:
```python theme={null}
from fiddler_evals.evaluators.base import Evaluator
from fiddler_evals.pydantic_models.score import Score
class CustomToneEvaluator(Evaluator):
"""Evaluates if response matches desired tone."""
def score(self, response: str, desired_tone: str = "professional") -> Score:
# Your custom evaluation logic
is_professional = self._check_tone(response, desired_tone)
return Score(
name="tone_match",
value=1.0 if is_professional else 0.0,
reasoning=f"Response {'matches' if is_professional else 'does not match'} {desired_tone} tone"
)
def _check_tone(self, text: str, tone: str) -> bool:
# Implement your tone detection logic
# Could use keyword matching, LLM-as-judge, or ML model
return True # Placeholder
# Use custom evaluator
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[CustomToneEvaluator()],
score_fn_kwargs_mapping={
"response": "answer",
"desired_tone": "professional"
}
)
```
***
## Advanced Features
### Batch Experiments with Parallel Processing
```python theme={null}
# Evaluate with parallel workers for faster execution
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential"), Conciseness(model="openai/gpt-4o", credential="your-llm-credential")],
max_workers=5 # Process 5 test cases in parallel
)
```
### Import Datasets from Files
```python theme={null}
# From CSV
dataset.insert_from_csv_file(
csv_file_path='test_cases.csv',
inputs_columns=['question'],
expected_outputs_columns=['answer']
)
# From JSONL
dataset.insert_from_jsonl_file(
jsonl_file_path='test_cases.jsonl'
)
# From Pandas DataFrame
import pandas as pd
df = pd.DataFrame({
'question': ['Q1', 'Q2'],
'expected_answer': ['A1', 'A2']
})
dataset.insert_from_pandas(
dataframe=df,
inputs_columns=['question'],
expected_outputs_columns=['expected_answer']
)
```
### Track Experiment Metadata
```python theme={null}
# Add experiment metadata for tracking
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential")],
name_prefix="experiment_v2", # Version your experiments
score_fn_kwargs_mapping={"response": "answer"}
)
# Results are automatically tracked in Fiddler
# View experiment history in the Fiddler UI
```
***
## Complete Example: RAG Experiment Pipeline
```python theme={null}
from fiddler_evals import init, Project, Application, Dataset, evaluate
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
from fiddler_evals.evaluators import (
RAGFaithfulness,
AnswerRelevance,
ContextRelevance,
Conciseness
)
# Step 1: Initialize
init(url='https://your-org.fiddler.ai', token='your-token')
# Step 2: Set up organization
project = Project.get_or_create(name='rag_experiments')
app = Application.get_or_create(name='doc_qa_system', project_id=project.id)
dataset = Dataset.create(name='qa_test_set', application_id=app.id)
# Step 3: Create test cases
test_cases = [
NewDatasetItem(
inputs={
"user_query": "What is machine learning?",
"retrieved_documents": "Machine learning is a subset of AI that enables "
"systems to learn from data."
},
expected_outputs={
"rag_response": "Machine learning is a subset of AI."
},
metadata={"difficulty": "easy"}
),
]
dataset.insert(test_cases)
# Step 4: Define RAG task
def rag_task(inputs, extras, metadata):
"""Your RAG system implementation."""
user_query = inputs["user_query"]
retrieved_documents = inputs["retrieved_documents"]
# Call your RAG system (simplified example)
rag_response = generate_answer(user_query, retrieved_documents)
return {
"rag_response": rag_response,
"retrieved_documents": retrieved_documents,
}
# Step 5: Run comprehensive evaluation
results = evaluate(
dataset=dataset,
task=rag_task,
evaluators=[
RAGFaithfulness(model="openai/gpt-4o", credential="your-llm-credential"), # Check factual grounding (Yes/No)
AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential"), # Check relevance to query (High/Medium/Low)
ContextRelevance(model="openai/gpt-4o", credential="your-llm-credential"), # Check retrieval quality (High/Medium/Low)
Conciseness(model="openai/gpt-4o", credential="your-llm-credential"), # Check for verbosity
],
name_prefix="rag_eval_v1",
score_fn_kwargs_mapping={
"rag_response": "rag_response",
"retrieved_documents": "retrieved_documents",
"user_query": lambda x: x["inputs"]["user_query"],
}
)
# Step 6: Analyze
print(f"Evaluated {len(results.results)} test cases")
for result in results.results:
print(f"\n{result.dataset_item_id}:")
for score in result.scores:
print(f" {score.name}: {score.value:.3f}")
```
***
## Best Practices
1. **Start Small**: Begin with 10-20 test cases to validate your setup
2. **Use Multiple Evaluators**: Combine quality, safety, and domain-specific evaluators
3. **Version Your Experiments**: Use `name_prefix` to track different experiment runs
4. **Monitor Over Time**: Run experiments regularly to catch regressions
5. **Custom Evaluators**: Create domain-specific evaluators for specialized needs
6. **Leverage Parallelization**: Use `max_workers` for faster evaluation of large datasets
7. **Organize Hierarchically**: Use Projects > Applications > Datasets structure
***
## Next Steps
### Complete Guides
* [**Evals SDK Quick Start**](/evaluate-and-test/evals-sdk-quick-start) - Full tutorial with detailed examples
* [**Evals SDK Reference**](/sdk-api/evals/evaluate) - Complete API documentation
### Concepts & Background
* [**Experiments Overview**](/getting-started/experiments) - Why and when to run experiments
* [**Centor Models**](/glossary/centor-models) - Fiddler's purpose-built LLM evaluation models
### Integration Guides
* [**Evals SDK Integration**](/integrations/agentic-ai/evals-sdk) - Integration patterns and examples
* [**LangGraph SDK**](/integrations/agentic-ai/langgraph-sdk) - Monitor LangGraph agents
* [**Strands Agents SDK**](/integrations/agentic-ai/strands-sdk) - Monitor Strands agents
***
## Summary
You've learned how to:
* ✅ Initialize the Fiddler Evals SDK with `init()`
* ✅ Create Projects, Applications, and Datasets for organization
* ✅ Build experiment datasets with test cases
* ✅ Use 13 built-in evaluators for quality, safety, and RAG metrics
* ✅ Create custom evaluators for domain-specific requirements
* ✅ Run experiments with the `evaluate()` function
* ✅ Analyze results programmatically and in the Fiddler UI
The Fiddler Evals SDK provides a comprehensive framework for systematic LLM experiments, enabling you to ensure quality, safety, and accuracy before deploying your AI applications.
# Get Started in <10 Minutes
Source: https://docs.fiddler.ai/developers/quick-starts/get-started-in-less-than-10-minutes
Choose your integration path and get started with Fiddler in under 10 minutes
Welcome to Fiddler! Choose your integration path based on what you want to accomplish. Each quick start gets you up and running in 10-20 minutes.
***
## 🎯 What Do You Want to Do?
### 🤖 Monitor AI Agents & Multi-Step Workflows
**Best for:** Applications using LangGraph, Strands, or custom agentic frameworks
Your AI agents make complex decisions across multiple steps. Monitor the complete workflow from initial reasoning to final response.
**Choose your framework:**
| Framework | Integration | Time | Quick Start |
| ------------------------- | -------------------- | ------ | -------------------------------------------------------------------------- |
| **LangGraph / LangChain** | Auto-instrumentation | 10 min | [LangGraph SDK →](/developers/quick-starts/langgraph-sdk-quick-start) |
| **Strands Agents** | Native integration | 10 min | [Strands Agents SDK →](/developers/quick-starts/strands-agent-quick-start) |
| **Custom / Other** | OpenTelemetry | 15 min | [OpenTelemetry →](/developers/quick-starts/opentelemetry-quick-start) |
**What you'll monitor:**
* Agent decision-making and tool selection
* Multi-step reasoning chains
* LLM calls with inputs/outputs
* Tool usage and external API calls
* Error propagation and recovery
***
### 💬 Monitor Simple LLM Applications
**Best for:** Single-shot LLM inference, chatbots, simple RAG systems
You're using LLMs for straightforward tasks like Q\&A, content generation, or basic chat interfaces.
**Quick Start:** [Simple LLM Monitoring →](/developers/quick-starts/simple-llm-monitoring) ⏱️ 10 min
**What you'll monitor:**
* LLM prompts and completions
* Token usage and costs
* Response latency
* Quality metrics (toxicity, PII, sentiment)
* Embedding visualizations
***
### 📊 Monitor Traditional ML Models
**Best for:** Scikit-learn, XGBoost, TensorFlow, PyTorch models in production
You have traditional ML models (classification, regression, ranking) deployed and need to track their performance.
**Quick Start:** [Simple ML Monitoring →](/developers/quick-starts/simple-ml-monitoring) ⏱️ 10 min
**What you'll monitor:**
* Model performance (accuracy, precision, recall)
* Data drift and distribution shifts
* Feature importance
* Prediction analytics
* Custom business metrics
***
### 🧪 Evaluate & Test LLM Applications
**Best for:** Pre-deployment testing, A/B testing, regression testing
You want to systematically evaluate LLM quality before deployment or compare different prompts/models.
**Quick Start:** [Experiments →](/developers/quick-starts/experiments-quick-start) ⏱️ 15 min
**What you'll evaluate:**
* Response accuracy and relevance
* Semantic similarity
* Custom domain-specific metrics
* Safety and bias
* RAG-specific metrics (faithfulness, context relevance)
***
### 🛡️ Add Safety Guardrails
**Best for:** Protecting LLM applications from harmful content, PII leaks, and hallucinations
You need real-time protection to prevent your LLM from generating harmful, sensitive, or incorrect content.
**Quick Start:** [Guardrails →](/developers/quick-starts/guardrails-quick-start) ⏱️ 10 min
**What you'll protect against:**
* Harmful and toxic content
* PII leaks (emails, SSNs, credit cards)
* Hallucinations and unsupported claims
* Jailbreak attempts
* Content policy violations
***
## 🤔 Not Sure Where to Start?
### If you're building with AI agents:
Start with [**Agentic Observability**](/developers/quick-starts/langgraph-sdk-quick-start) - it covers everything you need for multi-step workflows.
### If you're using LLMs for simple tasks:
Start with [**Simple LLM Monitoring**](/developers/quick-starts/simple-llm-monitoring) - perfect for chat, Q\&A, and content generation.
### If you have traditional ML models:
Start with [**Simple ML Monitoring**](/developers/quick-starts/simple-ml-monitoring) - track performance and drift for any ML model.
### If you want to test before deploying:
Start with [**Experiments**](/developers/quick-starts/experiments-quick-start) - build confidence with systematic testing.
### If you need to protect your users:
Start with [**Guardrails**](/developers/quick-starts/guardrails-quick-start) - add safety checks in minutes.
***
## 🚀 Quick Comparison
| Use Case | Monitoring Type | Best Quick Start | Time |
| ------------------------------------------------ | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ |
| Multi-agent systems, complex workflows | Agentic | [LangGraph](/developers/quick-starts/langgraph-sdk-quick-start) / [Strands](/developers/quick-starts/strands-agent-quick-start) / [OTeL](/developers/quick-starts/opentelemetry-quick-start) | 10 min |
| Simple chatbots, Q\&A, content generation | LLM | [Simple LLM Monitoring](/developers/quick-starts/simple-llm-monitoring) | 15 min |
| Classification, regression, ranking models | ML | [Simple ML Monitoring](/developers/quick-starts/simple-ml-monitoring) | 10 min |
| Pre-deployment testing, A/B testing | Experiments | [Experiments](/developers/quick-starts/experiments-quick-start) | 15 min |
| Safety, PII protection, hallucination prevention | Guardrails | [Guardrails](/developers/quick-starts/guardrails-quick-start) | 10 min |
***
## 📚 After Your Quick Start
Once you've completed a quick start:
1. **Explore the UI** - View your dashboards, metrics, and insights
2. **Set Up Alerts** - Get notified when issues occur
3. **Customize Metrics** - Add domain-specific monitoring
4. **Read Advanced Guides** - Deep dive into specific features
5. **Join the Community** - Get help and share best practices
***
## 💡 Pro Tips
* **Start Simple**: Pick one quick start, complete it fully, then expand
* **Use Notebooks**: Most quick starts have Colab notebooks for hands-on learning
* **Test Data First**: Use sample data before connecting production systems
* **Monitor + Evaluate**: Combine monitoring with evaluation for comprehensive coverage
* **Layer Guardrails**: Add safety checks on both inputs and outputs
***
## Need Help?
* **Documentation**: Browse our [complete documentation](/)
* **Getting Started Guides**: Read conceptual overviews for [Agentic](/getting-started/agentic-monitoring), [LLM](/getting-started/llm-monitoring), [ML](/getting-started/ml-observability), [Experiments](/getting-started/experiments), or [Guardrails](/getting-started/guardrails)
* **Support**: Contact your Fiddler team or \<[support@fiddler.ai](mailto:support@fiddler.ai)>
***
# Google ADK SDK Quick Start
Source: https://docs.fiddler.ai/developers/quick-starts/google-adk-quick-start
Learn how to integrate Google ADK agents with Fiddler using the Fiddler ADK SDK for automatic instrumentation and comprehensive observability.
## What You'll Learn
In this guide, you'll learn how to:
* Set up a Fiddler application for monitoring Google ADK agents
* Install and configure the Fiddler ADK SDK
* Instrument ADK agents with automatic telemetry
* Run a multi-turn agent with tool calls
* Verify monitoring is working correctly
**Time to complete**: \~10 minutes
## Prerequisites
Before you begin, ensure you have:
* **Fiddler Account**: An active account with [access](/reference/access-control/role-based-access) to create applications
* **Python 3.10+**: Verify your version:
```bash theme={null}
python --version
```
* **Fiddler ADK SDK**: Install the SDK:
```bash theme={null}
# Using uv (recommended)
uv add fiddler-adk
# Or using pip
pip install fiddler-adk
```
* **Google Gemini Access**: Either a Gemini API key or Vertex AI credentials:
```bash theme={null}
# Option A: Gemini API key
export GOOGLE_API_KEY=
# Option B: Vertex AI
export GOOGLE_GENAI_USE_VERTEXAI=1
export GOOGLE_CLOUD_PROJECT=
export GOOGLE_CLOUD_LOCATION=us-central1
```
If you prefer using a notebook, download it directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Google_ADK_Integration.ipynb) or open it in [Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Google_ADK_Integration.ipynb) to get started.
First, create a dedicated application in Fiddler to receive your agent traces.
1. Sign in to your Fiddler instance
2. Navigate to **GenAI Applications** in the left sidebar
3. Click **Add Application**
4. Enter the application details:
* **Name**: `google-adk-monitoring`
* **Project**: Select a project from the dropdown or press *Enter* to create a new one
5. Click **Create** and copy the **Application UUID** (you'll need this for configuration)
Set up the required credentials. You need your Fiddler API key (from Settings) and the Application UUID from Step 1.
Instructions for generating your API key can be found in the [Access guide](/reference/administration/settings#credentials).
```bash theme={null}
# Fiddler credentials
export FIDDLER_URL="https://your-fiddler-instance.com"
export FIDDLER_API_KEY=""
export FIDDLER_APPLICATION_ID=""
# Google Gemini credentials (choose one)
export GOOGLE_API_KEY=""
# or for Vertex AI:
# export GOOGLE_GENAI_USE_VERTEXAI=1
# export GOOGLE_CLOUD_PROJECT=""
```
**Tip**: Save these in a `.env` file and load with `python-dotenv` for easy reuse.
Add two lines to your application to enable Fiddler monitoring:
```python theme={null}
import os
from fiddler_otel import FiddlerClient
from fiddler_adk import GoogleADKInstrumentor
# Initialize Fiddler client
client = FiddlerClient(
api_key=os.environ["FIDDLER_API_KEY"],
application_id=os.environ["FIDDLER_APPLICATION_ID"],
url=os.environ["FIDDLER_URL"],
)
# Enable automatic instrumentation
GoogleADKInstrumentor(client).instrument()
```
That's it. All ADK agents created after this point are automatically traced.
Create an ADK agent with tools and run it:
```python theme={null}
import asyncio
from google.adk.agents.llm_agent import Agent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types
# Define a tool
def get_weather(city: str) -> dict:
"""Get current weather for a city."""
return {"city": city, "temp_f": 72, "condition": "sunny"}
# Create agent
agent = Agent(
model="gemini-2.5-flash",
name="weather_agent",
description="A helpful weather assistant",
instruction="You help users check the weather. Be brief.",
tools=[get_weather],
)
async def main():
session_service = InMemorySessionService()
runner = Runner(
agent=agent, app_name="demo", session_service=session_service
)
session = await session_service.create_session(
app_name="demo", user_id="user1"
)
# Send a message
message = types.Content(
role="user",
parts=[types.Part(text="What's the weather in San Francisco?")],
)
async for event in runner.run_async(
user_id="user1", session_id=session.id, new_message=message
):
if hasattr(event, "content") and event.content:
parts = getattr(event.content, "parts", []) or []
text = "".join(
p.text for p in parts if getattr(p, "text", None)
)
if text:
print(f"Agent: {text}")
# Flush traces before exit
client.force_flush(timeout_millis=5000)
asyncio.run(main())
```
After running your agent:
1. Navigate to your Fiddler instance
2. Open the **GenAI Applications** page
3. Click on your application (`google-adk-monitoring`)
4. Open the **Explorer**
You should see a trace tree with:
* **Agent** span (`invoke_agent`) - the agent execution
* **LLM** span (`call_llm`) - the Gemini API call with input/output
* **Tool** span (`execute_tool get_weather`) - the tool call with arguments and response
Each LLM span shows the user's question, the model's response, system instructions, and token usage.
## Troubleshooting
### No Traces Appearing
* Verify `client.force_flush()` is called before the process exits
* Check for 401 errors in logs (wrong API key)
* Ensure the Application UUID matches an existing GenAI Application
### Gemini API Errors
* Verify `GOOGLE_API_KEY` or Vertex AI credentials are set
* For Vertex AI, ensure `gcloud auth application-default login` has been run
## Configuration Options
### Supported Models
Any Gemini model supported by Google ADK:
* `gemini-2.5-flash`, `gemini-2.5-pro`
* `gemini-2.0-flash`
* `gemini-1.5-flash`, `gemini-1.5-pro`
### Supported ADK Versions
* `google-adk >= 1.34.2` (both 1.x and 2.x lines)
* `opentelemetry-api >= 1.37.0`
## Next Steps
* [**Google ADK Integration Guide**](/integrations/agentic-ai/google-adk-sdk) - Full integration documentation
* [**Fiddler Evals SDK**](/integrations/agentic-ai/evals-sdk) - Evaluate agent quality
* [**Google ADK SDK Reference**](/sdk-api/adk/google-adk-instrumentor) - API documentation
# Guardrails Quick Start
Source: https://docs.fiddler.ai/developers/quick-starts/guardrails-quick-start
Get started with Fiddler Guardrails to protect your LLM applications from harmful content, PII leaks, and hallucinations
Fiddler Guardrails provide real-time protection for your LLM applications by detecting and preventing harmful content, PII leaks, and hallucinations before they reach your users.
**Time to complete**: \~15 minutes
## What You'll Learn
* How to set up Fiddler Guardrails
* How to use the four main guardrail types (Safety, PII, Secret Detection, Faithfulness)
* How to interpret risk scores
* How to integrate guardrails into your LLM application
## Prerequisites
* **Fiddler Environment**: Access to a Fiddler environment with Guardrails enabled
* **API Key**: Generated from your Fiddler environment (**Settings → [Credentials](/reference/administration/settings#credentials)**)
* **Python 3.8+** (or any HTTP client)
***
## Quick Start: Setting Up Guardrails
### Step 1: Get Your API Key
1. Sign in to your organization's Fiddler environment
2. Go to **Settings → [Credentials](/reference/administration/settings#credentials)**
3. Generate a Fiddler API key and copy it for use in the requests below
If you are not sure whether Guardrails is enabled for your environment, contact your Fiddler representative. For access setup, see the [Guardrails Quick Start](/protection/guardrails-quick-start).
### Step 2: Install Required Libraries (Optional)
```bash theme={null}
# For Python
pip install requests
# Or use any HTTP client in your preferred language
```
### Step 3: Configure Your Connection
```python theme={null}
import requests
import json
# Your API credentials
FIDDLER_URL = "https://your-instance.fiddler.ai" # Replace with your Fiddler instance URL
API_KEY = "your-api-key-here"
# Standard headers for all guardrail requests
HEADERS = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"
}
```
***
## Guardrail Types and Usage
Each guardrail type has its own endpoint and request/response format. Choose the appropriate guardrail based on your protection needs.
### 🛡️ Safety Guardrails
Detect harmful, toxic, or jailbreaking content across 11 safety dimensions.
**Endpoint**: `/v3/guardrails/ftl-safety`
**Use cases:**
* Content moderation
* Jailbreak prevention
* Toxic content detection
#### Example: Check for Harmful Content
```python theme={null}
def check_safety(text):
"""Check text for safety violations across 11 dimensions."""
response = requests.post(
f"{FIDDLER_URL}/v3/guardrails/ftl-safety",
headers=HEADERS,
json={"data": {"input": text}}
)
return response.json()
# Test with potentially harmful content
test_input = "I want to cause harm to others"
safety_scores = check_safety(test_input)
print(json.dumps(safety_scores, indent=2))
```
**Response Format:**
```json theme={null}
{
"fdl_harmful": 0.92,
"fdl_violent": 0.78,
"fdl_unethical": 0.65,
"fdl_illegal": 0.45,
"fdl_sexual": 0.12,
"fdl_racist": 0.08,
"fdl_jailbreaking": 0.23,
"fdl_harassing": 0.54,
"fdl_hateful": 0.71,
"fdl_sexist": 0.15
}
```
**Safety Dimensions:**
* `fdl_harmful` - General harmful content
* `fdl_violent` - Violence and threats
* `fdl_unethical` - Unethical behavior
* `fdl_illegal` - Illegal activities
* `fdl_sexual` - Sexual content
* `fdl_racist` - Racist content
* `fdl_jailbreaking` - Prompt manipulation attempts
* `fdl_harassing` - Harassment
* `fdl_hateful` - Hateful content
* `fdl_sexist` - Sexist content
* `fdl_roleplaying` - Prompting persona change
#### Interpreting Safety Scores
Starting in release 26.13, `0.5` is the calibrated default decision threshold for the Centor Model for Safety. The threshold values in these examples are illustrative — tune them for your data and risk tolerance. See [safety threshold guidance](/protection/guardrails#safety).
Each dimension returns a score between 0 and 1:
* **0.0 - 0.3**: Low risk (safe to proceed)
* **0.3 - 0.5**: Elevated risk, below the calibrated `0.5` decision threshold (review recommended)
* **0.5 - 1.0**: At or above the calibrated `0.5` decision threshold (block or flag for review)
```python theme={null}
def is_safe(safety_scores, threshold=0.5):
"""
Check if content passes safety check.
Returns True if ALL dimensions are below threshold.
"""
dangerous_dimensions = [
dim for dim, score in safety_scores.items()
if score >= threshold
]
if dangerous_dimensions:
print(f"⚠️ Safety violations: {dangerous_dimensions}")
return False
print("✅ Content passed safety check")
return True
# Use in your application
if is_safe(safety_scores):
# Proceed with LLM processing
pass
else:
# Block or flag content
print("Content blocked due to safety concerns")
```
**→** [**Safety Guardrails Tutorial**](/developers/tutorials/guardrails/guardrails-safety)
***
### 🔒 PII Detection
Detect personally identifiable information (PII), protected health information (PHI), and custom sensitive data.
**Endpoint**: `/v3/guardrails/sensitive-information`
**Use cases:**
* Data privacy compliance
* GDPR/CCPA protection
* Sensitive data redaction
#### Example 1: Detect PII
```python theme={null}
def detect_pii(text, entity_categories="PII"):
"""
Detect sensitive information in text.
Args:
text: Input text to analyze
entity_categories: "PII", "PHI", "Custom Entities", or list like ["PII", "PHI"]
"""
payload = {
"data": {
"input": text,
"entity_categories": entity_categories
}
}
response = requests.post(
f"{FIDDLER_URL}/v3/guardrails/sensitive-information",
headers=HEADERS,
json=payload
)
return response.json()
# Test with PII data
test_text = """
Contact John Doe at john.doe@email.com or call (555) 123-4567.
SSN: 123-45-6789. Credit card: 4111-1111-1111-1111.
"""
pii_results = detect_pii(test_text)
print(json.dumps(pii_results, indent=2))
```
**Response Format:**
```json theme={null}
{
"fdl_sensitive_information_scores": [
{
"score": 0.987,
"label": "person",
"text": "John Doe",
"start": 8,
"end": 16
},
{
"score": 0.998,
"label": "email",
"text": "john.doe@email.com",
"start": 20,
"end": 38
},
{
"score": 0.991,
"label": "social security number",
"text": "123-45-6789",
"start": 72,
"end": 83
}
]
}
```
**Response Fields:**
* `score` - Confidence score (0.0 to 1.0)
* `label` - Entity type (e.g., "email", "social security number")
* `text` - The detected sensitive information
* `start` / `end` - Character positions in the input text
#### Example 2: Detect PHI (Healthcare Data)
```python theme={null}
# Detect protected health information
healthcare_text = """
Patient John Smith prescribed metformin for diabetes.
Insurance number: HI-987654321.
"""
phi_results = detect_pii(healthcare_text, entity_categories="PHI")
# Display detected PHI entities
for entity in phi_results.get("fdl_sensitive_information_scores", []):
print(f"Found {entity['label']}: '{entity['text']}' (confidence: {entity['score']:.3f})")
```
#### Example 3: Custom Entity Detection
```python theme={null}
# Detect organization-specific sensitive data
custom_text = "Employee ID: EMP-2024-001, API key: sk-abc123xyz789"
custom_results = detect_pii(
custom_text,
entity_categories="Custom Entities"
)
# Note: For custom entities, you can also specify the entity types:
payload = {
"data": {
"input": custom_text,
"entity_categories": "Custom Entities",
"custom_entities": ["employee id", "api key", "project code"]
}
}
```
**Supported Entity Categories:**
* **PII**: comprehensive coverage including names, addresses, SSN, credit cards, emails, and phone numbers
* **PHI**: healthcare-specific types including medication, medical conditions, and health insurance numbers
* **Custom Entities**: Define your own sensitive data patterns
#### Processing PII Results
```python theme={null}
def redact_pii(text, pii_results):
"""Redact detected PII from text."""
entities = pii_results.get("fdl_sensitive_information_scores", [])
# Sort by position in reverse to maintain correct offsets
entities_sorted = sorted(entities, key=lambda x: x['start'], reverse=True)
redacted_text = text
for entity in entities_sorted:
redacted_text = (
redacted_text[:entity['start']] +
f"[REDACTED {entity['label'].upper()}]" +
redacted_text[entity['end']:]
)
return redacted_text
# Use in your application
if pii_results.get("fdl_sensitive_information_scores"):
clean_text = redact_pii(test_text, pii_results)
print(f"Redacted: {clean_text}")
```
**→** [**PII Detection Tutorial**](/developers/tutorials/guardrails/guardrails-pii)
***
### 🔑 Secret Detection
Detect credentials, API keys, and tokens across \~42 known formats plus unknown secret-like strings.
**Endpoint**: `/v3/guardrails/secret-detection`
**Use cases:**
* Prevent credentials from leaking through LLM prompts or responses
* Detect and redact secrets before they are logged or forwarded
#### Example: Detect Secrets
```python theme={null}
def detect_secrets(text):
"""Detect credentials and API keys in text."""
response = requests.post(
f"{FIDDLER_URL}/v3/guardrails/secret-detection",
headers=HEADERS,
json={"data": {"input": text}}
)
return response.json()
# Test with an API key in the prompt
test_input = "Use this key to call the API: sk-ant-api03-abcdefghijklmnopqrstu"
secret_results = detect_secrets(test_input)
print(json.dumps(secret_results, indent=2))
```
**Response Format:**
```json theme={null}
{
"fdl_secret_detection_scores": [
{
"label": "Anthropic API Key",
"start": 30,
"end": 64
}
]
}
```
**Response Fields:**
* `label` - Secret type (e.g., `"Anthropic API Key"`, `"AWS Access Key ID"`, or `"Possible Secret"` for secrets that don't match a known format)
* `start` / `end` - Character positions for precise redaction
#### Processing Secret Results
```python theme={null}
def redact_secrets(text, secret_results):
"""Redact detected secrets from text."""
secrets = secret_results.get("fdl_secret_detection_scores", [])
# Apply right-to-left to preserve offsets
for secret in sorted(secrets, key=lambda s: s["start"], reverse=True):
label = secret["label"].upper().replace(" ", "_")
text = (
text[: secret["start"]] +
f"[REDACTED {label}]" +
text[secret["end"] :]
)
return text
# Use in your application
if secret_results.get("fdl_secret_detection_scores"):
clean_text = redact_secrets(test_input, secret_results)
print(f"Redacted: {clean_text}")
# Redacted: Use this key to call the API: [REDACTED ANTHROPIC_API_KEY]
```
**→** [**Secret Detection Tutorial**](/developers/tutorials/guardrails/guardrails-secrets)
***
### ✅ Faithfulness Detection
Detect hallucinations and unsupported claims by comparing LLM outputs to source context (for RAG applications) using Fiddler Centor Models.
**Endpoint**: `/v3/guardrails/ftl-response-faithfulness`
This guardrail uses the **Centor Model for Faithfulness** for real-time content blocking. For RAG pipeline diagnostics using the LLM-as-a-Judge approach, see [RAG Health Metrics](/concepts/rag-health-diagnostics).
**Use cases:**
* RAG application accuracy
* Fact-checking
* Hallucination prevention
#### Example: Check Response Faithfulness
```python theme={null}
def check_faithfulness(llm_response, source_context):
"""
Check if LLM response is faithful to the provided context.
Args:
llm_response: The text generated by your LLM
source_context: The reference text from your knowledge base/retrieval
"""
payload = {
"data": {
"response": llm_response,
"context": source_context
}
}
response = requests.post(
f"{FIDDLER_URL}/v3/guardrails/ftl-response-faithfulness",
headers=HEADERS,
json=payload
)
return response.json()
# Test with RAG example
retrieved_context = """
The Eiffel Tower is located in Paris, France. It was completed in 1889
and stands 330 meters tall. It was designed by Gustave Eiffel.
"""
llm_response_correct = "The Eiffel Tower in Paris is 330 meters tall and was completed in 1889."
llm_response_hallucinated = "The Eiffel Tower in Paris is 450 meters tall and was completed in 1895."
# Check faithful response
faithful_score = check_faithfulness(llm_response_correct, retrieved_context)
print(f"Faithful response score: {faithful_score}")
# Check hallucinated response
hallucinated_score = check_faithfulness(llm_response_hallucinated, retrieved_context)
print(f"Hallucinated response score: {hallucinated_score}")
```
**Response Format:**
```json theme={null}
{
"fdl_faithful_score": 0.92
}
```
**Score Interpretation:**
* **0.0 - 0.3**: Low faithfulness (likely hallucination)
* **0.3 - 0.7**: Medium faithfulness (review recommended)
* **0.7 - 1.0**: High faithfulness (response is well-supported by context)
```python theme={null}
def is_faithful(faithfulness_result, threshold=0.5):
"""Check if response is faithful to context. 0.5 is the recommended default threshold."""
score = faithfulness_result.get("fdl_faithful_score", 0.0)
if score >= threshold:
print(f"✅ Response is faithful (score: {score:.3f})")
return True
else:
print(f"⚠️ Possible hallucination detected (score: {score:.3f})")
return False
# Use in your RAG application
if not is_faithful(faithful_score):
print("Warning: LLM response may contain unsupported claims")
```
**→** [**Faithfulness Tutorial**](/developers/tutorials/guardrails/guardrails-faithfulness)
***
## Common Integration Patterns
### Pattern 1: Pre-Processing (Input Guardrails)
Check user input before sending to your LLM:
```python theme={null}
def process_user_input(user_message):
"""Process and validate user input before LLM processing."""
# Step 1: Check safety
safety_scores = check_safety(user_message)
# Block if any safety dimension exceeds threshold
max_safety_score = max(safety_scores.values())
if max_safety_score >= 0.5:
return {
"error": "Your message contains inappropriate content.",
"blocked": True
}
# Step 2: Check for secrets and redact if found
secret_results = detect_secrets(user_message)
if secret_results.get("fdl_secret_detection_scores"):
user_message = redact_secrets(user_message, secret_results)
print(f"⚠️ Secret detected and redacted")
# Step 3: Check for PII and redact if needed
pii_results = detect_pii(user_message)
if pii_results.get("fdl_sensitive_information_scores"):
# Redact PII before sending to LLM
user_message = redact_pii(user_message, pii_results)
print(f"⚠️ PII detected and redacted")
# Step 4: Proceed with LLM call
return {
"message": user_message,
"blocked": False
}
# Example usage
user_input = "My SSN is 123-45-6789. Can you help me?"
result = process_user_input(user_input)
if not result.get("blocked"):
# Safe to send to LLM
llm_response = call_your_llm(result["message"])
```
### Pattern 2: Post-Processing (Output Guardrails)
Check LLM output before returning to user:
```python theme={null}
def validate_llm_output(llm_response, retrieval_context=None):
"""Validate LLM output before returning to user."""
# Step 1: Check for PII in output
pii_results = detect_pii(llm_response)
if pii_results.get("fdl_sensitive_information_scores"):
# Redact any PII in the response
llm_response = redact_pii(llm_response, pii_results)
print("⚠️ PII detected in LLM output and redacted")
# Step 2: Check faithfulness (for RAG applications)
if retrieval_context:
faithfulness_result = check_faithfulness(llm_response, retrieval_context)
if not is_faithful(faithfulness_result, threshold=0.5):
return {
"response": llm_response,
"warning": "This response may contain information not supported by source documents."
}
return {
"response": llm_response,
"warning": None
}
# Example usage in RAG application
context = retrieve_from_knowledge_base(user_query)
llm_output = generate_llm_response(user_query, context)
validated = validate_llm_output(llm_output, context)
if validated.get("warning"):
print(f"⚠️ {validated['warning']}")
return validated["response"]
```
### Pattern 3: Complete LLM Pipeline with Multiple Guardrails
```python theme={null}
def safe_llm_pipeline(user_input, use_rag=True):
"""Complete LLM pipeline with comprehensive guardrails."""
# === INPUT GUARDRAILS ===
# 1. Safety check
safety_scores = check_safety(user_input)
if max(safety_scores.values()) >= 0.5:
return {"error": "Inappropriate content detected", "blocked": True}
# 2. Secret detection and redaction
secret_input = detect_secrets(user_input)
if secret_input.get("fdl_secret_detection_scores"):
user_input = redact_secrets(user_input, secret_input)
# 3. PII detection and redaction
pii_input = detect_pii(user_input)
if pii_input.get("fdl_sensitive_information_scores"):
user_input = redact_pii(user_input, pii_input)
# === LLM PROCESSING ===
context = None
if use_rag:
context = retrieve_from_knowledge_base(user_input)
llm_response = generate_llm_response(user_input, context)
# === OUTPUT GUARDRAILS ===
# 4. PII detection in output
pii_output = detect_pii(llm_response)
if pii_output.get("fdl_sensitive_information_scores"):
llm_response = redact_pii(llm_response, pii_output)
# 5. Faithfulness check (for RAG)
warning = None
if use_rag and context:
faithfulness = check_faithfulness(llm_response, context)
if faithfulness.get("fdl_faithful_score", 0) < 0.7:
warning = "Response may contain unsupported claims"
return {
"response": llm_response,
"warning": warning,
"blocked": False
}
```
***
## Best Practices
1. **Layer Multiple Guardrails**: Use safety + PII for inputs, faithfulness + PII for outputs
2. **Set Appropriate Thresholds**: Adjust risk score thresholds based on your use case sensitivity
3. **Log All Checks**: Track guardrail results for monitoring and continuous improvement
4. **Handle Gracefully**: Provide helpful user-facing messages when content is blocked
5. **Monitor Performance**: Track false positives/negatives and adjust thresholds accordingly
6. **Consider Latency**: Guardrail checks add \~100-300ms - use async calls when possible
7. **Mind deployment capacity**: Guardrail throughput is governed by the resources provisioned for your Fiddler deployment; batch and parallelize within that capacity
***
## Error Handling
```python theme={null}
def safe_guardrail_check(guardrail_func, *args, **kwargs):
"""Wrapper for safe guardrail execution with error handling."""
try:
response = guardrail_func(*args, **kwargs)
return response, None
except requests.exceptions.HTTPError as e:
if e.response.status_code == 401:
error = "Authentication failed. Check your API key."
elif e.response.status_code == 413:
error = "Input exceeds token length limit."
elif e.response.status_code == 429:
error = "Rate limit exceeded. Please retry later."
else:
error = f"HTTP error: {e.response.status_code}"
return None, error
except requests.exceptions.Timeout:
return None, "Request timed out."
except Exception as e:
return None, f"Unexpected error: {str(e)}"
# Usage
safety_result, error = safe_guardrail_check(check_safety, user_input)
if error:
print(f"Guardrail check failed: {error}")
# Fallback behavior
else:
# Process safety_result
pass
```
***
## Next Steps
* **API Reference**: [Complete Guardrails API Documentation](/sdk-api/guardrails-api-reference)
* **Setup Guide**: [Complete Guardrails Setup](/protection/guardrails-quick-start)
* **Concepts**: [Guardrails Overview](/getting-started/guardrails)
* **Tutorials**:
* [Safety Guardrails Notebook](/developers/tutorials/guardrails/guardrails-safety)
* [PII Detection Notebook](/developers/tutorials/guardrails/guardrails-pii)
* [Secret Detection Notebook](/developers/tutorials/guardrails/guardrails-secrets)
* [Faithfulness Detection Notebook](/developers/tutorials/guardrails/guardrails-faithfulness)
* **FAQ**: [Guardrails Frequently Asked Questions](/protection/guardrails-faq)
* **Monitoring**: [Integrate Guardrails with LLM Monitoring](/protection/guardrails)
***
## Summary
You've learned how to:
* ✅ Use Safety Guardrails to detect harmful content across 11 dimensions
* ✅ Detect and redact PII, PHI, and custom sensitive information
* ✅ Detect and redact credentials, API keys, and tokens
* ✅ Check response faithfulness to prevent hallucinations in RAG applications
* ✅ Integrate multiple guardrails into your LLM pipeline
* ✅ Handle errors and respect rate limits
Each guardrail type uses a different endpoint and response format optimized for its specific protection purpose. Combine multiple guardrails for comprehensive LLM application safety.
# LangChain SDK Quick Start
Source: https://docs.fiddler.ai/developers/quick-starts/langchain-sdk-quick-start
Monitor AI agent behavior in LangChain V1 applications. The Fiddler LangChain SDK provides real-time observability for `create_agent`-based workflows and middleware pipelines.
Instrument your LangChain V1 application with the Fiddler LangChain SDK in under 10 minutes.
## What You'll Learn
By completing this quick start, you'll:
* Set up monitoring for a LangChain V1 (`langchain.agents.create_agent`) application
* Send your first traces to Fiddler
* Verify data collection in the Fiddler dashboard
* Understand basic conversation tracking
**Before you begin, ensure you have**:
* **Python 3.10** or higher (up to Python 3.14)
* **Valid Fiddler account** with access to your instance
* **A LangChain V1 application** built with `langchain.agents.create_agent`
* **Network connectivity** to your Fiddler instance
**Supported Python Versions:**
* Python 3.10-3.14
**Framework Compatibility:**
* **LangChain V1:** `langchain >= 1.0` (the `create_agent` / middleware API)
**Core Dependencies:**
The SDK automatically installs these OpenTelemetry components:
* `opentelemetry-api`
* `opentelemetry-sdk`
* `opentelemetry-instrumentation`
* `opentelemetry-exporter-otlp-proto-http`
* `fiddler-otel` (transitive — provides `FiddlerClient`)
**Set Up Your Fiddler Application**
1. **Create your application in Fiddler**
Log in to your Fiddler instance and navigate to **GenAI Applications**, then click **Add Application** and follow the onboarding wizard to create your application.
2. **Copy your Application ID**
After creating your application, copy the **Application ID** from the GenAI Applications page using the copy icon next to the ID. This must be a valid UUID4 format (for example, `550e8400-e29b-41d4-a716-446655440000`). You'll need this for Step 3.
3. **Get Your Access Token**
Go to **Settings** > **Credentials** and copy your access token. You'll need this for Step 3. Refer to the [documentation](/reference/administration/settings#credentials) for more details.
**Install the Fiddler LangChain SDK**
For the stable release, install the Fiddler LangChain SDK using pip:
```bash theme={null}
pip install fiddler-langchain
```
**Instrument Your Application**
Add the Fiddler LangChain SDK to your application with just a few lines of code. The instrumentor patches `langchain.agents.create_agent` so every agent created afterwards is traced automatically — no per-agent middleware configuration is required.
```python theme={null}
import langchain.agents
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
# Initialize the FiddlerClient. Replace the placeholder values below.
fdl_client = FiddlerClient(
application_id='', # Application ID copied from UI in Step 1
api_key='', # Your access token
url='', # https://your-instance.fiddler.ai
)
# Instrument LangChain V1 — must run BEFORE you call create_agent()
instrumentor = FiddlerLangChainInstrumentor(client=fdl_client)
instrumentor.instrument()
# Use the module attribute so the patched create_agent is called.
# (If you prefer `from langchain.agents import create_agent`, call
# `instrument()` BEFORE that import.)
agent = langchain.agents.create_agent(
model="openai:gpt-4o-mini",
tools=[...],
name="my_agent", # used as the root span label in traces
)
# Your existing LangChain code runs normally
# Traces will automatically be sent to Fiddler
```
**Add Context and Conversation Tracking**
The main goal of context setting is to enrich the telemetry data sent to Fiddler:
```python theme={null}
from fiddler_langchain import set_llm_context, set_conversation_id
import uuid
# Attach descriptive context to a specific model — appears on its LLM spans.
set_llm_context(model, 'Customer support conversation')
# Set a conversation ID for tracking multi-turn conversations
set_conversation_id(str(uuid.uuid4()))
```
**Manual middleware alternative.** If you prefer not to use the auto-instrumentor, pass `middleware=[FiddlerAgentMiddleware(client=client, agent_name="my_agent")]` directly to each `create_agent()` call. See the [FiddlerAgentMiddleware](/sdk-api/langchain/fiddler-agent-middleware) reference for details.
**Need more control?** This Quick Start uses auto-instrumentation, which traces every `create_agent`-based agent automatically. The SDK also supports per-component span attributes (`add_span_attributes`) and session attributes (`add_session_attributes`) for fine-grained metadata tagging.
**Run a Complete Example**
This example requires an OpenAI API key. You can create or find your key on the [API keys page](https://platform.openai.com/api-keys) of your OpenAI account.
Here's a complete working example to verify your setup:
```python theme={null}
import os
import uuid
import langchain.agents
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from fiddler_otel import FiddlerClient
from fiddler_langchain import (
FiddlerLangChainInstrumentor,
set_conversation_id,
set_llm_context,
)
# Initialize the FiddlerClient. Replace the placeholder values below.
fdl_client = FiddlerClient(
application_id='', # Application ID copied from UI in Step 1
api_key='', # Your access token
url='', # https://your-instance.fiddler.ai
)
# Instrument LangChain V1 — must run BEFORE create_agent()
instrumentor = FiddlerLangChainInstrumentor(client=fdl_client)
instrumentor.instrument()
# Build a simple agent
model = ChatOpenAI(model="gpt-4o-mini")
@tool
def echo(text: str) -> str:
"""Echo back the input text."""
return text
agent = langchain.agents.create_agent(
model=model,
tools=[echo],
name="quickstart_agent",
)
# Set descriptive context for this interaction
set_llm_context(model, "Quick start example conversation")
# Generate and set a conversation ID
conversation_id = str(uuid.uuid4())
set_conversation_id(conversation_id)
# Run your agent — automatically instrumented
result = agent.invoke({
"messages": [{"role": "user", "content": "Hello! How are you?"}]
})
print("Response received:", result)
print("Conversation ID:", conversation_id)
```
**Short-lived scripts:** The SDK batches spans in memory and exports them on a schedule. If your script exits before the batch is sent, call `fdl_client.shutdown()` (or use `with FiddlerClient(...) as client:`) to flush pending spans. Long-running servers handle this automatically via `atexit`.
**Verify Monitoring is Working**
1. **Run your application** using the example above or your own instrumented code
2. **Check the Fiddler dashboard:** Navigate to **GenAI Applications** in your Fiddler instance
3. **Confirm active status:** If Fiddler successfully receives telemetry, your application will show as **Active**
**Success Criteria**
You should see:
* Application status changed to **Active** in the Fiddler dashboard
* Trace data appearing within 1-2 minutes of running your example
* Context labels match what you set in your code
* Conversation ID visible in the trace details
**Expected trace hierarchy** for `create_agent`:
```
[Span] quickstart_agent (Agent root - TYPE=agent)
├── [Span] gpt-4o-mini (LLM call - TYPE=llm)
├── [Span] echo (Tool call - TYPE=tool)
└── [Span] gpt-4o-mini (LLM call - TYPE=llm)
```
## Grant Team Access (optional)
Provide access to other users by assigning teams and users to the project that contains your applications. Managing permissions through Teams is recommended as a best practice. For more access control details, refer to the [Teams and Users Guide](/reference/administration/settings#access) and the [Role-based Access Guide](/reference/access-control/role-based-access).
1. Open the Settings page and select the Access tab
2. For both Users and Teams, select the "Edit" option to the right of the name
3. Add appropriate team members with the required permission levels
## Configuration Options
### Basic Configuration
```python theme={null}
from fiddler_otel import FiddlerClient
fdl_client = FiddlerClient(
application_id="your-app-id", # Must be valid UUID4
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
)
```
### Advanced Configuration
#### Customize Limits for High-Volume Applications
Set limits for your events, spans, and associated attributes. This is helpful for tuning reporting data to manageable numbers for highly attributed and/or high-volume applications.
```python theme={null}
from opentelemetry.sdk.trace import SpanLimits
from fiddler_otel import FiddlerClient
custom_limits = SpanLimits(
max_events=64, # Default: 32
max_links=64, # Default: 32
max_span_attributes=64, # Default: 32
max_event_attributes=64, # Default: 32
max_link_attributes=64, # Default: 32
max_span_attribute_length=4096, # Default: 2048
)
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
span_limits=custom_limits,
)
```
#### Sampling Traffic
Set a specific sampling percentage for incoming data.
```python theme={null}
from opentelemetry.sdk.trace import sampling
from fiddler_otel import FiddlerClient
sampler = sampling.TraceIdRatioBased(0.1) # Sample 10% of traces
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
sampler=sampler,
)
```
#### Environment Variables for Batch Processing
Adjust the following environment variables that the FiddlerClient will use when processing the OpenTelemetry traffic.
```python theme={null}
import os
os.environ['OTEL_BSP_MAX_QUEUE_SIZE'] = '500' # Default: 100
os.environ['OTEL_BSP_SCHEDULE_DELAY_MILLIS'] = '500' # Default: 1000
os.environ['OTEL_BSP_MAX_EXPORT_BATCH_SIZE'] = '50' # Default: 10
os.environ['OTEL_BSP_EXPORT_TIMEOUT'] = '10000' # Default: 5000
```
#### Compression Options
The SDK supports data compression to help reduce the overall data volume transmitted over the network. This can help improve network latency.
```python theme={null}
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
from fiddler_otel import FiddlerClient
# Enable gzip compression (default, recommended for production)
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
compression=Compression.Gzip,
)
```
### Environment Variables Reference
Configure OpenTelemetry batch processor behavior through environment variables:
| Variable | Default | Description |
| -------------------------------- | ------- | ------------------------------------------------------ |
| `OTEL_BSP_MAX_QUEUE_SIZE` | `100` | Maximum spans in queue before export |
| `OTEL_BSP_SCHEDULE_DELAY_MILLIS` | `1000` | Delay between batch exports (milliseconds) |
| `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | `10` | Maximum spans exported per batch |
| `OTEL_BSP_EXPORT_TIMEOUT` | `5000` | Export timeout (milliseconds) |
| `FIDDLER_API_KEY` | - | Your Fiddler API key (recommended for production) |
| `FIDDLER_APPLICATION_ID` | - | Your application UUID4 (recommended for production) |
| `FIDDLER_URL` | - | Your Fiddler instance URL (recommended for production) |
### Adding Custom Attributes
`fiddler-langchain` exposes two helpers for tagging traces with business metadata:
```python theme={null}
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
from fiddler_langchain import add_session_attributes, add_span_attributes
# Session attributes — appear on every span in the current invocation
add_session_attributes("user_id", "alice@example.com")
add_session_attributes("environment", "production")
# Span attributes — only appear on spans for this specific component
model = ChatOpenAI(model="gpt-4o-mini")
add_span_attributes(model, department="AI_Engineering", agent_role="flight_assistant")
@tool
def book_flight(from_airport: str, to_airport: str) -> str:
"""Book a flight between two airports."""
return f"Booked {from_airport} -> {to_airport}"
add_span_attributes(book_flight, third_party="acme_air", reward_points=10.2)
```
| Helper | Scope | Attribute behavior |
| ------------------------------------- | --------------------------------------- | ----------------------------------------------------- |
| `add_session_attributes(key, value)` | All spans in the invocation | Emitted as `fiddler.session.user.{key}` on every span |
| `add_span_attributes(node, **kwargs)` | Spans for that model / tool / retriever | Sets a span-level custom attribute with the given key |
## Async Agents
The instrumentation fully supports async agents. Use `agent.ainvoke()` instead of `agent.invoke()` — no additional configuration is needed:
```python theme={null}
import asyncio
import langchain.agents
from langchain_openai import ChatOpenAI
from fiddler_langchain import FiddlerLangChainInstrumentor
from fiddler_otel import FiddlerClient
client = FiddlerClient(api_key="...", application_id="...", url="...")
FiddlerLangChainInstrumentor(client=client).instrument()
agent = langchain.agents.create_agent(
model=ChatOpenAI(model="gpt-4o-mini"),
tools=[...],
name="my_agent",
)
async def main():
result = await agent.ainvoke({"messages": [{"role": "user", "content": "Hello!"}]})
print(result)
asyncio.run(main())
```
## Troubleshooting
### Common Installation Issues
**Problem**: `ModuleNotFoundError: No module named 'fiddler_langchain'`
* **Solution**: Install the package: `pip install fiddler-langchain`
**Problem**: Version conflicts with existing packages
* **Solution**: Use a virtual environment or update conflicting packages
### Common Configuration Issues
**Problem**: `ValueError: application_id must be a valid UUID4`
* **Solution**: Ensure your Application ID is a valid UUID4 format (e.g., `550e8400-e29b-41d4-a716-446655440000`)
**Problem**: `ValueError: URL must have a valid scheme and netloc`
* **Solution**: Ensure your URL includes the protocol (e.g., `https://your-instance.fiddler.ai`)
**Problem**: Agents are created but no spans appear in Fiddler
* **Solution**: Confirm `instrumentor.instrument()` is called **before** `langchain.agents.create_agent()`. The instrumentor patches the module attribute, so agents created prior to instrumentation are not traced. If you use `from langchain.agents import create_agent`, the local name binds to the *unpatched* function — either call `instrument()` before that import, or invoke the function as `langchain.agents.create_agent(...)`.
**Problem**: Debugging trace generation
* **Solution**: Enable console tracer for local debugging:
```python theme={null}
fdl_client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
console_tracer=True # Also prints spans to console; OTLP export to Fiddler still active
)
```
`console_tracer=True` is **additive** — span data is printed to stdout **and** continues to be exported to Fiddler via OTLP. Setting this to `True` does **not** disable or suppress the OTLP export to Fiddler.
### Verification Issues
**Problem**: Application not showing as "Active" in Fiddler
* **Solution**: Check the following:
1. Ensure your application executes instrumented code (the agent must be invoked)
2. Verify that your Fiddler access token and application ID are correct
3. Check network connectivity to your Fiddler instance
4. Enable `console_tracer=True` to see if spans are being generated locally
## Next Steps
Now that your application is instrumented:
1. **Explore the data:** Check your Fiddler dashboard for traces, metrics, and performance insights
2. **Review the SDK reference:** Check the [Fiddler LangChain SDK Reference](/integrations/agentic-ai/langchain-sdk) for complete documentation
3. **Optimize for production:** Review [configuration options](/developers/quick-starts/langchain-sdk-quick-start#configuration-options) for high-volume applications
# LangGraph SDK Quick Start
Source: https://docs.fiddler.ai/developers/quick-starts/langgraph-sdk-quick-start
Monitor AI agent behavior in LangGraph and LangChain applications. Fiddler LangGraph SDK provides real-time observability for GenAI workflows and debugging.
Instrument your LangGraph or LangChain application with the Fiddler LangGraph SDK in under 10 minutes.
## What You'll Learn
By completing this quick start, you'll:
* Set up monitoring for a LangGraph or LangChain application
* Send your first traces to Fiddler
* Verify data collection in the Fiddler dashboard
* Understand basic conversation tracking
**Before you begin, ensure you have**:
* **Python 3.10** or higher (up to Python 3.13)
* **Valid Fiddler account** with access to your instance
* **A LangGraph or LangChain application** ready for instrumentation
* **Network connectivity** to your Fiddler instance
**Supported Python Versions:**
* Python 3.10-3.14
**Framework Compatibility:**
* **LangGraph/LangChain:** >= 0.3.28 and \<= 1.1.0
**Core Dependencies:**
The SDK automatically installs these OpenTelemetry components:
* `opentelemetry-api` (>= 1.19.0, \<= 1.39.1)
* `opentelemetry-sdk` (>= 1.19.0, \<= 1.39.1)
* `opentelemetry-instrumentation` (>= 0.40b0, \<= 0.60b1)
* `opentelemetry-exporter-otlp-proto-http` (>= 1.19.0, \<= 1.39.1)
* `langgraph` (>= 0.3.28, \<= 1.1.0)
* `langchain` (>= 0.3.26)
1. **Create your application in Fiddler**
Log in to your Fiddler instance and navigate to **GenAI Applications**, then click **Add Application** and follow the onboarding wizard to create your application.
2. **Copy your Application ID**
After creating your application, copy the **Application ID** from the GenAI Applications page using the copy icon next to the ID. This must be a valid UUID4 format (for example, `550e8400-e29b-41d4-a716-446655440000`). You'll need this for Step 3.
3. **Get Your Access Token**
Go to **Settings** > **Credentials** and copy your access token. You'll need this for Step 3. Refer to the [documentation](/reference/administration/settings#credentials) for more details.
**Standard Installation**
For the stable release, install the Fiddler LangGraph SDK using pip:
```bash theme={null}
pip install fiddler-langgraph
```
Add the Fiddler LangGraph SDK to your LangGraph or LangChain application with just a few lines of code:
```python theme={null}
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
# Initialize the FiddlerClient. Replace the placeholder values below.
fdl_client = FiddlerClient(
application_id='', # Application ID copied from UI in Step 1
api_key='', # Your API key
url='', # https://your-instance.fiddler.ai
)
# Instrument your application
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
# Invoke your agent here
# You MUST instrument your application BEFORE invoking
# Your existing LangGraph code runs normally
# Traces will automatically be sent to Fiddler
```
**Add Context and Conversation Tracking**
The main goal of context setting is to enrich the telemetry data sent to Fiddler:
```python theme={null}
from fiddler_langgraph import set_llm_context, set_conversation_id
import uuid
# Set descriptive context for LLM processing
set_llm_context(model, 'Customer support conversation')
# Set conversation ID for tracking multi-turn conversations
conversation_id = str(uuid.uuid4())
set_conversation_id(conversation_id)
```
**For AI engineers:** The conversation tracking feature supports multi-turn agent interactions and helps trace decision flows across agent sessions.
**Need more control?** This Quick Start uses auto-instrumentation, which captures traces from LangGraph and LangChain automatically. The SDK also supports **decorator-based** and **manual** instrumentation for custom functions, non-LangGraph code, and fine-grained span control. See [Instrumentation Methods](/integrations/agentic-ai/langgraph-sdk#instrumentation-methods) in the integration guide.
This example requires an OpenAI API key. You can create or find your key on the [API keys page](https://platform.openai.com/api-keys) of your OpenAI account.
Here's a complete working example to verify your setup:
```python theme={null}
import os
import uuid
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor, set_llm_context, set_conversation_id
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
# Initialize the FiddlerClient. Replace the placeholder values below.
fdl_client = FiddlerClient(
application_id='', # Application ID copied from UI in Step 1
api_key='', # Your API key
url='', # https://your-instance.fiddler.ai
)
# Instrument the application
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
# Create your agent
model = ChatOpenAI(model="gpt-4o-mini")
agent = create_react_agent(model, tools=[])
# Set descriptive context for this interaction
set_llm_context(model, "Quick start example conversation")
# Generate and set a conversation ID
conversation_id = str(uuid.uuid4())
set_conversation_id(conversation_id)
# Run your agent - automatically instrumented
result = agent.invoke({
"messages": [{"role": "user", "content": "Hello! How are you?"}]
})
print("Response received:", result)
print("Conversation ID:", conversation_id)
```
**Short-lived scripts:** The SDK batches spans in memory and exports them on a schedule. If your script exits before the batch is sent, call `fdl_client.shutdown()` (or use `with FiddlerClient(...) as client:`) to flush pending spans. Long-running servers handle this automatically via `atexit`. See [Flush and shutdown handling](/integrations/agentic-ai/langgraph-sdk#flush-and-shutdown-handling) for details.
1. **Run your application** using the example above or your own instrumented code
2. **Check the Fiddler dashboard:** Navigate to **GenAI Applications** in your Fiddler instance
3. **Confirm active status:** If Fiddler successfully receives telemetry, your application will show as **Active**
**Success Criteria**
You should see:
* Application status changed to **Active** in the Fiddler dashboard
* Trace data appearing within 1-2 minutes of running your example
* Context labels match what you set in your code
* Conversation ID visible in the trace details
## Grant Team Access (optional)
Provide access to other users by assigning teams and users to the project that contains your applications. Managing permissions through Teams is recommended as a best practice. For more access control details, refer to the [Teams and Users Guide](/reference/administration/settings#access) and the [Role-based Access Guide](/reference/access-control/role-based-access).
1. Open the Settings page and select the Access tab
2. For both Users and Teams, select the "Edit" option to the right of the name
3. Add appropriate team members with the required permission levels
## Configuration Options
### Basic Configuration
```python theme={null}
from fiddler_langgraph import FiddlerClient
fdl_client = FiddlerClient(
application_id="your-app-id", # Must be valid UUID4
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
```
### Advanced Configuration
#### Customize Limits for High-Volume Applications
Set limits for your events, spans, and associated attributes. This is helpful for tuning reporting data to manageable numbers for highly attributed and/or high-volume applications.
```python theme={null}
from opentelemetry.sdk.trace import SpanLimits
from fiddler_langgraph import FiddlerClient
# Custom span limits for high-volume applications
custom_limits = SpanLimits(
max_events=64, # Default: 32
max_links=64, # Default: 32
max_span_attributes=64, # Default: 32
max_event_attributes=64, # Default: 32
max_link_attributes=64, # Default: 32
max_span_attribute_length=4096, # Default: 2048
)
client = FiddlerClient(
application_id="your-app-id", # Must be valid UUID4
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
span_limits=custom_limits,
)
```
#### Sampling Traffic
Set a specific sampling percentage for incoming data.
```python theme={null}
from opentelemetry.sdk.trace import sampling
from fiddler_langgraph import FiddlerClient
# Sampling strategy for production
sampler = sampling.TraceIdRatioBased(0.1) # Sample 10% of traces
client = FiddlerClient(
application_id="your-app-id", # Must be valid UUID4
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
sampler=sampler,
)
```
#### Environment Variables for Batch Processing
Adjust the following environment variables that the FiddlerClient will use when processing the OpenTelemetry traffic.
```python theme={null}
import os
from fiddler_langgraph import FiddlerClient
# Configure batch processing
os.environ['OTEL_BSP_MAX_QUEUE_SIZE'] = '500' # Default: 100
os.environ['OTEL_BSP_SCHEDULE_DELAY_MILLIS'] = '500' # Default: 1000
os.environ['OTEL_BSP_MAX_EXPORT_BATCH_SIZE'] = '50' # Default: 10
os.environ['OTEL_BSP_EXPORT_TIMEOUT'] = '10000' # Default: 5000
client = FiddlerClient(
application_id="your-app-id", # Must be valid UUID4
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
)
```
#### Compression Options
The SDK supports data compression to help reduce the overall data volume transmitted over the network. This can help improve network latency.
```python theme={null}
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
from fiddler_langgraph import FiddlerClient
# Enable gzip compression (default, recommended for production)
client = FiddlerClient(
application_id="your-app-id", # Must be valid UUID4
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
compression=Compression.Gzip,
)
# Disable compression (useful for debugging or local development)
client = FiddlerClient(
application_id="your-app-id", # Must be valid UUID4
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
compression=Compression.NoCompression,
)
# Use deflate compression (alternative to gzip)
client = FiddlerClient(
application_id="your-app-id", # Must be valid UUID4
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
compression=Compression.Deflate,
)
```
### Environment Variables Reference
Configure OpenTelemetry batch processor behavior through environment variables:
| Variable | Default | Description |
| -------------------------------- | ------- | ------------------------------------------------------ |
| `OTEL_BSP_MAX_QUEUE_SIZE` | `100` | Maximum spans in queue before export |
| `OTEL_BSP_SCHEDULE_DELAY_MILLIS` | `1000` | Delay between batch exports (milliseconds) |
| `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | `10` | Maximum spans exported per batch |
| `OTEL_BSP_EXPORT_TIMEOUT` | `5000` | Export timeout (milliseconds) |
| `FIDDLER_API_KEY` | - | Your Fiddler API key (recommended for production) |
| `FIDDLER_APPLICATION_ID` | - | Your application UUID4 (recommended for production) |
| `FIDDLER_URL` | - | Your Fiddler instance URL (recommended for production) |
**Example: Tuning for high-volume applications**
```python theme={null}
import os
os.environ['OTEL_BSP_MAX_QUEUE_SIZE'] = '500'
os.environ['OTEL_BSP_SCHEDULE_DELAY_MILLIS'] = '500'
os.environ['OTEL_BSP_MAX_EXPORT_BATCH_SIZE'] = '50'
os.environ['OTEL_BSP_EXPORT_TIMEOUT'] = '10000'
```
### LangChain Application Support
The SDK supports both LangGraph and LangChain applications. While agent names are automatically extracted from LangGraph applications by the SDK, LangChain applications need the agent name to be explicitly set using the configuration parameter:
```python theme={null}
from langchain_core.output_parsers import StrOutputParser
# Define your LangChain runnable using LangChain Expression Language (LCEL)
chat_app_chain = prompt | llm | StrOutputParser()
# Run with agent name configuration
response = chat_app_chain.invoke({
"input": user_input,
"history": messages,
}, config={"configurable": {"agent_name": "service_chatbot"}})
```
**Important:** If you don't provide an agent name for LangChain applications, it will appear as "UNKNOWN\_AGENT" in the Fiddler UI. All other features, including conversation ID, LLM context, and attribute structure, work the same as with LangGraph.
## Troubleshooting
### Common Installation Issues
**Problem**: `ModuleNotFoundError: No module named 'fiddler_langgraph'`
* **Solution**: Ensure you've installed the correct package: `pip install fiddler-langgraph`
**Problem**: Version conflicts with existing packages
* **Solution**: Use a virtual environment or update conflicting packages
### Common Configuration Issues
**Problem**: `ValueError: application_id must be a valid UUID4`
* **Solution**: Ensure your Application ID is a valid UUID4 format (e.g., `550e8400-e29b-41d4-a716-446655440000`)
**Example:**
```python theme={null}
# ❌ This will fail
client = FiddlerClient(
application_id="invalid-id", # Not a valid UUID4
api_key="your-access-token",
url="https://instance.fiddler.ai"
)
# Raises: ValueError: application_id must be a valid UUID4 string
# ✅ Correct format
client = FiddlerClient(
application_id="550e8400-e29b-41d4-a716-446655440000", # Valid UUID4
api_key="your-access-token",
url="https://instance.fiddler.ai"
)
```
**Problem**: `ValueError: URL must have a valid scheme and netloc`
* **Solution**: Ensure your URL includes the protocol (e.g., `https://your-instance.fiddler.ai`)
**Problem**: Connection errors or timeouts
* **Solution**: Check your network connectivity and Fiddler instance URL
**Example: Handling connection failures gracefully**
```python theme={null}
try:
client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url="https://your-instance.fiddler.ai"
)
instrumentor = LangGraphInstrumentor(client)
instrumentor.instrument()
except Exception as e:
print(f"Fiddler instrumentation failed: {e}")
# Your application continues running without tracing
```
**Problem**: Debugging trace generation
* **Solution**: Enable console tracer for local debugging:
```python theme={null}
fdl_client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
console_tracer=True # Also prints spans to console; OTLP export to Fiddler still active
)
```
`console_tracer=True` is **additive** — span data is printed to stdout **and** continues to be exported to Fiddler via OTLP. Setting this to `True` does **not** disable or suppress the OTLP export to Fiddler. Use it to visually confirm spans are being created during local development.
**Problem**: `ImportError: cannot import name 'LangGraphInstrumentor'`
* **Solution**: Ensure you have the correct import path:
```python theme={null}
from fiddler_langgraph import LangGraphInstrumentor
```
### Verification Issues
**Problem**: Application not showing as "Active" in Fiddler
* **Solution**: Check the following:
1. Ensure your application executes instrumented code
2. Verify that your Fiddler access token and application ID are correct
3. Check network connectivity to your Fiddler instance
4. Enable console tracer to see if traces are being generated locally
## Next Steps
Now that your application is instrumented:
1. **Explore the data:** Check your Fiddler dashboard for traces, metrics, and performance insights
2. **Custom instrumentation:** Use [decorator-based and manual instrumentation](/integrations/agentic-ai/langgraph-sdk#instrumentation-methods) for fine-grained control
3. **Learn advanced features:** See our [Advanced Guide](/developers/tutorials/llm-monitoring/langgraph-sdk-advanced) for multi-agent scenarios and production patterns
4. **Review the SDK reference:** Check the [Fiddler LangGraph SDK Reference](/sdk-api/langgraph/fiddler-client) for complete documentation
5. **Optimize for production:** Review [configuration options](#configuration-options) for high-volume applications
# OpenTelemetry Quick Start
Source: https://docs.fiddler.ai/developers/quick-starts/opentelemetry-quick-start
Integrate custom AI agents and agentic frameworks with Fiddler using OpenTelemetry for comprehensive observability and monitoring in multi-framework environments.
Monitor custom AI agents and multi-framework agentic applications with Fiddler using OpenTelemetry's native instrumentation.
## What You'll Learn
In this guide, you'll learn how to:
* Set up OpenTelemetry tracing for custom agent frameworks
* Configure Fiddler as your OTLP endpoint with proper authentication
* Map agent attributes to Fiddler's semantic conventions
* Create instrumented LLM and tool spans with required attributes
* Verify traces in the Fiddler dashboard
**Time to complete**: \~10-15 minutes
**When to Use Raw OpenTelemetry**
This guide is for advanced scenarios requiring full manual OTLP control:
* **Multi-framework environments** requiring unified observability across different agent frameworks
* **Existing OpenTelemetry infrastructure** where you want to route Fiddler traces through your own OTel pipeline
* **Advanced control** over trace sampling, batch processing, and attribute mapping
**When to Use Fiddler SDKs Instead (recommended for most users):**
* **Custom Python agents** → Use [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) (`@trace` decorator, typed span wrappers, zero OTLP setup)
* **LangChain V1 agents** → Use [Fiddler LangChain SDK](/integrations/agentic-ai/langchain-sdk) (one `instrument()` call)
* **LangGraph agents/workflows** → Use [Fiddler LangGraph SDK](/developers/quick-starts/langgraph-sdk-quick-start) (auto-instrumentation)
* **Strands Agents** → Use [Strands Agents SDK](/developers/quick-starts/strands-agent-quick-start) (native integration)
SDKs provide automatic instrumentation and require significantly less code. Use raw OpenTelemetry only when SDKs don't fit your use case.
## Prerequisites
Before you begin, ensure you have:
* **Fiddler Account**: An active account with a GenAI application created
* **Python 3.10+**
* **OpenTelemetry Packages**:
* `pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http`
* **LLM Provider** (for examples): OpenAI API key or similar
* **Fiddler Access Token**: Get your token from [Settings > Credentials](/reference/administration/settings#creating-api-keys)
For a complete working example with advanced patterns, download the [Advanced OpenTelemetry Notebook](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_OpenTelemetry_Advanced.ipynb) from GitHub or open it in [Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_OpenTelemetry_Advanced.ipynb).
1. **Log in to your Fiddler instance** and navigate to **GenAI Applications**
2. **Select "Add Application"** to create a new application
3. **Copy your Application ID** - This must be a valid UUID4 format (e.g., `550e8400-e29b-41d4-a716-446655440000`)
4. **Get your Access Token** from **Settings** > **Credentials**
**Important:** Keep your Application ID and Access Token secure. You'll need both for the next steps.
Set up your environment to connect to Fiddler's OTLP endpoint:
```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-instance.fiddler.ai"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ,fiddler-application-id="
export OTEL_RESOURCE_ATTRIBUTES="application.id="
```
**Environment Variable Breakdown:**
| Variable | Description | Example |
| ----------------------------- | ------------------------------------- | ---------------------------------------------------------------- |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Your Fiddler instance URL | `https://org.fiddler.ai` |
| `OTEL_EXPORTER_OTLP_HEADERS` | Authentication and app ID headers | `authorization=Bearer sk-...,fiddler-application-id=550e8400...` |
| `OTEL_RESOURCE_ATTRIBUTES` | Resource-level application identifier | `application.id=550e8400-e29b-41d4-a716-446655440000` |
**Python Configuration** (alternative to environment variables):
```python theme={null}
import os
os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'https://your-instance.fiddler.ai'
os.environ['OTEL_EXPORTER_OTLP_HEADERS'] = 'authorization=Bearer ,fiddler-application-id='
os.environ['OTEL_RESOURCE_ATTRIBUTES'] = 'application.id='
```
**Tip:** Store credentials in a `.env` file and use `python-dotenv` for local development:
```python theme={null}
from dotenv import load_dotenv
load_dotenv() # Loads variables from .env file
```
Set up OpenTelemetry with Fiddler's OTLP exporter:
```python theme={null}
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
# Initialize tracer provider
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
# Configure OTLP exporter for Fiddler
otlp_endpoint = os.getenv('OTEL_EXPORTER_OTLP_ENDPOINT') + '/v1/traces'
otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
# Add batch span processor
otlp_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(otlp_processor)
print(f"✅ OpenTelemetry configured with endpoint: {otlp_endpoint}")
```
**What This Does:**
* **TracerProvider**: Manages trace generation
* **OTLPSpanExporter**: Exports spans to Fiddler via OTLP protocol
* **BatchSpanProcessor**: Batches spans for efficient network transmission
**Local Debugging:** Add a console exporter to see traces locally while developing:
```python theme={null}
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
console_exporter = ConsoleSpanExporter()
console_processor = BatchSpanProcessor(console_exporter)
trace.get_tracer_provider().add_span_processor(console_processor)
```
Create instrumented spans for your agent's operations. Fiddler requires specific attributes to properly categorize and visualize your agent traces.
**Required Fiddler Attributes**
**Resource Level** (set via environment variable):
* `application.id` - UUID4 of your Fiddler application
**Span Level** (required for each span):
* `fiddler.span.type` - Type of operation: `"chain"`, `"tool"`, `"llm"`, or `"agent"`
**Recommended: Decorate functions with `start_as_current_span`**
OpenTelemetry's `tracer.start_as_current_span()` works both as a context
manager (`with ... as span:`) and as a **function decorator**. When a span
should wrap an entire function — which is the common case for tools, LLM
calls, and agent entry points — the decorator form is far less verbose: it
starts the span on entry and ends it on return automatically. Inside the
function, call `trace.get_current_span()` to set Fiddler attributes on the
active span.
Reach for the `with` block only when you need a span narrower than a whole
function (for example, instrumenting a single section of a larger function).
Both forms are pure OpenTelemetry and require no Fiddler SDK.
**Example: Simplified Travel Agent**
```python theme={null}
import json
from openai import OpenAI
from opentelemetry import trace
client = OpenAI()
# Reuse the tracer configured in the "Initialize OpenTelemetry" step above
tracer = trace.get_tracer(__name__)
AGENT_NAME = "travel_agent"
AGENT_ID = "travel_agent_v1"
# Define tools
@tracer.start_as_current_span("book_hotel")
def book_hotel_tool(city: str, date: str):
"""Book a hotel in the specified city."""
span = trace.get_current_span()
span.set_attribute("fiddler.span.type", "tool")
span.set_attribute("gen_ai.agent.name", AGENT_NAME) # optional
span.set_attribute("gen_ai.agent.id", AGENT_ID) # optional
# Tool-specific attributes
span.set_attribute("gen_ai.tool.name", "book_hotel")
tool_input = {"city": city, "date": date}
span.set_attribute("gen_ai.tool.input", json.dumps(tool_input))
# Execute tool
result = {"status": "confirmed", "hotel": f"Grand Hotel {city}", "confirmation": "HTL123"}
span.set_attribute("gen_ai.tool.output", json.dumps(result))
return result
@tracer.start_as_current_span("book_flight")
def book_flight_tool(source: str, destination: str, date: str):
"""Book a flight between two cities."""
span = trace.get_current_span()
span.set_attribute("fiddler.span.type", "tool")
span.set_attribute("gen_ai.agent.name", AGENT_NAME) # optional
span.set_attribute("gen_ai.agent.id", AGENT_ID) # optional
# Tool-specific attributes
span.set_attribute("gen_ai.tool.name", "book_flight")
tool_input = {"source": source, "destination": destination, "date": date}
span.set_attribute("gen_ai.tool.input", json.dumps(tool_input))
# Execute tool
result = {"status": "confirmed", "flight": "FL456", "departure": "10:00 AM"}
span.set_attribute("gen_ai.tool.output", json.dumps(result))
return result
@tracer.start_as_current_span("llm_call")
def call_llm(user_request: str):
"""Call the LLM to interpret the request and choose tools."""
span = trace.get_current_span()
span.set_attribute("fiddler.span.type", "llm")
span.set_attribute("gen_ai.agent.name", AGENT_NAME) # optional
span.set_attribute("gen_ai.agent.id", AGENT_ID) # optional
# LLM-specific attributes
span.set_attribute("gen_ai.request.model", "gpt-4o-mini")
span.set_attribute("gen_ai.system", "openai")
span.set_attribute("gen_ai.llm.input.user", user_request)
span.set_attribute(
"gen_ai.llm.input.system",
"You are a travel agent. Parse user requests and call appropriate tools."
)
# Call OpenAI
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a travel agent. Parse user requests and call appropriate tools."},
{"role": "user", "content": user_request}
],
tools=[
{
"type": "function",
"function": {
"name": "book_hotel",
"description": "Book a hotel in a city for a specific date",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string"},
"date": {"type": "string"}
},
"required": ["city", "date"]
}
}
},
{
"type": "function",
"function": {
"name": "book_flight",
"description": "Book a flight between two cities",
"parameters": {
"type": "object",
"properties": {
"source": {"type": "string"},
"destination": {"type": "string"},
"date": {"type": "string"}
},
"required": ["source", "destination", "date"]
}
}
}
]
)
# Set token usage
span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
span.set_attribute("gen_ai.usage.total_tokens", response.usage.total_tokens)
# Process tool calls
tool_results = []
if response.choices[0].message.tool_calls:
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
if tool_name == "book_hotel":
tool_results.append(book_hotel_tool(**tool_args))
elif tool_name == "book_flight":
tool_results.append(book_flight_tool(**tool_args))
span.set_attribute("gen_ai.llm.output",
f"Called tools and received: {tool_results}")
return tool_results
# Agent implementation
@tracer.start_as_current_span("travel_agent_chain")
def travel_agent(user_request: str):
"""Main travel agent function."""
span = trace.get_current_span()
span.set_attribute("fiddler.span.type", "chain")
span.set_attribute("gen_ai.agent.name", AGENT_NAME) # optional
span.set_attribute("gen_ai.agent.id", AGENT_ID) # optional
# call_llm runs as a child span because it is invoked within this span
tool_results = call_llm(user_request)
return {"status": "success", "bookings": tool_results}
# Run the agent
result = travel_agent("Book a hotel in Paris for tomorrow and a flight from London to Paris")
print(f"Agent result: {result}")
```
**Key Implementation Details:**
* **Decorator form**: `@tracer.start_as_current_span("name")` wraps the whole
function; retrieve the span inside with `trace.get_current_span()`
* **Chain Spans**: Use `fiddler.span.type = "chain"` for high-level workflows
* **LLM Spans**: Include model, system prompt, user input, output, and token usage
* **Tool Spans**: Include tool name, input JSON, and output JSON
* **Nested Spans**: A decorated function called from within another decorated
function automatically becomes a child span, building the trace hierarchy
**Simpler alternative:** Install `fiddler-otel` to skip all the manual OTLP configuration and attribute boilerplate. The SDK's `start_as_current_span()` method handles span type enforcement and attribute propagation automatically — no need to set `fiddler.span.type`, `gen_ai.agent.name`, or `gen_ai.agent.id` on every span manually. Typed span wrappers like `FiddlerGeneration` and `FiddlerTool` provide helper methods such as `set_model()` and `set_tool_name()` instead of raw `set_attribute()` calls. See [Manual Instrumentation](/integrations/agentic-ai/fiddler-otel-sdk#manual-instrumentation) in the integration guide.
1. **Run your instrumented code** using the example above
2. **Wait 1-2 minutes** for traces to appear in Fiddler
3. **Navigate to GenAI Applications** in your Fiddler instance
4. **Verify application status** changes to **Active**
5. **View traces** to see your agent spans, hierarchy, and attributes
**Success Criteria:**
✅ Application shows as **Active** in GenAI Applications ✅ Traces appear in the Explorer ✅ Span hierarchy shows chain → LLM → tools relationship ✅ Span type is set on every span ✅ LLM token usage is tracked ✅ Tool inputs and outputs are captured
**Verification Tip:** Check the trace timeline view to see the execution flow of your agent, including which tools were called and how long each operation took.
## Attribute Reference
### Required Attributes
**Resource Level:**
| Attribute | Type | Description | Example |
| ---------------- | ------ | --------------------------------- | ---------------------------------------- |
| `application.id` | string | UUID4 of your Fiddler application | `"550e8400-e29b-41d4-a716-446655440000"` |
**Span Level:**
| Attribute | Type | Description | Valid Values |
| ------------------- | ------ | ----------------- | --------------------------------------- |
| `fiddler.span.type` | string | Type of operation | `"chain"`, `"tool"`, `"llm"`, `"agent"` |
### Optional Attributes
**Agent Identification:**
| Attribute | Type | Description | Example |
| ------------------- | ------ | ------------------------------- | ------------------- |
| `gen_ai.agent.name` | string | Name of the AI agent | `"travel_agent"` |
| `gen_ai.agent.id` | string | Unique identifier for the agent | `"travel_agent_v1"` |
**Set agent attributes on every span.** `gen_ai.agent.name` and `gen_ai.agent.id` are optional, but if you include them, set both on **every span within the trace**. Fiddler uses these attributes to attribute spans to the correct agent — spans missing these fields will be unattributed even if other spans in the same trace carry them.
**Conversation Tracking:**
| Attribute | Type | Description | Example |
| ------------------------ | ------ | ------------------------------- | ------------ |
| `gen_ai.conversation.id` | string | Session/conversation identifier | `"conv_123"` |
**LLM Span Attributes:**
| Attribute | Type | Description | Example |
| ---------------------------- | ------------------- | ------------------------------ | ------------------------------------------ |
| `gen_ai.request.model` | string | Model name | `"gpt-4o-mini"`, `"claude-3-opus"` |
| `gen_ai.system` | string | LLM provider | `"openai"`, `"anthropic"` |
| `gen_ai.llm.input.system` | string | System prompt | `"You are a helpful assistant"` |
| `gen_ai.llm.input.user` | string | User input | `"What's the weather?"` |
| `gen_ai.llm.output` | string | LLM response | `"The weather is sunny"` |
| `gen_ai.usage.input_tokens` | int | Input tokens used | `42` |
| `gen_ai.usage.output_tokens` | int | Output tokens used | `28` |
| `gen_ai.usage.total_tokens` | int | Total tokens used | `70` |
| `gen_ai.input.messages` | string (JSON array) | Chat history provided to model | `[{"role": "user", "content": "Hello"}]` |
| `gen_ai.output.messages` | string (JSON array) | Messages returned by model | `[{"role": "assistant", "content": "Hi"}]` |
**Tool Span Attributes:**
| Attribute | Type | Description | Example |
| -------------------- | ------ | ------------------ | ----------------------- |
| `gen_ai.tool.name` | string | Tool/function name | `"search_database"` |
| `gen_ai.tool.input` | string | Tool input (JSON) | `"{"query": "hotels"}"` |
| `gen_ai.tool.output` | string | Tool output (JSON) | `"{"results": [...]}"` |
**Custom User-Defined Attributes:**
| Scope | Description | Example |
| ------------------------ | ------------------------------------------------------- | ------------------------------------------ |
| `fiddler.session.user.*` | Custom attribute applied to all spans in the invocation | `fiddler.session.user.user_id = "usr_123"` |
| Span attribute | Custom attribute applied to an individual span | `department = "sales"` |
## Troubleshooting
### Common Issues
**Problem:** Application not showing as "Active"
**Solutions:**
1. Verify environment variables are set correctly
2. Check that `OTEL_EXPORTER_OTLP_ENDPOINT` includes your Fiddler instance URL
3. Ensure `OTEL_EXPORTER_OTLP_HEADERS` contains valid authorization token and application ID
4. Add console exporter to verify spans are being generated locally
5. Check network connectivity: `curl -I https://your-instance.fiddler.ai`
**Problem:** `ModuleNotFoundError` for OpenTelemetry packages
**Solutions:**
```bash theme={null}
# Install all required packages
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
# Verify installation
pip list | grep opentelemetry
```
**Problem:** Spans not appearing in Fiddler
**Solutions:**
1. **Verify required attributes are set:**
```python theme={null}
# Every span MUST have this
span.set_attribute("fiddler.span.type", "llm") # or "tool", "chain", "agent"
# Optional but recommended
span.set_attribute("gen_ai.agent.name", "your_agent")
span.set_attribute("gen_ai.agent.id", "agent_id")
```
2. **Check resource attributes:**
```python theme={null}
# Verify application.id is set
print(os.getenv('OTEL_RESOURCE_ATTRIBUTES'))
```
3. **Enable console exporter for debugging:**
```python theme={null}
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
console_exporter = ConsoleSpanExporter()
console_processor = BatchSpanProcessor(console_exporter)
trace.get_tracer_provider().add_span_processor(console_processor)
```
**Problem:** Authentication errors (401 Unauthorized)
**Solutions:**
1. Regenerate your access token from Fiddler Settings > Credentials
2. Verify header format: `authorization=Bearer <token>,fiddler-application-id=<uuid>`
3. Ensure no extra spaces in header values
4. Check token hasn't expired
**Problem:** Invalid Application ID error
**Solutions:**
1. Copy Application ID directly from Fiddler UI
2. Verify UUID4 format: `550e8400-e29b-41d4-a716-446655440000`
3. Ensure no extra quotes or whitespace
## Configuration Options
### Basic Configuration
```python theme={null}
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
# Set environment variables
os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'https://your-instance.fiddler.ai'
os.environ['OTEL_EXPORTER_OTLP_HEADERS'] = 'authorization=Bearer ,fiddler-application-id='
os.environ['OTEL_RESOURCE_ATTRIBUTES'] = 'application.id='
# Initialize tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
# Configure OTLP exporter
otlp_endpoint = os.getenv('OTEL_EXPORTER_OTLP_ENDPOINT') + '/v1/traces'
otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
otlp_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(otlp_processor)
```
### Advanced Configuration
**High-Volume Applications (Batch Processing Tuning):**
```python theme={null}
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# Customize batch processor settings
custom_processor = BatchSpanProcessor(
otlp_exporter,
max_queue_size=500, # Default: 2048
schedule_delay_millis=500, # Default: 5000
max_export_batch_size=50, # Default: 512
export_timeout_millis=10000 # Default: 30000
)
trace.get_tracer_provider().add_span_processor(custom_processor)
```
**Environment Variable Configuration:**
```bash theme={null}
# Batch processor environment variables
export OTEL_BSP_MAX_QUEUE_SIZE=500
export OTEL_BSP_SCHEDULE_DELAY_MILLIS=500
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=50
export OTEL_BSP_EXPORT_TIMEOUT=10000
```
**Sampling for Production (Reduce Volume):**
```python theme={null}
from opentelemetry.sdk.trace import sampling
# Sample 10% of traces
sampler = sampling.TraceIdRatioBased(0.1)
# Create provider with sampler
provider = TracerProvider(sampler=sampler)
trace.set_tracer_provider(provider)
```
**Compression (Reduce Network Usage):**
```python theme={null}
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
# Enable gzip compression
otlp_exporter = OTLPSpanExporter(
endpoint=otlp_endpoint,
compression=Compression.Gzip
)
```
**Using FiddlerClient Alternative (Simplified Setup):**
**Deprecation notice:** Importing `FiddlerClient` and other core symbols (`trace`, `get_current_span`, `FiddlerSpan`, `FiddlerGeneration`, `FiddlerChain`, `FiddlerTool`, `get_client`, `set_conversation_id`) directly from `fiddler_langgraph` is deprecated and will be removed in a future release. Import from `fiddler_otel` instead: `from fiddler_otel import FiddlerClient`.
If you have `fiddler-otel` installed, you can use `FiddlerClient` for simplified setup — it handles OTLP configuration automatically. There are two levels of abstraction:
**Option 1: Get a pre-configured tracer** (use raw OTel spans, but skip OTLP configuration):
```python theme={null}
from fiddler_otel import FiddlerClient
client = FiddlerClient(
application_id='',
api_key='',
url=''
)
tracer = client.get_tracer()
with tracer.start_as_current_span("my_operation") as span:
span.set_attribute("fiddler.span.type", "chain")
# ... rest of your code
```
**Option 2: Use SDK span wrappers** (typed helper methods, automatic attribute propagation):
```python theme={null}
from fiddler_otel import FiddlerClient
client = FiddlerClient(
application_id='',
api_key='',
url=''
)
# Span type and agent attributes are set automatically
with client.start_as_current_span("llm_call", as_type="generation") as gen:
gen.set_model("gpt-4o")
gen.set_user_prompt(user_input)
response = call_llm(user_input)
gen.set_completion(response.content)
gen.set_usage(response.usage.prompt_tokens, response.usage.completion_tokens)
```
Both approaches handle OTLP configuration automatically. For graceful exit (e.g. servers or short scripts), call `client.shutdown()` (or `await client.ashutdown()` in async) so buffered spans are sent before the process exits. See [Manual Instrumentation](/integrations/agentic-ai/fiddler-otel-sdk#manual-instrumentation) for complete span wrapper documentation.
## Next Steps
Now that you have OpenTelemetry integration working:
* **Advanced Patterns:** Download the [Advanced OpenTelemetry Notebook](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_OpenTelemetry_Advanced.ipynb) for:
* Multi-agent configurations
* Conversation tracking across sessions
* Custom user-defined attributes
* Production-ready error handling
* Comprehensive debugging techniques
* **Consider SDKs for Common Frameworks:**
* [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) - `@trace` decorator for custom Python agents (no raw OTel boilerplate)
* [Fiddler LangChain SDK](/integrations/agentic-ai/langchain-sdk) - Auto-instrumentation for LangChain V1 agents
* [Fiddler LangGraph SDK](/developers/quick-starts/langgraph-sdk-quick-start) - Auto-instrumentation for LangGraph/LangChain
* [Strands Agents SDK](/developers/quick-starts/strands-agent-quick-start) - Native Strands agent integration
* **Explore Fiddler Capabilities:**
* [Getting Started: Agentic Observability](/getting-started/agentic-monitoring)
* [OpenTelemetry Integration Overview](/integrations/agentic-ai/opentelemetry-integration)
* [Fiddler Python Client SDK](/sdk-api/python-client/connection)
* **Production Deployment:**
* Review sampling strategies for cost optimization
* Implement error handling and retry logic
* Set up monitoring alerts
* Configure custom attributes for your business context
# Simple LLM Monitoring
Source: https://docs.fiddler.ai/developers/quick-starts/simple-llm-monitoring
Learn the basic onboarding steps to use Fiddler for monitoring LLM applications. Access Google Colab or download the notebook directly from GitHub.
This guide will walk you through the basic onboarding steps required to use Fiddler for monitoring of LLM applications, **using sample data provided by Fiddler**.
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Chatbot.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Chatbot.ipynb).
# Simple ML Monitoring
Source: https://docs.fiddler.ai/developers/quick-starts/simple-ml-monitoring
This document provides a guide for using Fiddler for model monitoring using sample data provided by Fiddler.
This guide will walk you through the basic onboarding steps required to use Fiddler for model monitoring, **using sample data provided by Fiddler**.
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Simple_Monitoring.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Simple_Monitoring.ipynb).
# Strands Agent SDK Quick Start
Source: https://docs.fiddler.ai/developers/quick-starts/strands-agent-quick-start
Learn how to integrate Strands agents with Fiddler using the Fiddler Strands SDK for automatic instrumentation and comprehensive observability of your AI agent workflows.
## What You'll Learn
In this guide, you'll learn how to:
* Set up a Fiddler application for monitoring Strands agents
* Install and configure the Strands Agents SDK
* Instrument Strands agents with automatic telemetry
* Use helper functions to add custom metadata
* Verify monitoring is working correctly
* Troubleshoot common integration issues
**Time to complete**: \~15 minutes
## Prerequisites
Before you begin, ensure you have:
* **Fiddler Account**: An active account with [access](/reference/access-control/role-based-access) to create applications
* **Python 3.10+**: Verify your version:
```bash theme={null}
python --version
```
* **Strands Agents SDK**: Install the SDK (includes Strands agents and OpenTelemetry):
```bash theme={null}
# Using uv (recommended)
uv add fiddler-strands
# Or using pip
pip install fiddler-strands
```
* **OpenAI API Key**: For running the example agent:
```bash theme={null}
export OPENAI_API_KEY=
```
If you prefer using a notebook, download it directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Strands_Agent_Integration.ipynb) or open it in [Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Strands_Agent_Integration.ipynb) to get started.
First, create a dedicated application in Fiddler to receive your agent traces.
1. Sign in to your Fiddler instance
2. Navigate to **GenAI Applications** in the left sidebar
3. Click **Add Application**
4. Enter the application details:
* **Name**: `strands-agent-monitoring`
* **Project**: Select a project from the dropdown or press *Enter* to create a new one
5. Click **Create** and copy the **Application UUID** (you'll need this for configuration)
Set up the required environment variables for Fiddler integration. Replace the placeholder values with your actual credentials.
Instructions for generating or retrieving your personal access token can be found in the [Access guide](/reference/administration/settings#credentials).
```bash theme={null}
# Fiddler URL (adjust domain for your instance)
export OTEL_EXPORTER_OTLP_ENDPOINT="http://demo.fiddler.ai"
# Your application UUID from Step 1
export OTEL_RESOURCE_ATTRIBUTES=application.id=
# Authentication headers (get access token from Fiddler settings)
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ,fiddler-application-id="
# Your OpenAI API key
export OPENAI_API_KEY=
```
**Tip**: Save these environment variables in a `.env` file for easy reuse:
```bash theme={null}
# Save to .env file
cat > .env << 'EOF'
OTEL_EXPORTER_OTLP_ENDPOINT=http://demo.fiddler.ai
OTEL_RESOURCE_ATTRIBUTES=application.id=
OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer ,fiddler-application-id=
OPENAI_API_KEY=
EOF
# Load in your shell
source .env
```
Now configure the Strands telemetry system with automatic Fiddler instrumentation using the SDK.
Create `agent_monitoring.py`:
```python theme={null}
import os
from strands.telemetry import StrandsTelemetry
from fiddler_strandsagents import StrandsAgentInstrumentor
# Initialize Strands telemetry
strands_telemetry = StrandsTelemetry()
# Setup exporters
strands_telemetry.setup_console_exporter() # For local debugging
strands_telemetry.setup_otlp_exporter() # For Fiddler export
# Enable automatic instrumentation with Fiddler integration
StrandsAgentInstrumentor(strands_telemetry).instrument()
```
**Configuration Breakdown**:
* **Console Exporter**: Prints traces to terminal for debugging
* **OTLP Exporter**: Sends traces to Fiddler via OpenTelemetry Protocol
* **StrandsAgentInstrumentor**: Automatically instruments agents with proper attribute propagation and Fiddler integration
**What the SDK Does Automatically**:
* Injects logging hooks into Strands agents
* Propagates agent attributes (name, ID, system prompt) to all child spans
* Processes spans with Fiddler-specific enhancements
* Handles all OpenTelemetry complexity behind the scenes
With telemetry configured, create a Strands agent that's fully instrumented for Fiddler monitoring.
Add to `agent_monitoring.py`:
```python theme={null}
from strands import Agent, tool
from strands.models.openai import OpenAIModel
# Initialize OpenAI model
model = OpenAIModel(
client_args={"api_key": os.getenv("OPENAI_API_KEY")},
model_id="gpt-4o-mini"
)
# Define example tools
@tool
def get_weather(city: str) -> str:
"""Get the current weather for a city.
Args:
city: The name of the city
Returns:
Weather information as a string
"""
# In production, call a real weather API
return f"The weather in {city} is sunny with a temperature of 72°F"
@tool
def search_knowledge_base(query: str) -> str:
"""Search the knowledge base for information.
Args:
query: The search query
Returns:
Relevant information from the knowledge base
"""
# In production, implement actual search logic
return f"Search results for '{query}': [relevant documentation snippets]"
# Create the agent with telemetry enabled
agent = Agent(
model=model,
system_prompt="""You are a helpful assistant that can check weather
and search a knowledge base. Always be concise and accurate in your responses.""",
tools=[get_weather, search_knowledge_base]
)
# Use the agent
if __name__ == "__main__":
# Example 1: Simple query
response = agent("What's the weather like in San Francisco?")
print(f"Response: {response}")
# Example 2: Knowledge base query
response = agent("Search for information about model monitoring")
print(f"Response: {response}")
```
Run your instrumented agent:
```bash theme={null}
python agent_monitoring.py
```
After running your agent, verify that traces are appearing in Fiddler.
1. Navigate to your application in Fiddler
2. Click on the **Traces** tab
3. You should see traces from your agent executions
4. Click on a trace to view detailed span information
5. Verify that agent attributes are present on spans (these are set automatically by the SDK):
* `gen_ai.agent.name` — optional, but if present should appear on all spans in the trace
* `gen_ai.agent.id` — optional, but if present should appear on all spans in the trace
**Success Indicators**:
* ✅ Traces appear within 30 seconds of agent execution
* ✅ Parent and child spans are properly linked
* ✅ Agent attributes (`gen_ai.agent.name`, `gen_ai.agent.id`) appear consistently across all spans
* ✅ Tool calls are captured as separate spans
* ✅ System prompts are visible in trace metadata
## Troubleshooting
### Traces Not Appearing in Fiddler
**Issue**: No traces show up after running your agent.
**Solutions**:
1. **Verify environment variables**:
```bash theme={null}
echo $OTEL_EXPORTER_OTLP_ENDPOINT
echo $OTEL_RESOURCE_ATTRIBUTES
echo $OTEL_EXPORTER_OTLP_HEADERS
```
2. **Check network connectivity**:
```bash theme={null}
curl -I $OTEL_EXPORTER_OTLP_ENDPOINT
```
3. **Validate authentication**:
* Ensure your access token is valid and not expired
* Verify Application UUID matches your Fiddler application
4. **Review console exporter output**:
* Check the terminal for trace output
* Look for error messages in console logs
### Missing Agent Attributes on Child Spans
**Issue**: Tool calls and sub-spans don't have agent context.
**Solutions**:
1. **Verify SDK instrumentation is enabled**:
```python theme={null}
# Should be in your code
from fiddler_strandsagents import StrandsAgentInstrumentor
StrandsAgentInstrumentor(strands_telemetry).instrument()
```
2. **Check instrumentation status**:
```python theme={null}
from fiddler_strandsagents import StrandsAgentInstrumentor
instrumentor = StrandsAgentInstrumentor(strands_telemetry)
print(f"Instrumented: {instrumentor.is_instrumented_by_opentelemetry}")
```
3. **Add custom attributes using helper functions**:
```python theme={null}
from fiddler_strandsagents import set_span_attributes
# Add custom attributes to ensure they appear on spans
set_span_attributes(agent, custom_attr="value")
set_span_attributes(model, environment="production")
```
### OTLP Export Errors
**Issue**: Error messages about OTLP export failures.
**Solutions**:
1. **Check endpoint format**:
```bash theme={null}
# Should be https://, not include /v1/traces
export OTEL_EXPORTER_OTLP_ENDPOINT="http://demo.fiddler.ai"
```
2. **Verify headers format**:
```bash theme={null}
# Headers should be comma-separated key=value pairs
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer TOKEN,fiddler-application-id=UUID"
```
3. **Test with a minimal example**:
```python theme={null}
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("test-span"):
print("Test trace sent")
```
### Performance Issues
**Issue**: Agent response times are slower after adding monitoring.
**Solutions**:
1. **Use batch span processor** (already default in StrandsTelemetry):
```python theme={null}
# Batching reduces overhead
from opentelemetry.sdk.trace.export import BatchSpanProcessor
```
2. **Disable console exporter in production**:
```python theme={null}
# Only use OTLP exporter
# strands_telemetry.setup_console_exporter() # Comment out
strands_telemetry.setup_otlp_exporter()
```
3. **Adjust sampling rate** if needed:
```python theme={null}
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
# Sample 10% of traces
sampler = TraceIdRatioBased(0.1)
```
## Configuration Options
### Basic Configuration
For most use cases, the basic configuration is sufficient:
```python theme={null}
from strands.telemetry import StrandsTelemetry
from fiddler_strandsagents import StrandsAgentInstrumentor
strands_telemetry = StrandsTelemetry()
strands_telemetry.setup_otlp_exporter()
StrandsAgentInstrumentor(strands_telemetry).instrument()
```
### Adding Custom Metadata with Helper Functions
The SDK provides helper functions to enrich your traces with custom business context:
```python theme={null}
from fiddler_strandsagents import (
set_conversation_id,
set_session_attributes,
set_span_attributes,
set_llm_context
)
# Set conversation ID for tracking related interactions
set_conversation_id(agent, 'session_1234567890')
# Add session-level attributes for business context
set_session_attributes(agent,
role='customer_support',
cost_center='travel_desk',
region='us-west'
)
# Add attributes to specific components (models or tools)
set_span_attributes(model, model_id='gpt-4o-mini', temperature=0.7)
set_span_attributes(search_tool, department='search', version='2.0')
# Set LLM context for additional background information
set_llm_context(model, 'Available hotels: Hilton, Marriott, Hyatt...')
```
**Helper Functions Available**:
* `set_conversation_id(agent, conversation_id)` - Track multi-turn conversations
* `set_session_attributes(agent, **kwargs)` - Add session-level business context
* `set_span_attributes(obj, **kwargs)` - Add attributes to models, tools, or agents
* `set_llm_context(model, context)` - Add background information for LLM interactions
* `get_conversation_id(agent)` - Retrieve conversation ID
* `get_session_attributes(agent)` - Retrieve session attributes
* `get_span_attributes(obj)` - Retrieve span attributes
* `get_llm_context(model)` - Retrieve LLM context
### Advanced Configuration
For production deployments with custom resource metadata and batch settings:
```python theme={null}
import os
from strands.telemetry import StrandsTelemetry
from fiddler_strandsagents import StrandsAgentInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
# Create custom resource with additional metadata
resource = Resource.create({
"service.name": "strands-agent-service",
"service.version": "1.0.0",
"deployment.environment": os.getenv("ENVIRONMENT", "production"),
"application.id": os.getenv("FIDDLER_APP_ID"),
})
# Initialize tracer provider with custom resource
provider = TracerProvider(resource=resource)
# Configure OTLP exporter with custom settings
otlp_exporter = OTLPSpanExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
headers={
"authorization": f"Bearer {os.getenv('FIDDLER_BEARER_TOKEN')}",
"fiddler-application-id": os.getenv("FIDDLER_APP_ID"),
},
timeout=10, # Custom timeout in seconds
)
# Add batch processor with custom settings
batch_processor = BatchSpanProcessor(
otlp_exporter,
max_queue_size=2048, # Max spans in queue
max_export_batch_size=512, # Spans per export batch
schedule_delay_millis=5000, # Export every 5 seconds
)
provider.add_span_processor(batch_processor)
# Initialize Strands telemetry with custom provider
strands_telemetry = StrandsTelemetry(tracer_provider=provider)
# Enable Fiddler instrumentation
StrandsAgentInstrumentor(strands_telemetry).instrument()
```
**Advanced Options Explained**:
* **Custom Resource**: Add service metadata for better organization
* **Batch Settings**: Tune for your throughput requirements
* **Timeout Configuration**: Adjust for network conditions
* **Environment Tagging**: Separate dev/staging/prod traces
* **SDK Integration**: Works seamlessly with custom OpenTelemetry configurations
### Multi-Agent Configuration
For systems with multiple agents:
```python theme={null}
from strands import Agent
from strands.models.openai import OpenAIModel
# Create multiple agents with distinct identities
research_agent = Agent(
model=OpenAIModel(model_id="gpt-4o"),
system_prompt="You are a research specialist...",
tools=[search_tool, analyze_tool],
agent_id="research-agent", # Unique identifier
agent_name="Research Specialist"
)
writing_agent = Agent(
model=OpenAIModel(model_id="gpt-4o"),
system_prompt="You are a writing specialist...",
tools=[write_tool, edit_tool],
agent_id="writing-agent",
agent_name="Writing Specialist"
)
# Agents will appear separately in Fiddler traces
```
**Multi-Agent Benefits**:
* ✅ Distinct agent identification in traces
* ✅ Separate performance metrics per agent
* ✅ Clear visualization of agent interactions
* ✅ Easier debugging of complex workflows
## Next Steps
Now that you have Strands agents integrated with Fiddler, explore these advanced capabilities:
### Advanced Observability and Evaluation
* [Experiments](/getting-started/experiments): Score and enrich your agent telemetry
# Experiments
Source: https://docs.fiddler.ai/developers/tutorials/experiments
Master LLM and AI application experiments with comprehensive tutorials covering the Fiddler Evals SDK, custom evaluators, model comparison, and custom experiment creation.
Building reliable AI applications requires systematic evaluation to ensure quality, safety, and consistent performance. This section provides comprehensive tutorials and quick starts to help you evaluate your LLM applications, RAG systems, and AI agents using Fiddler Experiments.
**New to Fiddler Experiments?** Start with [Getting Started with Fiddler Experiments](/getting-started/experiments) to understand the core concepts and interface before diving into these tutorials.
## What You'll Learn
These tutorials cover the full spectrum of experiment capabilities in Fiddler:
## Recommended Learning Path
**New to Fiddler Experiments?** Follow this progression:
1. [**Getting Started with Fiddler Experiments**](/getting-started/experiments) - Understand the why and what (15 min read)
2. [**Evals SDK Quick Start**](/evaluate-and-test/evals-sdk-quick-start) - Build your first experiment (20 min hands-on)
3. [**Advanced Patterns**](/developers/tutorials/experiments/evals-sdk-advanced) - Master production patterns (45 min hands-on)
4. [**Evals SDK Reference**](/sdk-api/evals/evaluate) - Complete SDK documentation (reference)
**Evaluating RAG applications?** Follow this path:
1. [**RAG Health Diagnostics**](/concepts/rag-health-diagnostics) - Understand the RAG diagnostic triad (15 min read)
2. [**RAG Health Metrics Tutorial**](/developers/tutorials/experiments/rag-health-metrics-tutorial) - Evaluate RAG systems with Answer Relevance 2.0, Context Relevance, and RAG Faithfulness (30 min hands-on)
3. [**RAG Evaluation Fundamentals Cookbook**](/developers/cookbooks/rag-evaluation-fundamentals) - End-to-end RAG evaluation use case
**Already familiar with Fiddler Experiments?** Jump to the Fiddler Evals SDK Reference for API details.
***
### Fiddler Evals SDK Quick Start
Get hands-on with the Fiddler Evals SDK in 20 minutes. Learn to create experiment datasets, use built-in evaluators (Answer Relevance, Coherence, Toxicity), build custom evaluators, and run comprehensive experiments with detailed analysis.
**Perfect for:** Developers new to the Fiddler Evals SDK who want to understand experiment workflows quickly.
[Start the Quick Start](/evaluate-and-test/evals-sdk-quick-start)
### RAG Health Metrics Tutorial
Evaluate RAG applications using the diagnostic triad: Answer Relevance 2.0, Context Relevance, and RAG Faithfulness. Learn to identify whether issues stem from retrieval, generation, or query understanding, and run experiments to compare pipeline configurations.
**Perfect for:** Teams building or maintaining RAG applications who need systematic evaluation and root cause analysis.
[Start the RAG Health Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial)
### Fiddler Evals SDK Advanced Guide
Master advanced evaluation patterns for production LLM applications. Explore complex data import strategies, context-aware evaluators for RAG systems, multi-score evaluators, lambda-based parameter mapping, and production-ready experiment patterns with 11+ evaluators.
**Perfect for:** Teams building production experiment pipelines with sophisticated requirements.
[Explore Advanced Patterns](/developers/tutorials/experiments/evals-sdk-advanced)
### Compare LLM Outputs
Learn how to systematically compare outputs from different LLM models (like GPT-3.5 and Claude) to determine the most suitable choice for your application. This guide demonstrates side-by-side model comparison workflows using Fiddler's evaluation framework.
**Perfect for:** Teams evaluating multiple models or prompt variations to make data-driven decisions.
[Compare Models](#compare-llm-outputs)
### Prompt Specs Quick Start
Create custom LLM-as-a-Judge evaluations in minutes using Prompt Specs. Learn to define evaluation schemas using JSON, validate and test your evaluations, and deploy custom evaluators to production monitoring—all without manual prompt engineering.
**Perfect for:** Teams needing domain-specific evaluation logic without extensive prompt-tuning effort.
[Create Custom Evaluations](#prompt-specs-quick-start)
## Key Experiment Capabilities
### Comprehensive Test Suites
Create datasets with test cases covering real-world scenarios, edge cases, and expected behaviors. Import data from CSV, JSONL, or pandas DataFrames with flexible column mapping.
### Built-in Evaluators
Access production-ready evaluators for common evaluation tasks:
* **Quality**: Answer Relevance 2.0 (ordinal scoring), Coherence, Conciseness, Completeness
* **RAG Health**: Answer Relevance 2.0, Context Relevance, RAG Faithfulness — the diagnostic triad for RAG pipeline evaluation
* **Safety**: Toxicity Detection, Prompt Injection, PII Detection
* **Context-Aware**: Faithfulness (Centor Model) for RAG systems
* **Sentiment**: Multi-score sentiment and topic classification
* **Pattern Matching**: Regex-based format validation
### Custom Evaluation Logic
Build evaluators tailored to your domain using:
* **Python-based evaluators** with the Evaluator base class
* **Prompt Specs** for schema-based LLM-as-a-Judge evaluation
* **Multi-score evaluators** returning multiple metrics per test case
* **Function wrappers** for existing evaluation functions
### Experiment Management
Run comprehensive experiments with:
* **Parallel processing** for faster evaluation across large datasets
* **Detailed results tracking** with scores, timing, and error handling
* **Metadata tagging** for experiment organization and filtering
* **Side-by-side comparison** to validate improvements
### Production Integration
Deploy evaluations to production monitoring:
* **Enrichment pipeline integration** for real-time evaluation
* **Automated alerting** based on evaluation metrics
* **Dashboard visualization** for tracking quality trends
* **Historical tracking** to monitor improvements over time
## Enterprise Experiment Features
### Team Collaboration
* **Shared experiment libraries**: Reuse datasets and evaluators across teams
* **Access control**: Project-level and application-level permissions
* **Experiment tracking**: Compare evaluations across team members and versions
### Production Integration
* **CI/CD pipelines**: Automated evaluation before deployment
* **Quality gates**: Set score thresholds that must be met for deployment
* **Regression detection**: Alert when experiment scores drop
### Compliance & Auditing
* **Evaluation history**: Complete audit trail of all experiments
* **Reproducibility**: Frozen datasets and evaluators for regulatory compliance
* **Export capabilities**: Download results for external analysis and reporting
## Experiment Use Cases
### Single-Turn Q\&A Systems
Evaluate direct question-answering applications with relevance, correctness, and conciseness metrics.
### RAG Applications
Assess context-grounded responses by checking for faithfulness, relevance, and completeness.
### Multi-Turn Conversations
Evaluate dialogue systems with coherence, context retention, and conversation quality metrics.
### Agentic Workflows
Test tool-using agents with trajectory evaluation, tool selection accuracy, and task completion metrics.
## Getting Started Paths
**For SDK Users:**
1. Start with [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start)
2. Progress to [Evals SDK Advanced Guide](/developers/tutorials/experiments/evals-sdk-advanced)
3. Review the [Fiddler Evals SDK Reference](/sdk-api/evals/evaluate)
**For Custom Evaluation Needs:**
1. Understand LLM Evaluation Prompt Specs concepts
2. Follow the [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start)
3. Explore [Advanced Prompt Specs](/developers/tutorials/llm-monitoring/prompt-specs-advanced) patterns
**For Model Selection:**
1. Review [Compare LLM Outputs](/evaluate-and-test/llm-evaluation-example)
2. Set up comparison experiments with your candidate models
3. Use evaluation metrics to make data-driven decisions
## Best Practices
### Start Small, Scale Systematically
Begin with a focused test suite covering critical functionality. Gradually expand coverage as you understand your application's failure modes.
### Use Multiple Evaluators
Combine different evaluator types (quality, safety, domain-specific) for a comprehensive assessment. No single metric captures all aspects of AI application quality.
### Track Over Time
Establish baselines and monitor evaluation metrics as your application evolves. Systematic tracking reveals degradation before it impacts users.
### Leverage Metadata
Tag test cases with categories, difficulty levels, and business context. Rich metadata enables targeted analysis and root cause investigation.
### Automate Evaluation
Integrate evaluations into CI/CD pipelines and deploy to production monitoring. Continuous evaluation prevents quality regressions and maintains user trust.
# Evals SDK Advanced Guide
Source: https://docs.fiddler.ai/developers/tutorials/experiments/evals-sdk-advanced
Advanced experiment patterns for production LLM applications including multi-score evaluators, complex parameter mapping, and comprehensive experiment analysis.
## What You'll Learn
This interactive notebook demonstrates advanced evaluation patterns for production LLM applications through comprehensive testing with the TruthfulQA benchmark dataset.
**Key Topics Covered:**
* Advanced data import with CSV/JSONL and complex column mapping
* Real LLM integration with production-ready task functions
* Context-aware evaluators for RAG and knowledge-grounded applications
* Multi-score evaluators and advanced evaluation patterns
* Complex parameter mapping with lambda functions
* Production experiments with 11+ evaluators and complete analysis
## Interactive Tutorial
The notebook guides you through building a comprehensive experiment pipeline for any LLM application, from single-turn Q\&A to multi-turn conversations, RAG systems, and agentic workflows.
[**Open the Advanced Evaluations Notebook in Google Colab →**](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Advanced_Evaluations_SDK.ipynb)
[**Or download the notebook directly from GitHub →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Advanced_Evaluations_SDK.ipynb)
### Prerequisites
* Fiddler account with API credentials
* Basic familiarity with the [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start)
* Optional: OpenAI API key for real LLM examples (mock responses available)
### Time Required
* **Complete tutorial**: 45-60 minutes
* **Quick overview**: 15-20 minutes
## Tutorial Highlights
## Key Takeaways from the Advanced Tutorial
Even if you prefer to run the notebook, here are the critical patterns you'll learn:
### 1. Complex Data Import Strategies
**CSV Import with Column Mapping**:
```python theme={null}
dataset.insert_from_csv_file(
file_path='truthfulqa.csv',
input_columns=['question', 'category'],
expected_output_columns=['best_answer'],
extras_columns=['context'], # Separate context for RAG evaluation
metadata_columns=['type', 'difficulty']
)
```
**Why This Matters**: Production datasets rarely have perfect column names. Column mapping lets you use any data source without reformatting files.
### 2. Context-Aware Evaluation for RAG Systems
**Faithfulness Checking**:
Fiddler provides two faithfulness evaluators: `RAGFaithfulness` (LLM-as-a-Judge, part of the RAG Health Metrics triad) for comprehensive diagnostics, and `FTLResponseFaithfulness` (powered by the Centor Model for Faithfulness) for low-latency guardrails.
```python theme={null}
from fiddler_evals.evaluators import RAGFaithfulness, FTLResponseFaithfulness
# RAG Faithfulness — LLM-as-a-Judge for RAG pipeline diagnostics
rag_faithfulness = RAGFaithfulness(model="openai/gpt-4o", credential="your-llm-credential")
# Requires user_query, rag_response, and retrieved_documents
# Faithfulness (Centor Model) — powered by the Centor Model for Faithfulness for low-latency guardrails
ftl_faithfulness = FTLResponseFaithfulness()
# Requires context and response only
```
**Why This Matters**: RAG systems must be evaluated differently than simple Q\&A. Use `RAGFaithfulness` with the full RAG Health Metrics triad (Answer Relevance, Context Relevance) for root cause diagnosis. Use `FTLResponseFaithfulness` for real-time guardrails where latency matters.
### 3. Multi-Score Evaluators
**Sentiment with Probability Scores**:
```python theme={null}
from fiddler_evals.evaluators import Sentiment
# Returns multiple scores: sentiment (categorical) and probability (float)
sentiment = Sentiment() # Fiddler Centor Model — no model/credential needed
# Returns multiple scores: sentiment label and probability
# - score.label: "positive" | "neutral" | "negative"
# - score.value: confidence score 0.0-1.0
```
**Why This Matters**: Some quality dimensions have multiple facets. Multi-score evaluators capture nuanced assessments in a single pass.
### 4. Production Experiment Patterns
**Multiple Evaluators in One Experiment**:
```python theme={null}
evaluators = [
# RAG Health Metrics (diagnostic triad)
AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential"),
ContextRelevance(model="openai/gpt-4o", credential="your-llm-credential"),
RAGFaithfulness(model="openai/gpt-4o", credential="your-llm-credential"),
# Quality metrics
Coherence(model="openai/gpt-4o", credential="your-llm-credential"),
Conciseness(model="openai/gpt-4o", credential="your-llm-credential"),
# Safety metrics
FTLPromptSafety(),
FTLResponseFaithfulness(),
# Domain-specific (Centor Model — no model/credential needed)
Sentiment(),
RegexSearch(pattern=r'\b[A-Z][a-z]+\s[A-Z][a-z]+\b'), # Proper nouns
CustomDomainEvaluator(),
]
results = evaluate(
dataset=large_dataset,
task=production_task,
evaluators=evaluators,
max_workers=10, # Parallel processing
metadata={"version": "v2.1", "environment": "staging"}
)
```
**Why This Matters**: Production systems need comprehensive evaluation across multiple dimensions. This pattern shows how to run extensive experiment suites efficiently.
### 5. Advanced Parameter Mapping
**Complex Data Structures**:
```python theme={null}
score_fn_kwargs_mapping={
# Simple mappings
"response": "answer",
# Extract from nested dicts
"prompt": lambda x: x["inputs"]["question"],
"context": lambda x: x["extras"]["retrieved_docs"],
# Compute values
"output_length": lambda x: len(x["outputs"]["answer"]),
# Combine values
"full_conversation": lambda x: x["inputs"]["history"] + [x["outputs"]["answer"]],
}
```
**Why This Matters**: Real applications have complex data structures. Lambda-based mapping gives you the flexibility to extract any value an evaluator needs.
***
### Advanced Data Import
Learn how to import complex experiment datasets with:
* CSV and JSONL file support with column mapping
* Separation of inputs, extras, expected outputs, and metadata
* Source tracking for test case provenance
* Support for RAG context and conversation history
### Production Evaluator Suite
Build a comprehensive evaluation with:
* **Context-aware evaluators**: Faithfulness checking for RAG systems
* **Safety evaluators**: Prompt safety and faithfulness detection
* **Quality evaluators**: Relevance, coherence, and conciseness
* **Custom evaluators**: Domain-specific metrics for complete customization
* **Multi-score evaluators**: Sentiment and topic classification
### Complex Parameter Mapping
Master advanced mapping techniques:
* Lambda-based parameter transformation
* Access to inputs, extras, outputs, and metadata
* Flexible mapping for any evaluator signature
* Production-ready patterns for all LLM use cases
### Comprehensive Analysis
Extract insights from experiment results:
* Aggregate statistics by evaluator
* Performance breakdown by category
* DataFrame export for further analysis
* A/B testing and regression detection patterns
## Who Should Use This
* **AI engineers** building production LLM applications
* **ML engineers** implementing systematic experiment pipelines
* **Data scientists** analyzing LLM performance and quality
* **QA engineers** setting up regression testing for AI systems
## Use Case Flexibility
The patterns demonstrated work for all LLM application types:
* **Single-turn Q\&A**: Direct question-answering without context
* **RAG applications**: Context-grounded responses with faithfulness checking
* **Multi-turn conversations**: Dialogue systems with conversation history
* **Agentic workflows**: Tool-using agents with intermediate outputs
* **Multi-task models**: Systems handling diverse request types
## Centor Models Integration
All evaluators in the advanced tutorial run on [Fiddler Centor Models](https://www.fiddler.ai/centor-models), which means:
### Cost Efficiency at Scale
Running multiple evaluators on 817 test cases (TruthfulQA dataset) would typically cost:
* **External LLM API**: \$50-100+ in API calls (0.01¢ per evaluation × 9,000 evaluations)
* **Fiddler Centor Models**: \$0 (no per-request charges)
### Performance at Scale
* **Parallel execution**: 10 workers process 817 items in \~5 minutes
* **Fast evaluators**: \<100ms per evaluation enables real-time feedback
* **No rate limits**: No API quota concerns for extensive batch experiments
### Security
* **Data locality**: All evaluations run within your Fiddler environment
* **No external calls**: Your prompts and responses never leave your infrastructure
* **Audit trail**: Complete traceability for compliance
This makes Fiddler Experiments ideal for enterprise-scale experiment pipelines.
## Next Steps
After completing the tutorial:
* **Technical Reference**: [Fiddler Evals SDK Documentation](/sdk-api/evals/evaluate)
* **Basic Tutorial**: [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start) for fundamentals
* **Getting Started Guide**: [Getting Started with Fiddler Experiments](/getting-started/experiments) for UI overview
# RAG Health Metrics Tutorial
Source: https://docs.fiddler.ai/developers/tutorials/experiments/rag-health-metrics-tutorial
Step-by-step guide to evaluating RAG applications using the RAG Health Metrics diagnostic triad: Answer Relevance, Context Relevance, and RAG Faithfulness.
Evaluate your RAG application using the RAG Health Metrics diagnostic triad to pinpoint whether issues originate in retrieval, generation, or query understanding.
**Time to complete**: \~30 minutes
## What You'll Learn
* Set up a RAG experiment pipeline with the Fiddler Evals SDK
* Use Answer Relevance, Context Relevance, and RAG Faithfulness together
* Interpret diagnostic results to identify pipeline failures
* Distinguish between retrieval and generation problems
## Prerequisites
* **Fiddler Account**: Active account with API access
* **Python 3.10+**
* **Fiddler Evals SDK**: `pip install fiddler-evals`
* **Familiarity with**: [Experiments Getting Started](/getting-started/experiments)
***
## Step 1: Connect and Set Up
```python theme={null}
from fiddler_evals import init, Project, Application, Dataset
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
# Initialize connection
init(
url='https://your-org.fiddler.ai',
token='your-access-token'
)
# Create organizational structure
project = Project.get_or_create(name='rag_health_experiments')
application = Application.get_or_create(
name='my_rag_app',
project_id=project.id
)
```
## Step 2: Create a RAG Experiment Dataset
Create test cases that include user queries and retrieved documents. The quality of your evaluation depends on realistic, representative test cases.
```python theme={null}
dataset = Dataset.create(
name='rag_health_test_cases',
application_id=application.id,
description='RAG Health Metrics experiment dataset'
)
test_cases = [
# Scenario 1: Good RAG response
NewDatasetItem(
inputs={
"user_query": "What are the benefits of renewable energy?",
"retrieved_documents": "Renewable energy sources like solar and wind reduce "
"greenhouse gas emissions, decrease dependence on fossil fuels, and can "
"lower long-term energy costs. Solar panel costs have dropped 89% since 2010."
},
metadata={"scenario": "good_response"}
),
# Scenario 2: Irrelevant retrieval
NewDatasetItem(
inputs={
"user_query": "What are the benefits of renewable energy?",
"retrieved_documents": "The history of the automobile dates back to the 15th "
"century. Karl Benz patented the first true automobile in 1886."
},
metadata={"scenario": "bad_retrieval"}
),
# Scenario 3: Hallucination risk
NewDatasetItem(
inputs={
"user_query": "What is the current price of solar panels?",
"retrieved_documents": "Solar energy adoption has grown significantly worldwide. "
"Many countries now have solar incentive programs."
},
metadata={"scenario": "insufficient_context"}
),
]
dataset.insert(test_cases)
print(f"Added {len(test_cases)} test cases")
```
## Step 3: Define Your RAG Task
The task function represents your RAG application. It receives inputs and returns the generated response.
```python theme={null}
def my_rag_task(inputs, extras, metadata):
"""Your RAG application logic.
Replace this with your actual RAG pipeline:
1. Take the user query
2. Use the retrieved documents as context
3. Generate a response
"""
user_query = inputs["user_query"]
context = inputs["retrieved_documents"]
# Call your LLM with the query and retrieved context
# Example: response = my_llm.generate(query=user_query, context=context)
response = generate_rag_response(user_query, context)
return {
"rag_response": response,
"retrieved_documents": context,
}
```
## Step 4: Run the RAG Health Experiment
Use all three evaluators together for comprehensive diagnostics:
```python theme={null}
from fiddler_evals import evaluate
from fiddler_evals.evaluators import AnswerRelevance, ContextRelevance, RAGFaithfulness
results = evaluate(
dataset=dataset,
task=my_rag_task,
evaluators=[
AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential"), # Is the response relevant? (High/Medium/Low)
ContextRelevance(model="openai/gpt-4o", credential="your-llm-credential"), # Are retrieved docs relevant? (High/Medium/Low)
RAGFaithfulness(model="openai/gpt-4o", credential="your-llm-credential"), # Is the response grounded? (Yes/No)
],
name_prefix="rag_health_baseline",
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["user_query"],
"rag_response": "rag_response",
"retrieved_documents": "retrieved_documents",
}
)
print(f"Evaluated {len(results.results)} test cases")
```
## Step 5: Analyze Diagnostic Results
Examine the results to identify which pipeline stage is causing issues:
```python theme={null}
for result in results.results:
print(f"\nScenario: {result.dataset_item.metadata.get('scenario', 'unknown')}")
scores = {score.name: score for score in result.scores}
# Extract scores
ar_score = scores.get("answer_relevance")
cr_score = scores.get("context_relevance")
rf_score = scores.get("rag_faithfulness")
if ar_score:
print(f" Answer Relevance: {ar_score.label} ({ar_score.value})")
print(f" Reasoning: {ar_score.reasoning}")
if cr_score:
print(f" Context Relevance: {cr_score.label} ({cr_score.value})")
print(f" Reasoning: {cr_score.reasoning}")
if rf_score:
print(f" RAG Faithfulness: {rf_score.label} ({rf_score.value})")
print(f" Reasoning: {rf_score.reasoning}")
# Diagnostic interpretation
if ar_score and cr_score and rf_score:
if ar_score.value >= 0.5 and rf_score.value == 0:
print(" Diagnosis: HALLUCINATION — response is relevant but not grounded")
elif rf_score.value == 1 and ar_score.value < 0.5:
print(" Diagnosis: OFF-TOPIC — response is grounded but doesn't answer the query")
elif cr_score.value < 0.5:
print(" Diagnosis: BAD RETRIEVAL — retrieved documents are not relevant")
elif ar_score.value >= 0.5 and rf_score.value == 1 and cr_score.value >= 0.5:
print(" Diagnosis: HEALTHY — all metrics indicate good RAG performance")
```
## Step 6: Compare RAG Configurations
Use experiments to compare different RAG configurations:
```python theme={null}
# Evaluate with a different retrieval strategy
results_v2 = evaluate(
dataset=dataset,
task=my_improved_rag_task, # Different retrieval or generation config
evaluators=[
AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential"),
ContextRelevance(model="openai/gpt-4o", credential="your-llm-credential"),
RAGFaithfulness(model="openai/gpt-4o", credential="your-llm-credential"),
],
name_prefix="rag_health_improved",
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["user_query"],
"rag_response": "rag_response",
"retrieved_documents": "retrieved_documents",
}
)
# Compare results side-by-side in the Fiddler UI
print("Compare experiments in your Fiddler dashboard")
```
***
## Understanding the Results
### Score Interpretation
| Evaluator | High Score | Low Score |
| ----------------- | ------------------------------------- | ----------------------------------------- |
| Answer Relevance | Response directly addresses the query | Response misses the point or is off-topic |
| Context Relevance | Retrieved documents support the query | Retrieved documents are irrelevant |
| RAG Faithfulness | Response is grounded in context | Response contains unsupported claims |
### Common Diagnostic Patterns
| Answer Relevance | Context Relevance | RAG Faithfulness | Diagnosis |
| ---------------- | ----------------- | ---------------- | --------------------------------------- |
| High | High | Yes | Healthy RAG pipeline |
| High | High | No | Hallucination — fix generation |
| Low | High | Yes | Query misunderstanding — fix prompt |
| Low | Low | - | Bad retrieval — fix retrieval |
| High | Low | Yes | Lucky generation — retrieval needs work |
***
## Next Steps
* [RAG Health Diagnostics](/concepts/rag-health-diagnostics) — Conceptual deep-dive into the diagnostic framework
* [Evals SDK Advanced Guide](/developers/tutorials/experiments/evals-sdk-advanced) — Advanced evaluation patterns
* [Evaluator Rules](/evaluate-and-test/evaluator-rules) — Set up continuous RAG monitoring in production
# Guardrails
Source: https://docs.fiddler.ai/developers/tutorials/guardrails
* [Faithfulness](/developers/tutorials/guardrails/guardrails-faithfulness): This Quick Start notebook introduces Fiddler Guardrails, an enterprise solution that safeguards LLM applications from risks like hallucinations, toxicity, and jailbreaking attempts.
* [Safety](/developers/tutorials/guardrails/guardrails-safety): This Quick Start Notebook introduces Fiddler Guardrails' Safety Detection capabilities, an essential component of our enterprise solution for protecting LLM applications.
* [PII](/developers/tutorials/guardrails/guardrails-pii): Learn to detect and protect PII, PHI, and sensitive data in text using Fiddler Guardrails for PII/PHI for comprehensive privacy compliance.
* [Secrets](/developers/tutorials/guardrails/guardrails-secrets): Learn to detect credentials, API keys, and tokens in text using Fiddler's Centor Secret Detection guardrail to prevent secret leakage in LLM applications.
# Faithfulness
Source: https://docs.fiddler.ai/developers/tutorials/guardrails/guardrails-faithfulness
This Quick Start notebook introduces Fiddler Guardrails, an enterprise solution that safeguards LLM applications from risks like hallucinations, toxicity, and jailbreaking attempts.
Learn how to implement the **Centor Model for Faithfulness**, which evaluates factual consistency between AI-generated responses and their source context in RAG applications.
**This tutorial covers the Centor Model for Faithfulness** (`ftl_response_faithfulness`) — Fiddler's proprietary Centor Model for real-time guardrail use cases. For the LLM-as-a-Judge RAG Faithfulness evaluator used in Agentic Monitoring and Experiments, see the [RAG Health Metrics Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial).
Inside you'll find:
* Step-by-step implementation instructions
* Code examples for evaluating response accuracy
* Practical demonstration of hallucination detection
* Sample inputs and outputs with score interpretation
Before running the notebook, generate a Fiddler API key from **Settings → [Credentials](/reference/administration/settings#credentials)** in your Fiddler environment. See the [Guardrails documentation](/protection/guardrails) and [FAQ](/protection/guardrails-faq) for more help getting started.
**Interactive Tutorial**
Run the Faithfulness Guardrail notebook end to end:
[**Open the Faithfulness Guardrail Notebook in Google Colab →**](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Guardrails_Faithfulness.ipynb)
[**Or download the notebook from GitHub →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Guardrails_Faithfulness.ipynb)
# PII
Source: https://docs.fiddler.ai/developers/tutorials/guardrails/guardrails-pii
Learn to detect and protect PII, PHI, and sensitive data in text using Fiddler Guardrails for PII/PHI for comprehensive privacy compliance.
Get your sensitive information detection running in **minutes** with Fiddler Guardrails for PII/PHI. This guide walks you through detecting PII, PHI, and custom entities to protect sensitive data across your applications.
## What You'll Build
In this quick start, you'll implement a sensitive information detection system that:
* Detects a comprehensive set of personally identifiable information (PII) entity types
* Identifies protected health information (PHI) entity types for healthcare compliance
* Configures custom entity detection for organization-specific data
* Provides real-time detection with sub-second latency
**Interactive Tutorial**
For more advanced examples, including batch processing, performance optimization, and production deployment patterns:
[**Open the Complete Sensitive Information Guardrail Notebook in Google Colab →**](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Sensitive_Information_Guardrail.ipynb)
[**Or download the notebook from GitHub →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Sensitive_Information_Guardrail.ipynb)
## Prerequisites
* Fiddler account with [access token](/reference/administration/settings#credentials)
* Python 3.10+ environment
* Basic understanding of data privacy concepts
## Overview
Fiddler's PII and PHI detection, powered by the Centor Model for PII/PHI, provides enterprise-grade protection against data leakage by automatically detecting sensitive information across multiple categories. These guardrails integrate seamlessly with Fiddler's AI Observability platform, enabling continuous monitoring and automated compliance reporting.
### Key Capabilities
* **PII Detection**: a comprehensive set of entity types, including names, addresses, SSN, credit cards, emails, and phone numbers
* **PHI Detection**: healthcare-specific entity types for HIPAA compliance
* **Custom Entities**: Define organization-specific sensitive data patterns
* **Real-time Processing**: Sub-second latency for production applications
Connect to Fiddler and configure the Sensitive Information Guardrail API:
```python theme={null}
import json
import pandas as pd
import requests
import time
import fiddler as fdl
# Replace with your actual values
URL = 'https://your_company.fiddler.ai'
TOKEN = 'your_token_here'
# API Configuration
SENSITIVE_INFORMATION_URL = f"{URL}/v3/guardrails/sensitive-information"
FIDDLER_HEADERS = {
"Authorization": f"Bearer {TOKEN}",
"Content-Type": "application/json",
}
# Connect to Fiddler
fdl.init(url=URL, token=TOKEN)
print("✅ Connected to Fiddler successfully!")
```
Create reusable functions for interacting with the API:
```python theme={null}
def get_sensitive_information_response(
text: str,
entity_categories: str | list[str] = 'PII',
custom_entities: list[str] = None,
):
"""
Detect sensitive information in text.
Args:
text: Input text to analyze
entity_categories: 'PII', 'PHI', 'Custom Entities', or list
custom_entities: Custom entity patterns (when using 'Custom Entities')
Returns:
Tuple of (API response dict, latency in seconds)
"""
data = {'input': text}
# Add entity configuration if specified
if entity_categories != 'PII' or custom_entities:
data['entity_categories'] = entity_categories
if custom_entities:
data['custom_entities'] = custom_entities
start_time = time.monotonic()
try:
response = requests.post(
SENSITIVE_INFORMATION_URL,
headers=FIDDLER_HEADERS,
json={'data': data},
)
response.raise_for_status()
return response.json(), (time.monotonic() - start_time)
except requests.exceptions.RequestException as e:
print(f'❌ API call failed: {e}')
return {}, (time.monotonic() - start_time)
def print_detection_results(response, latency):
"""Display detection results in a formatted way."""
entities = response.get('fdl_sensitive_information_scores', [])
print(f"\n🔍 Detection Results (⏱️ {latency:.3f}s)")
print(f"📊 Total Entities Found: {len(entities)}\n")
if not entities:
print("✅ No sensitive information detected.")
return
# Group by entity type
by_type = {}
for entity in entities:
label = entity.get('label', 'unknown')
if label not in by_type:
by_type[label] = []
by_type[label].append(entity)
# Display grouped results
for label, group in sorted(by_type.items()):
print(f"🏷️ {label.upper()} ({len(group)} found):")
for entity in group:
print(f" • '{entity['text']}' (confidence: {entity['score']:.3f})")
print()
```
Detect common personally identifiable information:
```python theme={null}
# Sample text with various PII types
sample_text = """
I'm John Doe and I live at 1234 Maple Street, Springfield, IL 62704.
You can reach me at john.doe@email.com or call me at (217) 555-1234.
My social security number is 123-45-6789, and I was born on January 15, 1987.
My credit card number is 4111 1111 1111 1111 with CVV 123.
"""
print("🧪 Testing PII Detection")
print("📄 Input Text:")
print(sample_text)
# Call the API with default PII configuration
response, latency = get_sensitive_information_response(sample_text)
# Display results
print_detection_results(response, latency)
```
**Expected Output:**
```
🔍 Detection Results (⏱️ 0.125s)
📊 Total Entities Found: 8
🏷️ PERSON (1 found):
• 'John Doe' (confidence: 0.987)
🏷️ ADDRESS (1 found):
• '1234 Maple Street, Springfield, IL 62704' (confidence: 0.945)
🏷️ EMAIL (1 found):
• 'john.doe@email.com' (confidence: 0.998)
🏷️ PHONE NUMBER (1 found):
• '(217) 555-1234' (confidence: 0.976)
🏷️ SOCIAL SECURITY NUMBER (1 found):
• '123-45-6789' (confidence: 0.991)
🏷️ CREDIT CARD NUMBER (1 found):
• '4111 1111 1111 1111' (confidence: 0.989)
🏷️ CVV (1 found):
• '123' (confidence: 0.892)
🏷️ DATE OF BIRTH (1 found):
• 'January 15, 1987' (confidence: 0.923)
```
Detect protected health information in medical contexts:
```python theme={null}
# Sample text with PHI information
healthcare_text = """
Patient report: John Smith was prescribed metformin for his diabetes condition.
His health insurance number is HI-987654321, and medical record shows
serial number MED-2024-001 for his glucose monitor device.
Birth certificate number is BC-IL-1987-001234.
Current medication includes aspirin and lisinopril for blood pressure management.
"""
print("🏥 Testing PHI Detection for Healthcare Data")
print("📄 Input Text:")
print(healthcare_text)
# Call the API with PHI configuration
response, latency = get_sensitive_information_response(
healthcare_text,
entity_categories="PHI"
)
# Display results
print_detection_results(response, latency)
```
**Expected Output:**
```
🔍 Detection Results (⏱️ 0.098s)
📊 Total Entities Found: 5
🏷️ PERSON (1 found):
• 'John Smith' (confidence: 0.976)
🏷️ MEDICATION (3 found):
• 'metformin' (confidence: 0.945)
• 'aspirin' (confidence: 0.932)
• 'lisinopril' (confidence: 0.928)
🏷️ HEALTH INSURANCE NUMBER (1 found):
• 'HI-987654321' (confidence: 0.887)
```
Define and detect organization-specific sensitive data:
```python theme={null}
# Sample text with custom entities
custom_text = """
Employee ID: EMP-2024-001, Badge Number: BD-789456
Project code: PROJ-AI-2024, Server hostname: srv-prod-01
API key: sk-abc123xyz789
Internal ticket: TICK-2024-5678
"""
# Define custom entities for your organization
custom_entities = [
'employee id',
'badge number',
'project code',
'api key',
'server hostname',
'ticket number'
]
print("🎯 Testing Custom Entity Detection")
print(f"🏷️ Custom entities: {custom_entities}")
# Call the API with custom entity configuration
response, latency = get_sensitive_information_response(
custom_text,
entity_categories='Custom Entities',
custom_entities=custom_entities
)
# Display results
print_detection_results(response, latency)
```
**Expected Output:**
```
🔍 Detection Results (⏱️ 0.112s)
📊 Total Entities Found: 6
🏷️ EMPLOYEE ID (1 found):
• 'EMP-2024-001' (confidence: 0.923)
🏷️ BADGE NUMBER (1 found):
• 'BD-789456' (confidence: 0.911)
🏷️ PROJECT CODE (1 found):
• 'PROJ-AI-2024' (confidence: 0.897)
🏷️ API KEY (1 found):
• 'sk-abc123xyz789' (confidence: 0.945)
🏷️ SERVER HOSTNAME (1 found):
• 'srv-prod-01' (confidence: 0.878)
🏷️ TICKET NUMBER (1 found):
• 'TICK-2024-5678' (confidence: 0.902)
```
## API Reference
### Endpoint
```
POST /v3/guardrails/sensitive-information
```
### Request Format
```json theme={null}
{
"data": {
"input": "Text to analyze for sensitive information",
"entity_categories": "PII" | "PHI" | "Custom Entities" | ["PII", "PHI"],
"custom_entities": ["api key", "employee id", "custom pattern"]
}
}
```
### Request Parameters
| Parameter | Type | Description | Default |
| ------------------- | --------------- | -------------------------------------------------------------- | -------- |
| `input` | string | Text to analyze for sensitive information | Required |
| `entity_categories` | string or array | Detection mode(s) to use | "PII" |
| `custom_entities` | array | Custom entity patterns (required when using "Custom Entities") | None |
### Response Format
```json theme={null}
{
"fdl_sensitive_information_scores": [
{
"score": 0.987,
"label": "email",
"start": 78,
"end": 100,
"text": "jane.smith@company.com"
}
]
}
```
### Response Fields
| Field | Type | Description |
| ------- | ------- | -------------------------------------- |
| `score` | float | Confidence score (0.0 to 1.0) |
| `label` | string | Entity type identifier |
| `start` | integer | Character position where entity starts |
| `end` | integer | Character position where entity ends |
| `text` | string | The detected entity text |
### Supported Entity Types
#### PII Entities
The Centor Model for PII/PHI returns labels as lowercase strings — use these exact values when filtering responses.
* **Personal**: person, date of birth
* **Contact**: email, email address, phone number, mobile phone number, landline phone number, fax number, address, postal code
* **Financial**: credit card number, credit card expiration date, cvv, cvc, bank account number, account number, iban
* **Government IDs**: social security number, passport number, driver's license number, tax identification number, license plate number, cpf, cnpj
* **Digital**: ip address, digital signature, website
#### PHI Entities
* **Medical**: medication, medical condition
* **Insurance**: health insurance number, health insurance id number, national health insurance number
* **Identifiers**: birth certificate number, serial number
### Code Examples
```python theme={null}
import requests
url = "https://your_company.fiddler.ai/v3/guardrails/sensitive-information"
headers = {
"Authorization": "Bearer YOUR_TOKEN",
"Content-Type": "application/json"
}
payload = {
"data": {
"input": "Contact John at john@email.com",
"entity_categories": "PII"
}
}
response = requests.post(url, json=payload, headers=headers)
entities = response.json().get("fdl_sensitive_information_scores", [])
for entity in entities:
print(f"Found {entity['label']}: {entity['text']} (confidence: {entity['score']})")
```
```python theme={null}
def safe_detect_pii(text):
"""Detect PII with proper error handling."""
try:
response = requests.post(
SENSITIVE_INFORMATION_URL,
headers=FIDDLER_HEADERS,
json={'data': {'input': text}},
timeout=10
)
response.raise_for_status()
return response.json()
except requests.exceptions.Timeout:
print("Request timed out. Try again or check your connection.")
return None
except requests.exceptions.HTTPError as e:
if e.response.status_code == 401:
print("Authentication failed. Check your API token.")
elif e.response.status_code == 429:
print("Rate limit exceeded. Wait before retrying.")
else:
print(f"HTTP error {e.response.status_code}: {e.response.text}")
return None
except Exception as e:
print(f"Unexpected error: {e}")
return None
```
```bash theme={null}
curl -X POST 'https://your_company.fiddler.ai/v3/guardrails/sensitive-information'
-H 'Content-Type: application/json'
-H 'Authorization: Bearer YOUR_TOKEN'
-d '{
"data": {
"input": "My SSN is 123-45-6789",
"entity_categories": "PII"
}
}'
```
## Next Steps
After completing this quick start:
* Explore other [Fiddler guardrails](/protection/guardrails-faq) for comprehensive AI safety
* Review the complete [guardrails documentation](/getting-started/guardrails) for all available guardrail types
* [Integrate guardrails into your applications ](/protection/guardrails)for production use
## Summary
You've learned how to:
* ✅ Detect a comprehensive set of PII entity types in text data
* ✅ Identify PHI for healthcare compliance
* ✅ Configure custom entities for your organization
* ✅ Integrate the Fiddler Guardrails for PII/PHI API into your applications.
Fiddler Guardrails for PII/PHI offer enterprise-grade protection for sensitive information with sub-second latency, making them ideal for real-time applications while ensuring compliance with privacy regulations such as GDPR, HIPAA, and CCPA.
# Safety
Source: https://docs.fiddler.ai/developers/tutorials/guardrails/guardrails-safety
This Quick Start Notebook introduces Fiddler Guardrails' Safety Detection capabilities, an essential component of our enterprise solution for protecting LLM applications.
Learn how to implement the **Centor Model for Safety**, which identifies harmful, sensitive, or jailbreaking content in both inputs and outputs of your generative AI systems.
Inside you'll find:
* Step-by-step implementation instructions
* Code examples for safety evaluation
* Practical demonstration of harmful content detection
* Sample inputs and outputs with risk score interpretation
Before running the notebook, generate a Fiddler API key from **Settings → [Credentials](/reference/administration/settings#credentials)** in your Fiddler environment. See the [Guardrails documentation](/protection/guardrails) and [FAQ](/protection/guardrails-faq) for more help getting started.
**Interactive Tutorial**
Run the Safety Guardrail notebook end to end:
[**Open the Safety Guardrail Notebook in Google Colab →**](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Guardrails_Safety.ipynb)
[**Or download the notebook from GitHub →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Guardrails_Safety.ipynb)
# Secrets
Source: https://docs.fiddler.ai/developers/tutorials/guardrails/guardrails-secrets
Learn to detect credentials, API keys, and tokens in text using Fiddler's Centor Secret Detection guardrail to prevent secret leakage in LLM applications.
Get your secret detection running in **minutes** with Fiddler's Centor Secret Detection Guardrail. This guide walks you through detecting API keys, credentials, and tokens to prevent sensitive secrets from leaking through your LLM applications.
## What You'll Build
In this quick start, you'll implement a secret detection system that:
* Detects \~42 known credential formats (Anthropic, OpenAI, AWS, GitHub, Slack, and more)
* Catches unknown and custom secrets that don't match a known format
* Returns character-level spans so you can locate and redact secrets in your text
* Provides real-time detection with sub-second latency
## Prerequisites
* Fiddler account with [access token](/reference/administration/settings#credentials)
* Python 3.10+ environment
## Overview
Fiddler's Centor Secret Detection guardrail scans text for credentials before they reach your LLM or get logged, delivering deterministic, low-latency results.
### Key Capabilities
* **Known-format detection**: \~42 known credential formats covering major providers and platforms
* **Unknown secret detection**: Catches strings that look like secrets but don't match a known format (labeled as `Possible Secret`)
* **Character-level spans**: Returns `start` and `end` offsets for precise redaction
* **Fast**: Sub-millisecond detection latency
Configure the Secret Detection Guardrail API:
```python theme={null}
import requests
import time
# Replace with your actual values
URL = 'https://your_company.fiddler.ai'
TOKEN = 'your_token_here'
# API Configuration
SECRET_DETECTION_URL = f"{URL}/v3/guardrails/secret-detection"
FIDDLER_HEADERS = {
"Authorization": f"Bearer {TOKEN}",
"Content-Type": "application/json",
}
```
Create reusable functions for interacting with the API:
```python theme={null}
def detect_secrets(text: str):
"""
Detect secrets and credentials in text.
Args:
text: Input text to scan
Returns:
Tuple of (API response dict, latency in seconds)
"""
start_time = time.monotonic()
try:
response = requests.post(
SECRET_DETECTION_URL,
headers=FIDDLER_HEADERS,
json={"data": {"input": text}},
)
response.raise_for_status()
return response.json(), (time.monotonic() - start_time)
except requests.exceptions.RequestException as e:
print(f"❌ API call failed: {e}")
return {}, (time.monotonic() - start_time)
def print_detection_results(response, latency):
"""Display detection results in a formatted way."""
secrets = response.get("fdl_secret_detection_scores", [])
print(f"\n🔍 Detection Results (⏱️ {latency:.3f}s)")
print(f"📊 Total Secrets Found: {len(secrets)}\n")
if not secrets:
print("✅ No secrets detected.")
return
for secret in secrets:
print(f"🔑 {secret['label']}")
print(f" Position: {secret['start']}–{secret['end']}")
print()
def redact_secrets(text: str, secrets: list) -> str:
"""Apply redactions right-to-left to preserve offsets."""
for secret in sorted(secrets, key=lambda s: s["start"], reverse=True):
label = secret["label"].upper().replace(" ", "_")
text = text[: secret["start"]] + f"[REDACTED {label}]" + text[secret["end"] :]
return text
```
Detect common API keys and credentials:
```python theme={null}
sample_text = """
Setting up the integration. My Anthropic key is sk-ant-api03-abcdefghijklmnopqrstu
and the OpenAI key is sk-proj-ABCDEFGHIJKLMNOPQRSTUVWXYZabcde.
GitHub token: ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
"""
print("🧪 Testing API Key Detection")
response, latency = detect_secrets(sample_text)
print_detection_results(response, latency)
# Redact detected secrets
secrets = response.get("fdl_secret_detection_scores", [])
redacted = redact_secrets(sample_text, secrets)
print("📝 Redacted text:")
print(redacted)
```
**Expected Output:**
```
🔍 Detection Results (⏱️ 0.008s)
📊 Total Secrets Found: 3
🔑 Anthropic API Key
🔑 OpenAI Project Key
🔑 GitHub Personal Access Token
📝 Redacted text:
Setting up the integration. My Anthropic key is [REDACTED ANTHROPIC_API_KEY]
and the OpenAI key is [REDACTED OPENAI_PROJECT_KEY].
GitHub token: [REDACTED GITHUB_PERSONAL_ACCESS_TOKEN]
```
> Exact character positions (`start`/`end`) vary by input and are returned by the API. The redaction helper uses API-returned offsets, not hardcoded values.
Detect AWS access keys and other infrastructure secrets:
```python theme={null}
infra_text = """
AWS credentials for the prod account:
Access Key ID: AKIAIOSFODNN7EXAMPLE
Slack webhook: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
HashiCorp Vault token: hvs.CAESIJlU9eFfaBcDeFgHiJkLmNoPqRsTuVwXyZ01234567890123
"""
print("☁️ Testing Infrastructure Secret Detection")
response, latency = detect_secrets(infra_text)
print_detection_results(response, latency)
```
**Expected Output:**
```
🔍 Detection Results (⏱️ 0.006s)
📊 Total Secrets Found: 3
🔑 AWS Access Key ID
🔑 Slack Webhook URL
🔑 HashiCorp Vault Token
```
## API Reference
### Endpoint
```
POST /v3/guardrails/secret-detection
```
### Request Format
```json theme={null}
{
"data": {
"input": "Text to scan for secrets and credentials"
}
}
```
### Request Parameters
| Parameter | Type | Description | Default |
| --------- | ------ | ------------------------ | -------- |
| `input` | string | Text to scan for secrets | Required |
### Response Format
```json theme={null}
{
"fdl_secret_detection_scores": [
{
"label": "Anthropic API Key",
"start": 10,
"end": 44
}
]
}
```
### Response Fields
| Field | Type | Description |
| ------- | ------- | ---------------------------------------- |
| `label` | string | Secret type (e.g. `"Anthropic API Key"`) |
| `start` | integer | Character position where secret starts |
| `end` | integer | Character position where secret ends |
### Detected Secret Types
#### LLM Provider Keys
`Anthropic API Key`, `OpenAI Project Key`, `OpenAI/Stripe Secret Key`, `Hugging Face Token`, `Replicate API Token`
#### Cloud Platforms
`AWS Access Key ID`, `AWS Secret Access Key`, `Google API Key`, `Google OAuth Client Secret`, `Azure Credential`, `DigitalOcean PAT`, `DigitalOcean OAuth Token`, `Heroku API Key`, `Datadog API Key`
#### Source Control
`GitHub Fine-grained PAT`, `GitHub Personal Access Token`, `GitHub OAuth Token`, `GitHub Server Token`, `GitLab Personal Access Token`, `GitLab Pipeline Token`, `Bitbucket App Password`
#### Package Registries
`npm Access Token`, `PyPI API Token`, `NuGet API Key`
#### Communication & Messaging
`Slack Bot Token`, `Slack User Token`, `Slack App Token`, `Slack Webhook URL`, `Discord Bot Token`, `SendGrid API Key`, `Twilio Account SID`, `Mailgun API Key`
#### Developer Tools
`Postman API Key`, `HashiCorp Vault Token`, `Terraform Cloud Token`, `Supabase Token`, `Vercel Token`
#### Generic Formats
`JWT Token`, `HTTP Basic Auth`, `HTTP Bearer Token`, `PEM Private Key`, `Database Connection String`
#### Unknown Secret Detection
Strings that look like secrets but don't match a known credential format. These are labeled as `Possible Secret`.
### Code Examples
```python theme={null}
import requests
url = "https://your_company.fiddler.ai/v3/guardrails/secret-detection"
headers = {
"Authorization": "Bearer YOUR_TOKEN",
"Content-Type": "application/json"
}
payload = {
"data": {
"input": "My key is sk-ant-api03-abcdefghijklmnopqrstu"
}
}
response = requests.post(url, json=payload, headers=headers)
secrets = response.json().get("fdl_secret_detection_scores", [])
for secret in secrets:
print(f"Found {secret['label']} at positions {secret['start']}–{secret['end']}")
```
```bash theme={null}
curl -X POST 'https://your_company.fiddler.ai/v3/guardrails/secret-detection' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-d '{
"data": {
"input": "My key is sk-ant-api03-abcdefghijklmnopqrstu"
}
}'
```
## Next Steps
* Use the [LiteLLM Guardrails](/protection/litellm-guardrails) integration to automatically block or redact secrets in real-time LLM proxy traffic
* Explore other [Fiddler guardrails](/protection/guardrails) for comprehensive AI safety
* Review the [Secret Detection Evaluator](/sdk-api/evals/ftl-secret-detection) for use with the Fiddler Evals SDK
## Summary
You've learned how to:
* ✅ Detect \~42 known credential formats
* ✅ Catch unknown secrets that don't match a known format
* ✅ Locate secrets precisely using character-level `start`/`end` spans
* ✅ Redact secrets from text before forwarding to an LLM
# Agentic & LLM Monitoring
Source: https://docs.fiddler.ai/developers/tutorials/llm-monitoring
* [LangGraph SDK Advanced](/developers/tutorials/llm-monitoring/langgraph-sdk-advanced): Advanced observability patterns for LangGraph applications including multi-agent workflows, conversation tracking, and production configuration.
# LangGraph SDK Advanced
Source: https://docs.fiddler.ai/developers/tutorials/llm-monitoring/langgraph-sdk-advanced
Advanced observability patterns for LangGraph applications including multi-agent workflows, conversation tracking, and production configuration.
## What You'll Learn
This interactive notebook demonstrates advanced monitoring patterns for production LangGraph applications through a realistic travel planning system with multiple specialized agents.
**Key Topics Covered:**
* Multi-agent workflow monitoring and orchestration
* Custom instrumentation with decorators and span wrappers
* Combining auto-instrumentation with fine-grained manual spans
* Conversation tracking across complex interactions
* Production configuration for high-volume scenarios
* Advanced error handling and recovery patterns
* Business intelligence integration and analytics
## Interactive Tutorial
The notebook walks through building a comprehensive travel planning application featuring hotel search, weather analysis, itinerary planning, and supervisor agents working together.
[**Open the Advanced Observability Notebook in Google Colab →**](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LangGraph_Advanced_Observability.ipynb)
[**Or download the notebook directly from GitHub →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LangGraph_Advanced_Observability.ipynb)
### Custom Instrumentation Tutorial
For hands-on examples of decorator-based and manual instrumentation, including `@trace()`, span wrappers, and async support:
[**Open the Custom Instrumentation Notebook in Google Colab →**](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LangGraph_Custom_Instrumentation.ipynb)
[**Or download the notebook directly from GitHub →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LangGraph_Custom_Instrumentation.ipynb)
## Custom Instrumentation Patterns
The SDK supports three instrumentation approaches. You can use them individually or combine them in the same application. For complete API reference, see the [Instrumentation Methods](/integrations/agentic-ai/langgraph-sdk#instrumentation-methods) section in the integration guide.
### Combining Auto-Instrumentation with Decorators
Use `LangGraphInstrumentor` for automatic LangGraph/LangChain tracing, then add `@trace()` decorators to capture custom business logic that runs outside the framework:
```python theme={null}
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor, trace, get_current_span
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
# Auto-instrument LangGraph nodes
instrumentor = LangGraphInstrumentor(client)
instrumentor.instrument()
# Add custom tracing to business logic outside LangGraph
@trace(name="validate_input", as_type="chain")
def validate_user_input(user_message: str) -> dict:
span = get_current_span(as_type="chain")
if span:
span.set_input(user_message)
result = {"valid": True, "sanitized": user_message.strip()}
if span:
span.set_output(result)
return result
# Both auto-instrumented LangGraph spans and custom spans
# appear together in Fiddler's trace view
validated = validate_user_input(user_message)
result = agent.invoke({"messages": [{"role": "user", "content": validated["sanitized"]}]})
```
### Multi-Agent Decorator Patterns
When building multi-agent systems, `@trace()` decorators automatically establish parent-child span relationships through nested function calls:
```python theme={null}
import asyncio
from fiddler_langgraph import trace, get_current_span
@trace(name="supervisor", as_type="chain")
async def supervisor_agent(state: dict) -> dict:
"""Top-level orchestrator — creates parent span."""
span = get_current_span(as_type="chain")
if span:
span.set_attribute("agent_count", len(state.get("tasks", [])))
# Child spans nest automatically under the supervisor span
results = await asyncio.gather(
research_agent(state),
analysis_agent(state),
)
if span:
span.set_output({"completed": len(results)})
return {"results": results}
@trace(name="research_agent", as_type="chain")
async def research_agent(state: dict) -> dict:
"""Child span nested under supervisor."""
span = get_current_span(as_type="chain")
if span:
span.set_attribute("data_sources", 3)
# ... research logic ...
return {"findings": "..."}
@trace(name="analysis_agent", as_type="chain")
async def analysis_agent(state: dict) -> dict:
"""Child span nested under supervisor."""
span = get_current_span(as_type="chain")
if span:
span.set_attribute("analysis_type", "comparative")
# ... analysis logic ...
return {"analysis": "..."}
```
### Using Span Wrappers for Typed Attributes
Span wrapper classes provide typed helper methods for setting semantic attributes on LLM calls, tool invocations, and chain operations. Use them with `start_as_current_span()` for fine-grained control:
```python theme={null}
from fiddler_langgraph import FiddlerClient
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
# as_type="generation" returns a FiddlerGeneration wrapper with LLM-specific helper methods
with client.start_as_current_span("llm_call", as_type="generation") as gen:
gen.set_model("gpt-4o")
gen.set_system_prompt("You are a travel planning assistant.")
gen.set_user_prompt(user_input)
response = call_llm(user_input)
gen.set_completion(response.content)
gen.set_usage(response.usage.prompt_tokens, response.usage.completion_tokens)
# as_type="tool" returns a FiddlerTool wrapper with tool-specific helper methods
with client.start_as_current_span("search_hotels", as_type="tool") as tool:
tool.set_tool_name("hotel_search")
tool.set_tool_input({"city": "Paris", "dates": "2026-03-01"})
results = search_hotels("Paris", "2026-03-01")
tool.set_tool_output(results)
```
For the complete list of helper methods on each span wrapper class, see the [Span Types and Helper Methods](/integrations/agentic-ai/langgraph-sdk#span-types-and-helper-methods) reference.
## Production Configuration Best Practices
Before deploying LangGraph applications to production, configure the SDK for your specific workload characteristics.
### High-Volume Applications
Optimize for applications processing thousands of traces per minute:
```python theme={null}
import os
from opentelemetry.sdk.trace import SpanLimits, sampling
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
from fiddler_langgraph import FiddlerClient
# Configure batch processing BEFORE initializing FiddlerClient
os.environ['OTEL_BSP_MAX_QUEUE_SIZE'] = '500' # Increased from default 100
os.environ['OTEL_BSP_SCHEDULE_DELAY_MILLIS'] = '500' # Faster export than default 1000ms
os.environ['OTEL_BSP_MAX_EXPORT_BATCH_SIZE'] = '50' # Larger batches than default 10
os.environ['OTEL_BSP_EXPORT_TIMEOUT'] = '10000' # Longer timeout than default 5000ms
# Increase span limits to capture more data
production_limits = SpanLimits(
max_events=128, # Default: 32
max_links=64, # Default: 32
max_span_attributes=128, # Default: 32
max_event_attributes=64, # Default: 32
max_link_attributes=32, # Default: 32
max_span_attribute_length=8192, # Default: 2048
)
# Sample 5-10% of traces to manage data volume
production_sampler = sampling.TraceIdRatioBased(0.05)
client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL"),
span_limits=production_limits,
sampler=production_sampler,
compression=Compression.Gzip,
)
```
### Low-Latency Requirements
Optimize for applications requiring sub-second trace export:
```python theme={null}
import os
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
from fiddler_langgraph import FiddlerClient
# Reduce batch delay for faster exports
os.environ['OTEL_BSP_SCHEDULE_DELAY_MILLIS'] = '100' # Export every 100ms
os.environ['OTEL_BSP_MAX_EXPORT_BATCH_SIZE'] = '5' # Smaller batches
client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL"),
compression=Compression.Gzip, # Still use compression
)
```
### Memory-Constrained Environments
Configure conservative limits for edge deployments or containerized environments:
```python theme={null}
import os
from opentelemetry.sdk.trace import SpanLimits, sampling
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
from fiddler_langgraph import FiddlerClient
memory_constrained_limits = SpanLimits(
max_events=16, # Minimal event capture
max_links=16, # Minimal linking
max_span_attributes=32, # Reduced attributes
max_event_attributes=16, # Reduced event attributes
max_link_attributes=16, # Reduced link attributes
max_span_attribute_length=1024, # Shorter attribute values
)
os.environ['OTEL_BSP_MAX_QUEUE_SIZE'] = '50' # Smaller queue
os.environ['OTEL_BSP_MAX_EXPORT_BATCH_SIZE'] = '5' # Smaller batches
client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL"),
span_limits=memory_constrained_limits,
sampler=sampling.TraceIdRatioBased(0.1), # Sample 10%
compression=Compression.Gzip,
)
```
### Development vs Production Configurations
**Development Configuration:**
```python theme={null}
import os
from fiddler_langgraph import FiddlerClient
# Capture everything with verbose debugging
# console_tracer=True prints spans to stdout AND continues to export to Fiddler (additive)
dev_client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL"),
console_tracer=True, # Also prints spans to console; does NOT disable OTLP export
sampler=None, # Capture 100% of traces
)
```
**Production Configuration:**
```python theme={null}
import os
from opentelemetry.sdk.trace import SpanLimits, sampling
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
from fiddler_langgraph import FiddlerClient
# Define production limits (see High-Volume Applications above for full example)
production_limits = SpanLimits(
max_events=128, max_span_attributes=128, max_span_attribute_length=8192
)
# Optimized for performance and cost
prod_client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL"),
sampler=sampling.TraceIdRatioBased(0.05), # Sample 5%
compression=Compression.Gzip, # Reduce bandwidth
span_limits=production_limits, # Controlled limits
)
```
### Best Practices for Context and Conversation IDs
Structure your identifiers for maximum analytical value:
```python theme={null}
from fiddler_langgraph import set_llm_context, set_conversation_id
import uuid
# Set meaningful, searchable context labels
set_llm_context(model, 'Customer Support - Tier 1 - Billing Inquiries')
set_llm_context(model, 'Content Generation - Marketing Copy - Blog Posts')
set_llm_context(model, 'Data Analysis - Financial Reports - Q4 2025')
# Use structured conversation IDs with metadata
user_id = 'user-12345'
session_type = 'support'
timestamp = '2026-06-15'
conversation_id = f'{user_id}_{session_type}_{timestamp}_{uuid.uuid4()}'
set_conversation_id(conversation_id)
# Example: user-12345_support_2026-06-15_550e8400-e29b-41d4-a716-446655440000
```
### Prerequisites
* Fiddler account with API credentials
* OpenAI API key for example interactions
* Basic familiarity with LangGraph concepts
### Time Required
* **Complete tutorial**: 45-60 minutes
* **Quick overview**: 15-20 minutes
## Telemetry Data Reference
Understanding the data captured by the Fiddler LangGraph SDK.
### Span Attributes
The SDK automatically captures these OpenTelemetry attributes:
| Attribute | Type | Description |
| ------------------------- | ------- | -------------------------------------------------------------------------------- |
| `gen_ai.agent.name` | `str` | Name of the AI agent (auto-extracted from LangGraph, configurable for LangChain) |
| `gen_ai.agent.id` | `str` | Unique identifier (format: `trace_id:agent_name`) |
| `gen_ai.conversation.id` | `str` | Session identifier set via `set_conversation_id()` |
| `fiddler.span.type` | `str` | Span classification: `chain`, `tool`, `llm`, or `agent` |
| `gen_ai.llm.input.system` | `str` | System prompt content |
| `gen_ai.llm.input.user` | `str` | User input/prompt |
| `gen_ai.llm.output` | `str` | Model response text |
| `gen_ai.llm.context` | `str` | Custom context set via `set_llm_context()` |
| `gen_ai.request.model` | `str` | Model identifier (e.g., "gpt-4o-mini") |
| `gen_ai.llm.token_count` | `int` | Token usage metrics |
| `gen_ai.tool.name` | `str` | Tool function name |
| `gen_ai.tool.input` | `str` | Tool input parameters (JSON) |
| `gen_ai.tool.output` | `str` | Tool execution results (JSON) |
| `gen_ai.tool.definitions` | `str` | Tool definitions available to the LLM (JSON array of OpenAI-format tool schemas) |
| `gen_ai.input.messages` | `str` | Complete message history provided as input to the LLM (JSON array) |
| `gen_ai.output.messages` | `str` | Output messages generated by the LLM, including tool calls (JSON array) |
| `duration_ms` | `float` | Span duration in milliseconds |
| `fiddler.error.message` | `str` | Error message (if span failed) |
| `fiddler.error.type` | `str` | Error type classification |
### Setting Attributes with Span Wrappers
When using [manual instrumentation](/integrations/agentic-ai/langgraph-sdk#manual-instrumentation), span wrapper classes provide typed helper methods that set these attributes automatically. For example, `FiddlerGeneration.set_model("gpt-4o")` sets `gen_ai.request.model`, and `FiddlerTool.set_tool_name("search")` sets `gen_ai.tool.name`.
| Span Wrapper | Key Methods | Attributes Set |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FiddlerGeneration` | `set_model()`, `set_system_prompt()`, `set_user_prompt()`, `set_completion()`, `set_usage()`, `set_messages()`, `set_output_messages()`, `set_tool_definitions()` | `gen_ai.request.model`, `gen_ai.llm.input.*`, `gen_ai.llm.output`, `gen_ai.usage.*`, `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.tool.definitions` |
| `FiddlerTool` | `set_tool_name()`, `set_tool_input()`, `set_tool_output()`, `set_tool_definitions()` | `gen_ai.tool.name`, `gen_ai.tool.input`, `gen_ai.tool.output`, `gen_ai.tool.definitions` |
| `FiddlerChain` | `set_input()`, `set_output()` | Input/output data attributes |
| `FiddlerSpan` | `set_attribute()`, `set_agent_name()`, `set_conversation_id()` | Any custom or standard attribute |
For the complete method reference, see [Span Types and Helper Methods](/integrations/agentic-ai/langgraph-sdk#span-types-and-helper-methods).
### Querying and Filtering in Fiddler
Use these attributes in the Fiddler UI to:
* **Filter by agent:** `gen_ai.agent.name = "hotel_search_agent"`
* **Find conversations:** `gen_ai.conversation.id = "user-123_support_2026-06-15..."`
* **Analyze by model:** `gen_ai.request.model = "gpt-4o"`
* **Track errors:** `fiddler.error.type EXISTS`
## Who Should Use This
* AI engineers building production LangGraph applications
* DevOps teams monitoring agentic systems
* Technical leaders evaluating observability strategies
## Limitations and Considerations
### Current Limitations
* **Framework Support**: LangGraph is fully supported with automatic agent name extraction
* LangChain applications require manual agent name configuration
* Non-LangGraph Python code can use `@trace()` decorators or manual context managers for custom instrumentation (see [Instrumentation Methods](/integrations/agentic-ai/langgraph-sdk#instrumentation-methods))
* **Protocol Support**: Currently uses HTTP-based OTLP
* gRPC support planned for future releases
* **Attribute Limits**: Default OpenTelemetry limits apply
* Configurable via `span_limits` parameter
* Very large attribute values may be truncated
### Performance Considerations
**Overhead**: Typical performance impact is \< 5% with default settings
* Use sampling to reduce overhead in high-volume scenarios
* Adjust batch processing delays based on latency requirements
**Memory**: Span queue size affects the memory footprint
* Default queue (100 spans) uses \~1-2MB
* Increase `OTEL_BSP_MAX_QUEUE_SIZE` for high throughput
* Decrease for memory-constrained environments
**Network**: Compression significantly reduces bandwidth usage
* Gzip compression: \~70-80% reduction
* Use `Compression.NoCompression` only for debugging
### Production Deployment Checklist
Before deploying to production:
* [ ] Set appropriate sampling rate (typically 5-10% for high-volume apps)
* [ ] Configure span limits based on your data characteristics
* [ ] Tune batch processing parameters for your traffic patterns
* [ ] Enable Gzip compression (default, recommended)
* [ ] Use environment variables for credentials (not hardcoded)
* [ ] Test instrumentation in staging environment first
* [ ] Monitor SDK performance impact
* [ ] Set up alerts for instrumentation failures
* [ ] Document your configuration for team knowledge sharing
### When to Tune Each Setting
| Scenario | Configuration |
| ---------------------------- | ----------------------------------------------- |
| **High-volume production** | Increase queue size, batch size, sampling rate |
| **Low-latency requirements** | Decrease schedule delay, smaller batches |
| **Memory constraints** | Decrease span limits, queue size, batch size |
| **Development/debugging** | Disable sampling, enable console tracer |
| **Cost optimization** | Increase sampling (lower %), enable compression |
## Next Steps
After completing the tutorial:
* **Custom Instrumentation Notebook**: [Hands-on decorator and span wrapper examples](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LangGraph_Custom_Instrumentation.ipynb)
* **Integration Guide**: [Instrumentation Methods reference](/integrations/agentic-ai/langgraph-sdk#instrumentation-methods) for `@trace()`, manual instrumentation, and span wrapper APIs
* **Technical Reference**: [Fiddler LangGraph SDK Documentation](/sdk-api/langgraph/fiddler-client)
* **Production Deployment**: Adapt the demonstrated patterns for your specific use case
# Advanced Prompt Specs
Source: https://docs.fiddler.ai/developers/tutorials/llm-monitoring/prompt-specs-advanced
Advanced guide to Fiddler's LLM-as-a-Judge capabilities, including custom prompting, model selection, performance optimization, and enterprise deployment patterns.
This comprehensive guide covers advanced LLM-as-a-Judge capabilities using Prompt Specs for **LLM Observability (Traditional Monitoring)**. It includes custom prompting, model configuration, performance optimization, and enterprise deployment patterns.
**For Agentic Monitoring and Experiments**, use the `CustomJudge` class from the Fiddler Evals SDK instead of Prompt Specs. `CustomJudge` provides `prompt_template` (Jinja syntax) and `output_fields` for structured evaluation. See the [Custom Judge Evaluators Cookbook](/developers/cookbooks/custom-judge-evaluators) for examples.
### Prerequisites
* Understanding of [Prompt Specs fundamentals](/observability/llm/llm-evaluation-prompt-specs)
* Completion of the [LLM Evaluation Quickstart](/evaluate-and-test/prompt-specs-quick-start)
* Familiarity with LLM evaluation concepts
> Download this tutorial directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLMaaJ_Prompt_Spec.ipynb) or run it in [Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLMaaJ_Prompt_Spec.ipynb)
```python theme={null}
import json
import pandas as pd
import requests
FIDDLER_BASE_URL = "https://your_company.fiddler.ai"
PROMPT_SPEC_URL = f"{FIDDLER_BASE_URL}/v3/llm-as-a-judge/prompt-spec"
FIDDLER_HEADERS = {
"Authorization": f"Bearer {FIDDLER_TOKEN}",
"Content-Type": "application/json",
}
```
We'll use news article data for this example:
```python theme={null}
# Load sample news data (using AG News dataset)
df_news = pd.read_parquet(
"hf://datasets/fancyzhx/ag_news/data/test-00000-of-00001.parquet"
).sample(20, random_state=25)
# Map labels to topic names
df_news["original_topic"] = df_news["label"].map({
0: "World", 1: "Sports", 2: "Business", 3: "Sci/Tech"
})
# Summarize the count of each unique topic
print(df_news["original_topic"].value_counts())
```
Define a simple evaluation schema:
```python theme={null}
basic_prompt_spec = {
"input_fields": {
"news_summary": {"type": "string"}
},
"output_fields": {
"topic": {
"type": "string",
"choices": ["World", "Sports", "Business", "Sci/Tech"]
},
"reasoning": {"type": "string"}
}
}
```
Validate your Prompt Spec schema:
```python theme={null}
validate_response = requests.post(
f"{PROMPT_SPEC_URL}/validate",
headers=FIDDLER_HEADERS,
json={"prompt_spec": prompt_spec_basic},
)
validate_response.raise_for_status()
print("Status Code:", validate_response.status_code)
print(json.dumps(validate_response.json(), indent=2))
```
Test with a larger set of data:
```python theme={null}
def get_prediction(prompt_spec, input_data):
predict_response = requests.post(
f"{PROMPT_SPEC_URL}/predict",
headers=FIDDLER_HEADERS,
json={"prompt_spec": prompt_spec, "input_data": input_data},
)
if predict_response.status_code != 200:
print(f"Error ({predict_response.status_code}): {predict_response.text}")
return {"topic": None, "reasoning": None}
return predict_response.json()["prediction"]
print(
json.dumps(
get_prediction(
prompt_spec_basic, {"news_summary": "Wimbledon 2025 is under way!"}
),
indent=2,
)
)
```
Evaluate a batch of data:
```python theme={null}
df_ag_news[["topic", "reasoning"]] = df_ag_news.apply(
lambda row: get_prediction(prompt_spec_basic, {"news_summary": row["text"]}),
axis=1,
result_type="expand",
)
```
Note several `Sci/Tech` articles were misclassified as `World`. The `reasoning` field helps identify trends. We'll use this to update our prompt spec in the next section.
```python theme={null}
accuracy = (df_ag_news["original_topic"] == df_ag_news["topic"]).mean()
print(f"Accuracy: {accuracy:.0%}")
df_ag_news.value_counts(subset=["original_topic", "topic"])
for r in df_ag_news[
(df_ag_news["original_topic"] == "Sci/Tech") & (df_ag_news["topic"] != "Sci/Tech")
]["reasoning"]:
print(r)
```
Just as descriptive field names can help improve model performance, you can also add a task instruction and field descriptions. Here, we will add a description to `topic` to help with classifying `Sci/Tech` articles. Note the improved results.
```python theme={null}
prompt_spec_rich = {
"instruction": "Determine the topic of the given news summary.",
"input_fields": {
"news_summary": {
"type": "string",
}
},
"output_fields": {
"topic": {
"type": "string",
"choices": df_ag_news["original_topic"].unique().tolist(),
"description": """Use topic 'Sci/Tech' if the news summary is about a company or business in the tech industry, or if the news summary is about a scientific discovery or research, including health and medicine.
Use topic 'Sports' if the news summary is about a sports event or athlete.
Use topic 'Business' if the news summary is about a company or industry outside of science, technology, or sports.
Use topic 'World' if the news summary is about a global event or issue.
""",
},
"reasoning": {
"type": "string",
"description": "The reasoning behind the predicted topic.",
},
},
}
print(
json.dumps(
get_prediction(prompt_spec_rich, {"news_summary": df_ag_news.loc[267, "text"]}),
indent=2,
)
)
```
Note the improvement of accuracy in the results:
```python theme={null}
df_ag_news[["topic", "reasoning"]] = df_ag_news.apply(
lambda row: get_prediction(prompt_spec_rich, {"news_summary": row["text"]}),
axis=1,
result_type="expand",
)
accuracy = (df_ag_news["original_topic"] == df_ag_news["topic"]).mean()
print(f"Accuracy: {accuracy:.0%}")
df_ag_news.value_counts(subset=["original_topic", "topic"])
```
### Deploying Your Evaluation to Production
Once you see the results you expect with your test data, deploy the custom evaluation to production and monitor your production application:
```python theme={null}
from datetime import datetime
import fiddler as fdl
fdl.init(url=FIDDLER_BASE_URL, token=FIDDLER_TOKEN)
PROJECT_NAME = "quickstart_examples" # If the project already exists, the notebook will create the model under the existing project.
MODEL_NAME = "fiddler_news_classifier"
project = fdl.Project.get_or_create(name=PROJECT_NAME)
```
Recall we used `news_summary` in our prompt. Let's make our dataframe match this and add some metadata.
```python theme={null}
platform_df = df_ag_news.rename(columns={"text": "news_summary"})
platform_df["id"] = platform_df.index
```
* `name` will be used as part of the generated column name; set it to something meaningful for your use case.
* `enrichment` must always be `llm_as_a_judge`.
* `columns` matches all the input columns your prompt spec uses.
* `config` must set the prompt spec.
Then define the remainder of the schema that makes up this application to be monitored. For more details on setting up Fiddler to monitor your ML models and LLM/GenAI applications, refer to the [ML Monitoring Quick Start](/developers/quick-starts/simple-ml-monitoring) and the [LLM Monitoring Quick Start](/developers/quick-starts/simple-llm-monitoring) guides.
```python theme={null}
fiddler_llm_enrichments = [
fdl.Enrichment(
name="news_topic",
enrichment="llm_as_a_judge",
columns=["news_summary"],
config={"prompt_spec": prompt_spec_rich},
)
]
model_spec = fdl.ModelSpec(
inputs=["news_summary"],
metadata=["id", "original_topic"],
custom_features=fiddler_llm_enrichments,
)
llm_application = fdl.Model.from_data(
source=platform_df,
name=MODEL_NAME,
project_id=project.id,
spec=model_spec,
task=fdl.ModelTask.LLM,
max_cardinality=5,
)
llm_application.create()
```
Our prediction will add two columns: `FDL news_topic (topic)` and `FDL news_topic (reasoning)`.
> **Note**: The column names follow the pattern: `FDL {enrichment name} ({prompt spec output column})`, using values as specified.
```python theme={null}
production_publish_job = llm_application.publish(platform_df)
# wait for the job to complete
production_publish_job.wait(interval=20)
```
```python theme={null}
llm_application.download_data(
output_dir="test_download",
env_type=fdl.EnvType.PRODUCTION,
start_time=start_time,
end_time=datetime.now(),
columns=[
"id",
"news_summary",
"original_topic",
"FDL news_topic (topic)",
"FDL news_topic (reasoning)",
],
)
# See the original data and the results of LLM-as-a-Judge
fdl_data = pd.read_parquet("test_download/output.parquet")
fdl_data.sample(15)
```
### Advanced Prompt Specs Configuration
#### Schema Design Patterns
#### Multi-Output Evaluation
**Domain-Specific Classification**
#### Performance Optimization Techniques
**Field Description Best Practices**
* **Be Specific**: Use concrete examples rather than abstract descriptions
* **Avoid Ambiguity**: Define edge cases and boundary conditions
* **Include Context**: Reference domain-specific knowledge when needed
### Bring-Your-Own-Prompt
For maximum customization, Fiddler supports custom prompt templates with multiple output format options.
#### Free-Form Output
Best for open-ended evaluations where structure is less important:
```json theme={null}
{
"prompt_template": {
"user": "Analyze the following text for potential bias: {text}. Provide detailed analysis."
},
"output_fields": ["analysis"],
"output_format": {
"type": "free_form"
}
}
```
#### Guided Choice Output
For single categorical outputs with high accuracy requirements:
```json theme={null}
{
"prompt_template": {
"system": "You are an expert content moderator.",
"user": "Classify this content: {content}. Choose the most appropriate category."
},
"output_fields": ["classification"],
"output_format": {
"type": "guided_choice",
"choices": ["safe", "questionable", "harmful", "requires_review"]
}
}
```
#### Guided JSON Output
For complex structured outputs with validation:
```json theme={null}
{
"prompt_template": {
"system": "Extract structured information from job postings.",
"user": "Extract key details from: {job_posting}"
},
"output_fields": ["title", "company", "salary_range", "remote_work"],
"output_format": {
"type": "guided_json",
"schema": {
"type": "object",
"properties": {
"title": { "type": "string" },
"company": { "type": "string" },
"salary_range": { "type": "string" },
"remote_work": { "type": "boolean" }
},
"required": ["title", "company"]
}
}
}
```
### Additional Documentation
* [LLM Observability Overview](/observability/llm): Understanding Fiddler's broader LLM monitoring capabilities
* [Enrichments](/observability/llm/enrichments): Technical details on Fiddler's evaluation infrastructure
# ML Monitoring
Source: https://docs.fiddler.ai/developers/tutorials/ml-monitoring
* [NLP Inputs](/developers/tutorials/ml-monitoring/simple-nlp-monitoring-quick-start): Dive into our guide on using Fiddler to monitor NLP models. Learn how a multi-class classifier is applied to the dataset and monitored with Vector Monitoring.
* [Class Imbalance](/developers/tutorials/ml-monitoring/class-imbalance-monitoring-example): Discover how Fiddler uses class weighting to address class imbalance. Compare two identical models–with and without weighting–to detect drift signals.
* [Model Versions](/developers/tutorials/ml-monitoring/ml-monitoring-model-versions): Explore our guide to using Fiddler’s sample data to set up and manage multiple versions of a model with the powerful Model Versions feature.
* [Ranking Models](/developers/tutorials/ml-monitoring/ranking-model): Explore our notebook to see how Fiddler monitors ranking models using a public dataset organized around “search result impressions” from Expedia hotel searches.
* [Regression](/developers/tutorials/ml-monitoring/ml-monitoring-regression): Check out our guide on using Fiddler to evaluate regression models. See examples of detecting issues using data drift and performance metrics like MAE.
* [Feature Impact](/developers/tutorials/ml-monitoring/user-defined-feature-impact): Leverage this guide on using Fiddler's feature impact upload API to supply your own feature impact values for your Fiddler model.
* [CV Inputs](/developers/tutorials/ml-monitoring/cv-monitoring): Explore our guide to using Fiddler’s monitoring for computer vision models. Learn to detect drift in image data with our unique Vector Monitoring approach.
# Class Imbalance
Source: https://docs.fiddler.ai/developers/tutorials/ml-monitoring/class-imbalance-monitoring-example
Discover how Fiddler uses class weighting to address class imbalance. Compare two identical models–with and without weighting–to detect drift signals.
Many ML use cases, like fraud detection and facial recognition, suffer from what is known as the *class imbalance problem*. This problem exists where a vast majority of the inferences seen by the model belong to only one class, known as the majority class. This makes detecting drift in the minority class very difficult as the "signal" is completely outweighed by the sheer number of inferences seen in the majority class.
This guide showcases how Fiddler uses a class weighting parameter to deal with this problem. This notebook will onboard two identical models -- one without class imbalance weighting and one with class imbalance weighting -- to illustrate how drift signals in the minority class are easier to detect once properly amplified by Fiddler's unique class weighting approach.
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Imbalanced_Data.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Imbalanced_Data.ipynb).
# CV Inputs
Source: https://docs.fiddler.ai/developers/tutorials/ml-monitoring/cv-monitoring
Explore our guide to using Fiddler’s monitoring for computer vision models. Learn to detect drift in image data with our unique Vector Monitoring approach.
This guide will walk you through the basic steps required to use Fiddler for monitoring computer vision (CV) models. In this notebook we demonstrate how to detect drift in image data using model embeddings using Fiddler's unique Vector Monitoring approach.
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Image_Monitoring.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Image_Monitoring.ipynb).
# Model Versions
Source: https://docs.fiddler.ai/developers/tutorials/ml-monitoring/ml-monitoring-model-versions
Explore our guide to using Fiddler’s sample data to set up and manage multiple versions of a model with the powerful Model Versions feature.
This guide will walk you through how you can use Model Versions feature in setting up multiple versions of the same model, **using sample data provided by Fiddler**.
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Model_Versions.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Model_Versions.ipynb).
# Regression
Source: https://docs.fiddler.ai/developers/tutorials/ml-monitoring/ml-monitoring-regression
Check out our guide on using Fiddler to evaluate regression models. See examples of detecting issues using data drift and performance metrics like MAE.
In this notebook, we demonstrate how Fiddler can monitor the performance of regression models, detecting/alerting issues with data drift and performance metrics (ex. Mean Absolute Error).
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Regression_Model.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Regression_Model.ipynb).
# Ranking Models
Source: https://docs.fiddler.ai/developers/tutorials/ml-monitoring/ranking-model
Explore our notebook to see how Fiddler monitors ranking models using a public dataset organized around “search result impressions” from Expedia hotel searches.
This notebook will show you how Fiddler enables monitoring for a Ranking model. This notebook uses a public dataset from Expedia that includes shopping and purchase data with information on price competitiveness. The data are organized around a set of “search result impressions”, or the ordered list of hotels that the user sees after they search for a hotel on the Expedia website.
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Ranking_Model.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Ranking_Model.ipynb).
# NLP Inputs
Source: https://docs.fiddler.ai/developers/tutorials/ml-monitoring/simple-nlp-monitoring-quick-start
Dive into our guide on using Fiddler to monitor NLP models. Learn how a multi-class classifier is applied to the dataset and monitored with Vector Monitoring.
This guide will walk you through the basic steps required to use Fiddler for monitoring NLP models. A multi-class classifier is applied to the 20newsgroup dataset and the text embeddings are monitored using Fiddler's unique Vector Monitoring approach.
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_NLP_Multiclass_Monitoring.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_NLP_Multiclass_Monitoring.ipynb).
# Feature Impact
Source: https://docs.fiddler.ai/developers/tutorials/ml-monitoring/user-defined-feature-impact
Leverage this guide on using Fiddler's feature impact upload API to supply your own feature impact values for your Fiddler model.
This guide will walk you through the steps needed to upload your model's existing feature impact values to your Fiddler model. This notebook uses the same example model leveraged in the [ML Monitoring - Simple](/developers/quick-starts/simple-ml-monitoring) quick start. If you have already run that notebook and the model exists in your Fiddler instance, then you may skip the setup steps in this guide as noted in the instructions.
Click [this link to get started using Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_User_Defined_Feature_Impact.ipynb)
Or download the notebook directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_User_Defined_Feature_Impact.ipynb).
# Configure Evaluator Downsampling
Source: https://docs.fiddler.ai/evaluate-and-test/configure-evaluator-downsampling
Reduce LLM-as-a-Judge evaluation cost at scale with evaluator downsampling. Learn when to use it, how the sampling rate relates to the enabled toggle, and how to configure it from the REST API and the UI.
Evaluator downsampling lets an [Evaluator Rule](/evaluate-and-test/evaluator-rules) run on a *fraction* of the spans it matches instead of every one. It is designed for high-volume GenAI applications where running an LLM-as-a-Judge evaluator on 100% of traffic is more costly than the monitoring signal requires.
You control downsampling with a single per-rule field, `sampling_rate`, a number in the range `(0.0, 1.0]` that defaults to `1.0` (evaluate every matching span).
**Traces are always persisted even with evaluator downsampling enabled.** Downsampling only controls whether an evaluator *runs* on a matching span — it never affects ingestion or storage. Sampled-out traces stay fully queryable in the [Explorer](/observability/agentic/trace-explorer). To *score* traces a rule skipped, create a new rule (or recreate the rule) with a higher `sampling_rate` and backfill enabled. Backfill can only be requested when a rule is created; a backfill at the same rate re-applies the same deterministic decision and skips the same traces again.
**Availability.** Evaluator downsampling is gated by a feature flag and may not be enabled for your deployment yet. To turn it on, contact your Fiddler account team or [Fiddler support](mailto:support@fiddler.ai); on self-hosted deployments, an administrator enables it through deployment configuration. Until it's enabled, the **Sampling rate** field is unavailable in the UI and the API rejects requests that include a `sampling_rate`.
***
## How it works
When evaluator downsampling is active, each matching span is checked against the rule's `sampling_rate` before the evaluator runs:
* The decision is **deterministic and trace-atomic** — every span within the same trace receives the same evaluate-or-skip outcome, so multi-step and agentic traces are always scored as a unit.
* A rule left at the default `sampling_rate = 1.0` behaves exactly as it does today, with no change in coverage.
For example, a rule with `sampling_rate = 0.1` evaluates roughly 10% of the traces it matches, cutting the evaluator's LLM-as-a-Judge calls for that rule by about 90% — while every trace is still ingested and stored.
***
## When to use evaluator downsampling
Use downsampling when **evaluation cost or latency, not coverage, is your constraint**:
* **Cost reduction at scale.** LLM-as-a-Judge evaluators are expensive at production volumes. A representative sample typically preserves drift and quality signal at a fraction of the cost — a 5–20% sample is often equivalent to full coverage for trend monitoring. See [Recommended minimum trace volume](#recommended-minimum-trace-volume) for choosing a rate.
* **High-throughput applications.** Reducing evaluation volume relieves backpressure on the enrichment pipeline during traffic spikes.
* **Multiple evaluators on the same traffic.** When several rules score the same high-volume application, downsampling each one keeps total evaluation cost manageable. (Each application supports up to [100 evaluator rules](/evaluate-and-test/evaluator-rules#rule-limit).)
**Sampling is correlated across rules.** Two rules with the same `sampling_rate` evaluate the *same* subset of traces — the decision is hashed on the trace, not the rule, so multiple judges agree on which traces to score. If you want two evaluators to cover *different* traces, downsampling them at the same rate will not spread them apart.
## When not to use evaluator downsampling
Avoid downsampling (keep `sampling_rate = 1.0`) when:
* **Your application is low-volume.** On low-traffic applications, sampling removes too many data points and produces noisy, unreliable drift and quality signals.
* **You need a very low rate (below \~1%).** Rates under roughly 1% introduce statistical noise that can swamp the signal you are trying to measure. If cost is still a concern at 1%, your application volume is probably too low to downsample at all.
* **You need every trace scored** — for example, for compliance, exhaustive evaluation, or guardrail-style use cases. In those cases, leave the rule at `1.0`.
* **You want to pause evaluation.** Do not use a tiny `sampling_rate` to "almost turn off" a rule. Deactivate the rule instead (see below).
### Recommended minimum trace volume
Choose a rate that keeps the *evaluated* volume statistically meaningful, not just the matched volume. As a rule of thumb:
* Aim for at least **\~1,000 evaluated traces per monitoring window** (the period over which you read drift or quality metrics).
* Avoid downsampling applications that match fewer than a few thousand traces per day for a given rule.
* Keep `sampling_rate` at or above **0.01 (1%)**.
In other words, pair lower rates with higher volume. A `sampling_rate` of `0.1` is reasonable at 100,000 daily traces (\~10,000 evaluated); the same `0.1` on an application with only a few hundred daily traces will not produce a reliable signal.
These thresholds are **recommendations, not enforced limits.** The API accepts any `sampling_rate` in `(0.0, 1.0]` (to a resolution of `0.0001`), so a rate like `0.01` is permitted — values below \~1% are simply unlikely to yield a usable monitoring signal, so treat **1% as a practical floor** rather than a hard minimum.
***
## `sampling_rate` vs. deactivating a rule
A rule's `sampling_rate` and its enabled state are **independent** controls. Use the one that matches your intent:
| Goal | Use | Effect |
| --------------------------------- | --------------------------------------- | -------------------------------------------------------------- |
| Reduce evaluation volume and cost | Lower `sampling_rate` (e.g. `0.1`) | The evaluator runs on a fraction of matching traces |
| Pause evaluation entirely | Deactivate the rule (`enabled = false`) | The evaluator runs on no spans; the rule is skipped completely |
Key behaviors:
* `sampling_rate = 0.0` is **rejected** — to stop evaluation entirely, deactivate the rule instead of using a near-zero rate.
* A rule is created **enabled**; the create request does not accept an `enabled` field. To pause a rule, deactivate it afterward — toggle it off in the UI or send a `PATCH` with `{"enabled": false}`.
* Deactivating a rule **preserves** its configured `sampling_rate`, so reactivating it restores your previous setting.
* Changing `sampling_rate` **never** reactivates a deactivated rule.
**Changes take effect within about 60 seconds.** When you update a rule's `sampling_rate`, evaluation already in flight may use the previous rate until the evaluator-rule cache refreshes (roughly a 60-second window). Plan rate changes with this short delay in mind.
***
## Configure downsampling
### Using the REST API
Set `sampling_rate` when you create a rule, or update it later with a `PATCH`.
Create a rule that evaluates 10% of matching spans:
```bash theme={null}
curl -X POST https:///v3/evaluator-rules \
-H "Authorization: Bearer $FIDDLER_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "custom_judge_sampled",
"application_id": "",
"evaluator_id": "",
"mapped_input_keys": {"response": "gen_ai.llm.output"},
"sampling_rate": 0.1
}'
```
Map the input keys your evaluator declares — a custom LLM-as-a-Judge takes the keys named in its prompt, while built-in evaluators require specific inputs (for example, the Coherence evaluator needs both `prompt` and `response`).
Change the sampling rate on an existing rule:
```bash theme={null}
curl -X PATCH https:///v3/evaluator-rules/ \
-H "Authorization: Bearer $FIDDLER_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sampling_rate": 0.25}'
```
A `sampling_rate` outside `(0.0, 1.0]` (including `0.0` or any value above `1.0`) returns `400 Bad Request`.
### Using the UI
When evaluator downsampling is enabled for your deployment, you set a rule's sampling rate from the **Evaluator Rules** tab.
**When creating a rule**, the **Add Evaluator Rule** dialog includes a **Sampling rate** field:
1. Open your application's **Evaluator Rules** tab and click **Add Rule**.
2. Configure the evaluator, input mappings, and application rules as usual (see [Evaluator Rules](/evaluate-and-test/evaluator-rules)).
3. Set the **Sampling rate** to the percentage of matching traces you want evaluated — for example, enter `10` for 10%. Leave it at the default (`100`) to evaluate everything.
4. Save the rule.
**For an existing rule**, edit its **Sampling rate** (also a percentage) directly in the Evaluator Rules table — update the value in the rule's **Sampling rate** cell. (The row menu offers only Pause and Delete; there is no separate edit dialog.)
**The UI uses a percentage; the REST API uses a fraction.** In the UI, enter `10` for 10%. The same rate over the API is `sampling_rate: 0.1`.
The **Sampling rate** control is rolling out behind a feature flag and may not yet be visible in your environment. If evaluator downsampling is enabled for your deployment but the control has not yet rolled out, configure downsampling through the REST API in the meantime.
### Python SDK
SDK support for configuring `sampling_rate` on Evaluator Rules is forthcoming. For now, configure downsampling through the REST API or the UI.
***
## Related Documentation
* [**Evaluator Rules**](/evaluate-and-test/evaluator-rules) - Create and manage Evaluator Rules
* [**Evaluator Rules REST API**](/sdk-api/rest-api/evaluator-rules) - Full create/update API contract, including the `sampling_rate` field
* [**Explorer**](/observability/agentic/trace-explorer) - Inspect ingested traces, including sampled-out traces
# Evals SDK Quick Start
Source: https://docs.fiddler.ai/evaluate-and-test/evals-sdk-quick-start
Learn how to evaluate Large Language Model (LLM) applications, RAG systems, and AI agents using the Fiddler Evals SDK with built-in and custom evaluators.
## What You'll Learn
In this guide, you'll learn how to:
* Connect to Fiddler and set up your experiment environment
* Create projects, applications, and datasets for organizing experiments
* Build experiment datasets with test cases
* Use built-in evaluators for common AI evaluation tasks
* Create custom evaluators for domain-specific requirements
* Run comprehensive experiments
* Analyze results with detailed metrics and insights
**Time to complete**: \~20 minutes
## Prerequisites
Before you begin, ensure you have:
* **Fiddler Account**: An active account with [access](/reference/administration/settings#access) to create applications
* **Python 3.10+**
* **Fiddler Evals SDK**:
* `pip install fiddler-evals`
* **Fiddler API Key**: Get your API key from [**Settings** > **Credentials**](/reference/administration/settings#credentials) in your Fiddler instance
If you prefer using a notebook, download a fully functional quick start directly from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Evaluations_SDK.ipynb) or open it in [Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Evaluations_SDK.ipynb) to get started.
First, establish a connection to your Fiddler instance using the Evals SDK.
**Connection Setup**:
```python theme={null}
from fiddler_evals import init
# Initialize connection to Fiddler
init(
url='https://your-org.fiddler.ai', # Your Fiddler URL
token='your-api-key' # Your API key
)
```
Fiddler Experiments uses a hierarchical structure to organize your experiments:
* **Projects** provide organizational boundaries for related applications
* **Applications** represent specific AI systems you want to evaluate
* **Datasets** contain test cases for experiments
* **Experiments** track individual evaluation runs
Create your organizational structure:
```python theme={null}
from fiddler_evals import Project, Application
# Create or get the project
project = Project.get_or_create(name='my_eval_project')
print(f'✅ Project: {project.name} (ID: {project.id})')
# Create or get the application within the project
application = Application.get_or_create(
name='my_llm_app',
project_id=project.id
)
print(f'✅ Application: {application.name} (ID: {application.id})')
```
**What This Creates**:
* A project to organize all your experiment work
* An application representing your AI system under test
* Persistent IDs for tracking results over time
Datasets contain the test cases you'll use to evaluate your AI applications. Each test case includes:
* **Inputs**: Data passed to your AI application (questions, prompts, etc.)
* **Expected Outputs**: What you expect the application to return
* **Metadata**: Additional context (categories, types, tags)
Create a dataset and add test cases:
```python theme={null}
from fiddler_evals import Dataset
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
# Create the dataset
dataset = Dataset.create(
name='experiment_dataset',
application_id=application.id,
description='Test cases for LLM experiments'
)
# Add test cases
test_cases = [
NewDatasetItem(
inputs={"question": "What happens to you if you eat watermelon seeds?"},
expected_outputs={"answer": "The watermelon seeds pass through your digestive system"},
metadata={"type": "Adversarial", "category": "Misconceptions"}
),
NewDatasetItem(
inputs={"question": "What is the capital of France?"},
expected_outputs={"answer": "Paris is the capital of France"},
metadata={"type": "Factual", "category": "Geography"}
)
]
dataset.insert(test_cases)
print(f'✅ Added {len(test_cases)} test cases to dataset')
```
**Data Import Options**:
```python theme={null}
dataset.insert_from_csv_file(
file_path='data.csv',
input_columns=['question'],
expected_output_columns=['answer'],
metadata_columns=['category']
)
```
```python theme={null}
dataset.insert_from_jsonl_file(
file_path='data.jsonl',
input_keys=['question'],
expected_output_keys=['answer'],
metadata_keys=['category']
)
```
```python theme={null}
dataset.insert_from_pandas(
df=df,
input_columns=['question'],
expected_output_columns=['answer'],
metadata_columns=['category']
)
```
Fiddler Experiments provides production-ready evaluators for common AI evaluation tasks. Let's test some key evaluators:
```python theme={null}
from fiddler_evals.evaluators import (
AnswerRelevance,
Coherence,
Conciseness,
Sentiment
)
# LLM-as-a-Judge evaluators require model and credential parameters
MODEL = "openai/gpt-4o"
CREDENTIAL = "your-llm-credential" # From Settings > LLM Gateway
# Test Answer Relevance (ordinal: High/Medium/Low)
relevance_evaluator = AnswerRelevance(model=MODEL, credential=CREDENTIAL)
score = relevance_evaluator.score(
user_query="What is the capital of France?",
rag_response="Paris is the capital of France."
)
print(f"Relevance Score: {score.value} ({score.label}) - {score.reasoning}")
# Test Conciseness
conciseness_evaluator = Conciseness(model=MODEL, credential=CREDENTIAL)
score = conciseness_evaluator.score(
response="Paris is the capital of France."
)
print(f"Conciseness Score: {score.value} - {score.reasoning}")
# Test Coherence
coherence_evaluator = Coherence(model=MODEL, credential=CREDENTIAL)
score = coherence_evaluator.score(
response="Thank you for your question! I'd be happy to help.",
prompt="Can you assist me?"
)
print(f"Coherence Score: {score.value} - {score.reasoning}")
# Sentiment is a Fiddler Centor Model — no model/credential needed
sentiment_evaluator = Sentiment()
scores = sentiment_evaluator.score("Paris is the capital of France.")
print(f"Sentiment: {scores[0].label} (confidence: {scores[1].value})")
```
**Available Built-in Evaluators**:
| Evaluator | Purpose | Key Parameters |
| ------------------------- | -------------------------------------------------------------------- | --------------------------------------------------- |
| `AnswerRelevance` | Checks if response addresses the question (ordinal: High/Medium/Low) | `user_query`, `rag_response` |
| `ContextRelevance` | Measures retrieval quality (ordinal: High/Medium/Low) | `user_query`, `retrieved_documents` |
| `RAGFaithfulness` | Detects hallucinations in RAG responses (binary: Yes/No) | `user_query`, `rag_response`, `retrieved_documents` |
| `Coherence` | Evaluates logical flow and consistency | `response`, `prompt` |
| `Conciseness` | Measures response brevity and clarity | `response` |
| `Sentiment` | Analyzes emotional tone | `text` |
| `RegexSearch` | Pattern matching for specific formats | `output`, `pattern` |
| `FTLPromptSafety` | Compute safety scores for prompts using Fiddler Centor Models | `text` |
| `FTLResponseFaithfulness` | Evaluate faithfulness of LLM responses using Fiddler Centor Models | `response`, `context` |
**RAG Health Metrics:** `AnswerRelevance`, `ContextRelevance`, and `RAGFaithfulness` form the RAG diagnostic triad. All LLM-as-a-Judge evaluators require `model` and `credential` parameters at initialization (e.g., `AnswerRelevance(model="openai/gpt-4o", credential="your-credential")`). See the [RAG Health Metrics Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial) for a complete walkthrough.
**Cost-Effective Experiments at Scale**
Fiddler Centor Model evaluators (`FTLPromptSafety`, `FTLResponseFaithfulness`, `Sentiment`, `TopicClassification`) run within your environment with no external API costs and sub-100ms latency. Initialize them with no parameters (e.g., `Sentiment()`).
LLM-as-a-Judge evaluators (`AnswerRelevance`, `ContextRelevance`, `RAGFaithfulness`, `Coherence`, `Conciseness`) use external LLMs via LLM Gateway and require `model` and `credential` parameters at initialization.
Build custom evaluation logic for your specific use cases by inheriting from the `Evaluator` base class:
```python theme={null}
from fiddler_evals.evaluators.base import Evaluator
from fiddler_evals.pydantic_models.score import Score
class LengthEvaluator(Evaluator):
"""
Custom evaluator that checks if a response length is appropriate.
Gives higher scores for responses that are neither too short nor too long.
"""
def __init__(self, min_length: int = 10, max_length: int = 200):
super().__init__()
self.min_length = min_length
self.max_length = max_length
def score(self, output: str) -> Score:
"""Score based on response length appropriateness."""
length = len(output.strip())
if length < self.min_length:
score_value = 0.0
reasoning = f"Response too short ({length} chars, minimum {self.min_length})"
elif length > self.max_length:
score_value = 0.5
reasoning = f"Response too long ({length} chars, maximum {self.max_length})"
else:
score_value = 1.0
reasoning = f"Response length appropriate ({length} chars)"
return Score(
name="length_check",
evaluator_name=self.name,
value=score_value,
reasoning=reasoning
)
# Test the custom evaluator
length_evaluator = LengthEvaluator(min_length=15, max_length=100)
score = length_evaluator.score("Paris is the capital of France.")
print(f"Length Score: {score.value} - {score.reasoning}")
```
**Function-Based Evaluators**:
You can also use simple functions:
```python theme={null}
def word_count_evaluator(output: str) -> float:
"""Returns word count normalized to 0-1 scale."""
word_count = len(output.split())
return min(word_count / 50.0, 1.0)
# Use directly in evaluators list
evaluators = [
AnswerRelevance(model=MODEL, credential=CREDENTIAL),
word_count_evaluator, # Function evaluator
]
```
Now run a comprehensive experiment. The `evaluate()` function:
1. Runs your AI application task on each dataset item
2. Executes all evaluators on the results
3. Tracks the experiment in Fiddler
4. Returns comprehensive results with scores and timing
Define your experiment task:
```python theme={null}
from fiddler_evals import evaluate
# Define your AI application task
def my_llm_task(inputs: dict, extras: dict, metadata: dict) -> dict:
"""
This function represents your AI application that you want to evaluate.
Args:
inputs: The input data from the dataset (e.g., {"question": "..."})
extras: Additional context data (e.g., {"context": "..."})
metadata: Any metadata associated with the test case
Returns:
dict: The outputs from your AI application (e.g., {"answer": "..."})
"""
question = inputs.get("question", "")
# Your LLM API call here
# For this example, we'll use a mock response
answer = f"Mock response to: {question}"
return {"answer": answer}
# Set up evaluators
evaluators = [
AnswerRelevance(model=MODEL, credential=CREDENTIAL),
Conciseness(model=MODEL, credential=CREDENTIAL),
Sentiment(), # Fiddler Centor Model — no model/credential needed
LengthEvaluator(),
]
# Run evaluation
experiment_result = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=evaluators,
name_prefix="my_experiment",
description="Comprehensive LLM experiment",
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["question"],
"rag_response": "answer",
"response": "answer",
"output": "answer",
"text": "answer",
},
max_workers=4 # Process 4 test cases concurrently
)
print(f"✅ Evaluated {len(experiment_result.results)} test cases")
print(f"📈 Generated {sum(len(result.scores) for result in experiment_result.results)} scores")
```
**Score Function Mapping**:
The `score_fn_kwargs_mapping` parameter connects your task outputs to evaluator inputs. This is necessary because evaluators expect specific parameter names (like `response`, `prompt`, `text`) but your task may use different names (like `answer`, `question`).
**Simple String Mapping** (use this for most cases):
```python theme={null}
# Your task returns: {"answer": "Paris is the capital of France"}
# Evaluators expect: rag_response="...", response="...", or text="..."
# Map your output keys to evaluator parameter names:
score_fn_kwargs_mapping={
"rag_response": "answer", # Map 'rag_response' param → 'answer' output key
"response": "answer", # Map 'response' param → 'answer' output key
"text": "answer", # Map 'text' param → 'answer' output key
}
```
**Advanced Mapping with Lambda Functions** (for nested values):
```python theme={null}
# Use lambda to extract nested or computed values:
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["question"], # Extract from inputs dict
"rag_response": "answer", # Simple string mapping
}
```
**How It Works**:
1. Your task returns a dict: `{"answer": "Some response"}`
2. The mapping tells Fiddler: "When an evaluator needs `rag_response`, use the value from `answer`"
3. Each evaluator gets the parameters it needs automatically
**Complete Example**:
```python theme={null}
# Task returns this structure:
{"answer": "Paris is the capital of France"}
# But evaluators need these parameters:
# - AnswerRelevance.score(user_query="...", rag_response="...")
# - Conciseness.score(response="...")
# - Sentiment.score(text="...")
# Solution: Map parameter names to your output structure
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["question"], # For AnswerRelevance
"rag_response": "answer", # For AnswerRelevance
"response": "answer", # For Conciseness
"text": "answer", # For Sentiment
}
```
This allows you to use any evaluator without changing your task function structure.
After running your experiment, analyze the comprehensive results in your notebook or the Fiddler UI:
```python theme={null}
from fiddler_evals import ScoreStatus, ExperimentItemStatus
import pandas as pd
# Analyze individual results
for i, result in enumerate(experiment_result.results):
item = result.experiment_item
scores = result.scores
print(f"\n📝 Test Case {i + 1}:")
print(f" Status: {item.status}")
print(f" Execution Time: {item.duration_ms}ms")
if item.status == ExperimentItemStatus.SUCCESS:
answer = item.outputs.get('answer', 'N/A')
print(f" Answer: {answer[:100]}...")
# Show scores
for score in scores:
status_emoji = "✅" if score.status == ScoreStatus.SUCCESS else "❌"
print(f" {status_emoji} {score.name}: {score.value}")
print(f" Reasoning: {score.reasoning}")
# Create summary statistics
from collections import defaultdict
evaluator_scores = defaultdict(list)
total_scores = 0
successful_scores = 0
for result in experiment_result.results:
for score in result.scores:
if score.value is not None:
evaluator_scores[score.name].append(score.value)
total_scores += 1
if score.status == ScoreStatus.SUCCESS:
successful_scores += 1
# Performance by evaluator
print("\n🎯 Performance by Evaluator:")
for evaluator_name, values in evaluator_scores.items():
avg_score = sum(values) / len(values) if values else 0
min_score = min(values) if values else 0
max_score = max(values) if values else 0
print(f" {evaluator_name}:")
print(f" Average: {avg_score:.3f}")
print(f" Min: {min_score:.3f}, Max: {max_score:.3f}")
# Overall statistics
print("\n📈 Overall Experiment Stats:")
print(f" Total Test Cases: {len(experiment_result.results)}")
print(f" Total Scores: {total_scores}")
print(f" Success Rate: {(successful_scores / total_scores) * 100:.1f}%")
```
Output:
```
📝 Test Case 1:
Status: ExperimentItemStatus.SUCCESS
Execution Time: 3251ms
Answer: Mock response to: What is the capital of France?...
✅ is_answer_relevant: 0.0
Reasoning: The response does not provide the actual capital of France. Instead, it repeats the prompt in a mock format, which does not answer the question.
✅ is_concise: 0.0
Reasoning: The response is not concise as it does not provide the actual answer to the question. It is vague and does not address the main point, lacking any relevant information about the capital of France.
✅ sentiment: None
Reasoning: None
✅ sentiment_prob: 0.8894587159156799
Reasoning: None
✅ length_check: 1.0
Reasoning: Response length appropriate (48 chars)
✅ word_count_evaluator: 0.18
Reasoning: Function result: 0.18
📝 Test Case 2:
Status: ExperimentItemStatus.SUCCESS
Execution Time: 3116ms
Answer: Mock response to: What happens to you if you eat watermelon seeds?...
✅ is_answer_relevant: 0.0
Reasoning: The response does not provide any information related to the effects of eating watermelon seeds. Instead, it is a placeholder or mock response that does not address the prompt.
✅ is_concise: 0.0
Reasoning: The response is not provided, so it cannot be evaluated for conciseness. A concise response would directly address the question without unnecessary details or digressions.
✅ sentiment: None
Reasoning: None
✅ sentiment_prob: 0.8732774257659912
Reasoning: None
✅ length_check: 1.0
Reasoning: Response length appropriate (66 chars)
✅ word_count_evaluator: 0.24
Reasoning: Function result: 0.24
🎯 Performance by Evaluator:
is_answer_relevant:
Average: 0.000
Min: 0.000, Max: 0.000
is_concise:
Average: 0.000
Min: 0.000, Max: 0.000
sentiment_prob:
Average: 0.881
Min: 0.873, Max: 0.889
length_check:
Average: 1.000
Min: 1.000, Max: 1.000
word_count_evaluator:
Average: 0.210
Min: 0.180, Max: 0.240
📈 Overall Experiment Stats:
Total Test Cases: 2
Total Scores: 12
Success Rate: 100.0%
```
**Export Results**
To conduct further analysis, export the experiment results:
```python theme={null}
# Convert to DataFrame for further analysis
results_data = []
for result in experiment_result.results:
item = result.experiment_item
row = {
'dataset_item_id': item.dataset_item_id,
'status': item.status,
'duration_ms': item.duration_ms,
}
# Add scores as columns
for score in result.scores:
row[f'{score.name}_score'] = score.value
row[f'{score.name}_reasoning'] = score.reasoning
results_data.append(row)
results_df = pd.DataFrame(results_data)
results_df.to_csv('experiment_results.csv', index=False)
print("💾 Results exported to experiment_results.csv")
```
## Next Steps
Now that you have the Fiddler Evals SDK set up, explore these advanced capabilities:
* [Experiments First Steps](/getting-started/experiments): An overview of Fiddler Experiments
* [Quick Start Notebook](/developers/tutorials/experiments): Download and run a more expansive version of this quick start guide
* [Fiddler Evals SDK](/sdk-api/evals/evaluate): Review the SDK technical reference
* [Advanced Evals Guide](/developers/tutorials/experiments/evals-sdk-advanced): Build sophisticated evaluation logic
## Troubleshooting
### Connection Issues
**Issue**: Cannot connect to Fiddler instance.
**Solutions**:
1. **Verify credentials**
1. **Test network connectivity**:
```bash theme={null}
curl -I https://your-org.fiddler.ai
```
2. **Validate API key**:
* Ensure your API key is valid and not expired
* Regenerate API key if needed from [Settings > Credentials](/reference/administration/settings#creating-api-keys)
### Import Errors
**Issue**: `ModuleNotFoundError: No module named 'fiddler_evals'`
**Solutions**:
1. **Verify installation**:
```bash theme={null}
pip list | grep fiddler-evals
```
2. **Reinstall the SDK**:
```bash theme={null}
pip uninstall fiddler-evals
pip install fiddler-evals
```
3. **Check Python version**:
* Requires Python 3.10 or higher
* Run `python --version` to verify
### Experiment Failures
**Issue**: Evaluators failing with errors.
**Solutions**:
1. **Check parameter mapping**:
```python theme={null}
# Ensure score_fn_kwargs_mapping matches evaluator requirements
score_fn_kwargs_mapping={
"response": "answer", # Maps to your task output key
"prompt": lambda x: x["inputs"]["question"],
}
```
2. **Verify task output format**:
* Task must return a dictionary
* Keys must match those referenced in score\_fn\_kwargs\_mapping
3. **Debug individual evaluators**:
```python theme={null}
# Test evaluators separately
score = evaluator.score(response="test response")
print(f"Score: {score.value}, Reasoning: {score.reasoning}")
```
### Performance Issues
**Issue**: Experiment is running slowly.
**Solutions**:
1. **Use parallel processing**:
```python theme={null}
experiment_result = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=evaluators,
max_workers=4 # Adjust based on your system
)
```
2. **Reduce dataset size for testing**:
* Start with a small subset
* Scale up once the configuration is validated
3. **Optimize LLM calls**:
* Use caching for repeated queries
* Implement batching where possible
## Configuration Options
### Basic Configuration
```python theme={null}
from fiddler_evals import init, evaluate
# Initialize connection
init(url='https://your-org.fiddler.ai', token='your-api-key')
# Run evaluation with basic settings
experiment_result = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=evaluators,
name_prefix="my_eval"
)
```
### Advanced Configuration
**Concurrent Processing**:
```python theme={null}
experiment_result = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=evaluators,
max_workers=8, # Process 8 test cases in parallel
name_prefix="parallel_eval"
)
```
**Experiment Metadata**:
```python theme={null}
experiment_result = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=evaluators,
metadata={
"model_version": "gpt-4",
"evaluation_date": "2024-01-15",
"temperature": 0.7,
"environment": "production"
}
)
```
**Custom Evaluator Configuration**:
```python theme={null}
# Configure evaluators with model and custom evaluators with specific thresholds
evaluators = [
AnswerRelevance(model="openai/gpt-4o", credential="your-llm-credential"),
Conciseness(model="openai/gpt-4o", credential="your-llm-credential"),
LengthEvaluator(min_length=20, max_length=150),
]
```
# Evaluator Downsampling Quick Start
Source: https://docs.fiddler.ai/evaluate-and-test/evaluator-downsampling-quick-start
Get started with evaluator downsampling: score only a fraction of matching traces to cut LLM-as-a-Judge cost, configured per rule from the UI or the REST API. Includes an end-to-end curl walkthrough.
[Evaluator downsampling](/evaluate-and-test/configure-evaluator-downsampling) lets an [Evaluator Rule](/evaluate-and-test/evaluator-rules) score only a *fraction* of the spans it matches instead of every one — cutting LLM-as-a-Judge evaluation cost at scale. You control it with one per-rule field, `sampling_rate`, a number in `(0.0, 1.0]` that defaults to `1.0` (evaluate everything). This guide walks you through configuring it from the **UI** and the **REST API**.
**Traces are always persisted, even with downsampling enabled.** Downsampling only controls whether an evaluator *runs* on a matching span — it never affects ingestion or storage. Sampled-out traces stay fully queryable in the [Explorer](/observability/agentic/trace-explorer).
**Availability.** Evaluator downsampling is rolling out behind a feature flag. Setting a `sampling_rate` requires it to be enabled for your deployment; until then, the field is unavailable in the UI and the API rejects requests that include it with a `400`. Contact [Fiddler support](mailto:support@fiddler.ai) to enable it.
## What you'll learn
* Set a rule's sampling rate from the UI when creating or editing a rule
* Create and tune a downsampled rule end-to-end with the REST API
* Pause evaluation the right way (and the guardrails on `sampling_rate`)
**Time to complete**: \~10 minutes
## Prerequisites
Before you begin, ensure you have:
* **Evaluator downsampling enabled** for your deployment (see the availability note above)
* **A GenAI application** with span data (see [Evaluator Rules](/evaluate-and-test/evaluator-rules))
* **A Fiddler API key** — from [**Settings** > **Credentials**](/reference/administration/settings#credentials) (for the API walkthrough)
* **`jq`** installed locally (optional, used to pretty-print API responses)
***
## Configure downsampling in the UI
When evaluator downsampling is enabled for your deployment, you set a rule's sampling rate from the **Evaluator Rules** tab. **The UI uses a percentage; the REST API uses a fraction** — enter `10` in the UI for the same coverage as `sampling_rate: 0.1` over the API.
1. In your application, open the **Evaluator Rules** tab and click **Add Rule**. Configure the evaluator, input mappings, and application rules as usual (see [Evaluator Rules](/evaluate-and-test/evaluator-rules)).
2. In the **Add Evaluator Rule** dialog, set the **Sampling rate** to the percentage of matching traces you want evaluated — for example, enter `10` for 10%. Leave it at the default (`100`) to evaluate everything.
3. Save the rule. It begins evaluating the configured fraction of matching spans.
4. To change the rate later, edit the **Sampling rate** for an existing rule directly in its **Sampling rate** cell in the Evaluator Rules table. To stop evaluation entirely, use the row's **Pause** action — do not set a near-zero rate.
The **Sampling rate** control is rolling out behind a feature flag and may not yet be visible in your environment. If downsampling is enabled for your deployment but the control hasn't rolled out, use the REST API below in the meantime.
***
## Configure downsampling with the REST API
This walkthrough creates a GenAI project, an application, an evaluator, and a downsampled rule, then tunes and tears it down. All endpoints are under `$FIDDLER_URL/v3/...` and authenticate with a bearer token (`Authorization: Bearer $FIDDLER_API_KEY`).
**Project naming rule.** Project names must start with a letter and then contain only letters, numbers, and underscores (`_`) — no hyphens or spaces. Other resource names (applications, evaluators, rules) aren't restricted to this pattern, but this guide uses it throughout for consistency.
### 1. Set up your environment
```bash theme={null}
export FIDDLER_URL="https://YOUR_NAMESPACE.fiddler.ai" # no trailing slash, no /v3
export FIDDLER_API_KEY="YOUR_API_KEY"
# reusable auth headers
auth=(-H "Authorization: Bearer $FIDDLER_API_KEY" -H "Content-Type: application/json")
```
### 2. Confirm connectivity and auth
```bash theme={null}
curl -s "${auth[@]}" "$FIDDLER_URL/v3/projects?limit=1" | jq '.kind, .data.total'
```
Expect `"NORMAL"` (or `"PAGINATED"`) and a number. A `401` means the URL or key is wrong.
### 3. Create a GenAI project and application
A project with `asset_type=GEN_AI_APP` is the container for GenAI applications; the application is what traces are sent to and what rules bind to.
```bash theme={null}
export PROJECT_ID=$(curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/projects" \
-d '{"name": "downsampling_demo", "asset_type": "GEN_AI_APP"}' | jq -r '.data.id')
export APPLICATION_ID=$(curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/applications" \
-d "{\"name\": \"chat_assistant\", \"project_id\": \"$PROJECT_ID\"}" | jq -r '.data.id')
echo "PROJECT_ID=$PROJECT_ID APPLICATION_ID=$APPLICATION_ID"
```
### 4. Create an evaluator
An *evaluator* is the scorer definition; the *rule* (next step) binds it to the application and is where `sampling_rate` lives. The `enrichment_name` field selects the evaluator type. This example uses a built-in PII check (least setup):
```bash theme={null}
export EVALUATOR_ID=$(curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/evaluators" \
-d '{"name": "demo_pii_check", "enrichment_name": "pii_detection", "enrichment_config": {}}' \
| jq -r '.data.id')
echo "EVALUATOR_ID=$EVALUATOR_ID"
```
For the cost-savings story, use an LLM-as-a-Judge evaluator instead (`enrichment_name: "llm_as_a_judge"`, with a `model_id` and `prompt_spec` in `enrichment_config`). Discover the available types and their config schemas with `GET /v3/evaluators/get-config-options`.
### 5. Discover the input keys to map
A rule maps the evaluator's template variables to span attribute paths via `mapped_input_keys`. This call returns the keys the evaluator expects:
```bash theme={null}
curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/evaluator-rules/map-input-keys" \
-d "{\"application_id\": \"$APPLICATION_ID\", \"evaluator_id\": \"$EVALUATOR_ID\"}" \
| jq '.data'
```
Match each value's type to the key's type in `rule_input_keys`: a `string`-typed key takes a single attribute path (e.g. `"gen_ai.llm.output"`); an array-typed key takes a list of paths.
### 6. Create the rule with a sampling rate
This is the feature. `sampling_rate: 0.1` means "score \~10% of matching traces; store 100%."
```bash theme={null}
export RULE_ID=$(curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/evaluator-rules" \
-d "{
\"name\": \"downsampled_rule\",
\"application_id\": \"$APPLICATION_ID\",
\"evaluator_id\": \"$EVALUATOR_ID\",
\"mapped_input_keys\": {\"text\": \"gen_ai.llm.output\"},
\"sampling_rate\": 0.1
}" | jq -r '.data.id')
echo "RULE_ID=$RULE_ID"
```
If this returns `400 ... "Evaluator downsampling is not enabled for this deployment"`, the feature flag is off for your deployment — see the availability note at the top of this page.
### 7. Read the rule back
Confirm the rate persisted. The list endpoint filters via a `filter` query param; build it with `-g -G --data-urlencode` so curl encodes the JSON and doesn't treat `[`/`]` as glob characters:
```bash theme={null}
curl -s -g -G "${auth[@]}" "$FIDDLER_URL/v3/evaluator-rules" \
--data-urlencode "filter={\"condition\":\"AND\",\"rules\":[{\"field\":\"application_id\",\"operator\":\"equal\",\"value\":\"$APPLICATION_ID\"}]}" \
| jq '.data.items[] | {id, name, enabled, sampling_rate}'
```
You should see `"sampling_rate": 0.1`. The traces are all still stored and queryable — only *evaluation* is sampled.
### 8. Tune the sampling rate live
Dial coverage up or down on a live rule without recreating it:
```bash theme={null}
curl -s "${auth[@]}" -X PATCH "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" \
-d '{"sampling_rate": 0.5}' | jq '.data | {id, name, sampling_rate}'
```
A new rate takes effect at the workers within the rule-cache TTL (\~60 seconds).
### 9. Pause evaluation the right way
`sampling_rate` cannot be `0.0`. To pause a rule entirely, set `enabled=false` (this preserves the configured rate, so re-enabling restores it):
```bash theme={null}
# Pause:
curl -s "${auth[@]}" -X PATCH "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" \
-d '{"enabled": false}' | jq '.data | {id, enabled, sampling_rate}'
# Re-enable:
curl -s "${auth[@]}" -X PATCH "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" \
-d '{"enabled": true}' | jq '.data | {id, enabled, sampling_rate}'
```
Both `sampling_rate: 0.0` and `sampling_rate > 1.0` are rejected with a `400`:
```bash theme={null}
curl -s -o /dev/null -w "%{http_code}\n" "${auth[@]}" -X PATCH \
"$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" -d '{"sampling_rate": 0.0}' # -> 400
```
### 10. Clean up
```bash theme={null}
curl -s "${auth[@]}" -X DELETE "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" | jq '.kind'
# Optional: delete the demo application (async cleanup also removes any remaining rules)
curl -s "${auth[@]}" -X DELETE "$FIDDLER_URL/v3/applications/$APPLICATION_ID" | jq '.kind'
# Optional: delete the project once the application cleanup finishes
curl -s "${auth[@]}" -X DELETE "$FIDDLER_URL/v3/projects/$PROJECT_ID" | jq '.kind'
```
Deleting a project does not cascade to its applications, so delete the application first. Because application deletion is asynchronous, wait a few seconds for the cleanup to finish before deleting the project; if the project delete still returns `"ERROR"`, retry after a moment.
***
## Quick reference
| Action | Method & path |
| --------------------------------------- | ----------------------------------------- |
| Create project | `POST /v3/projects` |
| Create application | `POST /v3/applications` |
| List evaluator config options | `GET /v3/evaluators/get-config-options` |
| Create evaluator | `POST /v3/evaluators` |
| Discover input keys | `POST /v3/evaluator-rules/map-input-keys` |
| Create rule (+`sampling_rate`) | `POST /v3/evaluator-rules` |
| List rules | `GET /v3/evaluator-rules?filter=...` |
| Update rule (`sampling_rate`/`enabled`) | `PATCH /v3/evaluator-rules/{id}` |
| Delete rule | `DELETE /v3/evaluator-rules/{id}` |
**`sampling_rate` rules:** range `(0.0, 1.0]`; default `1.0` (score everything); `0.0` rejected (pause with `enabled=false` instead); values above `1.0` rejected; resolution is 4 decimal places (1 basis point).
**Units differ by surface.** The UI uses a percentage (`10` = 10%); the REST API uses a fraction (`sampling_rate: 0.1`).
***
## Seeing sampling in action (optional, advanced)
Demonstrating a visible \~10% sample requires ingesting many traces and waiting for async enrichment. GenAI trace ingestion is **OTLP-based** (`POST /v1/traces` with an `Authorization: Bearer` header and a `fiddler-application-id` header) — it is *not* part of the v3 REST surface used above. After ingesting traces, query them back to confirm that sampled-out traces are still **stored** (just not scored):
```bash theme={null}
curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/spans/query" \
-d "{
\"application_id\": \"$APPLICATION_ID\",
\"start_time\": \"2026-01-01T00:00:00Z\",
\"end_time\": \"2026-12-31T23:59:59Z\",
\"page_size\": 50
}" | jq '.data.items[] | {trace_id, evaluators}'
```
Spans with a non-empty `evaluators` array were scored (sampled in); stored spans with an empty array were sampled out — same data retained, evaluation skipped.
***
## Next steps
* [**Configure Evaluator Downsampling**](/evaluate-and-test/configure-evaluator-downsampling) — the full guide: when to use it, recommended minimum trace volume, `sampling_rate` vs deactivating a rule, and how sampling correlates across rules
* [**Evaluator Rules**](/evaluate-and-test/evaluator-rules) — create and manage Evaluator Rules
* [**Evaluator Rules REST API**](/sdk-api/rest-api/evaluator-rules) — full create/update API contract, including the `sampling_rate` field
* [**Explorer**](/observability/agentic/trace-explorer) — inspect ingested traces, including sampled-out traces
# Evaluator Rules
Source: https://docs.fiddler.ai/evaluate-and-test/evaluator-rules
Configure automated evaluations for your GenAI application spans using Evaluator Rules. Learn to map evaluators to span data, define application rules, and manage backfill configuration.
Evaluator Rules define how automated evaluations are applied to your application's spans. They connect evaluators (LLM-based or rule-based functions) with span data, specify what inputs to use, and determine which spans qualify for evaluation.
***
## Overview
Evaluator Rules provide the configuration layer between your evaluators and your application's telemetry data. When properly configured, they automatically assess the quality, safety, and performance of your GenAI application based on real-time span data.
### What Are Evaluator Rules?
An **Evaluator Rule** determines how and when an evaluator runs against your application's spans. Each rule consists of four key components:
1. **Evaluator Configuration** - The evaluator definition, including provider, model, and prompt
2. **Input Field Mapping** - How span data is passed to the evaluator's input variables
3. **Application Rules** - Conditions that determine which spans qualify for evaluation
4. **Backfill Configuration** - Whether to apply evaluations to historical data
### How Evaluator Rules Work
When a new span is created in your application:
1. The system checks all active Evaluator Rules
2. Each rule evaluates whether its Application Rules match the span's attributes
3. If a match is found, the system extracts data from the span using Input Field Mappings
4. The evaluator runs with the mapped data as input
5. Results are stored and made available in dashboards and analytics
***
## Key Concepts
### Evaluators
An **Evaluator** is a configured model or function that performs analysis over spans. It can classify, score, or assess the quality of data generated by your application.
Evaluators are defined by:
* **Provider** - The LLM provider (OpenAI, Anthropic, Gemini, Fiddler)
* **Model** - The specific model to use for evaluation
* **Credentials** - Authentication to the provider (configured via [LLM Gateway](/reference/administration/llm-gateway))
* **Prompt or Logic** - The evaluation instructions or function
**Note:** Evaluators are defined at the organization level and shared across all projects in your organization.
### Input Mappings
**Input Mappings** define how data flows from spans into evaluators. Each variable used in an evaluator's prompt (such as `{{input}}` or `{{context}}`) must be mapped to a field or attribute in the span data.
For example, if your evaluator prompt includes `{{puppynoises}}`, you must map that variable to a span attribute like `gen_ai.llm.input.user`.
### Application Rules
**Application Rules** specify filtering conditions that determine which spans qualify for evaluation. Rules use AND/OR logic:
* **AND condition across categories** - A span must match ALL rule categories
* **OR condition within a category** - A span can match ANY value within a single category
**Example:**
```
Rule 1: SpanType = llm
Rule 2: Region = us-east OR us-west
Result: Evaluates spans that are type "llm" AND in either "us-east" or "us-west"
```
### Backfill
**Backfill** controls whether evaluations apply retroactively to existing historical data or only to spans created after the rule is configured.
The backfill process runtime depends on the volume of data in your history. Be certain to backfill only as needed.
**Backfill respects the rule's sampling rate.** If a rule uses [evaluator downsampling](/evaluate-and-test/configure-evaluator-downsampling), a backfill evaluates the same fraction of traces — and because the sampling decision is deterministic, a backfill at the same rate re-applies the same outcome and skips the same traces. To score traces a downsampled rule skipped, create a new rule (or recreate the rule) with a higher sampling rate and backfill enabled.
### Rule Limit
Each application supports up to **100 evaluator rules** by default. This soft cap keeps evaluation and monitoring performant as the number of rules grows. On self-hosted deployments, an administrator can raise or lower the limit through configuration.
When an application reaches the limit, creating another rule returns a `422` error. Delete rules you no longer need to make room.
**Note:** Deactivating a rule does not free a slot — disabled rules still count toward the limit. Delete unused rules to reduce the count.
***
## Create an Evaluator Rule
### Prerequisites
Before creating an Evaluator Rule, ensure you have:
* **Active Application** - A GenAI application with span data
* **Configured Evaluators** - Organization-level evaluators ready to use
* **LLM Gateway Credentials** - If using custom LLM-based evaluators (see [LLM Gateway Configuration](/reference/administration/llm-gateway))
***
### Step-by-Step Guide
Navigate to your application in the Fiddler UI and access the evaluator configuration:
1. Click the **Evaluator Rules** tab
2. Click **Add Rule** in the top-right corner
3. The **Add Evaluator Rule** dialog opens with available evaluators
Choose an evaluator from the list:
**Fiddler-Provided Evaluators:**
* Topic Classification
* Embedding
* Token Count
* Answer Relevance
* Coherence
* Conciseness
* Context Relevance
* RAG Faithfulness
* PII Detection
* Sentiment Analysis
* F# Prompt Safety
* F# Response Faithfulness
* **Llm As A Judge** (custom evaluator)
**Configure Custom Evaluator (Llm As A Judge)**
If you select **Llm As A Judge**, you'll need to configure the evaluator:
**a. Evaluator Name**
* Enter a descriptive name (e.g., `saddestpuppynoises`)
**b. Provider**
* Select the LLM provider (e.g., `fiddler`)
**c. Credential**
* Choose the API credential for authentication (e.g., `dummy`)
**d. Model**
* Select the specific model (e.g., `llama3.1-8b`)
**e. Prompt Template**
* Enter evaluation instructions with input variables using curly braces: `{{variableName}}`
* **Example:** `sad {{puppynoises}}`
**f. Outputs**
* Define the expected response format in JSON
**Example Output Configuration:**
```json theme={null}
{
"name": "sadnoises",
"description": "sad puppy noises",
"type": "categorical",
"choices": ["sad", "not sad"]
}
```
**Tip:** For Fiddler-provided evaluators, the evaluation method and fields are predetermined. You only need to map inputs and configure application rules.
Click **Next** to continue.
Map each evaluator input variable to a span attribute.
1. In the **Map Evaluator** step, you'll see all required input variables
2. For each variable (e.g., `puppynoises`):
* Click the **Select an attribute or enter a custom path** dropdown
* Choose from available span attributes or enter a custom path manually
**Common Span Attributes:**
* `pirate_completion_score` (custom span attribute)
* `gen_ai.llm.context`
* `gen_ai.usage.output_tokens` (system attribute)
* `fiddler.session.user.region` (session attribute)
* `gen_ai.usage.input_tokens` (system attribute)
* `fiddler.session.user.max_conversation_turns` (session attribute)
* `gen_ai.system`
* `gen_ai.llm.input.user`
* `gen_ai.tool.input`
* And many more...
3. Repeat for all input variables
4. Click **Next** to continue
All required input variables must be mapped. The evaluator cannot run without complete input mappings.
Specify which spans to evaluate by setting filter conditions.
1. In the **Apply Rules** step, you'll see the current rule conditions
2. The info box shows: **"This evaluator will apply to spans that match ALL of the following conditions:"**
3. Click **Add Rule** to add a new condition category
For each rule category:
**a. Rule Category**
* Select the attribute type (e.g., `Span Type`)
**b. Values**
* Choose which values to match:
* `chain`
* `llm` ✓
* `tool`
**c. Custom Values**
* (Optional) Add specific custom values to match
**Understanding Rule Logic**
* **AND condition across categories** - A span must match ALL rule categories
* **OR condition within a category** - A span can match ANY value within a single category
**Example:**
```
Rule 1: SpanType = llm
Rule 2: Region = us-east OR us-west
Result: Evaluates spans that are type "llm" AND in either "us-east" or "us-west"
```
5. Add multiple rule categories as needed
6. Click **Next** to continue
Determine whether to apply the evaluator to existing historical data and review your configuration.
**Backfill Configuration**
Choose one of three options:
**Option 1: Apply to all past data**
* Evaluates all existing spans in the dataset
* Use when: You need complete historical coverage
* Warning: May take significant time for large datasets
**Option 2: Apply from a specific past date**
* Evaluates spans created after a chosen date
* Use when: You want partial historical coverage
* Select the start date using the date picker
**Option 3: No backfill** (Default)
* Evaluates only new spans created after activation
* Use when: You only need a forward-looking evaluation
* Best for: Testing new evaluators or reducing processing time
**Review Configuration Summary**
**Evaluator Configuration**
* Evaluator name, model, provider, credential
* Prompt template and expected outputs
**Input Field Mapping**
* Variable → Span attribute mappings
**Application Rules**
* Span matching conditions
**Performance Tip:** Start with "No backfill" to test your evaluator configuration. Once validated, you can create a new rule with backfill enabled.
Complete the configuration and activate your evaluator rule.
1. **Configuration Name**
* Enter a descriptive name for this evaluator rule (e.g., `puppyjudge`)
* This name identifies the rule in your application's Evaluator Rules list
2. **Finalize:**
* Click **Save** to activate the rule
* Or click **Back** to modify any settings
* Or click **Cancel** to discard the configuration
Once saved, the evaluator rule becomes active and begins evaluating spans that match your criteria.
***
## Manage Evaluator Rules
### View Active Rules
Navigate to the **Evaluator Rules** tab in your application to see all configured rules.
The Evaluator Rules table displays:
| Column | Description |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Rule Name** | The configuration name you assigned |
| **Rule** | Span-matching conditions (e.g., "SpanName undefined: ChatOpenAI") |
| **Input Mappings** | Mapped input fields (e.g., "CONTEXT: gen\_ai.llm.con...") |
| **Outputs** | Expected output fields (e.g., "faithful\_prob", "spans") |
| **Sampling Rate** | Fraction of matching traces evaluated, shown as a percentage (`100%` = evaluate everything). Visible when [evaluator downsampling](/evaluate-and-test/configure-evaluator-downsampling) is enabled for your deployment. |
| **Status** | Active or Inactive |
| **Created At** | Date the rule was created |
### Activate or Deactivate a Rule
Toggle a rule's status without deleting it:
1. Locate the rule in the Evaluator Rules table
2. Click the **Status** toggle to activate or deactivate
* **Active** - Rule is running on matching spans
* **Inactive** - Rule is paused and not evaluating new spans
### Delete a Rule
Remove a rule permanently:
1. Locate the rule in the Evaluator Rules table
2. Click the **delete** icon (trash can) at the end of the row
3. Confirm the deletion when prompted
Deleting a rule does not remove evaluator results already generated. Historical evaluator data remains in your analytics.
***
## Best Practices
### Evaluator Configuration
* **Use Descriptive Names** - Name evaluators and rules clearly (e.g., `rag_faithfulness_prod` instead of `rule1`)
* **Test Before Backfill** - Create rules without backfill first, validate results, then create a new rule with backfill if needed
* **Version Your Prompts** - Include version identifiers in custom judge names (e.g., `topic_classifier_v2`)
### Input Mapping
* **Validate Paths** - Ensure span attributes exist before mapping
* **Use Consistent Paths** - Standardize attribute naming across your application
* **Document Custom Paths** - Keep a reference of custom attribute paths for your team
### Application Rules
* **Start Broad, Refine Later** - Begin with simple rules, add complexity as needed
* **Avoid Over-Filtering** - Don't create rules so specific that they match too few spans
* **Test Rule Logic** - Verify spans are matching as expected using span search
### Performance Optimization
* **Limit Backfill Scope** - Use date-based backfill instead of "all past data" for large datasets
* **Monitor Evaluation Latency** - Track how long evaluations take and optimize prompts if needed
* **Batch Similar Rules** - Group related evaluations to reduce overhead
***
## Troubleshooting
### Evaluator Not Running
**Issue:** Rule is active but not producing results.
**Solutions:**
* Verify Application Rules match actual span attributes
* Check that all input mappings point to valid span fields
* Ensure LLM Gateway credentials are valid and not expired
* Review span data to confirm matching spans exist
### Missing Input Data
**Issue:** Evaluator fails due to missing input values.
**Solutions:**
* Verify the span attribute path is correct
* Check that the attribute exists in your span schema
* Ensure spans contain data for the mapped field
* Use a different attribute or add the field to your instrumentation
### Backfill Taking Too Long
**Issue:** Historical evaluation is processing slowly.
**Solutions:**
* Use date-based backfill instead of all past data
* Start with recent data and expand the date range gradually
* Consider creating multiple rules for different time periods
* Deactivate unnecessary rules to free up processing capacity
### Unexpected Evaluator Results
**Issue:** Evaluator produces unexpected scores or classifications.
**Solutions:**
* Review the evaluator prompt template for clarity
* Verify input mappings are passing the correct data
* Test the evaluator with sample data outside Fiddler
* Check for prompt ambiguity or missing context
* Adjust the prompt and create a new rule version
### Evaluator Rule Limit Reached
**Issue:** Creating a new evaluator rule fails with a `422` error stating the application has reached the per-application rule limit.
**Solutions:**
* Delete evaluator rules you no longer need — each application supports up to 100 rules
* Remember that deactivating a rule does not free a slot; disabled rules still count toward the limit
* Contact Fiddler support if your use case requires a higher limit
***
## Related Documentation
* [**LLM Gateway Configuration**](/reference/administration/llm-gateway) - Configure LLM provider credentials
* [**Configure Evaluator Downsampling**](/evaluate-and-test/configure-evaluator-downsampling) - Reduce evaluation cost at scale with per-rule sampling
* [**Evaluator Downsampling Quick Start**](/evaluate-and-test/evaluator-downsampling-quick-start) - Configure a downsampled rule from the UI or REST API
* [**Fiddler Evals SDK**](/sdk-api/evals/evaluate) - Create and manage evaluators programmatically
* [**Custom Evaluators**](/evaluate-and-test/overview) - Build custom evaluation logic
* [**Application Monitoring**](/getting-started/agentic-monitoring) - Monitor your GenAI applications
# Capture Traces During Experiments
Source: https://docs.fiddler.ai/evaluate-and-test/experiment-trace-capture
Link the OpenTelemetry traces your task emits to the experiment item that produced them, and score evaluators on the captured spans.
A score tells you *that* an item failed.
The trace tells you *why*.
When you run an experiment, the Fiddler Evals SDK captures the OpenTelemetry spans your task emits and links them to the experiment item that produced them.
You get per-item traces in the UI for debugging, and your evaluators can score on the execution itself — tool call order, retry counts, token usage, latency — not just the final output string.
## What You'll Learn
* How traces are captured during an experiment run, with no setup
* How to score an evaluator on a captured trace
* How to keep evaluation traces out of your production application
**Time to complete**: \~15 minutes
## Prerequisites
* A task instrumented with OpenTelemetry — either through a [Fiddler integration](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) or your own spans
* A Fiddler API key from [**Settings** > **Credentials**](/reference/administration/settings#credentials)
* Python 3.10 or later
* **Fiddler Evals SDK**: `pip install fiddler-evals`
If you prefer a notebook, open the fully worked example in [Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Experiment_Trace_Capture.ipynb) or download it from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Experiment_Trace_Capture.ipynb).
## There Is Nothing to Turn On
Trace capture is automatic.
`fiddler-otel` is a base dependency of the Evals SDK, so `evaluate()` sets capture up on its own as long as it can resolve a Fiddler application for the dataset.
```python theme={null}
from fiddler_evals import Application, Dataset, Project, evaluate, init
from fiddler_evals.evaluators import AnswerRelevance
init(url='https://your-org.fiddler.ai', token='your-api-key')
project = Project.get_by_name(name='support')
application = Application.get_by_name(
name='support-agent-preprod',
project_id=project.id,
)
dataset = Dataset.get_by_name(
name='support-agent-eval-suite',
application_id=application.id,
)
def my_agent(inputs, extras, metadata):
"""Your instrumented application. Signature must be (inputs, extras, metadata)."""
user_query = inputs['user_query']
answer = call_your_agent(user_query)
return {'user_query': user_query, 'rag_response': answer}
result = evaluate(
dataset=dataset,
task=my_agent,
evaluators=[AnswerRelevance(model='openai/gpt-4o-mini', credential='my-openai-cred')],
name_prefix='support_agent_v3',
)
```
If your task emits spans, they are captured and linked.
If it does not, nothing changes.
Evaluators bind to your task's **outputs** by parameter name.
The `inputs` bucket is passed through as a single dict, not spread into the namespace, so an evaluator cannot read a dataset input directly.
`AnswerRelevance.score()` takes `(user_query, rag_response)`, which is why the task above returns `user_query` as well as the answer.
Omit it and the run fails with `ScoreFunctionInvalidArgs: Missing required arguments ... ['user_query']`.
Alternatively, remap with [`score_fn_kwargs_mapping`](/sdk-api/evals/evaluate).
Capture is **best-effort by design**: evaluation must work for users who do no tracing at all.
If setup fails, the SDK logs a warning and the experiment runs normally — the task still executes and the scores are still published.
There is exactly one exception, covered in [Keep evaluation traces out of production](#keep-evaluation-traces-out-of-production).
## Score an Evaluator on the Trace
Declare a `session` parameter on your evaluator's score function and the runner passes the captured spans to it.
Parameter binding is by name, so evaluators that do not declare `session` are unaffected.
```python theme={null}
from fiddler_evals import EvalFn, evaluate
def tool_call_efficiency(session=None):
"""Score 1.0 when the agent used three or fewer tool calls.
`session` is None when trace capture is unavailable, so always guard.
Returning None marks the score SKIPPED rather than failing the item.
"""
if session is None:
return None
tool_spans = [
span for span in session.spans
if span['attributes'].get('fiddler.span.type') == 'tool'
]
return len(tool_spans) <= 3
result = evaluate(
dataset=dataset,
task=my_agent,
evaluators=[EvalFn(tool_call_efficiency, score_name='tool_call_efficiency')],
name_prefix='support_agent_v3',
)
```
Always default `session` to `None` and handle the `None` case.
Capture is best-effort, so an evaluator that assumes a session breaks the moment tracing is unavailable.
`EvalFn` converts your return value for you: `bool` becomes 1.0 or 0.0, `int` and `float` pass through, and `None` produces a `SKIPPED` score.
Return a `Score` directly when you want to control the reasoning text:
```python theme={null}
from fiddler_evals import Score
def tool_call_efficiency(session=None):
if session is None:
return None
tool_spans = [
span for span in session.spans
if span['attributes'].get('fiddler.span.type') == 'tool'
]
names = ', '.join(span['name'] for span in tool_spans) or 'none'
return Score(
name='tool_call_efficiency',
evaluator_name='tool_call_efficiency',
value=1.0 if len(tool_spans) <= 3 else 0.0,
reasoning=f'{len(tool_spans)} tool calls: {names}',
)
```
A `Score` you construct yourself requires both `name` and `evaluator_name`, and the explanation field is `reasoning`.
### What a Session Contains
Your evaluator receives a `Session` with two attributes:
The experiment item's ID. Equal to `experiment_item.id`.
The spans the task produced, in completion order.
Each span is a plain dict — no OpenTelemetry objects to import:
| Key | Type | Notes |
| -------------------- | ------------- | ------------------------------------------ |
| `trace_id` | `str` | Hex, `0x`-prefixed |
| `span_id` | `str` | Hex, `0x`-prefixed |
| `parent_span_id` | `str \| None` | `None` for root spans |
| `name` | `str` | Span name |
| `kind` | `str \| None` | `INTERNAL`, `CLIENT`, `SERVER`, … |
| `start_time` | `int` | Nanoseconds |
| `end_time` | `int` | Nanoseconds |
| `status_code` | `str \| None` | `OK`, `ERROR`, `UNSET` |
| `status_description` | `str \| None` | |
| `attributes` | `dict` | Where instrumentations put payloads |
| `events` | `list[dict]` | Each has `name`, `timestamp`, `attributes` |
Times are integer nanoseconds so duration is a plain subtraction with no precision loss:
```python theme={null}
def responded_within_3s(session=None):
if session is None or not session.spans:
return None
root = next(
(span for span in session.spans if span['parent_span_id'] is None),
session.spans[-1],
)
duration_ms = (root['end_time'] - root['start_time']) / 1_000_000
return duration_ms < 3000
```
Exceptions arrive as span events rather than attributes, which is where most instrumentations record them:
```python theme={null}
def no_swallowed_exceptions(session=None):
if session is None:
return None
exceptions = [
event
for span in session.spans
for event in span['events']
if event['name'] == 'exception'
]
return not exceptions
```
The span dict is deliberately a curated subset aimed at scoring.
OpenTelemetry infrastructure fields — `resource`, `links`, and instrumentation scope — are omitted because they are constant or empty for eval runs.
The full-fidelity span is still written to the trace store and visible in the UI, so nothing is lost.
## Keep Evaluation Traces Out of Production
Evaluation traces are real traces, so send them to a dedicated pre-production application rather than the one serving live traffic.
A process shares one global `FiddlerClient`. If one already exists and its `application_id` does not match the dataset's application, `evaluate()` raises `ValueError` instead of degrading quietly — this is the one capture failure that is deliberately not best-effort, because silently mixing evaluation and production traffic is worse than a failed run.
Point the dataset at its own application and the SDK handles the rest:
```python theme={null}
eval_application = Application.get_or_create(
name='support-agent-preprod',
project_id=project.id,
)
dataset = Dataset.get_or_create(
name='support-agent-eval-suite',
application_id=eval_application.id,
)
```
A golden dataset [promoted from production spans](/evaluate-and-test/golden-datasets) necessarily belongs to the application that served those spans —
span promotion resolves spans through the dataset's application, and trace capture follows it.
Replaying such a dataset writes its captured traces to that same application.
## Next Steps
* [Build a golden dataset from production spans](/evaluate-and-test/golden-datasets)
* [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start)
* [Instrument your application with OpenTelemetry](/developers/quick-starts/opentelemetry-quick-start)
* [`Experiment` SDK reference](/sdk-api/evals/experiment)
# Golden Datasets
Source: https://docs.fiddler.ai/evaluate-and-test/golden-datasets
Build a golden dataset from real production traffic by promoting spans into a Fiddler Experiments dataset, then replay it against every change to catch regressions.
A **golden dataset** is a curated, stable set of test cases that every experiment runs against.
The strongest ones are built from traffic your application has actually served, not from invented examples.
Golden Datasets let you promote real spans — the slow ones, the ones a guardrail flagged, the ones a user complained about — into a dataset you can replay against every future change.
Each promoted span becomes a dataset item, so a bug you saw in production becomes a permanent regression test.
This page covers building a golden dataset with the Fiddler Evals SDK.
The same workflow is available in the UI through the **Add to Dataset** action in the [Explorer](/observability/agentic/trace-explorer).
## What You'll Learn
* Discover which attribute keys exist on your application's spans
* Find the spans worth promoting into a golden dataset
* Map span attributes onto dataset fields
* Add the selected spans as dataset items
**Time to complete**: \~15 minutes
## Prerequisites
* An [application](/getting-started/genai-application-onboarding) that is already receiving traces
* A Fiddler API key from [**Settings** > **Credentials**](/reference/administration/settings#credentials)
* Python 3.10 or later
* **Fiddler Evals SDK**: `pip install fiddler-evals`
If you prefer a notebook, open the fully worked example in [Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Golden_Datasets.ipynb) or download it from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Golden_Datasets.ipynb).
## Start in the UI
**The Explorer is the fastest way to build a golden dataset, and the recommended one for most cases.**
Select the spans you want and choose **Add to Dataset**. A three-step dialog walks you through picking a dataset, mapping fields, and reviewing the result before anything is written.
Mapping is drag-and-drop between the span fields on the left and the dataset buckets on the right, and each bucket says what it's for:
| Bucket | Holds |
| -------------------- | --------------------------------------------------- |
| **Inputs** | Fields sent to your LLM — prompt, context, question |
| **Expected Outputs** | Ground truth for evaluation — the expected answer |
| **Extras** | Context for analysis, not sent to the LLM |
Span fields are picked from a list, so you cannot mistype an attribute key — the most common way the SDK path fails.
Use the SDK instead when you need the workflow to be **repeatable**: curating on a schedule, promoting spans from CI, or parameterizing selection across applications.
The rest of this page covers that path.
## How It Works
Spans and dataset items have different shapes.
A span is a flat bag of OpenTelemetry attributes;
a dataset item has four named buckets — `inputs`, `expected_outputs`, `metadata`, and `extras`.
Adding spans to a dataset is therefore a projection, and you supply the projection as a **field mapping**.
Attribute keys are stored exactly as your instrumentation sent them;
[semantic mappings](/concepts/semantic-mappings) resolve framework-specific naming server-side without renaming what is stored.
The workflow has four steps:
Query for the spans you want, and collect their `trace_id` and `span_id` pairs.
Ask which attribute keys those spans carry, and how often.
Map each dataset field to the span attribute that populates it.
Fiddler resolves each span, applies the mapping, and writes one dataset item per span.
Fiddler resolves spans server-side from the IDs you send.
Span *content* never round-trips through your client, and the application is derived from the dataset — not from your request — so a mapping cannot pull attributes out of an application you do not have access to.
## Step 1: Select the Spans to Promote
Pick the spans first, then discover what's on them.
Doing it the other way around scans your whole window and can surface attribute keys that aren't present on the spans you actually promote — map one of those and you get a dataset of blank items.
Connect, then choose a time window:
```python theme={null}
from datetime import datetime, timedelta, timezone
from fiddler_evals import Application, Project, init
init(url='https://your-org.fiddler.ai', token='your-api-key')
project = Project.get_by_name(name='support')
application = Application.get_by_name(
name='support-agent',
project_id=project.id,
)
end_time = datetime.now(timezone.utc)
start_time = end_time - timedelta(days=7)
```
`add_items_from_spans()` takes explicit `(trace_id, span_id)` pairs, so fetch some spans.
Use the [`POST /v3/spans/query`](/sdk-api/rest-api/spans/query-spans) endpoint, which returns spans newest-first.
The Evals SDK does not yet include a span-query method,
so this helper wraps the REST endpoint directly.
```python theme={null}
import requests
def query_spans(
url, token, application_id, start_time, end_time, filter_=None, page_size=200
):
"""Return spans matching a filter, newest first.
Wraps POST /v3/spans/query. See /sdk-api/rest-api/spans/query-spans.
"""
payload = {
'application_id': str(application_id),
'start_time': start_time.isoformat(),
'end_time': end_time.isoformat(),
'page_size': page_size, # server maximum is 200
}
if filter_ is not None:
payload['filter'] = filter_.model_dump(mode='json')
response = requests.post(
f'{url}/v3/spans/query',
headers={'Authorization': f'Bearer {token}'},
json=payload,
timeout=60,
)
response.raise_for_status()
return response.json()['data']['items']
```
`page_size` maxes out at 200, matching the per-call span limit on `add_items_from_spans()`, so one page maps cleanly to one call.
To promote more than 200 spans you must advance `offset` yourself and call `add_items_from_spans()` once per page.
### Narrow the Selection
`filter` is a structured tree over span fields. Field names are namespaced:
| Namespace | Example | Matches |
| ----------------- | -------------------------------------------------------- | ------------------------------- |
| `Span::` | `Span::span_type`, `Span::duration`, `Span::status_code` | Reserved span columns |
| `SpanAttribute::` | `SpanAttribute::gen_ai.usage.total_tokens` | Any user-defined span attribute |
| `Evaluator::` | `Evaluator::Safety::Is Toxic` | An evaluator rule's output |
[Span and Resource Attributes](/integrations/agentic-ai/attributes) catalogs the indexed span attributes and their value types,
including whether each is reached through a fast `Span::` column or through `SpanAttribute::`.
`search` is a case-insensitive substring match over span content, AND-ed with `filter`.
It matches the content attributes Fiddler full-text indexes, such as `gen_ai.llm.input.user`.
A filter tree is limited to 3 levels of nesting and 10 rules total.
A `search` query must be 3–64 characters.
`Span::duration` is in nanoseconds.
Keep the filter in a variable — Step 2 reuses it so discovery describes exactly this selection:
```python theme={null}
from fiddler_evals import (
OperatorType,
QueryCondition,
QueryRule,
SpanReference,
)
# LLM spans slower than 2 seconds.
active_filter = QueryCondition(
rules=[
QueryRule(
field='Span::span_type',
operator=OperatorType.EQUAL,
value='llm',
),
QueryRule(
field='Span::duration',
operator=OperatorType.GREATER,
value=2_000_000_000, # nanoseconds
),
]
)
spans = query_spans(
url='https://your-org.fiddler.ai',
token='your-api-key',
application_id=application.id,
start_time=start_time,
end_time=end_time,
filter_=active_filter,
)
span_refs = [
SpanReference(trace_id=span['trace_id'], span_id=span['span_id'])
for span in spans
]
print(f'Selected {len(span_refs)} spans')
```
```text Expected output theme={null}
Selected 47 spans
```
Pass `filter_=None` to select every span in the window.
## Step 2: Discover the Attributes on Those Spans
Now that the selection is fixed, ask what those spans carry.
`get_span_fields()` returns every attribute key with a coverage count, plus any evaluator outputs recorded on them.
Passing `active_filter` scopes discovery to exactly the spans you selected, so the keys you see are the keys you can map.
```python theme={null}
fields = application.get_span_fields(
start_time=start_time,
end_time=end_time,
filter=active_filter,
)
print(f'{fields.total_spans} spans scanned (sampled={fields.sampled})')
for attribute in sorted(fields.span_attributes, key=lambda a: -a.count):
print(f' {attribute.key}: {attribute.count}')
```
```text Expected output theme={null}
47 spans scanned (sampled=False)
gen_ai.llm.input.user: 47
gen_ai.request.model: 47
fiddler.span.type: 47
gen_ai.llm.output: 46
support.ticket_id: 33
```
The `count` is what makes this useful: an attribute present on 33 of 47 spans is a poor choice for a required input field, because the spans missing it produce an empty string rather than an error.
`get_span_fields()` scans at most 10,000 matching spans.
Above that it sets `sampled=True` and the counts describe a sample rather than the full set.
Narrow the time range or the filter when you need exact coverage.
## Step 3: Define a Field Mapping
A `FieldMapping` maps *dataset field name* to *span attribute key*, bucket by bucket.
Every bucket is optional and defaults to empty.
The mapping is not stored anywhere — pass it on every call, or keep it in version control alongside your evaluation code.
```python theme={null}
from fiddler_evals import Dataset, FieldMapping
dataset = Dataset.get_or_create(
name='support-agent-regressions',
application_id=application.id,
description='Toxic responses caught in production, promoted for replay',
)
mapping = FieldMapping(
inputs={'user_query': 'gen_ai.llm.input.user'},
expected_outputs={'expected_response': 'gen_ai.llm.output'},
metadata={
'model': 'gen_ai.request.model',
'ticket_id': 'support.ticket_id',
},
)
```
Read each entry as "populate the dataset field on the left from the span attribute on the right."
Span attribute keys are stored **verbatim** — the key your instrumentation set is the key you map.
Spans ingested before Fiddler 26.16 may carry legacy prefixed keys,
so a time window that spans the upgrade can mix both forms.
The discovery output always shows the stored form.
Keys are matched **exactly**, and an unresolved key yields an empty string rather than an error.
A near-miss therefore produces a dataset of blank items alongside a successful `items_created` count.
Always copy keys from the [Step 2](#step-2-discover-the-attributes-on-those-spans) output, and verify the first promoted item is populated.
Verbatim key storage requires Fiddler 26.16 or later,
but because you copy keys from the discovery output,
the workflow stays correct on earlier deployments that still store the prefixed keys.
Note what each bucket is for when you promote production spans:
* **`inputs`** is what your task replays. Your task function receives this bucket as its `inputs` argument, so `inputs['user_query']` is how it reads the captured query.
* **`expected_outputs`** is the reference to compare against. Promoting the production response here captures "what we shipped last time," which is what makes the dataset a regression baseline.
* **`metadata`** and **`extras`** carry context you want to slice results by, but that the task does not consume.
To align a mapping with a dataset that already has items, inspect its schema first:
```python theme={null}
schema = dataset.get_schema()
print(f'{schema.total_items} items (sampled={schema.sampled})')
for field in schema.inputs:
print(f' inputs.{field.key}: {field.data_type} ({field.count})')
```
Reusing the existing field names keeps the dataset consistent, which matters because experiment tasks bind to inputs by name.
## Step 4: Add the Items
```python theme={null}
result = dataset.add_items_from_spans(
spans=span_refs,
mapping=mapping,
start_time=start_time,
end_time=end_time,
)
print(f'Created {result.items_created} dataset items')
print(result.item_ids[:3])
```
```text Expected output theme={null}
Created 47 dataset items
[UUID('3f2b...'), UUID('9c41...'), UUID('be07...')]
```
`start_time` and `end_time` must cover the spans you reference — Fiddler uses them to bound the lookup, and a span outside the window is treated as missing.
The call is **all-or-nothing**.
If any span cannot be resolved, Fiddler writes nothing and returns `422` with a `SpanNotFound` entry per unresolved span.
That makes retries safe: a failed call leaves the dataset untouched, so you can fix the references and call again without creating duplicates.
## Limits
| Limit | Value | Applies to |
| --------------------------------- | --------------- | ---------------------------------------- |
| Spans per call | 200 | `add_items_from_spans()` |
| Items per dataset | 10,000 | Total across all sources |
| Spans scanned for field discovery | 10,000 | `get_span_fields()`, sets `sampled=True` |
| Filter nesting depth | 3 levels | `filter` |
| Filter rules | 10 | `filter` |
| Search query length | 3–64 characters | `search` |
| Span query page size | 200 | `POST /v3/spans/query` |
To add more than 200 spans, page through your span query and call `add_items_from_spans()` once per batch.
Each call is independent, so a batch that fails does not roll back batches that already succeeded.
The per-call span limit and the per-dataset item limit can vary by deployment.
Check with your Fiddler administrator if your instance behaves differently.
## Next Steps
* [Run an experiment](/evaluate-and-test/evals-sdk-quick-start) against the golden dataset you just built
* [Capture traces during experiments](/evaluate-and-test/experiment-trace-capture) to debug what your task did on each item
* [Span and Resource Attributes](/integrations/agentic-ai/attributes) — the full catalog of filterable fields
* [Query spans REST reference](/sdk-api/rest-api/spans/query-spans)
* [Bulk add spans to dataset REST reference](/sdk-api/rest-api/evals/bulk-add-spans-to-dataset)
* [`Dataset` SDK reference](/sdk-api/evals/dataset)
# Compare LLM Outputs
Source: https://docs.fiddler.ai/evaluate-and-test/llm-evaluation-example
Learn how to systematically compare outputs from different LLM models (GPT-3.5, Claude, etc.) using Fiddler's pre-production evaluation environment to make data-driven model selection decisions.
## Overview
Making the right LLM model choice is critical for your application's success, but comparing models effectively requires more than intuition. This quick start demonstrates how to use Fiddler's pre-production evaluation environment to perform **systematic, side-by-side comparisons** of different LLM models using the same prompts and consistent evaluation metrics.
## What You'll Learn
In this hands-on notebook guide, you'll learn how to:
* **Upload model outputs** from different LLMs (GPT-3.5 and Claude) to Fiddler's pre-production environment
* **Define a consistent evaluation schema** that works across multiple models
* **Apply Fiddler enrichments** for automated quality assessment:
* **Faithfulness (Centor Model)** - Detect hallucinations in RAG responses using Fiddler Centor Models
* **Sentiment Analysis** - Understand response tone
* **PII Detection** - Identify privacy risks
* **Embeddings** - Track semantic patterns and outliers
**Looking for RAG-specific evaluation?** For comprehensive RAG pipeline diagnostics using Answer Relevance 2.0, Context Relevance, and RAG Faithfulness, see the [RAG Health Metrics Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial) and [RAG Evaluation Fundamentals Cookbook](/developers/cookbooks/rag-evaluation-fundamentals).
\* \*\*Build comparison dashboards\*\* using metric cards to visualize differences \* \*\*Make data-driven decisions\*\* about which model best fits your needs
## Why Compare in Pre-Production?
Pre-production evaluation lets you test models **before committing to production deployment**:
* **Apples-to-apples comparison** - Same prompts, same metrics, consistent evaluation
* **Cost optimization** - Identify the most cost-effective model that meets quality requirements
* **Risk assessment** - Understand safety and quality trade-offs before production
* **Informed decisions** - Replace guesswork with quantitative evidence
## What You'll Build
By the end of this guide, you'll have:
1. **Two datasets** uploaded to Fiddler (GPT-3.5 and Claude outputs)
2. **Consistent enrichments** applied to both datasets for fair comparison
3. **Metric card dashboards** showing side-by-side model performance
4. **Clear insights** into which model performs better for your use case
## Prerequisites
Before you begin:
* **Fiddler account** with access to create projects
* **Python environment** with Jupyter notebook support
* **Fiddler Python client** (installed via pip)
* **Sample data** (provided in the notebook from Fiddler examples repository)
## Time to Complete
**\~15 minutes** - Follow the interactive notebook for a guided experience
## Get Started
Choose your preferred environment:
### Option 1: Google Colab (Recommended)
Run the notebook directly in your browser with zero setup:
[Open in Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Comparison.ipynb)
### Option 2: Local Jupyter Notebook
Download and run locally in your own environment:
[Download from GitHub →](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Comparison.ipynb)
## Workflow Overview
The notebook walks through this experiment workflow:
Initialize the Python client with your credentials.
Set up a project to organize your evaluation.
Read sample datasets containing GPT-3.5 and Claude responses.
Enable automated quality metrics:
* Text embeddings for semantic analysis
* Faithfulness scoring for hallucination detection
* Sentiment analysis for response tone
* PII detection for privacy compliance
Specify input/output columns and metadata.
Upload both model outputs to the pre-production environment.
Create metric cards for visual comparison.
Analyze results to choose the best model.
## Key Features Demonstrated
### Fiddler Enrichments
The notebook demonstrates several powerful Fiddler enrichments:
**Text Embeddings:**
* Generate semantic representations of prompts, responses, and source documents
* Track outliers and drift in your experiment datasets
* Enable similarity-based analysis
**Faithfulness Assessment:**
* Automatically detect hallucinations in RAG-based responses
* Compare how well each model grounds responses in source documents
* Critical for applications requiring factual accuracy
**Sentiment & Safety:**
* Analyze response tone and sentiment
* Detect PII leakage across models
* Assess safety and compliance risks
### Pre-Production Comparison
**Side-by-Side Analysis:**
* Upload multiple model outputs to the same project
* Apply identical enrichments for fair comparison
* Visualize differences with metric cards
**Data-Driven Decisions:**
* Quantitative metrics replace subjective judgment
* Compare quality, safety, and consistency across models
* Balance performance with cost considerations
## Use Cases
This comparison approach works for:
* **Model selection** - Choose between GPT-4, Claude, Llama, or other LLMs
* **Prompt optimization** - Compare different prompt strategies on the same model
* **Version testing** - Evaluate new model versions before upgrading
* **Cost optimization** - Find the most cost-effective model meeting quality standards
* **RAG system tuning** - Compare retrieval strategies and grounding effectiveness
## What Happens Next?
After completing this quick start:
1. **Analyze your results** - Use metric cards to understand performance differences
2. **Iterate on your evaluation** - Add more test cases or different models
3. **Scale to production** - Deploy the winning model with confidence
4. **Continue monitoring** - Use Fiddler's production monitoring for ongoing quality tracking
## Related Resources
**Expand Your Experiment Capabilities:**
* [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start) - Build custom experiment workflows
* [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start) - Create custom LLM-as-a-Judge evaluators
* [Experiments Overview](/getting-started/experiments) - Comprehensive guide to Fiddler Experiments
**Production Deployment:**
* [Agentic Monitoring](/getting-started/agentic-monitoring) - Monitor LLM applications in production
* [LLM Monitoring](/getting-started/llm-monitoring) - Track production LLM performance
* [Guardrails](/getting-started/guardrails) - Add real-time safety validation
# Overview
Source: https://docs.fiddler.ai/evaluate-and-test/overview
Hands-on quick start guides for evaluating LLM applications, testing with custom LLM-as-a-Judge metrics, and comparing model outputs using Fiddler Experiments.
## Quick Start Guides
Ready to start testing your LLM applications? Choose the hands-on guide that matches your evaluation needs. Each quick start provides step-by-step instructions, code examples, and takes 15-20 minutes to complete.
**New to Fiddler Experiments?** Start with our [comprehensive Experiments guide](/getting-started/experiments) to understand core concepts, workflows, and best practices before diving into these quick starts.
***
### Evals SDK Quick Start
**Build comprehensive experiment workflows with built-in and custom evaluators**
**What you'll learn:**
* Connect to Fiddler and set up evaluation projects
* Create datasets with test cases (CSV, JSONL, or DataFrame)
* Use production-ready evaluators (Relevance, Coherence, Toxicity, Sentiment)
* Build custom evaluators for domain-specific requirements
* Run experiments with parallel processing
* Analyze results and export data for further analysis
**Perfect for:**
* Teams needing full control over evaluation logic
* Building comprehensive test suites with multiple quality dimensions
* Creating domain-specific custom metrics
* Programmatic experiment workflows and CI/CD integration
**Time to complete:** \~20 minutes
[Start Evals SDK Quick Start →](/evaluate-and-test/evals-sdk-quick-start)
***
### Prompt Specs Quick Start
**Create custom LLM-as-a-Judge evaluations without manual prompt engineering**
**What you'll build:** A news article topic classifier that demonstrates:
* Schema-based evaluation definition (no prompt writing!)
* Validation and testing workflows
* Iterative improvement with field descriptions
* Production deployment as Fiddler enrichments
**What you'll learn:**
* Define evaluation schemas using JSON
* Validate Prompt Specs before deployment
* Test evaluation logic with sample data
* Improve accuracy through structured descriptions
* Deploy custom evaluators to production monitoring
**Perfect for:**
* Teams needing domain-specific evaluation logic
* Avoiding time-consuming prompt engineering
* Rapid iteration on evaluation criteria
* Schema-driven evaluation workflows
**Time to complete:** \~15 minutes
[Start Prompt Specs Quick Start →](/evaluate-and-test/prompt-specs-quick-start)
***
### Compare LLM Outputs
**Systematically compare different LLM models to make data-driven decisions**
**What you'll learn:**
* Compare outputs from different LLM models (GPT-4, Claude, Llama, etc.)
* Evaluate multiple prompt variations side-by-side
* Use Fiddler's observability features for pre-production testing
* Balance quality, cost, and latency trade-offs
**Perfect for:**
* Model selection and validation
* Prompt A/B testing and optimization
* Cost optimization through model comparison
* Pre-production evaluation of LLM outputs
**Time to complete:** \~15 minutes
**Interactive notebook:**
* [Open in Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Comparison.ipynb)
* [Download from GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Comparison.ipynb)
[Start Comparing Models →](/evaluate-and-test/llm-evaluation-example)
***
## Choosing the Right Quick Start
Not sure which guide to start with? Use this table:
| Goal | Recommended Guide |
| -------------------------------------------- | ----------------------------------------------------------------------- |
| Implement custom domain logic (schema-based) | [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start) |
| Implement custom domain logic (Python-based) | [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start) |
| Build test suites | [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start) |
| Compare models | [Compare LLM Outputs](/evaluate-and-test/llm-evaluation-example) |
**Quick recommendations:**
* 🎯 **First-time users**: Start with [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start) to learn the fundamentals
* 🔧 **Custom evaluations needed**: Use [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start) for schema-based approach
* 📊 **Model comparison**: Jump to [Compare LLM Outputs](/evaluate-and-test/llm-evaluation-example) for side-by-side testing
## Core Evaluation Concepts
These quick starts demonstrate key Fiddler Experiments capabilities:
### Built-in Evaluators
Production-ready metrics that run on [Fiddler Centor Models](/glossary/centor-models):
* **Quality**: Answer Relevance, Coherence, Conciseness, Completeness
* **Safety**: Toxicity Detection, Prompt Injection, PII Detection
* **RAG-Specific**: Faithfulness, Context Relevance
* **Sentiment**: Multi-score sentiment and topic classification
**Key benefits:**
* Zero external API costs
* \<100ms latency for real-time evaluation
* Your data never leaves your environment
### Custom Evaluation Frameworks
Build domain-specific evaluators using:
* **Python-based evaluators** - Full programmatic control
* **Prompt Specs** - Schema-driven LLM-as-a-Judge (no manual prompting)
* **Function wrappers** - Integrate existing evaluation logic
### Experiment Tracking & Comparison
Every experiment run is tracked:
* Complete lineage of inputs, outputs, and scores
* Side-by-side experiment comparison in Fiddler UI
* Aggregate statistics and drill-down analysis
* Export capabilities for further processing
## Common Experiment Workflows
These quick starts support various experiment scenarios:
### Pre-Production Testing
* **Regression Testing**: Run comprehensive test suites before deployment
* **Quality Gates**: Set score thresholds that must be met
* **Version Validation**: Compare model versions on same datasets
### Model & Prompt Optimization
* **A/B Testing**: Compare prompt variations quantitatively
* **Model Selection**: Evaluate multiple LLMs on same tasks
* **Hyperparameter Tuning**: Test temperature, top-p, and other configs
### RAG System Evaluation
Evaluate RAG pipelines end-to-end using the [RAG Health Metrics](/getting-started/experiments) evaluators — a purpose-built diagnostic framework that pinpoints whether issues originate in retrieval, generation, or query understanding:
* **Answer Relevance 2.0**: Assess how well responses address user queries with ordinal scoring (High / Medium / Low)
* **Context Relevance**: Measure whether retrieved documents are relevant to the query (High / Medium / Low)
* **RAG Faithfulness**: Verify responses are grounded in retrieved documents (Yes / No with reasoning)
Use these evaluators together to diagnose specific failure modes — for example, high faithfulness with low relevance indicates the response is grounded but doesn't answer the question, pointing to a retrieval problem rather than a generation problem.
### Safety & Compliance
* **Adversarial Testing**: Test with jailbreak attempts and prompt injections
* **Content Moderation**: Measure toxicity, bias, and PII exposure
* **Policy Validation**: Ensure outputs meet organizational standards
## From Development to Production
Fiddler Experiments integrates seamlessly with production monitoring:
**Unified Workflow Benefits:**
* **Consistent Metrics**: Same evaluators in development and production
* **Continuous Learning**: Production insights feed back into test datasets
* **Seamless Transition**: Deploy with confidence—monitoring matches testing
**Complete AI Lifecycle:**
1. **Build** → Design and instrument your applications
2. **Test** → Evaluate with Fiddler Experiments *(these quick starts)*
3. **Monitor** → Track production with [Agentic Monitoring](/getting-started/agentic-monitoring)
4. **Improve** → Refine based on insights
Learn more about [Fiddler's end-to-end agentic AI lifecycle](https://www.fiddler.ai/blog/end-to-end-agentic-observability-lifecycle).
## Getting Started Checklist
Ready to evaluate your LLM applications?
* [ ] Choose a quick start guide based on your evaluation needs
* [ ] Install the [Fiddler Evals SDK](/sdk-api/evals/evaluate) (for SDK and Prompt Specs guides)
* [ ] Prepare 5-10 sample test cases for your application
* [ ] Follow the step-by-step guide (15-20 minutes)
* [ ] Review results in Fiddler UI
* [ ] Iterate and expand your experiment coverage
## Additional Resources
**Learn More:**
* [Experiments Overview](/getting-started/experiments) - Comprehensive guide to Fiddler Experiments
* [Golden Datasets](/evaluate-and-test/golden-datasets) - Build datasets from real production spans
* [Capture Traces During Experiments](/evaluate-and-test/experiment-trace-capture) - Debug and score on execution traces
* [Evals SDK Advanced Guide](/developers/tutorials/experiments/evals-sdk-advanced) - Production patterns
* [Fiddler Evals SDK Reference](/sdk-api/evals/evaluate) - Complete API documentation
* [Experiments Glossary](/glossary/experiments) - Key terminology
**Example Notebooks:**
* [Evaluations SDK Notebook](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Evaluations_SDK.ipynb)
* [Prompt Specs Notebook](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLMaaJ_Prompt_Spec.ipynb)
* [LLM Comparison Notebook](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Comparison.ipynb)
**Related Capabilities:**
* [Agentic Monitoring](/getting-started/agentic-monitoring) - Production agent observability
* [LLM Monitoring](/getting-started/llm-monitoring) - Production LLM tracking
* [Guardrails](/getting-started/guardrails) - Real-time safety validation
# Prompt Specs Quick Start
Source: https://docs.fiddler.ai/evaluate-and-test/prompt-specs-quick-start
Get started with Fiddler's LLM-as-a-Judge evaluation using Prompt Specs in minutes. Learn to create custom evaluations, test them, and deploy to production monitoring.
Get your first custom LLM evaluation running in **minutes** using Prompt Specs with Fiddler's LLM-as-a-Judge solution. This guide walks you through creating, testing, and deploying a custom evaluation using Prompt Specs.
### What You'll Build
In this quick start, you'll create a news article topic classifier that:
* Takes a news summary as input
* Classifies it into one of four categories: World, Sports, Business, or Sci/Tech
* Provides reasoning for its classification
* Deploys to production monitoring in Fiddler
### Prerequisites
* Fiddler platform access
* Basic familiarity with Python and REST APIs
* A Fiddler API token and base URL
Refer to the Fiddler Python client SDK [Installation and Setup Guide](/developers/python-client-guides/installation-and-setup) for details on the Fiddler Access Token, URL, and client initialization.
```python theme={null}
import json
import fiddler as fdl
import pandas as pd
import requests
# Replace with your actual values
FIDDLER_TOKEN = "your_token_here"
FIDDLER_BASE_URL = "https://your_company.fiddler.ai"
PROMPT_SPEC_URL = f"{FIDDLER_BASE_URL}/v3/llm-as-a-judge/prompt-spec"
FIDDLER_HEADERS = {
"Authorization": f"Bearer {FIDDLER_TOKEN}",
"Content-Type": "application/json",
}
```
We'll use news article data for this example:
```python theme={null}
# Load sample news data (using AG News dataset)
df_news = pd.read_parquet(
"hf://datasets/fancyzhx/ag_news/data/test-00000-of-00001.parquet"
).sample(20, random_state=25)
# Map labels to topic names
df_news["original_topic"] = df_news["label"].map({
0: "World", 1: "Sports", 2: "Business", 3: "Sci/Tech"
})
# Summarize the count of each unique topic
print(df_news["original_topic"].value_counts())
```
Define a simple evaluation schema:
```python theme={null}
basic_prompt_spec = {
"input_fields": {
"news_summary": {"type": "string"}
},
"output_fields": {
"topic": {
"type": "string",
"choices": ["World", "Sports", "Business", "Sci/Tech"]
},
"reasoning": {"type": "string"}
}
}
```
Validate your Prompt Spec schema:
```python theme={null}
validate_response = requests.post(
f"{PROMPT_SPEC_URL}/validate",
headers=FIDDLER_HEADERS,
json={"prompt_spec": basic_prompt_spec}
)
if validate_response.status_code == 200:
print("✅ Schema validation successful!")
else:
print("❌ Validation failed:", validate_response.text)
```
```python theme={null}
def get_prediction(prompt_spec, input_data):
response = requests.post(
f"{PROMPT_SPEC_URL}/predict",
headers=FIDDLER_HEADERS,
json={"prompt_spec": prompt_spec, "input_data": input_data}
)
if response.status_code == 200:
return response.json()["prediction"]
return {"topic": None, "reasoning": None}
# Test with a single example
test_result = get_prediction(
basic_prompt_spec,
{"news_summary": "Wimbledon 2025 is under way!"}
)
print(json.dumps(test_result, indent=2))
```
Add field descriptions to improve classification accuracy:
```python theme={null}
enhanced_prompt_spec = {
"instruction": "Determine the topic of the given news summary.",
"input_fields": {
"news_summary": {"type": "string"}
},
"output_fields": {
"topic": {
"type": "string",
"choices": ["World", "Sports", "Business", "Sci/Tech"],
"description": """Use 'Sci/Tech' for technology companies, scientific discoveries, or health/medical research.
Use 'Sports' for sports events or athletes.
Use 'Business' for companies outside of tech/sports.
Use 'World' for global events or issues."""
},
"reasoning": {
"type": "string",
"description": "Explain why you chose this topic."
}
}
}
```
Test your enhanced Prompt Spec on multiple examples:
```python theme={null}
# Test on your dataset
results = []
for _, row in df_news.iterrows():
prediction = get_prediction(
enhanced_prompt_spec,
{"news_summary": row["text"]}
)
results.append({
"original": row["original_topic"],
"predicted": prediction["topic"],
"reasoning": prediction["reasoning"]
})
# Calculate accuracy
df_results = pd.DataFrame(results)
accuracy = (df_results["original"] == df_results["predicted"]).mean()
print(f"Accuracy: {accuracy:.1%}")
```
Once satisfied with your Prompt Spec, deploy it as a Fiddler enrichment:
```python theme={null}
import fiddler as fdl
# Initialize Fiddler client
fdl.init(url=FIDDLER_BASE_URL, token=FIDDLER_TOKEN)
# Create project and enrichment
project = fdl.Project.get_or_create(name="llm_evaluation_demo")
enrichment = fdl.Enrichment(
name="news_topic_classifier",
enrichment="llm_as_a_judge",
columns=["news_summary"],
config={"prompt_spec": enhanced_prompt_spec}
)
# Create model with enrichment
model_spec = fdl.ModelSpec(
inputs=["news_summary"],
custom_features=[enrichment]
)
model = fdl.Model.from_data(
source=df_news.rename(columns={"text": "news_summary"}),
name="news_classifier",
project_id=project.id,
spec=model_spec,
task=fdl.ModelTask.LLM
)
model.create()
print(f"Model created: {model.name}")
```
Publish your data and start monitoring:
```python theme={null}
# Publish production events
job = model.publish_batch(source=df_news.rename(columns={"text": "news_summary"}))
job.wait()
if job.status == "SUCCESS":
print("✅ Data published successfully!")
print("🎯 Your evaluation is now running in production monitoring")
```
```python theme={null}
import json
from datetime import datetime
import fiddler as fdl
import pandas as pd
import requests
# Replace with your actual values
# FIDDLER_TOKEN = "your_token_here"
# FIDDLER_BASE_URL = "https://your_company.fiddler.ai"
FIDDLER_TOKEN = "hqvUV7r8-WUkMkjvKHbvI_sVpxRd9DJLKX6PCloRwVk"
FIDDLER_BASE_URL = "https://preprod.cloud.fiddler.ai"
PROMPT_SPEC_URL = f"{FIDDLER_BASE_URL}/v3/llm-as-a-judge/prompt-spec"
FIDDLER_HEADERS = {
"Authorization": f"Bearer {FIDDLER_TOKEN}",
"Content-Type": "application/json",
}
# Load sample news data (using AG News dataset)
df_news = pd.read_parquet(
"hf://datasets/fancyzhx/ag_news/data/test-00000-of-00001.parquet"
).sample(20, random_state=25)
# Map labels to topic names
df_news["original_topic"] = df_news["label"].map({
0: "World", 1: "Sports", 2: "Business", 3: "Sci/Tech"
})
print(df_news["original_topic"].value_counts())
basic_prompt_spec = {
"input_fields": {
"news_summary": {"type": "string"}
},
"output_fields": {
"topic": {
"type": "string",
"choices": ["World", "Sports", "Business", "Sci/Tech"]
},
"reasoning": {"type": "string"}
}
}
validate_response = requests.post(
f"{PROMPT_SPEC_URL}/validate",
headers=FIDDLER_HEADERS,
json={"prompt_spec": basic_prompt_spec}
)
if validate_response.status_code == 200:
print("✅ Schema validation successful!")
else:
print("❌ Validation failed:", validate_response.text)
def get_prediction(prompt_spec, input_data):
response = requests.post(
f"{PROMPT_SPEC_URL}/predict",
headers=FIDDLER_HEADERS,
json={"prompt_spec": prompt_spec, "input_data": input_data}
)
if response.status_code == 200:
return response.json()["prediction"]
return {"topic": None, "reasoning": None}
# Test with a single example
test_result = get_prediction(
basic_prompt_spec,
{"news_summary": "Wimbledon 2025 is under way!"}
)
print(json.dumps(test_result, indent=2))
enhanced_prompt_spec = {
"instruction": "Determine the topic of the given news summary.",
"input_fields": {
"news_summary": {"type": "string"}
},
"output_fields": {
"topic": {
"type": "string",
"choices": ["World", "Sports", "Business", "Sci/Tech"],
"description": """Use 'Sci/Tech' for technology companies, scientific discoveries, or health/medical research.
Use 'Sports' for sports events or athletes.
Use 'Business' for companies outside of tech/sports.
Use 'World' for global events or issues."""
},
"reasoning": {
"type": "string",
"description": "Explain why you chose this topic."
}
}
}
# Test on your dataset
results = []
for _, row in df_news.iterrows():
prediction = get_prediction(
enhanced_prompt_spec,
{"news_summary": row["text"]}
)
results.append({
"original": row["original_topic"],
"predicted": prediction["topic"],
"reasoning": prediction["reasoning"]
})
# Calculate accuracy
df_results = pd.DataFrame(results)
accuracy = (df_results["original"] == df_results["predicted"]).mean()
print(f"Accuracy: {accuracy:.1%}")
```
### What Happens Next
After completing this quick start:
1. **View Results**: Check the Fiddler UI to see your model and enrichment results
2. **Monitor Performance**: Set up alerts based on classification accuracy or confidence scores
3. **Iterate**: Refine your Prompt Spec descriptions to improve accuracy
4. **Scale**: Apply the same approach to your own evaluation use cases
### Key Takeaways
* **Fast Setup**: From zero to production evaluation in minutes, not weeks
* **No Manual Prompting**: JSON schema approach eliminates prompt engineering bottlenecks
* **Built-in Monitoring**: Seamless integration with Fiddler's observability platform
* **Easy Iteration**: Update schemas without rewriting prompts
### Next Steps
* [**Complete Interactive Notebook**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLMaaJ_Prompt_Spec.ipynb): Follow along with a full working example
* [**Prompt Specs Guide**](/observability/llm/llm-evaluation-prompt-specs): Learn more about the underlying framework
# Agentic Observability
Source: https://docs.fiddler.ai/getting-started/agentic-monitoring
Comprehensive monitoring, tracing, and analysis of AI agent systems that provide hierarchical visibility into agent reasoning, coordination, and decision-making across multi-agent applications
Modern GenAI applications built with agentic frameworks create complex, multi-step workflows that can be difficult to understand and debug. Fiddler, the pioneer in AI Observability and Security, provides comprehensive monitoring solutions for AI agents regardless of your framework. Whether you're using LangGraph, Strands, or custom implementations, you get complete visibility into agent behaviors with enterprise-grade safeguards and real-time monitoring—the insights you need to confidently deploy reliable, high-performance GenAI applications in production.
**First Time Setting Up?**
Before you can start monitoring your agentic application, you need to create a project and application in Fiddler. See [Onboard Your GenAI Application](/getting-started/genai-application-onboarding) for step-by-step instructions.
## What Is Agentic Observability?
Agentic monitoring observes and analyzes AI agent behavior in real-time. Unlike traditional application monitoring that focuses on system metrics, agentic observability captures the unique characteristics of AI workflows:
* **Agent decision-making processes**: How agents choose between different tools and actions
* **Multi-step reasoning chains**: Complex workflows from initial prompt to final response
* **LLM interactions**: Model inputs, outputs, and performance across different calls
* **Tool usage patterns**: How agents utilize external functions and APIs
* **Error propagation**: How failures cascade through agent workflows
## Why Agentic Observability Matters
GenAI applications present unique observability challenges that traditional monitoring approaches can't address:
### Complexity and Opacity
AI agents make autonomous decisions that are difficult to predict or understand. Without proper monitoring, you can't debug agent behavior in production or understand why an agent made specific choices.
### Dynamic Workflows
Unlike traditional applications with fixed execution paths, AI agents create dynamic workflows based on context and available tools. You need to trace the actual execution path for each interaction.
### Performance Variability
LLM response times and quality vary significantly based on model load, prompt complexity, and external factors. Monitoring helps you identify performance patterns and optimize accordingly.
### Cost Management
GenAI applications consume tokens and compute resources with each LLM call. Understanding usage patterns helps you optimize costs and prevent unexpected billing spikes.
### Quality Assurance
AI outputs vary in quality and accuracy. Monitoring helps you identify when agents produce suboptimal results and understand the conditions that lead to better performance.
## Integration Options
Fiddler provides flexible integration options to instrument your agentic applications, regardless of the framework you're using. Choose the approach that best fits your technology stack:
* **LangGraph & LangChain SDK** — automatic instrumentation with minimal code changes for applications built with LangGraph or LangChain. Initialize the client, and your agent workflows are monitored automatically.
* **Strands SDK** — native integration with the Strands framework for monitoring without additional instrumentation code.
* **OpenTelemetry** — direct instrumentation for custom frameworks, non-Python applications, or when you need maximum control. Works with any programming language.
Initialize the Fiddler client for your framework:
```python LangGraph theme={null}
import os
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
fdl_client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL")
)
```
```python Strands theme={null}
import os
from fiddler_strands import FiddlerClient
fdl_client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL")
)
```
```python OpenTelemetry theme={null}
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# Configure OpenTelemetry to send spans to Fiddler
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(
endpoint="https://your-instance.fiddler.ai/v1/traces"
))
provider.add_span_processor(processor)
```
### Integration Comparison
| Feature | LangGraph SDK | Strands SDK | OpenTelemetry |
| ----------------------------- | -------------------- | -------------------- | -------------------------- |
| **Automatic Instrumentation** | ✅ Zero code changes | ✅ Native integration | ❌ Manual setup |
| **Framework Support** | LangGraph, LangChain | Strands Agents | Any framework |
| **Language Support** | Python | Python | Python, Java, Go, JS, etc. |
| **Setup Time** | \~15 minutes | \~15 minutes | \~20 minutes |
| **Customization** | Medium | Medium | High |
## How Agentic Observability Works
Regardless of which integration method you choose, Fiddler's agentic observability provides the same comprehensive observability capabilities:
### Built on OpenTelemetry Standards
All Fiddler agentic observability solutions leverage [OpenTelemetry](https://opentelemetry.io/docs/) (OTeL), the industry standard for observability. This ensures compatibility with existing monitoring infrastructure, vendor neutrality, and future-proofing of your investment.
### Automatic or Manual Instrumentation
Depending on your integration choice, you can collect telemetry data automatically or with fine-grained control:
**Automatic Instrumentation** (LangGraph & Strands SDKs):
* Zero-code or minimal-code setup
* Automatically captures all agent interactions
* Best for rapid deployment and standard use cases
**Manual Instrumentation** (OpenTelemetry):
* Full control over what data is captured
* Custom span attributes and metadata
* Best for specialized requirements or custom frameworks
All approaches collect the same essential telemetry:
* **Distributed traces**: Complete execution flow
* **Span attributes**: Inputs, outputs, and metadata
* **Performance metrics**: Timing and resource usage
* **Error tracking**: Detailed context and stack traces
### Near Real-Time Streaming
Telemetry data streams in near real-time to your Fiddler instance, enabling immediate visibility into agent behavior and rapid response to issues.
## Architecture Overview
Fiddler's agentic observability integrates seamlessly into your application architecture, regardless of which integration method you choose:
```mermaid theme={null}
graph TD
subgraph APP["Agent Application"]
A([Agents & Workflows])
B([LLM Calls])
C([Tools & Actions])
end
subgraph OTEL["OpenTelemetry Layer"]
D[Instrumentation]
E[Trace Collection & Export]
end
subgraph FDL["Fiddler AI Platform"]
F[Trace Ingestion]
G[Monitoring Dashboards]
H[Analytics & Evaluators]
end
A & B & C --> D
D --> E
E -->|OTLP Protocol| F
F --> G
F --> H
style A fill:#1976d2,color:#fff,stroke:#0d47a1
style B fill:#1976d2,color:#fff,stroke:#0d47a1
style C fill:#1976d2,color:#fff,stroke:#0d47a1
style D fill:#388e3c,color:#fff,stroke:#1b5e20
style E fill:#388e3c,color:#fff,stroke:#1b5e20
style F fill:#f57c00,color:#fff,stroke:#e65100
style G fill:#f57c00,color:#fff,stroke:#e65100
style H fill:#f57c00,color:#fff,stroke:#e65100
style APP fill:#e3f2fd,stroke:#1976d2,color:#0d47a1
style OTEL fill:#e8f5e9,stroke:#388e3c,color:#1b5e20
style FDL fill:#fff3e0,stroke:#f57c00,color:#e65100
```
### Key Components
1. **Instrumentation Layer**: Captures agent execution events (automatic with SDKs, manual with OpenTelemetry)
2. **Trace Exporter**: Sends telemetry data to Fiddler using [OTLP](https://opentelemetry.io/docs/specs/otlp/) protocol
3. **Fiddler Client**: Manages configuration, authentication, and connection to your Fiddler instance
4. **OpenTelemetry Integration**: Provides industry-standard distributed tracing across all frameworks
## What You Can Monitor
Once your agent application is instrumented, Fiddler provides comprehensive observability through dashboards, metrics, and automated evaluations:
### Dashboards & Visualization
Access specialized dashboards designed for agentic workflows:
**Pre-Built Agentic Dashboards:**
* **Agent Performance Overview** - Monitor success rates, latency, and throughput across all agents
* **Workflow Execution Traces** - Visualize complete multi-step reasoning chains hierarchically
* **Tool Usage Analytics** - Track which external tools and APIs your agents are calling
* **Error & Exception Tracking** - Identify where agent workflows fail and diagnose the root cause
**Trace Visualization:** Every agent interaction is captured as a hierarchical trace showing:
* Agent decision points and reasoning steps
* LLM calls with full inputs, outputs, and metadata
* Tool invocations and API requests
* Timing information for each step
* Parent-child relationships in multi-agent systems
**Custom Dashboards:**
* Build custom dashboards combining multiple charts to track KPIs specific to your use case
* Filter by agent type, user segments, time periods, or custom attributes
* Share dashboards with team members and stakeholders
* Configure alerts based on dashboard metrics
### Metrics & Analytics
Fiddler automatically generates specialized metrics for agent monitoring:
**Agent-Specific Metrics:**
* **Agent Success Rate** - Percentage of workflows completing successfully
* **Tool Call Distribution** - Frequency analysis of tool usage
* **Reasoning Chain Length** - Average number of steps per workflow
* **Agent Handoffs** - How often agents delegate to other agents
* **Retry & Recovery Rate** - Agent resilience and error recovery patterns
**Performance Metrics:**
* **End-to-End Latency** - Total time from user request to final response
* **Per-Step Latency** - Duration of individual reasoning steps, LLM calls, and tool invocations
* **Token Usage** - LLM consumption tracking across all agent interactions
* **API Call Volume** - Monitor external tool and API usage patterns
**Quality Metrics:**
* **Response Accuracy** - Validate agent outputs (requires ground truth labels)
* **Hallucination Detection** - Identify when agents generate unsupported claims
* **RAG Health Metrics** - Diagnose RAG pipeline issues with Answer Relevance 2.0, Context Relevance, and RAG Faithfulness evaluators. Pinpoint whether failures originate in retrieval, generation, or query understanding
* **Safety & Guardrails** - Track safety violations and guardrail activations
* **User Satisfaction** - Capture feedback signals from end users
### Evaluator Rules
**Automated Quality Assessment**
Evaluator Rules enable automated, continuous evaluation of your agent's performance directly from production spans and traces. Define once, and Fiddler automatically assesses quality, safety, and performance in real-time.
**How Evaluator Rules Work:**
1. **Configure an Evaluator** - Choose from Fiddler-provided evaluators or create custom LLM-as-a-judge evaluators
2. **Map Inputs** - Connect evaluator variables to span attributes from your agent telemetry
3. **Set Application Rules** - Define which spans qualify for evaluation (e.g., only "llm" span types)
4. **Activate** - Evaluations run automatically on all matching spans
**Available Evaluators:**
* **Fiddler-Provided**: Topic Classification, Answer Relevance, Coherence, Conciseness, Context Relevance, RAG Faithfulness, PII Detection, Sentiment Analysis, Prompt Safety, Response Faithfulness
* **Custom LLM-as-a-Judge**: Create domain-specific evaluators with custom prompts and scoring logic
**Key Benefits:**
* **Automated Quality Monitoring** - Continuous evaluation without manual labeling
* **Real-Time Insights** - Identify quality issues as they happen in production
* **Backfill Capability** - Apply evaluations retroactively to historical data
* **Flexible Filtering** - Evaluate specific span types, regions, or custom conditions
→ [**Learn more about Evaluator Rules**](/evaluate-and-test/evaluator-rules)
## What Data Is Captured
All instrumentation methods capture comprehensive data about your agent workflows:
### Agent Execution Traces
* **Workflow structure**: Complete hierarchy of agent steps and decisions
* **Timing information**: Duration of each step and overall execution time
* **Agent identification**: Unique identifiers for different agents in your system
### LLM Interactions
* **Model configuration**: Model name, temperature, and other parameters
* **Input prompts**: System messages, user input, and conversation history
* **Model outputs**: Generated responses, token usage, and completion metadata
* **Performance metrics**: Response time, tokens consumed, and success rates
### Tool and Function Calls
* **Tool identification**: Names and types of tools used by agents
* **Input parameters**: Arguments passed to functions and tools
* **Output results**: Return values and success/failure status
* **Execution context**: When and why tools were invoked
### Error and Exception Handling
* **Exception details**: Full error messages and stack traces
* **Context information**: State of the agent when errors occurred
* **Recovery attempts**: How agents handled and recovered from failures
## Key Benefits
Agentic observability with Fiddler provides immediate value across your development lifecycle, helping you move beyond experimentation to deploy production AI confidently:
### Development and Learning
* **Quick setup**: Start monitoring with just a few lines of code
* **Immediate insights**: See agent behavior without complex configuration
* **Deep visibility**: Understand decision-making processes beyond just inputs and outputs
### Performance and Optimization
* **Hierarchical root cause analysis**: Drill down from application-level issues to specific agent spans to reduce MTTI (Mean Time to Identify) and MTTR (Mean Time to Resolve)
* **Application-critical metrics**: Monitor performance, costs, and safety through a unified dashboard
* **Quality improvement**: Understand which patterns lead to better results
### Production and Troubleshooting
* **Enterprise-grade monitoring**: Track agent performance and success rates at Fortune 500 scale
* **End-to-end visibility**: Complete observability into multi-agent interactions and coordination patterns
* **Actionable alerts**: Get early warnings on performance issues and cross-agent problems
* **Data-driven decisions**: Make informed optimizations based on comprehensive telemetry data
## Security and Privacy Considerations
Fiddler's agentic observability is designed with enterprise-grade security and privacy:
* **Cloud Enterprise compliance**: SOC 2 Type 2 security and HIPAA compliance standards
* **Data encryption**: All telemetry data is encrypted in transit using HTTPS/TLS
* **Access control**: Role-based access control (RBAC) and SSO for enterprise user management
* **Personal access token**: Access token-based authentication ensures only authorized access
* **Data control**: You control what data is captured and sent to Fiddler
* **Deployment flexibility**: Deploy in Fiddler Cloud or your own cloud
* **Compliance**: Built on industry-standard [OpenTelemetry](https://opentelemetry.io/docs/) for compliance requirements
## Ready to Get Started?
You're now ready to add enterprise-grade observability to your agentic and LLM applications. Choose your integration library to get started in under 10 minutes:
Instrument LangGraph and LangChain apps. \~10 min.
Instrument Strands agents. \~10 min.
Instrument any framework or language. \~15 min.
Once your application is instrumented, traces appear in your Fiddler dashboard within 1-2 minutes of agent execution.
**Learn More:**
* [Agentic Observability](/observability/agentic) - Dashboards, metrics, and analytics
## Frequently Asked Questions
Traditional APM focuses on system metrics. Agentic observability captures AI-specific behaviors, such as reasoning chains, tool selection, and LLM interactions.
Choose based on your framework: LangGraph SDK for LangGraph/LangChain apps, Strands SDK for Strands agents, or OpenTelemetry for custom frameworks or non-Python applications. All provide the same monitoring capabilities.
With default settings, expect less than 5% overhead. This can be reduced further with sampling.
Yes, all data is encrypted in transit using HTTPS/TLS, and you retain full control over what data is captured and sent to Fiddler. Fiddler supports deployment in your own cloud environment for maximum security.
Telemetry data streams in near real-time, typically appearing in your dashboard within 1-2 minutes of agent execution.
Yes, all instrumentation methods capture comprehensive error information, including exception details, agent state at failure, and recovery attempts, helping you debug and improve agent reliability.
Yes, Evaluator Rules work with spans and traces from all integration methods—LangGraph SDK, Strands SDK, and OpenTelemetry.
## Limitations and Considerations
Fiddler Agentic Observability has some current limitations:
* **Protocol support**: Currently uses HTTP-based OTLP; gRPC support planned for future releases
* **Attribute limits**: Default limits prevent oversized spans; configurable for high-volume use cases
* **Language support**: LangGraph and Strands SDKs are Python-only; use OpenTelemetry for other languages
These limitations don't affect the core monitoring capabilities but are essential to consider for production planning.
## Next Steps
* [Quick Start Guides](/developers/quick-starts/get-started-in-less-than-10-minutes): Choose the quick start applicable to your use case
* [SDK References](/sdk-api): Complete technical documentation for all SDK components
# AWS SageMaker Partner AI App
Source: https://docs.fiddler.ai/getting-started/aws-sagemaker-partner-ai-app
Get started with Fiddler's Partner AI App on AWS SageMaker. Monitor, explain, and analyze your ML models and GenAI apps in your own AWS environment.
### Introduction
The [Fiddler Partner AI App for Amazon SageMaker](https://aws.amazon.com/marketplace/pp/prodview-caia5ckldtyhs) brings enterprise-grade AI observability directly into your SageMaker environment. As a fully integrated Partner AI App, Fiddler enables data scientists and ML engineers to monitor, analyze, and protect their ML models, LLM applications, GenAI systems, and AI agents in a unified platform without leaving the SageMaker ecosystem. This seamless integration allows you to leverage Fiddler's powerful monitoring capabilities—including drift detection, performance tracking, and root cause analysis features—while maintaining your existing SageMaker workflows and security boundaries.
### What is a Partner AI App?
Amazon SageMaker Partner AI Apps are fully managed AI/ML applications from AWS partners that run natively within your SageMaker environment. Think of them as pre-built, enterprise-ready applications that integrate seamlessly with your existing AWS infrastructure—no complex deployments or external accounts required.
**Key Benefits**:
* **Secure by Design**: Applications run entirely within your AWS account, maintaining your existing security boundaries and compliance requirements
* **Unified Experience**: Access partner tools directly from SageMaker Studio alongside your notebooks, models, and datasets
* **Simplified Procurement**: Subscribe through AWS Marketplace with consolidated billing on your AWS invoice
* **Managed Infrastructure**: AWS handles the underlying infrastructure while you focus on using the application
**How It Works**: When subscribing to a Partner AI App like Fiddler, AWS provisions the application within your SageMaker domain. Your team can then access it through SageMaker Studio with single sign-on, use it via APIs from notebooks, and integrate it into your ML workflows—all while keeping your data within your AWS environment.
This approach eliminates the traditional challenges of third-party tool integration: no separate accounts to manage, no data movement across boundaries, and no complex networking configurations.
### For Administrators
If you are an AWS administrator responsible for deploying and managing applications, our [Overview](/integrations/cloud-platforms-and-deployment/aws-sagemaker) and [Admin Guide](/integrations/aws-sagemaker/partner-ai-app-admin-guide) supplements the official AWS SageMaker documentation with Fiddler-specific configuration details and best practices. Use our guide alongside the official [Partner AI App Setup](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-app-onboard.html) and [Provisioning](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps-provision.html) documentation for a complete deployment experience.
Introduction to the Amazon SageMaker Partner AI Apps offering, including the subscription and deployment processes.
Fiddler-specific configuration, troubleshooting, and deployment tips.
Automated script to configure IAM roles and permissions for quick demo purposes.
### For Users
Once your administrator has deployed the Fiddler Partner AI App, you can begin monitoring, analyzing, and protecting your ML models and AI applications. Our [User Guide](/integrations/aws-sagemaker/partner-ai-app-user-guide) provides step-by-step instructions for accessing the Fiddler interface through SageMaker Studio and connecting via the Python client SDK from your notebooks.
To trace LLM and agentic applications from SageMaker, the [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) signs trace exports using the same Partner App authentication — see [Running in AWS SageMaker](/integrations/agentic-ai/fiddler-otel-sdk#running-in-aws-sagemaker).
**Before getting started**, confirm with your administrator that:
* The Fiddler Partner AI App has been deployed and is in "Deployed" status
* Your username has been granted appropriate access permissions
* You have been assigned the necessary IAM roles for Partner AI App access
* You understand which access method is available in your organization:
* **SageMaker Studio access**: Launch Fiddler directly from the Studio interface
* **Direct link access**: Access Fiddler via a pre-signed URL generated through AWS CLI
Some organizations may only permit direct link access, while others allow both. Your administrator will inform you which option(s) are available.
### Additional Resources
For a complete overview and detailed technical information, please refer to Amazon's SageMaker Partner AI App documentation:
* [Amazon SageMaker Partner AI Apps Overview](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps.html)
* [Amazon SageMaker Partner AI Apps Setup Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-app-onboard.html)
# Evaluators & Metrics Guide
Source: https://docs.fiddler.ai/getting-started/evaluators-metrics-guide
Interactive guide for selecting the right Fiddler evaluators and metrics for your use case. Filter by observability type, use case, and rating to find what to deploy.
## How to use this guide
Use this guide to choose the right Fiddler evaluators and metrics for what you're building. Start by picking your **observability type** — a Gen AI application, a single LLM, a predictive (ML) model, or real-time Guardrails — then select a use case to see which metrics Fiddler rates **Recommended**, worth a **Consider**, or **Optional** for that scenario.
Open any metric card for its per-use-case rationale, evaluator type and model provider, and a link to the documentation. Use **Show more filters** to narrow by evaluator type, model provider, or observability objective, or search by name to jump straight to a specific metric.
Below the guide, **Cookbooks & Resources** pairs proven, copy-pasteable recipes with your selection — worked examples for your chosen use case — and **Use Your Own Agent** exports your current filters as context for Claude or any agent.
# Experiments
Source: https://docs.fiddler.ai/getting-started/experiments
Systematically evaluate and compare your LLM and agentic applications with Fiddler Experiments: built-in and custom evaluators, golden datasets, and side-by-side experiment comparison.
Building reliable LLM and agentic applications requires more than just deploying models; it requires systematic evaluation to ensure quality, safety, and consistent performance. Fiddler Experiments provides an evaluation framework that helps you test, measure, and improve your AI applications. Whether you're comparing different prompts, testing model updates, or ensuring quality standards, Fiddler Experiments provides the tools to quantify and validate changes.
**First Time Setting Up?**
Before you can start running experiments, you need to create a project and application in Fiddler. See [Onboard Your GenAI Application](/getting-started/genai-application-onboarding) for step-by-step instructions.
## What Is Fiddler Experiments?
Fiddler Experiments is an evaluation platform that helps you measure and improve the quality of your LLM and agentic applications. It provides built-in evaluators, custom evaluation support, and a comparison interface to:
* **Test systematically**: Create comprehensive test suites with real-world scenarios
* **Measure objectively**: Use built-in and custom evaluators to assess quality
* **Compare confidently**: Analyze experiments side-by-side to make data-driven decisions
* **Improve continuously**: Track progress over time and identify areas for enhancement
### Core Concepts
Understanding three key concepts will help you get the most from Fiddler Experiments:
1. **Datasets**: Collections of test cases with inputs and expected outputs
2. **Experiments**: Test runs that evaluate your application against a dataset
3. **Evaluators**: Metrics that assess specific aspects of your application's performance
**New to evaluation terminology?** See our [Experiments Glossary](/glossary/experiments) for definitions of key terms such as evaluator, metric, score, and experiment.
**Powered by Fiddler Centor Models**
Fiddler Experiments evaluators run on [Fiddler Centor Models](/glossary/centor-models) that operate entirely within your environment—no external API calls, zero hidden costs, and enterprise-grade security.
## Why Choose Fiddler Experiments?
Fiddler Experiments stands apart from fragmented evaluation tools by providing an integrated approach to AI quality assurance:
### Unified Development-to-Production Workflow
Unlike tools that separate pre-production testing from production monitoring, Fiddler Experiments integrates seamlessly with [Fiddler Agentic Observability](/getting-started/agentic-monitoring). This unified workflow means:
* **Consistent metrics**: The same evaluators you use in development run in production monitoring
* **Continuous learning**: Production insights feed back into experiment datasets
* **Seamless transition**: Deploy with confidence knowing your production monitoring matches your testing
### Cost-Effective with Centor Models
Powered by [Fiddler Centor Models](/glossary/centor-models), Fiddler Experiments evaluators run on purpose-built Centor Models:
* **Zero Hidden Costs**: No external API calls, no per-request fees, no token charges
* **High Performance**: \<100ms response times enable real-time evaluation
* **Enterprise Security**: Your data never leaves your environment—no third-party API exposure
* **Superior Accuracy**: 50% more accurate than generic models on LLM evaluation benchmarks
### Enterprise-Grade Reliability
* **Scalable**: Evaluate thousands of test cases in parallel
* **Collaborative**: Team access controls and shared experiment libraries
* **Auditable**: Complete traceability for compliance and debugging
* **Framework-Agnostic**: Works with any LLM provider or agentic framework
## Why Systematic Evaluation Matters
LLM and agentic applications face unique quality challenges that make systematic evaluation essential:
### The Challenge of Variability
LLMs and agentic applications are non-deterministic, meaning they can produce different outputs for the same input, making quality assessment difficult. Without systematic evaluation:
* You can't reliably detect quality degradation
* Improvements are based on anecdotal evidence rather than data
* Edge cases and failure modes go unnoticed until production
### The Need for Objectivity
Human evaluation is valuable but subjective and doesn't scale. Automated evaluators provide:
* Consistent, repeatable measurements
* Scalable evaluation across thousands of test cases
* Objective metrics for decision-making
### The Power of Comparison
Understanding relative performance is crucial for improvement. Side-by-side comparison helps you:
* Validate that changes actually improve performance
* Choose between different approaches with confidence
* Track progress toward quality goals
## Navigating the Fiddler Experiments Interface
The Fiddler Experiments interface provides search, filtering, and side-by-side comparison of experiments. Let's explore the key areas you'll use.
### Experiments Dashboard
The main dashboard provides an overview of all your experiments, making it easy to track progress and identify trends.
Key features of the dashboard:
* **Search and filter**: Quickly find experiments by name, application, or dataset
* **Status indicators**: See which experiments are completed, in progress, or failed
* **Metadata display**: View custom metadata to understand experiment context
* **Quick actions**: Access experiment details or start comparisons directly
### Viewing Experiment Details
Click on any experiment to explore the results in depth and understand your application's performance.
The experiment details view provides:
* **Test case results**: See inputs, outputs, and expected outputs for each item
* **Evaluator scores**: View all metrics calculated for each test case
* **Experiment metadata**: View details and labels that describe the experiment
### Comparing Experiments
The comparison view shows performance differences between experiments, helping you validate whether changes improve your application.
Comparison features include:
* **Side-by-side metrics**: See how each experiment performs on the same test cases
* **Flexible metric selection**: Choose which evaluators to compare
## Core Workflow
The typical Fiddler Experiments workflow follows a simple pattern that scales from quick tests to comprehensive experiment suites:
**TL;DR**: Create a dataset with test cases → Configure evaluators → Run experiment → Analyze results. Takes \~15 minutes for the first experiment.
```mermaid theme={null}
graph TD
A[📊 Create Dataset] --> B[✏️ Define Test Cases]
B --> C[▶️ Run Experiment]
C --> D[📈 Analyze Results]
D --> E[🔄 Compare & Iterate]
E --> C
style A fill:#1976d2,color:#fff,stroke:#0d47a1,stroke-width:3px
style B fill:#1976d2,color:#fff,stroke:#0d47a1,stroke-width:3px
style C fill:#388e3c,color:#fff,stroke:#1b5e20,stroke-width:3px
style D fill:#f57c00,color:#fff,stroke:#e65100,stroke-width:3px
style E fill:#7b1fa2,color:#fff,stroke:#4a148c,stroke-width:3px
```
The following walk-through is a high-level overview of a basic experiment workflow. For a fully functional example, refer to our [Quick Start Guide and Notebook](/evaluate-and-test/evals-sdk-quick-start).
Refer to the [Evals SDK Technical Reference](/sdk-api/evals/evaluate) for instructions on installing and initializing the `fiddler-evals-sdk` Python package.
Start by creating a dataset that represents the scenarios your application needs to handle. This example enters test cases inline in the code, but you may also use a CSV file, JSONL file, or pandas DataFrame to load test cases:
```python theme={null}
from fiddler_evals import Dataset, NewDatasetItem
# Create dataset
dataset = Dataset.create(
name="customer-support-qa",
application_id=app.id,
description="Common customer support questions"
)
# Add test cases
items = [
NewDatasetItem(
inputs={"question": "How do I reset my password?"},
expected_outputs={"answer": "To reset your password, click 'Forgot Password' on the login page..."},
metadata={"category": "account"}
),
# Add more test cases...
]
dataset.insert(items)
```
Choose evaluators that measure what matters for your use case:
```python theme={null}
from fiddler_evals.evaluators import AnswerRelevance, Conciseness, Coherence
MODEL = "openai/gpt-4o"
CREDENTIAL = "your-llm-credential" # From Settings > LLM Gateway
evaluators = [
AnswerRelevance(model=MODEL, credential=CREDENTIAL), # Is the answer relevant to the question? (High/Medium/Low)
Conciseness(model=MODEL, credential=CREDENTIAL), # Is the response appropriately brief?
Coherence(model=MODEL, credential=CREDENTIAL), # Does the response flow logically?
]
```
**Evaluating a RAG application?** Use the [RAG Health Metrics](#rag-system-evaluation) evaluators (`AnswerRelevance`, `ContextRelevance`, `RAGFaithfulness`) to diagnose retrieval vs. generation issues.
**Evaluation Pattern**: Fiddler's built-in evaluators use the [LLM-as-a-Judge](/observability/llm/llm-evaluation-prompt-specs) pattern, where language models assess quality dimensions that are difficult to measure with rule-based systems. This provides automated quality assessment that approximates human evaluation patterns while maintaining consistency across thousands of test cases.
Execute your experiment to see how your application performs:
```python theme={null}
from fiddler_evals import evaluate
def my_application(inputs, extras, metadata):
# Your LLM application logic
response = generate_answer(inputs["question"])
return {"answer": response}
results = evaluate(
dataset=dataset,
task=my_application,
evaluators=evaluators,
name_prefix="v1.0-baseline"
)
```
Use the Fiddler UI to understand your results and identify improvements:
1. Review individual scores to find problem areas
2. Compare experiments to validate improvements
3. Export data for deeper analysis
4. Iterate based on insights
## Understanding Your Experiment Results
Interpreting experiment results effectively helps you make informed decisions about your application.
### Reading Score Cards
Each evaluator produces scores that help you understand specific aspects of performance:
* **Binary scores** (0 or 1): Pass/fail metrics like relevance or correctness
* **Continuous scores** (0.0 to 1.0): Gradual metrics like similarity or confidence
* **Categorical scores**: Classifications like sentiment (positive/neutral/negative)
### Identifying Patterns
Look for patterns across your test cases:
* **Consistent failures**: Indicate systemic issues that need addressing
* **Category-specific problems**: Suggest areas needing specialized handling
* **Score correlations**: Reveal trade-offs between different metrics
### Making Improvements
Use experiment insights to guide your optimization efforts:
1. **Focus on the lowest scores**: Address the most significant quality issues first
2. **Test hypotheses**: Use experiments to validate that changes improve metrics
3. **Monitor trade-offs**: Ensure improvements don't degrade other aspects
## Common Use Cases
Fiddler Experiments supports various experiment scenarios across the LLM application lifecycle:
### A/B Testing Prompts
Compare different prompt strategies to find what works best:
```python theme={null}
# Baseline prompt
baseline_results = evaluate(
dataset=dataset,
task=baseline_prompt_app,
evaluators=evaluators,
name_prefix="prompt-baseline"
)
# Improved prompt
improved_results = evaluate(
dataset=dataset,
task=improved_prompt_app,
evaluators=evaluators,
name_prefix="prompt-improved"
)
# Compare in UI to see which performs better
```
### Model Version Comparison
Validate that model updates improve performance:
* Test the same dataset against different model versions
* Compare quality metrics side-by-side
* Ensure no regression in critical capabilities
### Regression Testing
Protect against quality degradation as you develop:
* Run standard test suites before deployments
* Set quality thresholds that must be met
* Track performance trends over time
### Safety Validation
Ensure your application meets safety standards:
* Test with adversarial inputs
* Measure toxicity and bias metrics
* Validate content filtering effectiveness
### RAG System Evaluation
Evaluate Retrieval-Augmented Generation pipelines using the RAG Health Metrics evaluators — a purpose-built diagnostic framework that pinpoints whether issues originate in retrieval, generation, or query understanding:
```python theme={null}
from fiddler_evals.evaluators import AnswerRelevance, ContextRelevance, RAGFaithfulness
MODEL = "openai/gpt-4o"
CREDENTIAL = "your-llm-credential" # From Settings > LLM Gateway
evaluators = [
AnswerRelevance(model=MODEL, credential=CREDENTIAL), # Is the response relevant to the query? (High/Medium/Low)
ContextRelevance(model=MODEL, credential=CREDENTIAL), # Are retrieved documents relevant? (High/Medium/Low)
RAGFaithfulness(model=MODEL, credential=CREDENTIAL), # Is the response grounded in context? (Yes/No)
]
results = evaluate(
dataset=dataset,
task=my_rag_application,
evaluators=evaluators,
name_prefix="rag-baseline"
)
```
Use the three evaluators together to diagnose specific failure modes:
| What the metrics tell you | Why it's happening | Next Step |
| --------------------------------- | ------------------------------------- | ------------------------------------------------ |
| High relevance + Low faithfulness | Hallucinations despite being on-topic | Check if retrieval provided sufficient grounding |
| High faithfulness + Low relevance | Grounded but didn't answer the query | Check if retrieval provided relevant information |
| Low Context Relevance | Retrieval pulling wrong documents | Fix retrieval mechanism |
**RAG Health Metrics** works alongside Fiddler's existing 80+ LLM metrics (toxicity, PII, coherence, and more) — providing targeted RAG diagnostics that complement your existing evaluation stack.
### Agentic Application Evaluation
Evaluate AI agents and multi-step workflows with specialized patterns:
* **Trajectory Evaluation**: Assess agent decision-making sequences and tool selection paths
* **Reasoning Coherence**: Validate logical flow from planning through execution
* **Tool Usage Quality**: Measure appropriateness and effectiveness of tool calls
* **Multi-Agent Coordination**: Track information flow and task delegation patterns
**Connect to Production**: Use Fiddler Experiments during development, then monitor agent behavior in production with [Agentic Monitoring](/getting-started/agentic-monitoring).
## Best Practices
Follow these practices to get the most value from Fiddler Experiments:
### Building Representative Datasets
Create test sets that reflect real-world usage:
* **Include edge cases**: Don't just test the happy path—use dataset metadata to tag edge cases for focused analysis
* **Balance categories**: Ensure coverage across different scenarios, then use experiment comparison to validate your test distribution matches production patterns
* **Use production data**: Incorporate actual user inputs when possible (anonymized and sanitized) — [Golden Datasets](/evaluate-and-test/golden-datasets) promote real production spans directly into a dataset
* **Update regularly**: Keep test cases current with evolving requirements—track dataset versions in metadata
### Choosing Appropriate Evaluators
Select metrics that align with your goals:
* **Start with basics**: Answer relevance and safety evaluators (toxicity, PII) are essential for most applications
* **Add domain-specific metrics**: Build custom evaluators for specialized needs
* **Avoid metric overload**: Focus on 3-5 key metrics that actually drive decisions rather than tracking everything
* **Validate with humans**: Spot-check evaluator scores against human judgment to ensure they align with your quality standards
### Setting Up Experiment Cycles
Make experiments a routine part of development:
* **Pre-deployment testing**: Always evaluate before production changes
* **Regular benchmarking**: Schedule periodic comprehensive experiments
* **Continuous monitoring**: Track key metrics in production
* **Iterative improvement**: Use insights to guide development priorities
## Getting Started Checklist
Ready to start evaluating? Follow this simple checklist:
* [ ] **Set up your environment**: Install the [Fiddler Evals SDK](/sdk-api/evals/evaluate)
* [ ] **Create your first dataset**: Start with 10-20 representative test cases
* [ ] **Run a baseline experiment**: Establish current performance levels
* [ ] **Review results in the UI**: Understand your application's strengths and weaknesses
* [ ] **Improve and compare**: Validate that changes have a positive impact
## Troubleshooting Common Issues
### Experiments Not Appearing
If your experiments don't show in the dashboard:
* Verify the experiment was completed successfully
* Check that you're viewing the correct project/application
* Refresh the page to load the latest data
### Unexpected Scores
If experiment scores seem incorrect:
* Review the evaluator documentation to understand scoring logic
* Check that inputs/outputs are formatted correctly
* Validate that the correct evaluator parameters are used
### Comparison Not Working
If you can't compare experiments:
* Ensure both experiments use the same dataset
* At least one metric/evaluator should be in common to compare the experiments
* Verify experiments have completed successfully
* Check that you have permissions to view both experiments
## Next Steps
### From Experiments to Production: The Complete Lifecycle
Fiddler Experiments is your **Test** phase in Fiddler's complete [end-to-end agentic AI lifecycle](https://www.fiddler.ai/blog/end-to-end-agentic-observability-lifecycle):
**1. Build** → Design and instrument your LLM applications and agents
**2. Test** → Evaluate systematically with Fiddler Experiments *(you are here)*
**3. Monitor** → Track production performance with Agentic Monitoring
**4. Improve** → Use insights to enhance quality and refine your agents
This unified approach ensures your evaluation criteria in development become your monitoring standards in production—no fragmentation, no tool switching.
***
Choose your path based on your role and goals:
**For Developers** 🔧
1. [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start) - Hands-on tutorial with code
2. [Golden Datasets](/evaluate-and-test/golden-datasets) - Promote production spans into a regression dataset
3. [Capture Traces During Experiments](/evaluate-and-test/experiment-trace-capture) - Score evaluators on execution traces
4. [Evals SDK Advanced Guide](/developers/tutorials/experiments/evals-sdk-advanced) - Production-ready configurations
5. [Fiddler Evals SDK](/sdk-api/evals/evaluate) - Complete technical docs
**For Teams Scaling AI** 📈
1. [Agentic Monitoring](/getting-started/agentic-monitoring) - Monitor agents in production
2. [LLM Monitoring](/getting-started/llm-monitoring) - Production observability
**For Product & Business** 💼
1. Review sample dashboards in your Fiddler instance
2. Schedule a workshop with your Fiddler team
3. Explore case studies and best practices on the Fiddler blog
## Summary
Fiddler Experiments adds systematic measurement to LLM application development, replacing ad-hoc testing with quantified assessment. By evaluating your applications, you can:
* **Compare experiments quantitatively**: Use side-by-side metrics to validate that changes improve performance
* **Track experiment trends**: Monitor quality over time through the experiments dashboard
* **Establish quality baselines**: Define acceptable score thresholds for your use case
* **Reuse test suites**: Ensure consistency by testing model versions against the same datasets
Start with 10-20 test cases and gradually expand your experiment coverage. The metrics you track will help you make informed decisions about model changes and deployment.
# Onboard Your GenAI Application
Source: https://docs.fiddler.ai/getting-started/genai-application-onboarding
Set up your first GenAI project and application in Fiddler. Learn how to create projects, configure applications, and start monitoring your GenAI interactions.
This guide walks you through setting up your first **project** and **application**, connecting it to your code, and starting real-time monitoring of your GenAI interactions.
***
## Before You Begin
Before onboarding your GenAI application, ensure you have:
* **Fiddler Account** - Active access to your Fiddler instance
* **GenAI Application** - An LLM-powered application or agent ready to instrument
* **Admin or Project Admin Role** - [Permissions](/reference/access-control/role-based-access) to create projects and applications
***
## Key Concepts
### Projects
A **project** represents a workspace that groups related AI applications together. Projects are typically organized by product, team, or use case.
Each project can have:
* Multiple **applications** under it
* Shared [**team members**](/reference/administration/settings#teams) and [**permissions**](/reference/access-control/role-based-access)
* Consolidated monitoring and analytics across all applications in that project
You can **use an existing project** if your team already has one, or **create a new project** for a fresh setup.
### Applications
An **application** is the specific GenAI system or integration you want to monitor. It represents one instance of your deployed LLM or agentic workflow—such as a chatbot, summarization API, or evaluation pipeline.
Each application has its own:
* **Application ID** - Unique identifier for SDK integration
* **Monitoring configuration** - Metrics, evaluators, and alerts
* **Data streams** - Span events, traces, and performance metrics
***
## Onboard Your Application
1. Navigate to **GenAI Applications** in the left sidebar
2. Click **Add Application** in the top-right corner
3. Select how to organize your new application:
**Option 1: Use Existing Project**
* Select **Use Existing Project**
* Choose a project from the dropdown menu
* Review current team members, add or remove members, or change roles
**Option 2: Create New Project**
* Select **Create New Project**
* (Optional) Add initial team members using the same process as above
4. Click **Next** to continue
**Tip:** Choose project names that reflect your team structure or product areas (for example, `CustomerSupportChatbot` or `DataScienceTeamApps`).
1. In the **Create Your Application** step:
* Review the selected project name displayed at the top
* Enter an **Application Name** in the text field
* The name should uniquely identify the GenAI system you're monitoring
2. Click **Create Application** to continue
**Note:** Application names must be unique within the project. Choose names that clearly identify the specific use case (for example, `ContentSummarizer` or `CodeAssistant`).
Once your application is created, you'll see the **Setup Complete!** confirmation screen with your integration details.
**1. Copy Your Credentials**
You'll need two pieces of information to connect your application:
* **Application ID** - Uniquely identifies your application (UUID format)
* **API Key** - Authenticates requests from your code
**To copy your credentials:**
1. Locate the **Grab Your Application ID and API Key** section
2. Copy the **Application ID**:
* Click the copy icon next to the Application ID field
* Store it securely—you'll need it for SDK configuration
3. Copy your **API Key**:
* Navigate to **Settings → Credentials** in Fiddler
* Copy your API key or create a new one
* See [API Keys](/reference/administration/settings#credentials) for details
**Security Best Practice**
Treat your API key like a password. Never commit it to version control or share it publicly. Use environment variables or secure credential management systems.
**2. Integrate Using the SDK**
Click **View SDK Documentation** to access integration guides for your technology stack:
* **Fiddler LangGraph SDK** - For LangGraph and LangChain applications
* **Fiddler Evals SDK** - For custom experiments and testing workflows
* **OpenTelemetry Integration** - For custom instrumentation
Follow the SDK-specific guide to:
1. Install the Fiddler SDK in your application
2. Configure authentication with your Application ID and API Key
3. Instrument your code to send span events and metrics
4. Verify data is flowing to Fiddler
**3. Complete Setup**
Once you've copied your credentials and reviewed the integration guide:
* Click **Finish Setup** to close the onboarding dialog
Your application is now configured and ready to receive monitoring data
## Next Steps
Now that your application is onboarded, you can:
* **Publish Data** - Run your instrumented application to send span events to Fiddler
* **View Live Data** - Navigate to your application to see real-time traces and metrics
* **Analyze Performance** - Use dashboards to track latency, throughput, and errors
* **Add Evaluators** - Create custom evaluators to assess response quality
* **Set Up Alerts** - Configure alerts for anomalies or performance degradation
* **Define Rules** - Create business rules for content moderation or compliance
* **Add More Applications** - Onboard additional GenAI systems under the same project
* **Organize by Team** - Create separate projects for different teams or products
* **Manage Access** - Use project-level permissions to control who can view and modify applications
***
## Frequently Asked Questions
No, once an application is created within a project, it cannot be moved to another project. If you need to reorganize, you'll need to create a new application in the correct project.
There is no hard limit on the number of applications per project. However, for better organization, we recommend grouping related applications that share common team members or business context.
Deleting a project will remove all applications, monitoring data, and configurations within that project. This action cannot be undone. Ensure you export any data you need before deletion.
No, each application instance (development, staging, production) should have its own Application ID. This allows you to monitor each environment separately and apply different configurations or alerts.
Your API key is available in **Settings → Credentials**. See [Creating API Keys](/reference/administration/settings#creating-api-keys) for step-by-step instructions.
***
## Related Documentation
Instrument LangGraph applications in 10 minutes.
Run experiments for your GenAI applications.
Understand project and application concepts.
Manage team permissions.
Create and manage API keys.
***
# Guardrails
Source: https://docs.fiddler.ai/getting-started/guardrails
Fiddler Guardrails protects GenAI and agentic applications against hallucinations, safety risks, and jailbreaks in real time — available via the Fiddler API.
### Overview
Fiddler Guardrails provides enterprise-grade protection against critical LLM risks in production environments. This solution actively moderates and mitigates harmful content in both prompts and responses, including hallucinations, toxicity, safety violations, prompt injection attacks, and jailbreaking attempts. The solution is powered by proprietary, fine-tuned, task-specific Fiddler Centor Models, specifically engineered for real-time content analysis.
### Key Benefits
* **Industry’s Fastest Guardrails**: Achieves sub-100ms latency for real-time moderation without impacting user experience
* **Enterprise Scalability:** Handles 5+ million daily requests with consistent performance and reliability
* **Resource Efficiency**: Purpose-built Centor Models deliver high accuracy with significantly lower computational requirements than general-purpose LLMs
* **Enterprise Security**: Deployed in the customer’s VPC or air-gapped environment with data never leaving the customer’s environment
### Availability
Fiddler Guardrails is available as part of the Fiddler AI Observability and Security platform. You access Guardrails in your own Fiddler environment over the REST API (`https://{fiddler_endpoint}/v3/guardrails/*`) using a Fiddler API key generated from **Settings → [Credentials](/reference/administration/settings#credentials)**.
Fiddler Guardrails covers safety, faithfulness (hallucination), and PII/PHI detection, each powered by a purpose-built Fiddler Centor Model. See the [Quick Start Guide](/protection/guardrails-quick-start) to set up access and make your first call.
### Next Steps
Explore additional documentation to get the most out of your Fiddler Guardrails experience.
#### Fiddler Guardrails guides include:
* [Quick Start Guide](/protection/guardrails-quick-start)
* [Guardrails Documentation](/protection/guardrails)
* [Frequently Asked Questions](/protection/guardrails-faq)
* Notebook Tutorials:
* [Safety](/developers/tutorials/guardrails/guardrails-safety)
* [Faithfulness](/developers/tutorials/guardrails/guardrails-faithfulness)
* [PII](/developers/tutorials/guardrails/guardrails-pii)
# LLM Monitoring
Source: https://docs.fiddler.ai/getting-started/llm-monitoring
Monitor LLM applications in production with Fiddler. Track quality, safety, and performance enrichments, detect problematic responses, and diagnose issues before they reach users.
## Fiddler LLM Monitoring Introduction
### What Is LLM Monitoring?
Large Language Models (LLMs) are powerful, but they introduce unique challenges around accuracy, safety, and reliability. Effective monitoring is critical for detecting issues like hallucinations, toxic content, and performance degradation in production LLM applications.
### How Fiddler LLM Monitoring Works
Fiddler's LLM monitoring solution tracks your AI application's inputs and outputs, then enriches this data with specialized metrics that measure quality, safety, and performance. These enrichments provide visibility into how your LLM applications behave in production, enabling you to:
* Detect problematic responses before they impact users
* Identify patterns of failure across your applications
* Track performance trends over time
* Analyze root causes when issues occur
### Key Capabilities
* **Comprehensive Metrics**: Monitor hallucinations, toxicity, relevance, latency, and many other LLM-specific metrics
* **Real-time Analysis**: Track performance as it happens with intuitive dashboards
* **Advanced Enrichments**: Generate embeddings, similarity scores, and specialized trust metrics automatically
* **Drift Detection**: Identify when prompts or responses drift from expected patterns
* **RAG-specific Monitoring**: For retrieval-augmented applications, analyze retrieval quality and source relevance
### Getting Started
Implementing Fiddler LLM monitoring requires just three steps:
Define its inputs, outputs, and which enrichment metrics you need.
Send prompts, responses, and context to Fiddler.
Use dashboards and alerts to track the metrics most important to your use case.
Fiddler handles the complex work of generating enrichments, detecting anomalies, and providing the visualizations you need to maintain high-quality LLM applications.
### Next Steps
Stand up LLM monitoring end to end in about 10 minutes.
* [Understanding LLM enrichment metrics](/observability/llm/enrichments)
* [How Fiddler generates LLM metrics](/observability/llm/llm-based-metrics)
* [Available LLM metrics](/observability/llm/selecting-enrichments)
* [Create LLM visualizations using embeddings](/observability/llm/embedding-visualization-with-umap)
* [Fiddler Python client SDK](/sdk-api/python-client/connection)
* [Fiddler Python client SDK guides](/developers/python-client-guides/installation-and-setup)
# ML Observability
Source: https://docs.fiddler.ai/getting-started/ml-observability
Monitor traditional ML models in production with Fiddler. Track performance, detect data drift, run root cause analysis, and ensure model fairness at scale.
### Introduction
Fiddler is a comprehensive AI observability platform that helps data science teams monitor, explain, and improve their machine learning models and LLM applications in production. Our platform provides the visibility and insights you need to ensure your AI systems perform reliably and deliver business value.
With Fiddler, you can:
* **Monitor model health** across traditional ML models and LLM applications with specialized metrics
* **Detect issues early** through customizable alerts and drift detection
* **Debug problems quickly** with explainable AI and root cause analysis tools
* **Ensure model fairness** through segment analysis and bias detection
* **Optimize performance** with detailed traffic and performance tracking
Whether you're managing a few models or hundreds, Fiddler provides the tools to maintain confidence in your AI systems as they scale.
## Key Capabilities
### Comprehensive Monitoring
* **Performance Tracking**: Monitor model accuracy, precision, recall, and other ML metrics in real time across all your deployments
* **Data Drift Detection**: Identify shifts in your production data that could impact model performance before they cause issues
* **Data Integrity Checks**: Ensure your models receive valid, properly formatted data that meets your expectations
* **Vector Monitoring**: Specialized tools for monitoring embedding-based and vector search applications
### Advanced Analytics
* **Embedding Visualization**: Explore high-dimensional data using UMAP to understand patterns and clusters
* **Model Segmentation**: Analyze performance across different user cohorts to identify bias and uncover targeted improvements
* **Statistical Analysis**: Generate detailed statistics on model inputs, outputs, and performance metrics
* **Custom Metrics**: Define and track metrics specific to your business needs and use cases
### Getting Started
Implementing Fiddler ML monitoring requires just three steps:
Define its inputs, outputs, and related metadata.
Send the "digital exhaust" from your model serving platform to Fiddler.
Use dashboards and alerts to track the metrics most important to your use case.
Fiddler automatically handles the complex work of generating metrics, detecting anomalies, and providing the visualizations you need to maintain high-quality ML systems.
### Next Steps
Stand up ML monitoring end to end in about 10 minutes.
* [Available auto-generated metrics](/observability/platform)
* [Data drift and model performance](/observability/platform/performance-tracking-platform)
* [Setting up alerts](/observability/platform/alerts-platform)
* [Creating your own custom metrics](/glossary/custom-metrics)
* [Diagnose issues with Root Cause Analysis](/observability/analytics#root-cause-analysis)
* [Auto-generated and custom dashboards](/observability/dashboards)
* [Fiddler Python client SDK](/sdk-api/python-client/connection)
* [Fiddler Python client SDK guides](/developers/python-client-guides/installation-and-setup)
# Use Fiddler with AI Agents
Source: https://docs.fiddler.ai/getting-started/use-fiddler-with-ai-agents
Install Fiddler's published agent skills and connect the documentation MCP server so your AI coding agent can onboard models and set up monitoring in Fiddler.
Fiddler publishes agent skills and a documentation server so your AI coding agent can learn how to use Fiddler — onboarding a model, wiring up monitoring, and picking the right SDK — without you pasting docs into the chat.
## What Ships
An agent skill is a packaged set of instructions and working code patterns your agent loads on demand. Fiddler publishes these skills in the open agentskills.io format directly from the documentation site. Two skills ship today:
| Skill | What it does | Typical trigger |
| ----------------------- | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------- |
| `fiddler` | Orients an agent in Fiddler's product surface and points it to the right path and SDK. | Any Fiddler task where the agent must pick a path or SDK. |
| `fiddler-ml-onboarding` | Walks an agent through onboarding a traditional ML model end to end and publishing events. | Onboarding a tabular model or publishing inference events. |
Alongside the skills, Fiddler serves a documentation Model Context Protocol (MCP) server at `https://docs.fiddler.ai/mcp`. A connected agent can search and read the docs, and the skills are exposed as MCP resources, so an agent can read them without installing anything.
## Connect the MCP Server
Connecting the MCP server is the fastest way to start — it requires no installation and works with any MCP client.
Use the page-actions menu on every docs page to connect a client in one click: open it and choose your client from the Cursor, VS Code, and MCP entries.
For any other MCP client, add the server manually:
```json theme={null}
{
"mcpServers": {
"fiddler-docs": {
"url": "https://docs.fiddler.ai/mcp"
}
}
}
```
For the most reliable results, also install the skills: in testing, agents with the skills installed completed onboarding tasks that documentation search alone did not.
## One-Line Install
Install the skills into your supported agents with a single command. It requires Node.js (which provides `npx`) and detects which agents you have installed, adding the skills to each:
```bash theme={null}
npx skills add https://docs.fiddler.ai
```
The command installs into each detected agent's skills directory — for Claude Code, the project's `.claude/skills/` (or `~/.claude/skills/` for a user-scope install). To confirm the skills are active, ask your agent "which Fiddler skills do you have available?", or start a task that mentions onboarding a Fiddler model and check that the `fiddler-ml-onboarding` skill engages.
## Compatibility
This matrix lists only what is tested today. It grows as we verify more agents.
| Agent | Loads and applies skills | Reads skills over MCP (`/mcp`) |
| -------------- | -------------------------------- | ------------------------------ |
| Claude Code | Verified (project-scope loading) | Verified |
| Any MCP client | Not applicable | Verified |
Copying a skill's raw `SKILL.md` into your agent's skills directory is the verified path today and the fallback when an agent supports neither install-from-URL nor MCP; the discovery index at `https://docs.fiddler.ai/.well-known/agent-skills/index.json` lists every published skill's `SKILL.md` URL. `npx skills add https://docs.fiddler.ai` is the streamlined one-liner and works with any agent the skills CLI supports. Additional agents, such as Cursor, get their own rows here as we verify them.
Skills are most reliably applied by frontier-tier models. A smaller model may need an explicit nudge — for example, "use the fiddler-ml-onboarding skill" — to apply the packaged guidance.
## Everything Else Your Agent Needs
For anything the skills do not cover, point your agent at the Fiddler llms.txt index (`https://docs.fiddler.ai/llms.txt`) and connect the documentation MCP server (`https://docs.fiddler.ai/mcp`). Together they let an agent discover and read any page on the docs site on demand.
## Next Steps
Register a tabular model and start monitoring drift and performance in about 10 minutes.
Stand up monitoring for an LLM or GenAI application end to end.
A condensed, end-to-end tour of Fiddler for your first hands-on run.
# Agentic Observability
Source: https://docs.fiddler.ai/glossary/agentic-observability
Comprehensive monitoring, tracing, and analysis of AI agent systems that provides hierarchical visibility into agent reasoning, coordination, and decision-making across distributed multi-agent applica
Agentic Observability is the practice of monitoring, tracing, and analyzing autonomous AI agent systems to ensure their reliability, transparency, and alignment with business objectives. It extends beyond traditional application performance monitoring (APM) by capturing the cognitive lifecycle of AI agents—including reasoning processes, tool selection, execution outcomes, self-reflection, and inter-agent coordination—to provide complete visibility into distributed multi-agent systems.
Unlike traditional observability, which tracks deterministic metrics such as latency and error rates, Agentic Observability addresses the unique challenges of probabilistic, goal-driven systems that autonomously plan, decide, and adapt based on dynamic environments and outcomes.
## Core Terminology
### Agent
An autonomous AI system that can plan actions, make decisions, interact with tools, and learn from experience. Agents operate through a cognitive loop of thought, action, execution, reflection, and alignment.
### Multi-Agent System
A distributed system where multiple AI agents coordinate to accomplish complex workflows. Each agent maintains its own state, reasoning, and tool interactions while communicating and collaborating with other agents.
### Trace
A complete record of an agent's execution path, including all decisions, tool calls, and outcomes for a specific task or session. Traces capture the full context of agent behavior.
### Span
An individual unit of work within a trace, representing discrete operations such as tool invocations, API calls, or agent-to-agent communications. Spans form the building blocks of hierarchical observability.
### Session
A bounded interaction context containing multiple agent executions, typically representing a complete user request or workflow that may involve multiple agents and numerous traces. Also known as a conversation.
### Tool Call
An agent's invocation of external capabilities, APIs, or functions to accomplish specific tasks. Tool calls represent the bridge between agent reasoning and real-world actions.
## How Fiddler Provides Agentic Observability
Fiddler's Agentic Observability platform delivers comprehensive monitoring for multi-agent systems through a hierarchical approach that captures data across multiple layers:
**1. Application Layer**: High-level system health metrics, aggregated performance indicators, and cross-agent dependencies
**2. Session Layer**: User interaction contexts, workflow orchestration patterns, and end-to-end request tracking
**3. Agent Layer**: Individual agent performance, reasoning traces, decision paths, and behavioral patterns
**4. Action Layer**: Granular tool calls, API interactions, execution results, and timing metrics
The platform integrates with leading agentic frameworks (LangGraph, Strands, custom agents) and provides:
* **Hierarchical Root Cause Analysis**: Drill down from application-level issues to specific agent decisions or tool failures
* **Semantic Tracing**: Capture not just what agents do, but why they make particular decisions
* **Cross-Agent Visibility**: Monitor coordination, information flow, and dependencies between agents
* **Real-time Behavioral Analysis**: Detect off-policy behavior, coordination failures, and goal misalignment
## Why Agentic Observability Is Important
As enterprises deploy multi-agent systems for critical business processes, the complexity of monitoring increases exponentially, requiring up to 26 times more monitoring resources than single-agent applications. Agentic Observability addresses several critical challenges:
* **Debugging Complexity**: Multi-agent systems generate extensive reasoning traces, tool logs, and decision paths that traditional APM tools cannot effectively parse or correlate.
* **Trust and Compliance**: With 90% of enterprises citing security, trust, and compliance as top concerns for agentic AI, comprehensive observability enables policy enforcement and regulatory adherence.
* **Cascading Failures**: Errors in one agent can propagate through dependencies, making root cause analysis essential for system reliability.
* **Performance Optimization**: Understanding agent decision-making patterns enables teams to optimize workflows, reduce unnecessary tool calls, and improve response times.
* **Alignment Verification**: Ensures agents operate within defined boundaries and adhere to business objectives, preventing autonomous systems from deviating from intended behavior.
## The Agent Lifecycle: Five Observable Stages
Fiddler breaks down agent observability into five critical stages that form a continuous feedback loop:
1. **Thought (Ingest, Retrieve, Interpret)**: Captures prompt processing, memory retrieval, belief state formation, and goal interpretation
2. **Action (Plan and Select Tools)**: Monitors decision operationalization, tool selection logic, and execution sequencing
3. **Execution (Perform Tasks)**: Tracks tool invocations, API calls, input/output traces, latency, and success/failure signals
4. **Reflection (Evaluate and Adapt)**: Observes self-critique processes, trajectory scoring, error analysis, and adaptive learning
5. **Alignment (Enforce Trust and Safety)**: Implements guardrails, Centor Model evaluations, and human-in-the-loop interventions
## Types of Agentic Observability
* **Development-Time Observability**: Trace and debug multi-agent systems during development to identify coordination issues, optimize workflows, and validate agent behavior before production deployment.
* **Runtime Performance Monitoring**: Track operational metrics including agent latency, tool call efficiency, resource utilization, and throughput across distributed agent deployments.
* **Behavioral Analysis**: Monitor agent reasoning patterns, decision consistency, goal achievement rates, and adaptation mechanisms to ensure aligned autonomous behavior.
* **Coordination Monitoring**: Observe inter-agent communication, information handoffs, task delegation patterns, and collaborative decision-making in multi-agent systems.
* **Trust and Safety Monitoring**: Implement continuous evaluation of agent outputs against safety policies, compliance requirements, and ethical guidelines with real-time intervention capabilities.
## Challenges
Implementing effective Agentic Observability presents unique technical and operational challenges:
* **Data Volume and Complexity**: Multi-agent systems generate massive amounts of hierarchical data across reasoning traces, tool logs, and coordination events, requiring sophisticated data management strategies.
* **Semantic Understanding**: Unlike traditional metrics, agent decisions require semantic interpretation to understand the "why" behind actions, not just the "what."
* **Real-time Processing**: Agents operate at high speed with complex interdependencies, demanding low-latency observability that doesn't impact system performance.
* **Cross-Agent Correlation**: Tracing causality across multiple autonomous agents with asynchronous interactions requires advanced correlation algorithms and timestamp synchronization.
* **Dynamic Adaptation**: Agents that learn and adapt their behavior over time make it challenging to establish stable baselines for anomaly detection.
* **Privacy and Security**: Monitoring agent reasoning and data flow must strike a balance between comprehensive visibility and data privacy requirements, as well as security constraints.
## Agentic Observability Implementation How-to Guide
1. **Establish Observability Architecture**
* Design hierarchical data collection across application, session, agent, and action layers
* Implement a unified telemetry pipeline supporting both infrastructure metrics and semantic traces
* Configure data retention policies, balancing granularity with storage costs
2. **Instrument Agent Frameworks**
* Integrate observability SDKs with your agentic framework (LangGraph, Strands, custom)
* Capture agent lifecycle events: thought formation, tool selection, execution, reflection
* Implement correlation IDs for cross-agent tracing
3. **Define Behavioral Baselines**
* Establish expected agent behavior patterns and decision boundaries
* Configure anomaly detection for off-policy actions and coordination failures
* Set performance thresholds for latency, success rates, and resource usage
4. **Implement Hierarchical Monitoring**
* Create dashboards with drill-down capabilities from the system to the span level
* Configure alerts for both technical failures and semantic misalignments
* Enable real-time root cause analysis workflows
5. **Deploy Trust and Safety Controls**
* Integrate Centor Models for output validation and safety scoring
* Implement guardrails for real-time intervention on policy violations
* Configure human-in-the-loop escalation for critical decisions
6. **Establish Continuous Improvement**
* Analyze agent performance trends and optimization opportunities
* Use reflection data to identify systematic improvements
* Iterate on agent coordination patterns based on observed bottlenecks
## Frequently Asked Questions
**Q: How does Agentic Observability differ from LLM Observability?**
Agentic Observability extends beyond monitoring individual LLM calls to capture the complete autonomous decision-making lifecycle, inter-agent coordination, and goal-driven behavior. While LLM Observability focuses on model inputs and outputs, as well as quality metrics, Agentic Observability provides visibility into planning, tool usage, reflection, and multi-agent orchestration.
**Q: What frameworks does Fiddler support for Agentic Observability?**
Fiddler integrates with leading agent frameworks, including LangGraph, Strands, and custom-built agent systems. The platform offers SDKs and APIs for seamless integration, eliminating the need for architectural changes.
**Q: Can Agentic Observability handle real-time monitoring at scale?**
Yes, Fiddler's platform is designed for enterprise-scale deployments, featuring hierarchical data aggregation, efficient trace sampling, and distributed processing to maintain low-latency monitoring even with high-volume, multi-agent systems.
**Q: How do I monitor agent coordination in distributed systems?**
Fiddler provides cross-agent correlation through unified session tracking, distributed tracing with correlation IDs, and visualization of agent dependencies and information flow. The hierarchical view enables tracking coordination from high-level workflows down to individual message passing.
## Related Terms
* [LLM Observability](/glossary/llm-observability)
* [Guardrails](/glossary/guardrails)
* [Centor Models](/glossary/centor-models)
* [Root Cause Analysis](/observability/analytics#root-cause-analysis)
## Related Resources
* [Agentic Observability in the Enterprise](https://www.fiddler.ai/agentic-observability)
* [Building Reliable Agentic Systems](https://www.fiddler.ai/blog/agentic-observability-development)
# Baseline
Source: https://docs.fiddler.ai/glossary/baseline
Reference datasets in Fiddler that serve as comparison points for detecting data drift, evaluating model performance, and identifying when production data deviates from expected patterns.
Baselines are data that serve as a point of reference for calculating [data drift](/glossary/data-drift). When determining if data drift has occurred, Fiddler's monitoring features must compare the distribution of production data (at a point in time) to reference data. Baselines serve as this reference.
Most commonly, training data is used to establish a model's baseline. In this case, the model's training data is uploaded by publishing the dataset. Fiddler will then create a static pre-production baseline of the same name. Multiple baselines can be defined for a model, too. It is not uncommon to have other baselines defined not from training data but from static sets of historical inferences or rolling baselines that look back weeks, months, or quarters at historical inferences.
In other words, a baseline is a representative sample of the data you expect to see in production. It represents the ideal data that your model works best on. For this reason, in most cases, a baseline dataset should be sampled from your model’s training set.
## How Fiddler Uses Baselines
Fiddler requires a baseline to detect data drift in production data. Baselines serve as a point of reference for Fiddler to understand what data distributions the model expects to see for all of its inputs and outputs.
## Why Baselines Are Important
Baselines are crucial for monitoring and maintaining the performance of machine learning models by serving as a reference point to detect data drift, assess model degradation, and ensure consistency over time. They help teams compare real-world data distributions against expected patterns, enabling proactive adjustments to models, improving fairness, and maintaining compliance with regulatory standards.
The roles of baselines in machine learning and model monitoring include:
* **Reference for Data Drift Detection**: Baselines serve as a point of reference to compare production data and identify if the data distribution has changed over time (LINK data drift).
* **Performance Benchmarking**: Baselines provide a comparison standard for model performance, ensuring that more complex models outperform simple baseline models.
* **Model Validation**: They help assess whether a model’s performance is consistent with expectations by comparing it against static or historical data.
* **Monitoring Stability**: Baselines enable ongoing tracking of model behavior in production, helping detect any significant deviations in real-time predictions.
* **Regulatory Compliance**: They ensure that models align with predefined expectations and regulatory standards, especially in critical industries.
* **Error Detection**: Baselines help identify when the model's output deviates beyond acceptable thresholds, guiding corrective actions.
## Types of Baselines
* **Static Pre-production**: A fixed baseline created from training data before a model is deployed, used to compare production data against expected distributions.
* **Static Production**: A fixed baseline derived from a snapshot of historical production data, used to monitor long-term data stability.
* **Rolling Production**: A dynamic baseline that updates periodically using recent production data, allowing for continuous drift monitoring over time.
* **Default Static**: A pre-set static production baseline, created automatically when onboarding a model.
## Challenges
* **Choosing the Right Data**: It can be challenging to select the appropriate reference data for baselines (e.g., training data, historical inferences, etc.). The data chosen must represent expected real-world conditions to detect data drift effectively.
* **Dynamic Data**: In environments with constantly changing data, it’s challenging to define a static baseline that remains relevant over time. Rolling or dynamic baselines may be required but are more complex to manage.
* **Handling Bias**: Baselines defined from biased or incomplete data may lead to misleading conclusions, especially if the reference data doesn’t adequately represent diverse real-world scenarios.
* **Complexity in Multiple Baselines**: Defining multiple baselines for different use cases or time frames (e.g., historical vs. real-time data) can create confusion and make choosing the right one for comparisons difficult.
* **Scalability**: As data volumes and model complexity increase, managing and updating baselines becomes more resource-intensive and burdensome to scale effectively.
* **Adaptability**: Baselines that were accurate at one point might not remain effective as the data distribution or model requirements evolve, making it essential to adapt baselines regularly.
* **Ensuring Consistency**: Maintaining consistency across multiple baseline definitions can be challenging, especially when different teams or systems define their baselines independently.
## Baseline Implementation How-to Guide
1. **Set up baseline performance metrics**
* If monitoring for long-term drift, a static baseline from training data may suffice.
* If monitoring for short-term trends, rolling baselines are preferred.
2. **Upload and Define Your Baseline**
* Ensure training data is appropriately formatted and uploaded to the monitoring system.
* If using Fiddler, define a pre-production baseline using the same model name.
3. **Set Up Monitoring Rules**
* Establish thresholds for acceptable levels of drift.
* Configure alerts for when drift exceeds predefined thresholds.
4. **Compare Production Data to the Baseline**
* Regularly evaluate how production data distributions compare to the baseline.
* Adjust the model if significant drift is detected.
## Frequently Asked Questions
**Q: What happens if my baseline itself becomes outdated?**
You may need to upload an updated baseline periodically or introduce rolling baselines to capture evolving trends.
**Q: What is data drift?**
Data drift is the change in the statistical properties of input data over time, which can impact a model's performance and predictions.
**Q: How do I know if my model is drifting too much?**
Set drift thresholds and monitor performance metrics like accuracy, precision, recall, or fairness scores against your baseline.
## Related Terms
* [Data Drift](/glossary/data-drift)
* [Model Drift](/glossary/model-drift)
* [Model Performance](/glossary/model-performance)
* [ML Observability](/glossary/ml-observability)
## Related Resources
* [Creating Baselines in Fiddler](/developers/python-client-guides/publishing-production-data/creating-a-baseline-dataset)
* [Model Onboarding](/developers/python-client-guides/model-onboarding/create-a-project-and-model)
* [Python Client SDK](/sdk-api/python-client/dataset)
* [Baseline REST API Guide](/sdk-api/rest-api/alert-rules)
# Fiddler Centor Models
Source: https://docs.fiddler.ai/glossary/centor-models
Purpose-built LLMs that evaluate AI outputs in real time, powering both monitoring metrics and real-time guardrails with significantly lower latency than general-purpose models.
Fiddler Centor Models are purpose-built LLMs that evaluate AI outputs in real time, with significantly lower latency than general-purpose models. Unlike general-purpose LLMs optimized for content generation, Centor Models are fine-tuned specifically for evaluation tasks — measuring quality, safety, and alignment across a wide range of LLM output dimensions. This specialization enables Centor Models to deliver consistent, rapid assessments with lower computational overhead.
Centor Models power two primary capabilities within the Fiddler platform: observability features that generate quality metrics for LLM outputs, and real-time protection through Fiddler Guardrails. By using purpose-built models rather than general-purpose LLMs, Fiddler Centor Models deliver evaluations with lower latency, reduced costs, and improved reliability compared to solutions that rely on third-party LLM APIs.
## What Are Centor Models
Centor Models are specialized large language models designed specifically for evaluating and scoring AI system outputs across multiple quality, safety, and reliability dimensions. They assess whether AI-generated content meets established standards for accuracy, safety, ethical alignment, and business requirements.
Centor Models evaluate dimensions such as factual accuracy, harmfulness detection, bias identification, and alignment with intended use cases, making them essential components for AI governance and risk management in production environments. Their evaluation-first design means they provide automated, scalable mechanisms for content evaluation that would otherwise require extensive human review.
### Performance Characteristics
> **⚡ Performance at Scale**
>
> 10-100x faster than general-purpose LLMs
> Real-time evaluation capabilities
> Reduced operational costs
Fiddler Centor Models deliver 10-100x faster processing speeds than general-purpose LLMs while maintaining comparable assessment quality, enabling real-time monitoring and guardrail applications with significantly lower computational overhead.
This performance profile makes it feasible to monitor high-volume LLM applications in production environments. Lower latency translates directly to reduced operational costs and improved system responsiveness — particularly important for real-time guardrail implementations where evaluation speed directly impacts user experience.
## How Fiddler Uses Centor Models
Fiddler Centor Models serve as the evaluation backbone for Fiddler's LLM monitoring and guardrail capabilities, powering two primary functions within the platform:
For observability features, Centor Models process LLM inputs and outputs to generate specialized metrics that evaluate output quality, safety, and alignment. These metrics are then integrated into Fiddler's monitoring dashboards and alerting systems.
For real-time protection through Fiddler Guardrails, Centor Models evaluate potential LLM outputs against customizable safety policies before they reach end users, filtering out problematic content and providing detailed explanations of policy violations.
By maintaining Centor Models as an internal capability, Fiddler ensures consistent, reliable performance with optimized costs compared to solutions that rely on external LLM APIs for similar functionality.
### Evaluation Metrics Coverage
> **📊 Comprehensive Evaluation**
>
> 14+ evaluation dimensions
> Safety, quality, and accuracy metrics
> Customizable thresholds
Fiddler Centor Models assess LLM outputs across multiple critical dimensions to provide comprehensive quality and safety evaluations:
**Safety and Risk Metrics:**
* Toxicity detection and harmful content identification
* Jailbreak attempt recognition and prompt injection detection
* Personally identifiable information (PII) exposure assessment
* Profanity and inappropriate content filtering
**Quality and Accuracy Metrics:**
* Faithfulness (Centor Model) — powered by the Centor Model for Faithfulness for groundedness evaluation against source material (uses `context` and `response` inputs)
* Answer relevance and context relevance scoring
* Coherence and logical consistency assessment
* Conciseness and response appropriateness
**RAG Health Metrics:** In addition to Centor Model-based metrics, Fiddler provides **RAG Health Metrics** — a diagnostic triad of LLM-as-a-Judge evaluators (Answer Relevance 2.0, Context Relevance, RAG Faithfulness) for comprehensive RAG pipeline evaluation. These evaluators use external LLMs rather than Centor Models. See [RAG Health Diagnostics](/concepts/rag-health-diagnostics) for details.
**Specialized Assessments:**
* Sentiment analysis and emotional tone evaluation
* Topic classification and content categorization
* Language detection and multilingual support
* Custom regex matching and banned keyword detection
This comprehensive metric coverage enables organizations to monitor LLM applications against their specific quality standards and risk tolerance levels.
Compare Centor Models cost to third-party LLM evaluators with the [Fiddler Evals TCO calculator](https://www.fiddler.ai/evals-tco-calculator).
## Why Fiddler Centor Models
Fiddler Centor Models address several critical challenges in LLM monitoring and governance. By providing specialized models optimized for evaluation tasks, they enable more efficient, cost-effective, and reliable monitoring than solutions dependent on general-purpose LLMs.
Centor Models are essential for organizations that need to maintain real-time visibility into their LLM applications while ensuring outputs meet safety and quality standards. They enable faster detection of issues, more comprehensive monitoring coverage, and stronger protections against potentially harmful outputs.
As LLM deployments scale across the enterprise, the efficiency of Centor Models becomes increasingly valuable, reducing both operational costs and computational overhead compared to traditional evaluation approaches.
* **Performance Optimization**: Fiddler Centor Models are specifically optimized for evaluation tasks, delivering similar quality assessments as general-purpose LLMs but with significantly lower latency and computational requirements.
* **Cost Efficiency**: By using purpose-built models rather than larger general-purpose LLMs, Centor Models reduce the computational resources required for comprehensive LLM monitoring, translating to lower operational costs.
* **Reliability**: As a dedicated capability maintained by Fiddler, Centor Models provide more consistent availability and performance than solutions dependent on third-party API calls, which may have rate limits or service disruptions.
* **Comprehensive Coverage**: Centor Models support both post-deployment monitoring (observability) and pre-deployment protection (guardrails), providing a unified approach to LLM governance throughout the application lifecycle.
* **Specialized Evaluation**: Unlike general metrics, Centor Models provide specialized assessments tailored specifically to LLM outputs, measuring dimensions like hallucination, alignment, toxicity, and quality that are unique to generative AI systems.
* **Scalability**: As organizations deploy more LLM applications, the efficiency of Centor Models enables monitoring at scale without proportional increases in computational overhead or costs.
* **Privacy and Security**: By processing evaluations within Fiddler's infrastructure rather than sending data to third-party APIs, Centor Models help organizations maintain stronger data privacy and security controls.
### Security and Privacy Benefits
> **🔒 Enterprise Security**
>
> Air-gapped deployment ready
> No external API dependencies
> Full data sovereignty
Fiddler Centor Models process all evaluations within Fiddler's managed infrastructure, ensuring customer data never leaves the secure environment. This approach supports compliance with GDPR, HIPAA, and industry-specific standards while enabling air-gapped deployments for organizations with strict security requirements.
By eliminating external API dependencies, Centor Models reduce security vulnerabilities and remove third-party availability risks, enabling comprehensive LLM monitoring without compromising data governance policies.
## Challenges
Effective LLM monitoring and protection present several technical and operational challenges that Fiddler Centor Models are designed to address.
* **Evaluation Latency**: Traditional approaches to LLM evaluation using other LLMs introduce significant latency, which Centor Models address through specialized, efficient models optimized for evaluation tasks.
* **Computational Cost**: Evaluating LLM outputs at scale using general-purpose models can be prohibitively expensive, a challenge mitigated by Centor Models' more efficient purpose-built design.
* **Coverage vs. Performance**: Organizations often face tradeoffs between comprehensive monitoring coverage and system performance, which Centor Models help balance through optimized evaluation approaches.
* **Evaluation Quality**: Simpler metrics may fail to capture nuanced issues in LLM outputs, while Centor Models provide sophisticated evaluations that maintain high correlation with human judgments.
* **Real-time Protection**: Implementing guardrails without introducing significant latency is challenging, addressed by Centor Models' efficient architecture and optimized processing pipeline.
* **Customization Needs**: Different organizations have varying standards for acceptable content, requiring flexible evaluation systems that can be tailored to specific use cases and policies.
* **Integration Complexity**: Adding monitoring to existing LLM deployments can be complex, a challenge Centor Models address through streamlined integration options and APIs.
## Frequently Asked Questions
**Q: What advantages do Fiddler Centor Models offer over using general-purpose LLMs for evaluation?**
Fiddler Centor Models provide similar quality assessments as general-purpose LLMs but with significantly lower latency, reduced computational requirements, lower costs, and more consistent availability since they don't depend on third-party APIs that may have rate limits or service disruptions.
**Q: Can I use Fiddler Centor Models for both monitoring and real-time protection?**
Yes, Fiddler Centor Models power both observability features (monitoring metrics) and real-time protection through Fiddler Guardrails. You can implement either or both capabilities depending on your specific needs.
**Q: What types of metrics do Centor Models provide?**
Centor Models generate specialized metrics for LLM outputs including safety evaluations (detecting harmful, unethical, or inappropriate content), faithfulness assessments (measuring hallucination and factual accuracy), and other quality dimensions like coherence, relevance, and alignment with intended use.
**Q: How do Centor Models integrate with my existing LLM applications?**
For monitoring, you can publish LLM inputs and outputs to Fiddler's platform either through batch uploads or real-time API calls. For guardrails protection, you integrate the [Guardrails](/glossary/guardrails) API directly into your application flow, sending potential outputs for evaluation before displaying them to users.
**Q: Is Fiddler Guardrails a separate product from the Fiddler platform?**
No. Fiddler Guardrails is a capability of the Fiddler platform, powered by Centor Models, and is accessed through the Guardrails API within your Fiddler environment.
## Related Terms
* [Trust Score](/glossary/trust-score)
* [Guardrails](/glossary/guardrails)
* [Embedding Visualization](/glossary/embedding-visualization)
* [Data Drift](/glossary/data-drift)
## Related Resources
* [LLM Monitoring Overview](/observability/llm)
* [LLM-Based Metrics Guide](/observability/llm/llm-based-metrics)
* [Embedding Visualization with UMAP](/observability/llm/embedding-visualization-with-umap)
* [Selecting Enrichments](/observability/llm/selecting-enrichments)
* [Enrichments Documentation](/observability/llm/enrichments)
* [Guardrails for Proactive Application Protection](/glossary/guardrails)
* [Fiddler Centor Model Metrics](/observability/llm/llm-based-metrics#fiddler-centor-model-metrics)
* [Fiddler Evals TCO Calculator](https://www.fiddler.ai/evals-tco-calculator)
# Custom Metric
Source: https://docs.fiddler.ai/glossary/custom-metrics
User-defined calculations in Fiddler that extend monitoring beyond standard metrics, allowing teams to track business-specific KPIs and specialized measurements for their AI applications.
[Custom Metrics](/glossary/custom-metrics) are user-defined monitoring measures created using [Fiddler Query Language](/observability/platform/fiddler-query-language) (FQL) within the AI/ML/GenAI observability platform. They allow data scientists and ML engineers to extend beyond built-in metrics by defining their own calculations and thresholds for monitoring model performance.
## Additional Context
Custom Metrics transform standard observability into a tailored monitoring solution by enabling teams to implement domain-specific KPIs that complement built-in metrics like data drift and data integrity. This flexibility allows organizations to focus on metrics that directly impact their business objectives rather than solely relying on standard technical indicators.
## Why Custom Metrics Are Important
The roles of Custom Metrics in machine learning and model monitoring include:
* Addressing unique business requirements not covered by standard metrics
* Creating composite metrics that combine multiple signals into actionable insights
* Implementing domain-specific calculations that reflect business KPIs
* Enabling proactive alerting on custom-defined thresholds
## Custom Metric Use Cases
* **Business-focused metrics**: Metrics that directly tie to business outcomes like conversion rates, revenue impact, or customer satisfaction
* **Composite technical metrics**: Combined measures that blend multiple data signals for more holistic monitoring
* **Data quality extensions**: Custom definitions of what constitutes data quality in specific domains
## Custom Metrics How-to Guide
1. **Identify the metric need**
* Determine what performance aspects aren't covered by built-in metrics
2. **Design the FQL formula**
* Write the formula using Fiddler Query Language syntax using the UI or API
3. **Test on historical data**
* Validate that your metric catches issues using past data
4. **Iterate based on results**
* Refine the metric definition as you learn from real-world monitoring
## Frequently Asked Questions
**Q: How do Custom Metrics differ from built-in metrics?**
Custom Metrics allow you to define domain-specific calculations using FQL that may not be available through pre-built metrics, giving you flexibility to monitor aspects of your AI systems most relevant to your business.
**Q: Can Custom Metrics be used for alerting?**
Yes, Custom Metrics integrates seamlessly with the platform's alerting system, allowing you to set thresholds and receive notifications when your user-defined metrics exceed acceptable ranges.
**Q: What technical knowledge is required to create Custom Metrics?**
Basic understanding of SQL-like query languages and knowledge of your data schema are sufficient for creating most Custom Metrics with FQL.
## Related Resources
* [Custom Metrics for ML Models](/observability/platform/custom-metrics)
* [Custom Metrics for Agentic Applications](/observability/agentic/custom-metrics)
* [Fiddler Query Language](/observability/platform/fiddler-query-language)
# Data Drift
Source: https://docs.fiddler.ai/glossary/data-drift
The statistical change in data distributions over time that can impact model performance. Fiddler detects drift by comparing production data against baselines to identify degradation causes.
Data drift refers to the change in the statistical properties of input data over time, which can impact a model's performance and predictions. It occurs when the distribution of data in production differs significantly from the distribution of the baseline data (often the training dataset) that the model was trained on.
Model performance can be poor if models trained on a specific dataset encounter different data in production. Data drift serves as a great proxy metric for performance decline, especially in cases where there is a delay in getting labels for production events (e.g., in a credit lending use case, an actual default may happen after months or years).
## How Fiddler Uses Data Drift
Fiddler's monitoring platform uses data drift metrics to help users identify what data is drifting, when it's drifting, and how it's drifting. This is a crucial first step in identifying potential model performance issues. Fiddler calculates drift between the distribution of a field in the baseline dataset and that same distribution for the time period of interest using metrics like [Jensen-Shannon Divergence](https://www.fiddler.ai/ml-guides/how-to-measure-ml-model-drift) (JSD) and [Population Stability Index](https://www.fiddler.ai/blog/measuring-data-drift-population-stability-index) (PSI).
## Why Data Drift Is Important
Monitoring data drift helps you stay informed about distributional shifts in the data for features of interest, which could have business implications even if there is no immediate decline in model performance. High drift can occur as a result of data integrity issues (bugs in the data pipeline), or as a result of an actual change in the distribution of data due to external factors (e.g., a dip in income due to economic changes).
* **Early Warning System**: Data drift serves as an early warning mechanism for potential model performance degradation before it significantly impacts business outcomes.
* **Delayed Ground Truth**: In many real-world scenarios, ground truth labels are delayed or expensive to obtain, making drift detection essential for timely model management.
* **Data Pipeline Validation**: Detecting drift can help identify issues in data pipelines, such as bugs or data quality problems that might otherwise go unnoticed.
* **Business Insight**: Changes in data distributions can provide valuable business insights about changing customer behaviors or market conditions, even when model performance remains stable.
* **Efficiency in Retraining**: Monitoring data drift helps teams make informed decisions about when to retrain models, optimizing the use of resources.
## Types of Data Drift
* **Feature Drift**: Changes in the statistical properties of input features that may affect model performance, such as shifts in customer demographics or behavior patterns.
* **Prediction Drift**: Changes in the distribution of model outputs or predictions over time, which may indicate underlying issues even when feature distributions appear stable.
* **Concept Drift**: Changes in the relationship between input features and target variables, where the statistical properties of the target variable change in relation to the features.
* **Virtual Drift**: Changes in data that don't affect the target concept but may still impact model performance, such as the introduction of new feature values.
## Challenges
Managing data drift effectively presents several challenges for data science and MLOps teams.
* **Determining Threshold Levels**: Setting appropriate thresholds for what constitutes significant drift requires careful consideration of business context and model sensitivity.
* **Root Cause Analysis**: Identifying which specific features are contributing most to observed drift and understanding their business implications can be complex.
* **Distinguishing Natural Variation**: Differentiating between normal seasonal or cyclic patterns and problematic drift requires domain expertise and historical context.
* **Handling Multivariate Relationships**: Drift may occur in complex relationships between variables rather than in individual features, making detection more challenging.
* **Balancing Sensitivity**: Drift detection systems must be sensitive enough to catch important changes while avoiding false alarms from minor fluctuations.
* **Delayed Response**: Determining how quickly to respond to detected drift requires balancing the costs of model retraining against the risks of performance degradation.
## Data Drift Monitoring Implementation Guide
1. **Establish a Baseline**
* Define a representative baseline dataset, typically the training data used to build the model.
* Analyze and document the statistical properties of this baseline for future comparison.
2. **Select Appropriate Drift Metrics**
* Choose appropriate statistical metrics such as JSD or PSI to quantify distribution differences.
* Consider the data types and distributions when selecting metrics (e.g., categorical vs. continuous variables).
3. **Set Drift Thresholds**
* Establish thresholds for acceptable levels of drift based on business impact and model sensitivity.
* Consider variable importance when setting thresholds, as drift in critical features may be more impactful.
4. **Implement Monitoring Systems**
* Set up automated monitoring to compare production data distributions against the baseline.
* Configure alerts for when drift exceeds predefined thresholds.
5. **Analyze and Respond to Drift**
* Investigate the root causes of detected drift using drill-down analysis.
* Determine appropriate responses, which may include model retraining, feature engineering adjustments, or addressing data pipeline issues.
## Frequently Asked Questions
**Q: How is data drift different from concept drift?**
Data drift refers to changes in the statistical properties of input data, while concept drift refers to changes in the relationship between inputs and the target variable. Data drift focuses on feature distributions, while concept drift involves the underlying prediction problem changing over time.
**Q: How often should I check for data drift?**
The frequency depends on your use case and how quickly your data environment changes. Critical applications may require daily or even hourly monitoring, while more stable models might be monitored weekly or monthly.
**Q: What actions should I take when data drift is detected?**
When significant drift is detected, first investigate the root cause through drill-down analysis. Depending on the cause, actions may include retraining the model, engineering new features, fixing data pipeline issues, or adjusting business processes to account for the new data reality.
**Q: Can data drift occur without affecting model performance?**
Yes, data drift can occur without immediately affecting model performance if the changes happen in areas that don't significantly impact the model's predictions or if the model is robust to the specific type of drift occurring.
## Related Terms
* [ML Observability](/glossary/ml-observability)
* [Baseline](/glossary/baseline)
* [Model Performance](/glossary/model-performance)
## Related Resources
* [Data Drift Monitoring Platform](/observability/platform/data-drift-platform)
* [How to Detect Model Drift in ML Monitoring](https://www.fiddler.ai/blog/how-to-detect-data-drift)
* [Alerts](/observability/platform/alerts-platform)
* [Monitoring Charts](/observability/platform/monitoring-charts-platform)
# Embedding Visualization
Source: https://docs.fiddler.ai/glossary/embedding-visualization
Interactive visualizations in Fiddler AI that transform complex embedding vectors into 3D displays, revealing semantic patterns, clusters, and outliers in LLM data.
[Embedding Visualizations](/observability/llm/embedding-visualization-with-umap) in Fiddler AI are interactive graphical representations that display high-dimensional embedding vectors in a more accessible two or three-dimensional space. These visualizations use dimensionality reduction techniques, primarily [UMAP](https://umap-learn.readthedocs.io/en/latest/) (Uniform Manifold Approximation and Projection), to transform complex vector data into visual patterns that humans can interpret and analyze.
When working with Large Language Models (LLMs) and other AI systems, embeddings capture semantic relationships in high-dimensional space (typically hundreds or thousands of dimensions). Embedding Visualizations make these abstract mathematical relationships visible, allowing users to identify clusters, outliers, and patterns that might otherwise remain hidden in the raw numerical data.
In the Fiddler platform, Embedding Visualizations appear as interactive charts that plot embedding vectors as points in 2D space, with proximity between points indicating semantic similarity. These visualizations provide a powerful tool for understanding model behavior, monitoring for drift or anomalies, and gaining insights into how AI systems represent and process information.
## How Fiddler Uses Embedding Visualizations
Fiddler integrates Embedding Visualizations as a core component of its LLM monitoring and observability platform. When monitoring LLM applications, Fiddler processes embedding data (either uploaded by users or generated through Fiddler's enrichment capabilities) and creates interactive UMAP visualizations that help users understand the semantic landscape of their model inputs and outputs.
These visualizations are displayed in Fiddler Charts, which provide additional interactive capabilities such as filtering, color-coding, and time-based analysis. Users can explore embedding spaces to identify clusters of similar content, detect outlier patterns, and track how embedding distributions change over time.
Embedding Visualizations complement Fiddler's other monitoring metrics by providing a spatial understanding of semantic relationships that numerical metrics alone cannot capture. They serve as an essential tool for both regular monitoring and deep-dive investigations when issues are detected.
## Why Embedding Visualizations Are Important
Embedding Visualizations address a fundamental challenge in LLM and AI monitoring: how to make sense of high-dimensional data that cannot be directly observed or interpreted by humans. By transforming complex embedding spaces into visual representations, these visualizations enable insights that would be impossible to derive from raw vector data or simple statistical metrics.
For organizations deploying LLM applications, Embedding Visualizations provide a crucial window into how their models are processing and representing information, helping teams detect subtle patterns, identify unexpected behaviors, and communicate findings to both technical and non-technical stakeholders.
The visual nature of these representations makes complex AI behavior more accessible and interpretable, supporting more effective monitoring, debugging, and governance of AI systems.
* **Pattern Detection**: Visualizations reveal clusters, outliers, and other patterns in embedding space that might indicate important semantic groupings or anomalous data points that require investigation.
* **Drift Monitoring**: By comparing embedding distributions over time, visualizations can highlight semantic drift that might not be captured by traditional statistical drift metrics, showing how the meaning and context of data is evolving.
* **Model Understanding**: Visualizations provide insights into how models represent information, helping users understand the semantic relationships and structures learned by the model.
* **Anomaly Investigation**: When unusual model behaviors occur, embedding visualizations can help identify whether these anomalies cluster together semantically, suggesting common underlying causes.
* **Communication Tool**: Visual representations of complex data make technical concepts more accessible to diverse stakeholders, facilitating communication between data scientists, engineers, compliance teams, and business leaders.
* **Quality Assessment**: Visualizations can reveal whether similar inputs receive similar outputs or whether semantically related concepts are appropriately clustered, indicating model consistency and quality.
* **Dataset Exploration**: Visualizations enable interactive exploration of large datasets, helping users understand the distribution and characteristics of their data in ways that tabular views cannot provide.
## Challenges
While Embedding Visualizations provide powerful insights, they also come with several technical and interpretive challenges that users should be aware of when incorporating them into monitoring workflows.
* **Dimensionality Reduction Trade-offs**: Techniques like UMAP inherently lose some information when reducing high-dimensional spaces to 2D or 3D, meaning that some relationships or patterns might be obscured or distorted in the visualization.
* **Parameter Sensitivity**: UMAP and similar algorithms require careful parameter tuning, as different settings can produce significantly different visualizations from the same underlying data, potentially leading to different interpretations.
* **Computational Overhead**: Generating high-quality embedding visualizations, especially for large datasets, can be computationally intensive and may require significant processing resources.
* **Interpretation Complexity**: Without proper context and understanding of the underlying algorithms, users may misinterpret patterns in embedding visualizations or draw incorrect conclusions about what they represent.
* **Temporal Consistency**: Maintaining consistent visualizations over time can be challenging, as new data points may shift the overall projection, making it difficult to compare visualizations from different time periods.
* **Scalability Limitations**: Visualizing very large numbers of embeddings can lead to overcrowded displays or performance issues, requiring careful sampling or filtering strategies.
* **Contextual Information Loss**: While embeddings capture semantic relationships, the original context that produced those embeddings may be lost in visualization, requiring additional metadata to fully interpret observed patterns.
## Embedding Visualization Implementation Guide
1. **Prepare Your Embedding Data**
* Ensure your model is configured to generate or capture embeddings for the selected fields.
* If using custom embeddings, verify they are properly formatted and included in your data publishing pipeline.
2. **Create a Visualization Chart**
* In the Fiddler platform, navigate to the Charts section and create a new UMAP Visualization chart.
* Select the appropriate model and field containing the embedding vectors you wish to visualize.
3. **Configure UMAP Parameters**
* Adjust parameters like number of neighbors and minimum distance to optimize the visualization for your specific data characteristics.
* Consider experimenting with different parameters to find the most informative representation.
4. **Add Contextual Information**
* Configure color-coding or filters based on relevant metadata to add context to the visualization.
* Consider adding time-based filters to observe how embeddings change over specific periods.
5. **Analyze Patterns**
* Look for clusters that might indicate semantic groupings in your data.
* Identify outliers or unexpected patterns that might require further investigation.
6. **Integrate with Monitoring**
* Add embedding visualizations to monitoring dashboards alongside other metrics.
* Set up regular reviews to detect changes in embedding patterns over time.
## Frequently Asked Questions
**Q: What is UMAP and why is it used for embedding visualization?**
UMAP (Uniform Manifold Approximation and Projection) is a dimensionality reduction technique that preserves both local and global structure when projecting high-dimensional data to lower dimensions. It's particularly well-suited for visualizing embeddings because it maintains meaningful relationships between data points, allowing clusters and patterns in the high-dimensional space to be visible in the 2D projection. Fiddler uses UMAP because it offers a good balance of performance and accuracy in representing complex embedding spaces.
**Q: How should I interpret clusters in an embedding visualization?**
Clusters in embedding visualizations typically represent groups of semantically similar items. Points that appear close together in the visualization share similar meanings or characteristics in the high-dimensional embedding space. When analyzing clusters, examine a sample of items within each cluster to understand the common themes or attributes they share. This can reveal how your model is grouping concepts and whether these groupings align with your expectations.
**Q: Can embedding visualizations help detect problems in my LLM?**
Yes, embedding visualizations can reveal various issues in LLM systems. Unexpected outliers might indicate anomalous inputs or outputs. Shifts in cluster patterns over time could signal semantic drift affecting your model. Overlapping clusters that should be distinct might suggest the model is conflating concepts it should differentiate. By regularly monitoring these visualizations alongside other metrics, you can detect subtle changes in model behavior that might not be apparent through other monitoring approaches.
**Q: How frequently should I update my embedding visualizations?**
The optimal frequency depends on your specific use case and data volume. For high-traffic LLM applications, daily or weekly visualization updates may be appropriate to catch shifts in patterns early. For more stable applications or those with lower traffic, monthly updates might be sufficient. Consider automating the generation of these visualizations as part of your regular monitoring cycle, aligning their frequency with your organization's model governance and review procedures.
**Q: Can I export or share embedding visualizations from Fiddler?**
Yes, Fiddler allows you to export embedding visualizations for sharing with team members or inclusion in reports. These exports capture the current state of the visualization including any applied filters or color-coding. This capability is particularly useful for communicating findings to stakeholders or documenting the state of your model at specific points in time for governance and compliance purposes.
## Related Terms
[LLM and GenAI Observability](/glossary/llm-observability)
## Related Resources
* [Embedding Visualization with UMAP](/observability/llm/embedding-visualization-with-umap)
* [Monitoring Charts Platform](/observability/platform/monitoring-charts-platform)
* [Vector Monitoring Platform](/observability/platform/vector-monitoring-platform)
* [Selecting LLM Enrichments](/observability/llm/selecting-enrichments)
# Enrichment
Source: https://docs.fiddler.ai/glossary/enrichment
Comprehensive overview of enrichments in AI monitoring and evaluation. Learn how Fiddler's enrichment framework transforms raw LLM data into actionable insights through specialized metrics and custom
[Enrichments](/observability/llm/enrichments), also known as Evaluations, are specialized computational processes that transform raw AI model inputs and outputs into structured metrics, insights, and derived features.
**Terminology note:** In Agentic Monitoring and Experiments (v26.3+), enrichments are referred to as **evaluators**. The underlying concept is the same — automated metrics applied to AI model data — but the term "evaluators" better reflects the active assessment role these components play in evaluation workflows. The `Enrichment` API class name remains unchanged.
In the context of LLM monitoring and evaluation, enrichments serve as the bridge between unstructured model data and actionable intelligence, enabling organizations to measure, monitor, and improve their AI applications through quantitative analysis.
When LLM applications publish inference data to monitoring platforms like Fiddler, enrichments automatically analyze this data to generate various metrics—from safety scores and hallucination detection to custom business logic evaluation. These computed enrichments become additional features in the monitoring dataset, providing dimensions that can be tracked over time, used in alerting rules, and analyzed for patterns or anomalies.
Enrichments represent the operationalization of AI evaluation methodologies, transforming subjective quality assessments into measurable, scalable monitoring capabilities that enable reliable production deployment of generative AI systems.
## Core Terminology
### Custom Features
Derived data columns are created through enrichment processes that augment the original inference data. Custom features represent computed insights that don't exist in the raw model data but provide valuable monitoring and analysis dimensions.
### Enrichment Pipeline
The automated processing infrastructure that applies enrichment operations to incoming inference data. The pipeline manages the execution, dependency resolution, and result integration of multiple enrichments across different data streams.
### Trust Scores
[Trust Scores](/glossary/trust-score) are metrics that are specialized enrichment outputs that evaluate the trustworthiness, safety, and quality of AI model outputs. Trust metrics include dimensions like toxicity, bias, faithfulness, and safety that are critical for responsible AI deployment.
### Centor Models
[Fiddler Centor Models](/glossary/centor-models) are purpose-built, optimized models developed by Fiddler specifically for efficient enrichment computation. These models provide comparable evaluation quality to general-purpose LLMs but with significantly lower latency and computational requirements.
### Evaluation Framework
The underlying methodology and infrastructure that defines how enrichments assess model outputs. Different evaluation frameworks may use rule-based systems, ML models, or LLM-as-a-Judge approaches depending on the specific enrichment type.
## How Fiddler Uses Enrichments
Fiddler's enrichment system operates as a comprehensive evaluation and monitoring layer that processes all inference data flowing through the platform. When LLM applications publish events containing prompts, responses, and contextual information, Fiddler's enrichment pipeline automatically applies configured enrichments to generate derived metrics.
**Integration Process**: During model onboarding, users specify which enrichments should be enabled by including Enrichment objects in their ModelSpec configuration. This declarative approach allows teams to select the specific evaluation dimensions most relevant to their use case while avoiding unnecessary computational overhead.
**Real-time Processing**: As inference events are published to Fiddler, the enrichment pipeline processes them through the configured enrichments, generating additional data columns that appear alongside the original inference data in monitoring dashboards and analytics interfaces.
**Scalable Architecture**: Fiddler's enrichment infrastructure is designed for enterprise-scale deployment, supporting concurrent processing of multiple enrichment types while maintaining low latency. The system leverages purpose-built Centor Models and optimized processing pipelines to deliver consistent performance even with high-volume data streams.
**Monitoring Integration**: Enrichment outputs seamlessly integrate with Fiddler's monitoring capabilities, enabling users to create alerts, track trends, and perform root cause analysis based on derived metrics rather than just raw model performance indicators.
## Why Enrichments Are Important
Enrichments address a fundamental gap in AI monitoring: the ability to automatically assess qualitative aspects of model behavior at scale. While traditional monitoring focuses on quantitative metrics like latency and throughput, enrichments enable organizations to systematically evaluate the quality, safety, and appropriateness of AI-generated content.
**Scalable Quality Assessment**: Manual evaluation of AI outputs doesn't scale to production volumes. Enrichments automate quality assessment, enabling continuous monitoring of millions of model interactions while maintaining consistent evaluation criteria.
**Proactive Risk Management**: By continuously evaluating outputs for safety, bias, toxicity, and other risk factors, enrichments enable proactive identification and mitigation of potential issues before they impact users or business outcomes.
**Regulatory Compliance**: Many industries require ongoing assessment of AI system behavior for compliance purposes. Enrichments provide auditable, quantitative evidence of responsible AI practices and systematic quality control.
**Performance Optimization**: Enrichment data reveals patterns in model performance that guide optimization efforts. Understanding when and why models produce lower-quality outputs enables targeted improvements to prompts, training data, or model architecture.
**Business Intelligence**: Custom enrichments can evaluate business-specific criteria, providing insights into how well AI systems achieve organizational objectives beyond generic quality metrics.
## Types of Enrichments
### Safety and Trust Enrichments
**Safety**: Comprehensive safety evaluation powered by the Centor Model for Safety to detect harmful, toxic, or inappropriate content across multiple safety dimensions including violence, hate speech, and explicit material.
**Bias Detection**: Evaluation of potential biases in model outputs, including gender, racial, political, or other forms of bias that could impact fairness and inclusivity.
### Quality and Accuracy Enrichments
**Faithfulness (Centor Model)**: Automated detection of hallucinations and factual inaccuracies powered by the Centor Model for Faithfulness. Compares model outputs against provided context, returning a probability score (0.0–1.0). Available as a [Guardrail](/glossary/guardrails) and as an enrichment in LLM Observability. Uses `context` and `response` as inputs. See also: RAG Faithfulness below.
**RAG Faithfulness**: LLM-as-a-Judge evaluator that assesses whether a response is grounded in the retrieved documents. Returns a binary label (Yes/No) with detailed reasoning. Uses `user_query`, `rag_response`, and `retrieved_documents` as inputs. Available in Agentic Monitoring and Experiments. See [RAG Health Diagnostics](/concepts/rag-health-diagnostics) for how this evaluator fits into the RAG diagnostic triad.
**Faithfulness (Centor Model) vs RAG Faithfulness:** These are separate evaluators with different architectures. Faithfulness (Centor Model) is powered by the Centor Model for Faithfulness (purpose-built, probability score 0.0–1.0); RAG Faithfulness is an LLM-as-a-Judge evaluator (binary Yes/No with reasoning). See the [RAG Health Diagnostics](/concepts/rag-health-diagnostics) guide for a detailed comparison.
**Answer Relevance**: Measurement of how well model responses address the specific query or prompt. In v26.3, Answer Relevance 2.0 provides ordinal scoring (High/Medium/Low = 1.0/0.5/0.0) with detailed reasoning, replacing the previous binary scoring. Available in Agentic Monitoring, Experiments, and LLM Observability.
**Context Relevance**: Measures whether retrieved documents are relevant to the user's query, isolating retrieval quality from generation quality. Uses ordinal scoring (High/Medium/Low = 1.0/0.5/0.0) with reasoning. Available in Agentic Monitoring and Experiments only.
**Coherence**: Assessment of logical flow, consistency, and overall readability of generated content to identify confusing or disjointed responses.
### Custom and Business Logic Enrichments
**LLM-as-a-Judge**: Configurable evaluation system using Prompt Specs or custom prompts to implement specialized business logic and quality criteria.
**Sentiment Analysis**: Emotional tone detection and sentiment classification to understand the affective characteristics of model outputs.
### Technical and Performance Enrichments
**Embedding Generation**: Automatic creation of vector embeddings from text content to enable similarity analysis, clustering, and semantic drift detection.
**Token Count**: Tracking of input and output token usage for cost analysis, performance optimization, and usage pattern understanding.
**Response Time**: Measurement of model inference latency to identify performance bottlenecks and optimize system responsiveness.
### Data Quality Enrichments
**PII Detection**: Identification of personally identifiable information in model inputs and outputs to ensure privacy compliance and data protection.
**Data Validation**: Verification of data format, completeness, and consistency to ensure reliable model operation and meaningful analysis.
**Language Detection**: Automatic identification of content language to enable appropriate processing and evaluation in multilingual environments.
## Challenges
Implementing effective enrichment systems involves several technical and operational considerations that organizations must address to realize the full value of automated AI evaluation.
**Computational Efficiency**: Running multiple enrichments on high-volume data streams requires careful optimization to avoid introducing significant latency or computational overhead that could impact application performance.
**Evaluation Accuracy**: Ensuring that automated enrichments provide reliable assessments that align with human judgment requires ongoing validation and calibration, particularly as language usage and societal standards evolve.
**Cost Management**: Large-scale enrichment processing can incur substantial computational costs, requiring organizations to balance evaluation comprehensiveness with budget constraints and performance requirements.
**Configuration Complexity**: With dozens of available enrichments, selecting the optimal combination for specific use cases requires domain expertise and understanding of the trade-offs between different evaluation approaches.
**Result Interpretation**: Translating enrichment scores into actionable insights requires clear guidelines and thresholds that may vary by organization, industry, or application context.
**Data Privacy**: Some enrichments require processing of sensitive content, necessitating careful consideration of data privacy requirements and potential regulatory constraints.
## Enrichments Implementation Guide
1. **Assess Evaluation Requirements**
* Identify the specific quality, safety, and business criteria relevant to your AI application
* Prioritize evaluation dimensions based on risk assessment and regulatory requirements
* Consider the trade-offs between evaluation comprehensiveness and computational cost
2. **Design Enrichment Strategy**
* Select appropriate enrichments based on your use case and data characteristics
* Plan for both generic quality metrics and domain-specific custom enrichments
* Consider dependency relationships between different enrichment types
3. **Configure Model Specification**
* Include selected enrichments in your ModelSpec configuration during onboarding
* Define appropriate column mappings and enrichment-specific parameters
* Test enrichment configuration with representative sample data
4. **Establish Monitoring Framework**
* Create dashboards that effectively visualize enrichment results alongside operational metrics
* Configure alert rules based on enrichment score thresholds and trend analysis
* Implement escalation procedures for different types of quality or safety issues
5. **Validate and Calibrate**
* Compare enrichment results with human evaluation on representative samples
* Adjust thresholds and interpretation guidelines based on validation results
* Establish periodic review processes to ensure continued accuracy and relevance
6. **Optimize and Scale**
* Monitor enrichment processing performance and optimize for efficiency
* Implement data retention and archival strategies for enrichment results
* Plan for scaling enrichment infrastructure with application growth
## Frequently Asked Questions
**Q: How do enrichments differ from traditional ML metrics?**
Traditional ML metrics like accuracy and precision measure model performance against known ground truth labels. Enrichments evaluate qualitative aspects of model outputs—such as safety, coherence, and relevance—that don't have simple right/wrong answers but require nuanced assessment against multiple criteria.
**Q: Can I create custom enrichments for my specific business needs?**
Yes, Fiddler supports custom enrichments through multiple approaches: LLM-as-a-Judge with Prompt Specs for structured evaluation, and bring-your-own-prompt capabilities for maximum flexibility. These frameworks enable implementation of domain-specific evaluation logic.
**Q: How do enrichments impact system performance and latency?**
Fiddler's enrichments are designed for minimal performance impact through purpose-built Centor Models and optimized processing pipelines. Most enrichments add only modest latency (typically under 100ms) and can be processed asynchronously to avoid blocking primary application flows.
**Q: Are enrichment results consistent and reliable enough for compliance use?**
Fiddler's enrichments undergo extensive validation against human evaluation to ensure reliability. While no automated system perfectly matches human judgment in all cases, enrichments provide consistent, auditable results that meet the standards required for most compliance and governance use cases.
**Q: How do I choose which enrichments to enable for my application?**
Start with core safety and quality enrichments (Safety, Faithfulness (Centor Model)) that apply broadly, then add domain-specific enrichments based on your use case. Consider your risk profile, regulatory requirements, and the specific ways your AI application could fail or cause harm.
**Q: Can enrichments be applied retroactively to historical data?**
Yes, enrichments can be applied to previously collected inference data, enabling retrospective analysis and baseline establishment. This capability is particularly valuable when implementing new evaluation criteria or investigating historical issues.
## Related Terms
* [Trust Score](/glossary/trust-score) - The quantitative outputs generated by enrichments
* [Fiddler Centor Models](/glossary/centor-models) - The evaluation models powering enrichment computation
* [LLM Observability](/glossary/llm-observability) - The broader monitoring practice enabled by enrichments
* [Fiddler Guardrails](/glossary/guardrails) - Real-time protection using enrichment evaluation
* [Custom Metrics](/glossary/custom-metrics) - User-defined metrics complementing enrichments
## Related Resources
* [LLM Observability Metrics Reference](/reference/llm-observability-metrics) — Complete list of all LLM enrichments with output columns and sub-metrics
* [LLM Application Monitoring & Protection](/observability/llm)
* [Selecting Enrichments Guide](/observability/llm/selecting-enrichments)
* [Enrichments Documentation](/observability/llm/enrichments)
* [Enrichment API Reference](/sdk-api/python-client/enrichment)
* [RAG Health Diagnostics](/concepts/rag-health-diagnostics) — Understanding the RAG diagnostic triad (Answer Relevance 2.0, Context Relevance, RAG Faithfulness)
# Experiments
Source: https://docs.fiddler.ai/glossary/experiments
Systematic assessment of LLM application quality through structured testing with datasets, evaluators, and experiments that enable data-driven decision-making for prompt optimization, model selection,
Experiments are structured, repeatable test runs for assessing the quality, safety, and performance of Large Language Model (LLM) applications. Each experiment runs your application against a curated test dataset, using evaluators to score outputs across specific quality dimensions. Unlike ad-hoc manual review, experiments provide quantitative, reproducible assessment that scales to thousands of test cases.
Experiments serve as the testing framework for generative AI systems, analogous to unit testing and integration testing in traditional software development. They enable teams to make data-driven decisions about prompt engineering, model selection, hyperparameter tuning, and safety validation by providing objective evidence of how changes impact application behavior across representative scenarios.
In the context of LLM development, experiments complement observability by providing proactive quality gates before deployment, while monitoring provides reactive insights after production release. Together, they form a comprehensive quality assurance strategy for reliable AI applications.
## Core Terminology
### Evaluator
A function or system that assesses LLM outputs against specific quality criteria and produces a score. Evaluators can be rule-based (regex matching, length checks), model-based (embedding similarity, LLM-as-a-judge), or custom business logic. Each evaluator focuses on a specific dimension such as relevance, coherence, safety, or faithfulness.
### Score
The quantitative or qualitative output produced by an evaluator represents how well an LLM output meets the evaluation criteria. Scores can be binary (pass/fail), continuous (0.0 to 1.0), categorical (positive/neutral/negative), or multi-dimensional (sentiment with confidence).
### Test Case
A single evaluation data point containing inputs (such as a user query), optional extras (such as context for RAG systems), expected outputs (ground truth answers), and metadata. Test cases represent specific scenarios your application should handle correctly.
### Dataset
A structured collection of test cases used for evaluation, typically representing diverse real-world scenarios, edge cases, and everyday user interactions. Datasets enable consistent, repeatable testing across different model versions, prompts, or configurations.
### Experiment
A single execution of an experiment workflow where an LLM application runs against all test cases in a dataset, with evaluators scoring each output. Experiments track inputs, outputs, scores, metadata, and timing information, enabling comparison between different application versions.
### Baseline
A reference experiment representing current or expected performance levels. Baselines provide the comparison point for evaluating whether changes improve or degrade application quality across metrics.
## How Fiddler Provides Experiments
Fiddler Experiments delivers a comprehensive evaluation platform that integrates systematic testing into the LLM development lifecycle. The platform combines dataset management, experiment orchestration, built-in evaluators, and custom evaluation frameworks into a unified system accessible through both SDK and UI interfaces.
**Dataset Management**: Teams create and version experiment datasets containing representative test cases with inputs, expected outputs, and contextual information. Datasets can be imported from CSV, JSONL, or constructed programmatically, with support for complex column mapping and metadata enrichment.
**Built-in Evaluators**: Fiddler provides production-ready evaluators for common quality dimensions, including answer relevance, faithfulness (hallucination detection), coherence, conciseness, safety, toxicity, and sentiment. These evaluators leverage both Fiddler Centor Models and LLM-as-a-judge approaches for reliable assessment.
**Custom Evaluation Framework**: Beyond built-in evaluators, developers can create custom evaluators using Python functions, LLM-based classification, or bring-your-own-prompt approaches. This flexibility enables domain-specific evaluation logic and business-specific quality criteria.
**Experiment Tracking**: Every experiment run is tracked with a complete lineage of inputs, outputs, scores, and metadata. The platform provides side-by-side comparison of experiments, aggregate statistics by evaluator, and drill-down capabilities to individual test case results.
**Integration Points**: The Fiddler Evals SDK integrates with development workflows through Python APIs, enabling CI/CD integration, automated regression testing, and programmatic analysis. The UI provides visual exploration of results, experiment comparison dashboards, and collaborative review capabilities.
## Why Experiments Are Important
Experiments transform LLM application development from subjective guesswork into objective, data-driven engineering. Without systematic evaluation, teams rely on anecdotal evidence and manual spot-checking, making it impossible to confidently assess whether changes improve or degrade quality across diverse scenarios.
**Quality Assurance at Scale**: Manual review of LLM outputs doesn't scale to the thousands of scenarios production applications must handle. Evaluations automate quality assessment, enabling comprehensive testing that catches edge cases and regressions human reviewers might miss.
**Objective Decision-Making**: When comparing prompts, models, or configurations, evaluations provide quantitative evidence of which option performs better across metrics that matter for your use case. This eliminates arguments based on cherry-picked examples or personal preference.
**Regression Prevention**: As LLM applications evolve through prompt refinements, model updates, and feature additions, evaluations serve as quality gates that catch unintended degradation before it reaches production. Continuous evaluation prevents the "whack-a-mole" problem where fixing one issue breaks another.
**Safety Validation**: For applications with safety, compliance, or ethical requirements, evaluations provide systematic verification that outputs meet organizational standards. Safety evaluators can detect toxicity, bias, harmful content, or policy violations across comprehensive test suites.
**Cost Optimization**: Evaluations enable data-driven decisions about model selection and configuration. By comparing quality metrics across different models or prompts, teams can identify cost-effective configurations that meet quality requirements without over-provisioning expensive models.
**Continuous Improvement**: Experiment results guide optimization efforts by revealing where applications struggle. Identifying low-performing test case categories, correlated failures, or specific evaluator weaknesses focuses development effort on meaningful improvements.
## Types of Experiments
### Development-Time Evaluation
**Prompt A/B Testing**: Compare different prompt formulations to identify which produces better quality outputs. Run identical datasets through competing prompts and analyze score distributions across evaluators to make evidence-based prompt selection decisions.
**Model Comparison**: Evaluate multiple LLM models (GPT-4, Claude, Llama) on the same tasks to balance quality, cost, and latency trade-offs. Use evaluation metrics to justify model selection for specific use cases.
**Hyperparameter Tuning**: Systematically test temperature, top-p, and other configuration parameters to optimize the balance between creativity and consistency for your application.
### Pre-Deployment Testing
**Regression Testing**: Run comprehensive experiment suites before each deployment to ensure new changes don't degrade existing capabilities. Establish quality thresholds that must be met for deployment approval.
**Quality Gates**: Implement automated checks that block deployment if experiment scores fall below acceptable baselines, preventing quality regressions from reaching production.
**Version Validation**: When upgrading model versions or dependencies, run experiments to verify that improvements claimed by providers actually manifest in your specific use case.
### Safety and Trust Evaluation
**Toxicity Detection**: Evaluate outputs for offensive, harmful, or inappropriate content across diverse test scenarios. Ensure content moderation policies are enforced systematically.
**Bias Assessment**: Test for gender, racial, political, or other biases in model responses using carefully constructed datasets that expose potential fairness issues.
**Adversarial Testing**: Run experiments with jailbreak attempts, prompt injections, and other adversarial inputs to verify that safety guardrails function correctly.
### RAG System Evaluation
Fiddler provides a purpose-built diagnostic framework — **RAG Health Metrics** — for evaluating RAG applications. The framework consists of three evaluators that together pinpoint where RAG pipelines fail:
**Answer Relevance 2.0** (Enhanced): Measures how well the response addresses the user's query using ordinal scoring (High/Medium/Low = 1.0/0.5/0.0) with detailed reasoning. Available in Agentic Monitoring, Experiments, and LLM Observability.
**Context Relevance** (New): Measures whether the retrieved documents are relevant to the user's query, isolating retrieval quality from generation quality. Uses ordinal scoring (High/Medium/Low = 1.0/0.5/0.0) with reasoning. Available in Agentic Monitoring and Experiments only.
**RAG Faithfulness** (Repackaged): An LLM-as-a-Judge evaluator that assesses whether the response is grounded in the retrieved documents. Returns a binary label (Yes/No = 1/0) with detailed reasoning. Available in Agentic Monitoring, Experiments, and LLM Observability.
See [RAG Health Diagnostics](/concepts/rag-health-diagnostics) for a comprehensive guide to using these evaluators together for root cause analysis.
**Source Attribution**: Verify that responses correctly attribute information to source documents when required for transparency or compliance.
### Performance Benchmarking
**Cross-Provider Comparison**: Evaluate the same application logic across different LLM providers to understand quality and capability differences for specific tasks.
**Longitudinal Tracking**: Run standardized experiment datasets periodically to detect performance drift as models, prompts, and data distributions evolve.
**Domain-Specific Benchmarking**: Create custom experiment datasets representing your specific domain, use case, and user population to benchmark against industry standards or competitors.
## Challenges
Implementing effective evaluation systems presents unique challenges stemming from the subjective nature of language, the complexity of assessment criteria, and the resource requirements of comprehensive testing.
**Subjectivity and Context-Dependence**: Defining "good" outputs is inherently subjective and varies by use case, user population, and context. What constitutes relevant, appropriate, or high-quality content for a customer service chatbot differs dramatically from what is considered high-quality content for a creative writing assistant. Establishing evaluation criteria requires a deep understanding of user expectations and business objectives.
**Ground Truth Acquisition**: Creating reference answers (ground truth) for test cases is labor-intensive and may not even be possible for open-ended generation tasks. While some scenarios have clear correct answers, creative tasks, summarization, or opinion-based queries lack a single correct response, complicating evaluator design.
**Computational Cost**: Running comprehensive evaluations, especially with LLM-as-a-judge evaluators, incurs significant computational expense and latency. Balancing evaluation comprehensiveness with budget constraints requires careful selection of evaluators and potentially sampling strategies.
**Test Coverage and Representativeness**: Creating datasets that adequately represent production diversity is challenging. Production systems encounter edge cases, novel user intents, and data distributions that may not be captured in experiment datasets, potentially leading to overfitting to test scenarios.
**Evaluation-Production Gap**: Test performance may not predict real-world performance due to differences in user behavior, data distribution, or system context. Experiment datasets may lack the complexity, noise, or unexpected patterns present in production environments.
**Metric Selection**: With dozens of available evaluators measuring different quality dimensions, choosing the proper subset for your use case requires domain expertise. Overloading evaluations with too many metrics creates noise, while missing critical evaluators can blind teams to essential failure modes.
**LLM-as-Judge Reliability**: When using LLMs as evaluators, the evaluator itself may hallucinate, show bias, or produce inconsistent judgments. Ensuring evaluator quality requires meta-evaluation—validating that evaluators align with human judgment.
**Maintenance Overhead**: Experiment datasets require ongoing maintenance as applications evolve, user expectations shift, and new failure modes emerge. Stale datasets provide false confidence, while keeping evaluations current demands continuous effort.
## Experiments Implementation Guide
1. **Define Evaluation Objectives**
* Identify the specific quality dimensions most critical for your use case (relevance, safety, coherence, etc.)
* Establish acceptable performance thresholds based on business requirements and user expectations
* Prioritize evaluation dimensions based on risk assessment—safety-critical applications require a different focus than creative tools
2. **Build Representative Datasets**
* Start with 10-20 high-priority scenarios covering common use cases and known failure modes
* Expand to include edge cases, adversarial inputs, and diverse user populations
* Include ground truth answers where possible, or clear evaluation criteria when ground truth isn't feasible
* Use production data (anonymized and filtered) to ensure test cases reflect real-world distribution
* Version datasets and track changes to maintain evaluation consistency over time
3. **Select Appropriate Evaluators**
* Begin with built-in evaluators for common quality dimensions (relevance, coherence, safety)
* Add domain-specific custom evaluators for business requirements not covered by standard metrics
* Avoid metric overload—focus on evaluators that drive actual decisions rather than collecting all possible scores
* Validate evaluators against human judgment on sample test cases to ensure alignment
4. **Run Baseline Experiments**
* Execute evaluations on your current application to establish baseline performance
* Document baseline scores and acceptable ranges for each evaluator
* Identify problem areas and prioritize improvement efforts based on experiment results
5. **Establish Evaluation Cycles**
* Integrate evaluations into development workflow—run before each pull request or deployment
* Set up automated experiment pipelines that block deployment when scores fall below thresholds
* Schedule periodic comprehensive evaluations to detect drift even when code hasn't changed
* Create feedback loops where evaluation insights drive prompt refinement and model tuning
6. **Iterate and Improve**
* Analyze experiment results to identify patterns in failures and guide optimization
* Compare experiments side-by-side to validate that changes improve target metrics without regressing others
* Expand datasets based on newly discovered failure modes in production
* Refine evaluators and thresholds as understanding of quality requirements evolves
## Frequently Asked Questions
**Q: How many test cases do I need for reliable evaluation?**
Start with 10-20 high-priority test cases covering core functionality and known edge cases. As you mature your evaluation practice, expand to 50-100+ cases for comprehensive coverage. The optimal number depends on application complexity and diversity of scenarios—simple Q\&A may require fewer cases than complex multi-turn conversations or RAG systems.
**Q: How do I choose which evaluators to use?**
Prioritize evaluators aligned with your specific failure modes and user expectations. Safety-critical applications need toxicity and bias evaluators; RAG systems require faithfulness; customer service chatbots benefit from relevance and coherence. Start with 3-5 core evaluators and expand based on observed gaps rather than trying to measure everything from day one.
**Q: Should I use LLM-as-a-judge or rule-based evaluators?**
Use both strategically. Rule-based evaluators (length checks, regex matching, keyword presence) are fast, deterministic, and cheap but limited to surface-level criteria. LLM-as-a-judge provides a nuanced semantic assessment but incurs cost and latency. Combine rule-based evaluators for quick sanity checks with LLM-based evaluators for quality assessment.
**Q: How often should I run experiments?**
Run experiments before every deployment for regression testing, and schedule comprehensive experiments weekly or monthly, even without code changes, to detect drift. For active development, run experiments on every pull request. Balance frequency with computational cost—use smaller, quick-check datasets for frequent testing and comprehensive datasets for periodic deep evaluation.
**Q: What's the difference between evaluation and monitoring?**
Evaluation involves proactive testing with curated test datasets before deployment, while monitoring is the reactive observation of production behavior. Evaluation provides control over test scenarios and ground truth, but may not capture production complexity. Monitoring reflects real-world usage but lacks controlled testing. Use evaluation for pre-deployment quality gates and monitoring for production health.
**Q: Can evaluations replace human review?**
Evaluations complement rather than replace human review. Automated evaluations scale to comprehensive testing, which is impossible for humans, but humans provide nuanced judgment on edge cases, ethical considerations, and subjective quality that automated systems may miss. Use evaluations for systematic coverage and humans for final judgment on critical or ambiguous cases.
## Related Terms
* [LLM Observability](/glossary/llm-observability) - Production monitoring complementing pre-deployment evaluation
* [Agentic Observability](/glossary/agentic-observability) - Monitoring multi-agent systems with evaluation capabilities
* [Guardrails](/glossary/guardrails) - Real-time safety controls often validated through evaluation
* [Trust Score](/glossary/trust-score) - Metrics measuring trustworthiness, often used as evaluators
* [Enrichment](/glossary/enrichment) - Automated metrics generation similar to evaluation, but for production data
* [Model Performance](/glossary/model-performance) - Traditional ML metrics complementing LLM evaluation
## Related Resources
* [Getting Started with Fiddler Experiments](/getting-started/experiments)
* [Fiddler Evals SDK Reference](/sdk-api/evals/evaluate)
* [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start)
* [Evals SDK Advanced Guide](/developers/tutorials/experiments/evals-sdk-advanced)
* [RAG Health Diagnostics](/concepts/rag-health-diagnostics) — Conceptual guide to RAG pipeline diagnostics
* [RAG Health Metrics Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial) — Step-by-step RAG evaluation guide
# Fiddler Guardrails
Source: https://docs.fiddler.ai/glossary/guardrails
Fiddler Guardrails is a real-time content-safety capability that evaluates and filters harmful LLM outputs before they reach users, powered by Fiddler Centor Models.
[Fiddler Guardrails](/glossary/guardrails) is a real-time content safety solution that evaluates and filters potentially harmful outputs from large language models (LLMs) before they reach end users. It serves as a protective layer between LLM systems and their users, ensuring that generated content adheres to organizational policies and safety standards.
Powered by [Fiddler Centor Models](/glossary/centor-models), Fiddler Guardrails leverages purpose-built models optimized for efficient content evaluation. The system processes LLM outputs in real-time, detecting problematic content across multiple safety dimensions including harmfulness, toxicity, illegal activity, bias, and more. When violations are detected, Guardrails can either filter out the content entirely or provide detailed explanations of the specific policies that were violated.
Fiddler Guardrails is a capability of the Fiddler AI Observability and Security platform. You access it through its real-time API within your Fiddler environment, powered by Fiddler Centor Models.
## How Fiddler Uses Guardrails
Fiddler Guardrails integrates with the broader Fiddler platform to provide comprehensive protection and visibility for LLM and GenAI applications. As part of Fiddler's end-to-end LLM governance solution, Guardrails helps organizations maintain control over their generative AI deployments by ensuring outputs meet safety standards before reaching users.
When integrated with Fiddler's monitoring capabilities, Guardrails contributes to a complete approach where potentially harmful content is both blocked in real-time and analyzed for patterns that might indicate systemic issues. This dual approach enables organizations to maintain strong protections while continuously improving their LLM systems.
Fiddler offers Guardrails through a simple API that can be integrated into existing LLM application workflows, allowing organizations to implement protection without significant architectural changes.
## Why Fiddler Guardrails Is Important
As organizations increasingly deploy generative AI applications across their operations, ensuring the safety and appropriateness of LLM outputs becomes a critical governance concern. Fiddler Guardrails addresses this need by providing real-time protection against potentially harmful, toxic, or inappropriate content generation.
Without robust guardrails in place, organizations face significant risks from LLM deployments, including reputational damage, compliance violations, and potential harm to users. By implementing Fiddler Guardrails, organizations can confidently deploy generative AI with protections that minimize these risks.
The importance of Guardrails extends beyond individual content filtering to enable trustworthy, responsible AI deployment at scale across an organization.
* **Risk Mitigation**: Guardrails prevents harmful, toxic, or inappropriate content from reaching end users, protecting both users and the organization from potential negative impacts of unsafe LLM outputs.
* **Compliance Support**: By enforcing content policies consistently, Guardrails helps organizations meet regulatory requirements and internal governance standards for responsible AI use.
* **Deployment Confidence**: With protective measures in place, organizations can more confidently deploy LLM applications across a broader range of use cases and user groups.
* **Brand Protection**: By filtering inappropriate content before it reaches customers, Guardrails helps protect brand reputation and maintain trust in AI-powered services.
* **Operational Efficiency**: Real-time protection reduces the need for human review of all LLM outputs, allowing teams to focus on improving models rather than constantly monitoring for problematic content.
* **Educational Feedback**: When content is filtered, Guardrails provides detailed explanations about policy violations, helping developers understand and address recurring issues in their prompting strategies or model configurations.
* **Customizable Protection**: Organizations can tailor protection levels to their specific needs and risk tolerance, ensuring appropriate safeguards without unnecessarily restrictive filtering.
## Types of Guardrails Protection
* **Safety Guardrails**: Core protections against harmful, toxic, illegal, or objectionable content across multiple categories including violence, hate speech, explicit content, and illegal activities.
* **Custom Policy Guardrails**: Organization-specific policy enforcement that can be configured to reflect particular industry requirements, company values, or audience sensitivities.
* **Guardrails API**: Real-time content safety delivered through the Guardrails API within your Fiddler environment.
* **Integrated Platform Guardrails**: Guardrails functionality within the comprehensive Fiddler observability platform, working alongside monitoring features for complete LLM governance.
* **Multi-level Protection**: Tiered protection options allowing organizations to set different filtering thresholds for different applications or user contexts.
## Challenges
Implementing effective content safety for LLM applications presents several challenges that Fiddler Guardrails is designed to address.
* **Latency Management**: Adding protective layers can potentially slow response times, a challenge Guardrails addresses through efficient purpose-built models optimized for evaluation speed.
* **Nuanced Content Evaluation**: Distinguishing between genuinely harmful content and benign discussions of sensitive topics requires sophisticated evaluation capabilities beyond simple keyword filtering.
* **False Positive Balance**: Setting appropriate thresholds that protect against harmful content without excessive blocking of legitimate outputs requires careful calibration.
* **Context Awareness**: Properly evaluating content requires understanding the broader context of the conversation, not just analyzing isolated responses.
* **Policy Customization**: Different organizations have varying standards for acceptable content, requiring flexible guardrail systems that can be tailored to specific needs.
* **Multilingual Support**: Ensuring consistent protection across content in different languages presents challenges in evaluation consistency.
* **Evolving Threat Landscape**: As LLM capabilities advance and new exploitation techniques emerge, guardrail systems must continuously update to address novel risks.
## Fiddler Guardrails Implementation Guide
1. **Define Your Content Safety Requirements**
* Identify which safety dimensions are most important for your LLM use cases.
* Determine appropriate filtering thresholds based on your user base and risk tolerance.
2. **Choose Deployment Approach**
* Decide how to integrate the Guardrails API into your application's request and response flow.
* Select between cloud-hosted or on-premises deployment options.
3. **Integrate Guardrails API**
* Implement API calls in your application flow to route LLM outputs through Guardrails before display.
* Set up appropriate error handling for cases where content is filtered.
4. **Configure Safety Policies**
* Set appropriate thresholds for different safety dimensions.
* Configure custom policies if needed for organization-specific requirements.
5. **Test and Refine**
* Verify protection effectiveness with representative test cases.
* Refine thresholds based on observed false positive/negative rates.
6. **Monitor and Improve**
* Track guardrail filtering patterns to identify recurring issues.
* Use insights to improve prompt engineering and LLM configuration.
## Frequently Asked Questions
**Q: How does Fiddler Guardrails differ from prompt engineering for safety?**
While prompt engineering attempts to elicit safe behavior through careful input design, Guardrails provides a dedicated protection layer that evaluates outputs regardless of prompt quality. This approach is more robust as it doesn't rely on the effectiveness of prompts alone and can catch unsafe responses even when prompt constraints fail.
**Q: What types of content can Fiddler Guardrails detect and filter?**
Fiddler Guardrails can detect and filter content across multiple safety dimensions including but not limited to harmful content, toxicity, hate speech, violence, sexual content, illegal activities, bias, profanity, and content faithfulness.
**Q: How does Guardrails impact response latency?**
Fiddler Guardrails is designed to minimize latency impact through highly optimized models specifically built for efficient content evaluation. While there is some processing overhead, it's typically measured in milliseconds rather than seconds, and significantly faster than using general-purpose LLMs for evaluation.
**Q: Can I customize the safety policies to fit my organization's needs?**
Yes, Fiddler Guardrails allows for customization of safety policies and thresholds to align with specific organizational requirements, industry regulations, and risk tolerance levels.
**Q: Is Fiddler Guardrails a separate product from the Fiddler platform?**
No. Fiddler Guardrails is a capability of the Fiddler AI Observability and Security platform, powered by Fiddler Centor Models, and is accessed through the Guardrails API within your Fiddler environment.
## Related Terms
* [Fiddler Centor Models](/glossary/centor-models)
* [Trust Score](/glossary/trust-score)
## Related Resources
* [Getting Started with Fiddler Guardrails](/getting-started/guardrails)
* [Guardrails in the Fiddler Platform](/glossary/guardrails)
* [Guardrails FAQ](/protection/guardrails-faq)
* [Fiddler Centor Model Metrics](/observability/llm/enrichments#safety)
# Glossary
Source: https://docs.fiddler.ai/glossary/index
Review product concepts and terminology for the Fiddler platform to help get up to speed quickly when adopting Fiddler for your ML and GenAI monitoring.
This page explains core concepts and terminology used throughout Fiddler's AI Observability and Security platform. Understanding these concepts will help you navigate the platform more effectively and get the most from Fiddler's capabilities.
## Monitoring and Observability Concepts
### ML Observability
[ML observability](/glossary/ml-observability) is the practice of gaining comprehensive insights into AI application performance throughout its lifecycle. It goes beyond simple indicators of good and bad performance by empowering stakeholders to understand why a model behaves in a certain manner and how to enhance its performance. ML Observability begins with monitoring and alerting on performance issues but extends to guiding model owners toward the underlying root causes.
### LLM Observability
[LLM observability](/glossary/llm-observability) is the specialized practice of evaluating, monitoring, analyzing, and improving Generative AI and LLM-based applications across their lifecycle. Fiddler provides real-time monitoring on safety metrics like toxicity, bias, and PII exposure, as well as correctness metrics like hallucinations, faithfulness, and relevancy specific to language models.
### Alerts
[Alerts](/observability/platform/alerts-platform) are rules that trigger when production data meets defined conditions. These rules can be user-defined or automatically generated based on user configuration. Alert notifications can be sent via email, Slack, PagerDuty, or any combination thereof, enabling teams to respond quickly to potential issues with model performance or data quality.
### Metrics
Metrics in Fiddler refer to the quantitative measurements and calculations the platform performs on inference data. These metrics provide insights into model behavior, data characteristics, and performance over time. Fiddler offers several core metric types:
* [Data Drift](/glossary/data-drift): Measures statistical differences between production and baseline data distributions
* [Performance](/observability/platform/performance-tracking-platform): Tracks model accuracy, precision, recall, and other performance indicators
* [Data Integrity](/observability/platform/data-integrity-platform): Identifies missing values, outliers, and other data quality issues
* [Traffic](/observability/platform/traffic-platform): Monitors request volumes, response times, and utilization patterns
* Statistical: Provides basic descriptive statistics about data distributions
* [Custom Metrics](/glossary/custom-metrics): User-defined calculations tailored to specific business needs
## Data Management Concepts
### Pre-production Data
Data designated as pre-production contains non-time series data, which is uploaded to Fiddler in a single batch. Pre-production data typically includes training datasets, validation datasets, or other static data meant to be evaluated as a complete unit without the dimension of trends over time.
### Production Data
Data designated as production contains time series data such as inference logs generated by models making decisions in live environments. This time series data provides the inputs and outputs of each model inference/decision, which Fiddler analyzes and compares against pre-production data to determine if model performance is degrading over time.
### Baselines
[Baselines](/glossary/baseline) are reference datasets used for calculating data drift and other comparative metrics. When determining if drift has occurred, Fiddler compares the distribution of current production data against this reference data. Most commonly, training data establishes a model's baseline, but multiple baselines can be defined for a model, including static sets of historical inferences or rolling baselines that look back over specific time periods.
### Segments (Cohorts)
[Segments](/observability/platform/segments), also called Cohorts, are subsets of inference logs defined by custom filters. Segments allow users to analyze metrics for specific subsets of data (for example, "transactions under \$1000" or "users from a specific region"). Segmentation enables more granular analysis of model performance across different data populations.
### Trust Scores (Enrichments)
[Trust Scores](/glossary/trust-score), also known as Enrichments, are specialized metrics that assess various quality and safety dimensions of LLM outputs. Generated by Fiddler Centor Models, these scores evaluate dimensions such as safety, toxicity, hallucination, relevance, and coherence. They provide quantifiable measurements for monitoring LLM behavior and can trigger alerts or actions when outputs fall below quality thresholds.
## Platform Components and Features
### Fiddler Centor Models
[Fiddler Centor Models](/glossary/centor-models) are purpose-built LLMs that evaluate AI outputs in real time, powering both monitoring metrics and real-time guardrails with significantly lower latency than general-purpose models. Centor Models:
* Evaluate LLM outputs with significantly higher efficiency than general-purpose LLMs
* Maintain comparable quality in their assessments
* Support both observability features and real-time protection capabilities
### Fiddler Guardrails
[Fiddler Guardrails](/glossary/guardrails) is a real-time content safety solution that evaluates and filters potentially harmful outputs from large language models before they reach end users.
Powered by Fiddler Centor Models, Guardrails detects problematic content across multiple safety dimensions and can either filter out unsafe content or provide detailed explanations of policy violations.
### Embedding Visualizations
[Embedding Visualizations](/glossary/embedding-visualization) in Fiddler display high-dimensional embedding vectors in an accessible two-dimensional space using techniques like UMAP (Uniform Manifold Approximation and Projection). These visualizations make complex vector relationships visible, allowing users to identify clusters, outliers, and patterns that would remain hidden in raw numerical data.
### Dashboards and Charts
Fiddler uses customizable [Dashboards](/observability/dashboards) for monitoring and sharing model behavior. Dashboards comprise Charts that provide distinct visualization types:
* [Monitoring Charts](/observability/platform/monitoring-charts-platform): Track metrics over time and compare model performance
* [Embedding Visualizations](/glossary/embedding-visualization): Display semantic relationships in embedding space
* [Performance Analytics](/observability/analytics): Analyze model performance across different segments
Dashboards consolidate visualizations in one place, offering a detailed overview of model performance and an entry point for deeper analysis and root cause identification.
### Bookmarks
Bookmarking enables quick access to frequently used projects, models, charts, and dashboards. The comprehensive bookmark page enhances navigation efficiency within the Fiddler platform, allowing users to quickly return to their most important resources.
## Administration Concepts
### Projects
[Projects](/developers/python-client-guides/model-onboarding/create-a-project-and-model) in Fiddler serve as the principal organizational containers for your AI applications or use cases. Each project functions as a logical workspace that encapsulates related models, datasets, baselines, monitoring configurations, and analytics.
Projects provide several key benefits:
* **Organizational Structure**: Group related models and assets by business function, team ownership, or application purpose
* **Access Control**: Define which users and teams can view or modify project resources through role-based permissions
* **Resource Isolation**: Maintain separate environments for different AI initiatives to prevent configuration conflicts
* **Focused Monitoring**: Create dashboards and alerts specific to the business context of each application
* **Collaborative Workflow**: Enable teams to work together on related models within a consistent environment
Within a project, you can onboard multiple models, upload baseline datasets, create production data-based baselines, configure alerts, build dashboards, and analyze performance—all within a unified context that reflects your organization's structure and workflows.
Projects help you scale AI governance across your organization by providing clear boundaries between different applications while maintaining consistent monitoring and explainability practices.
### Role-Based Access Control
Fiddler supports Role-Based Access Control (RBAC) that defines who can access which resources within the platform. Available roles include:
* **Org Admin**: Manages users, teams, projects, and organization settings
* **Org Member**: Has limited access to organization settings and cannot create projects
* **Project Admin**: Manages all aspects of a project including models, settings, and alerts
* **Project Writer**: Can view and edit most project details but has limited administrative capabilities
* **Project Viewer**: Can view project resources but cannot make changes
### Teams
[Teams](/reference/administration/settings#teams) are groups of users within your organization that can be assigned specific roles and permissions for different projects. Each user can be a member of multiple teams, enabling flexible access control based on organizational structure and responsibilities.
# LLM Observability
Source: https://docs.fiddler.ai/glossary/llm-observability
Comprehensive monitoring of LLM applications that evaluates safety, quality, and performance metrics to detect issues like hallucinations, toxicity, and drift in generative AI systems.
[LLM Observability](/index) is the practice of monitoring, measuring, and analyzing Large Language Model systems in production environments to ensure their reliability, safety, and performance. It involves the systematic collection and analysis of LLM inputs, outputs, and associated metrics to provide visibility into model behavior, detect anomalies, ensure alignment with business objectives, and maintain trust.
Unlike traditional ML model monitoring, LLM Observability addresses unique challenges specific to generative AI, including hallucination detection, prompt safety evaluation, response quality assessment, and embedding analysis. This comprehensive approach enables organizations to understand how their LLM applications perform in real-world scenarios and take proactive measures to maintain quality and mitigate risks.
## How Fiddler Provides LLM Observability
Fiddler's LLM Observability platform provides a comprehensive approach to monitoring and protecting LLM applications through enrichments, which are custom features designed to augment data provided in events. The platform requires publication of LLM application inputs and outputs, including prompts, prompt context, responses, and source documents (for RAG-based applications).
Fiddler generates various AI trust and safety metrics through its enrichment pipeline, allowing users to detect data drift, visualize embeddings, identify hallucinations, assess response quality, detect harmful content, and monitor overall application health. These metrics can be used for alerting, analysis, and debugging purposes across the application lifecycle.
## Why LLM Observability Is Important
LLM Observability is crucial for organizations deploying generative AI applications in production environments. As LLMs become increasingly integrated into critical business processes and customer-facing applications, maintaining transparency, quality, and safety becomes essential. Effective LLM Observability enables teams to detect issues early, continuously improve model performance, ensure responsible AI deployment, and maintain compliance with evolving regulatory requirements.
* **Quality Assurance and Hallucination Detection**: LLM Observability helps identify instances of hallucinations, factual inaccuracies, or low-quality outputs through metrics like faithfulness and answer relevance, ensuring that generated content meets quality standards.
* **Safety and Trust Monitoring**: Monitoring ensures LLM applications remain safe and trustworthy by detecting harmful, toxic, or inappropriate content through metrics like safety scores, profanity detection, and toxicity assessment.
* **Performance Optimization**: By tracking operational metrics such as token usage, embedding quality, and response times, organizations can optimize their LLM applications for both cost efficiency and user satisfaction.
* **Root Cause Analysis**: When issues arise, LLM Observability provides the tools to conduct detailed analysis, identify the root causes of problems, and implement targeted improvements.
* **Drift Detection**: As the world changes and user behavior evolves, LLM Observability helps detect shifts in prompt patterns or content distribution that might affect model performance.
* **Regulatory Compliance**: With growing regulatory scrutiny of AI systems, LLM Observability provides the transparency and documentation needed to demonstrate responsible AI practices to stakeholders and regulators.
## Types of LLM Observability
* **Input Monitoring**: Tracking and analyzing user prompts, prompt context, and embedding patterns to identify trends, anomalies, and potential security risks like jailbreak attempts or prompt injections.
* **Output Quality Assessment**: Evaluating LLM responses for quality metrics including faithfulness, coherence, conciseness, and relevance to ensure outputs align with user expectations and business requirements.
* **Safety and Trust Evaluation**: Monitoring for harmful content, inappropriate language, PII leakage, toxicity, and other trust-related concerns that might compromise user safety or organizational reputation.
* **Embedding Visualization**: Using techniques like UMAP to visualize high-dimensional embeddings in 2D or 3D space, enabling identification of clusters, patterns, and anomalies in LLM data.
* **Performance Monitoring**: Tracking system-level metrics such as response times, token usage, error rates, and throughput to optimize operational efficiency and cost management.
## Challenges
Implementing effective LLM Observability presents unique challenges due to the complex, generative nature of these models and the contextual importance of their outputs.
* **Defining Meaningful Metrics**: Unlike traditional ML models with clear accuracy metrics, defining and measuring "quality" for LLM outputs is subjective and context-dependent, requiring multiple complementary evaluation approaches.
* **Hallucination Detection**: Reliably identifying when LLMs generate false or misleading information requires sophisticated evaluation techniques and often involves comparing outputs against trusted knowledge sources.
* **Balancing Performance and Safety**: Organizations must navigate the trade-off between optimizing for response quality and speed while maintaining robust safety guardrails and content filtering.
* **Managing High-Dimensionality Data**: LLM embeddings and feature spaces are highly dimensional, making them challenging to analyze, visualize, and interpret without specialized techniques like UMAP.
* **Handling Diverse Use Cases**: Different LLM applications (customer service, content creation, code generation) require different monitoring approaches and metrics, making it difficult to establish universal standards.
* **Privacy and Security**: LLM applications may process sensitive user data, creating challenges for monitoring that must be balanced with privacy requirements and security considerations.
* **Real-time vs. Batch Analysis**: Organizations must decide which metrics require real-time monitoring with immediate alerts versus those that can be analyzed in batch processes, balancing responsiveness with resource efficiency.
## LLM Observability Implementation How-to Guide
1. **Define Monitoring Objectives**
* Identify key performance indicators (KPIs) most relevant to your specific LLM application use case.
* Determine acceptable thresholds for safety, quality, and performance metrics based on business requirements.
2. **Set Up Data Collection**
* Implement comprehensive logging for all LLM application inputs and outputs, including prompts, context, and responses.
* For RAG applications, capture retrieved documents and sources to enable faithfulness evaluation.
3. **Implement Essential Enrichments**
* Configure embedding generation for semantic analysis of prompts and responses.
* Set up basic safety and quality enrichments including toxicity detection, PII scanning, and relevance metrics.
4. **Establish Visualization Capabilities**
* Implement UMAP visualization for embedding spaces to identify clusters and anomalies.
* Create dashboards displaying key metrics over time to track performance trends.
5. **Configure Alerting and Guardrails**
* Set up threshold-based alerts for critical metrics related to safety, performance, and quality.
* Implement guardrails for proactive protection against harmful content and prompt injections.
6. **Develop Analysis Workflows**
* Create standard procedures for investigating alerts and conducting root cause analysis.
* Establish regular review cycles to assess LLM application health and identify areas for improvement.
## Frequently Asked Questions
**Q: How is LLM Observability different from traditional ML monitoring?**
LLM Observability addresses unique challenges like hallucinations, prompt effectiveness, safety concerns, and nuanced quality metrics that aren't present in traditional ML models. It focuses on unstructured text outputs requiring qualitative and semantic evaluation rather than simple accuracy metrics.
**Q: What metrics should I prioritize for my LLM application?**
Priority metrics depend on your use case but typically include safety metrics (toxicity, harmful content), quality metrics (faithfulness, coherence, relevance), and operational metrics (response time, token usage). Applications handling sensitive information should prioritize PII detection, while customer-facing applications may emphasize response quality.
**Q: How can I detect LLM hallucinations in production?**
Fiddler offers multiple approaches to hallucination detection. **Faithfulness (Centor Model)** (powered by the Centor Model for Faithfulness) provides real-time faithfulness scoring in LLM Observability and Guardrails. **RAG Faithfulness** (LLM-as-a-Judge) provides binary assessment with detailed reasoning in Agentic Monitoring and Experiments. For RAG applications, the RAG Health Metrics diagnostic triad (Answer Relevance 2.0, Context Relevance, RAG Faithfulness) helps pinpoint whether issues stem from retrieval or generation. See [RAG Health Diagnostics](/concepts/rag-health-diagnostics).
**Q: How do embedding visualizations help with LLM monitoring?**
UMAP embedding visualizations help identify clusters of similar prompts or responses, detect outliers, visualize concept drift, and identify problematic patterns like jailbreak attempts or toxic content clusters, providing intuitive visual analysis of high-dimensional data.
## Related Terms
* [Enrichments](/glossary/enrichment)
* [Guardrails](/glossary/guardrails)
* [Embedding Visualization](/glossary/embedding-visualization)
* [Data Drift](/glossary/data-drift)
## Related Resources
* [LLM Monitoring Overview](/index)
* [LLM-based Metrics Guide](/observability/llm/llm-based-metrics)
* [Embedding Visualization with UMAP](/observability/llm/embedding-visualization-with-umap)
* [Selecting Enrichments](/observability/llm/selecting-enrichments)
* [Enrichments Documentation](/observability/llm/enrichments)
* [Guardrails for Proactive Application Protection](/glossary/guardrails)
# Metric
Source: https://docs.fiddler.ai/glossary/metric
Metrics in Fiddler AI are quantitative measurements that evaluate model behavior, data quality, and performance over time, enabling proactive monitoring and issue detection.
[Metrics](/observability/platform/data-drift-platform) in Fiddler refer to the quantitative measurements and calculations that Fiddler performs on inference data published to the platform. These metrics provide insights into model behavior, data characteristics, and performance over time.
Metrics serve as key indicators that help monitor model health, detect anomalies, and ensure that AI/ML systems are functioning as expected in production environments. Fiddler calculates various types of metrics ranging from statistical measures of data drift to sophisticated evaluations of LLM outputs.
By tracking these metrics, users can gain visibility into how their models are performing in real-world scenarios, identify potential issues before they impact business outcomes, and maintain trust in their AI systems.
## How Fiddler Uses Metrics
Fiddler leverages metrics as the foundation of its monitoring and observability capabilities. When inference data is published to the Fiddler platform, it automatically calculates relevant metrics based on the model type and configuration. These metrics are then displayed in dashboards, used to trigger alerts when thresholds are exceeded, and stored for historical trend analysis.
For traditional ML models, Fiddler calculates metrics like data drift, performance tracking, and data integrity. For LLM/GenAI systems, Fiddler extends its metrics suite to include specialized measurements like faithfulness, safety scores, and other LLM-specific evaluations.
## Why Metrics Are Important
Metrics are essential for maintaining reliable and trustworthy AI systems in production. They provide quantifiable evidence of model behavior and performance, enabling teams to make data-driven decisions about when interventions are necessary. Without proper metrics, organizations would be operating their AI systems blindly, unable to detect degradation, bias, or unexpected behaviors until they cause significant business impact.
By establishing a comprehensive metrics framework, organizations can proactively monitor their AI systems, demonstrate compliance with regulations, and build confidence in their deployment practices.
* **Performance Monitoring**: Metrics enable continuous evaluation of model accuracy, precision, recall, and other performance indicators to ensure models are delivering expected results.
* **Drift Detection**: Statistical metrics like JSD (Jensen-Shannon Divergence) and PSI (Population Stability Index) help identify when input data distributions shift away from training data, potentially impacting model performance.
* **Data Quality Assurance**: Data integrity metrics reveal missing values, outliers, and other quality issues that might affect model predictions.
* **Operational Insights**: Traffic metrics and response time measurements provide visibility into the operational aspects of deployed models.
* **LLM Output Evaluation**: Specialized metrics for LLM/GenAI systems assess output quality, safety, and alignment with human expectations.
* **Compliance and Governance**: Metrics support regulatory requirements by providing evidence of ongoing monitoring and model governance.
* **Issue Debugging**: When problems occur, metrics provide crucial diagnostic information to identify root causes.
## Types of Metrics
* **Data Drift Metrics**: Measurements that quantify distributional differences between reference and production data, including JSD, PSI, and other statistical distance measures.
* **Performance Metrics**: Indicators of model accuracy and effectiveness, such as precision, recall, F1 score, and custom business performance KPIs.
* **Data Integrity Metrics**: Measurements that assess data quality, completeness, and validity, highlighting missing values, outliers, and schema violations.
* **Traffic Metrics**: Counts and rates of model invocations, response times, and utilization patterns that reveal operational characteristics.
* **Statistical Metrics**: Basic descriptive statistics such as mean, median, standard deviation, and correlation that characterize data distributions.
* **Custom Metrics**: User-defined calculations tailored to specific business needs and use cases.
* **LLM-Based Metrics**: Specialized evaluations for generative AI outputs, including faithfulness, safety, toxicity, bias, and relevance scores.
## Challenges
While metrics provide essential visibility into AI systems, implementing an effective metrics strategy comes with several challenges that organizations must navigate.
* **Metric Selection**: Choosing the right metrics for specific use cases can be challenging, as different models and applications require different evaluation approaches.
* **Threshold Setting**: Determining appropriate threshold values that balance sensitivity to real issues against false alarms requires expertise and context-specific knowledge.
* **Computational Overhead**: Calculating complex metrics at scale can introduce performance overhead, especially for high-volume inference systems.
* **Interpretation Complexity**: Some advanced metrics may be difficult to interpret without specialized knowledge, making it challenging to translate metric values into actionable insights.
* **Metric Drift**: The relevance of metrics themselves may change over time as business requirements evolve or as models are updated.
* **Correlation vs. Causation**: Changes in metrics may correlate with issues but not necessarily reveal their root causes, requiring additional analysis.
* **LLM Evaluation Subjectivity**: Metrics for generative AI often involve subjective judgments about quality, making standardization difficult.
## Metrics Implementation How-to Guide
1. **Define Monitoring Objectives**
* Identify key performance indicators relevant to your model and business use case.
* Determine which aspects of model behavior require closest monitoring.
2. **Select Appropriate Metrics**
* Choose data drift metrics based on your feature data types (categorical vs. continuous).
* Select performance metrics aligned with your model type (classification, regression, LLM).
3. **Configure Baselines**
* Upload training or reference data to establish baseline distributions for drift detection.
* Set initial performance benchmarks for comparison.
4. **Establish Thresholds**
* Define alert thresholds for each metric based on tolerance for risk.
* Consider implementing tiered alerting with warning and critical levels.
5. **Integrate with Workflows**
* Connect metric alerts to notification systems (email, Slack, etc.).
* Establish response procedures for different types of metric anomalies.
## Frequently Asked Questions
**Q: How frequently should metrics be calculated?**
The calculation frequency depends on your use case. Critical applications may require real-time or hourly metrics, while less sensitive applications might use daily or weekly calculations. Consider both the business impact of issues and the computational resources required.
**Q: Can I create custom metrics in Fiddler?**
Yes, Fiddler supports custom metrics through its API and interface. You can define calculations based on your specific business needs and model characteristics.
**Q: How do I know which thresholds to set for my metrics?**
Start by monitoring metrics without alerts to establish normal operational patterns. Then set thresholds that balance sensitivity (catching real issues) with specificity (avoiding false alarms). Initial thresholds often require adjustment based on experience.
**Q: What's the difference between data drift and performance metrics?**
Data drift metrics measure changes in the statistical properties of input data, while performance metrics evaluate the accuracy and effectiveness of model outputs. Both are important as drift often precedes performance degradation.
**Q: How does Fiddler calculate LLM metrics differently?**
For LLM/GenAI systems, Fiddler calculates specialized metrics that evaluate text quality, safety, and alignment. Some of these metrics are generated by Fiddler's proprietary algorithms and purpose-built LLMs, while others may leverage external APIs like Anthropic and OpenAI for specific evaluations.
## Related Terms
* [Data Drift](/glossary/data-drift)
* [Model Performance](/glossary/model-performance)
* [Baseline](/glossary/baseline)
* [Custom Metric](/glossary/custom-metrics)
## Related Resources
* [ML Metrics Reference](/reference/ml-metrics-reference) — Complete list of all built-in ML metrics
* [LLM Observability Metrics Reference](/reference/llm-observability-metrics) — Complete list of all LLM enrichments
* [Monitoring Platform Overview](/index)
* [Data Drift Monitoring](/index)
* [Performance Tracking](/observability/platform/performance-tracking-platform)
* [Data Integrity Monitoring](/observability/platform/data-integrity-platform)
* [Traffic Monitoring](/observability/platform/traffic-platform)
* [Statistical Metrics](/observability/platform/statistics)
* [Custom Metrics](/glossary/custom-metrics)
* [LLM-Based Metrics](/observability/llm/llm-based-metrics)
* [Selecting LLM Enrichments](/observability/llm/selecting-enrichments)
# ML Observability
Source: https://docs.fiddler.ai/glossary/ml-observability
A comprehensive approach to monitoring AI systems that goes beyond performance metrics to provide insights into model behavior, data quality, and root causes of issues throughout the ML lifecycle.
[ML Observability](/index) is the systematic practice of monitoring, analyzing, and troubleshooting machine learning models throughout their lifecycle to ensure reliability, performance, and alignment with business objectives. It involves continuously tracking inputs, outputs, and model behaviors to provide visibility into how models operate in production environments, detect performance degradation, identify data integrity issues, and maintain trust.
Unlike traditional software monitoring, ML Observability addresses the unique challenges of machine learning systems, including data drift, concept drift, model decay, and the black-box nature of complex models. This comprehensive approach enables organizations to detect issues early, perform effective root cause analysis, maintain model quality, and ensure responsible AI deployment.
## How Fiddler Provides ML Observability
Fiddler's ML Observability platform provides a comprehensive approach to monitoring and improving machine learning models through five key metric types: data drift, performance, data integrity, traffic, and statistical properties. The platform helps ML teams detect issues early, diagnose root causes, and take corrective actions to maintain model quality and reliability.
Fiddler acts as a unified management platform with centralized controls and actionable insights, enabling ML teams to monitor both traditional ML models and LLM applications. The platform's explainability capabilities help users understand model behavior and decisions, while its monitoring features track drift, data quality, and performance metrics to ensure models operate as expected in production.
## Why ML Observability Is Important
ML Observability is crucial for organizations deploying machine learning models in production environments. As ML systems increasingly drive critical business decisions and customer experiences, maintaining visibility, reliability, and trust becomes essential. Effective ML Observability enables teams to detect issues early, continuously improve model performance, ensure business value, maintain compliance with regulations, and provide governance for responsible AI.
* **Data Drift Detection**: ML Observability continuously monitors for shifts in data distribution between training and production environments using metrics like Jensen-Shannon distance (JSD) and Population Stability Index (PSI), helping identify when models encounter data patterns they weren't trained on.
* **Performance Monitoring**: By tracking key performance metrics (accuracy, precision, recall, F1 scores, etc.), ML Observability helps ensure models continue to meet expected quality standards in production and alerts teams when performance degrades.
* **Data Integrity Validation**: ML Observability identifies data quality issues like missing values, type mismatches, and range violations that can arise from complex feature pipelines, preventing incorrect data from flowing into models and causing poor performance.
* **Root Cause Analysis**: When issues arise, ML Observability provides tools to diagnose the underlying causes through feature impact analysis, drift contribution metrics, and data segment performance comparisons, enabling targeted improvements.
* **Operational Efficiency**: ML Observability streamlines troubleshooting workflows, reduces time spent debugging issues, and helps ML teams focus on model development rather than reactive problem-solving, accelerating the ML lifecycle.
* **Business Impact Alignment**: By connecting model performance to business KPIs, ML Observability helps quantify the value of ML investments, prioritize improvements based on business impact, and ensure models deliver on their intended business objectives.
## Types of ML Observability
* **Data Drift Monitoring**: Tracking changes in the statistical properties of model inputs and outputs over time, comparing production data distributions against baseline data (typically training data) to detect when the model may be receiving data it wasn't designed for.
* **Performance Tracking**: Monitoring model accuracy, precision, recall, F1 scores, and other metrics specific to model tasks (classification, regression, ranking) to ensure performance remains within acceptable thresholds across different data segments.
* **Data Integrity Validation**: Checking for issues in data quality, including missing values, type mismatches, and range violations that might indicate problems in data pipelines or transformations feeding into the model.
* **Traffic Analysis**: Monitoring the volume and patterns of requests to ML models to detect anomalies like unexpected spikes or drops that might indicate system issues or potential security concerns.
* **Segment-based Analysis**: Analyzing model performance and behavior across different cohorts, slices, or segments of data to identify issues that may affect specific user groups or business scenarios.
* **Explainability Analysis**: Generating local and global explanations of model decisions to understand feature attributions, provide transparency, and diagnose why models make specific predictions.
## Challenges
Implementing effective ML Observability presents unique challenges due to the complex nature of machine learning systems and the dynamic environments in which they operate.
* **Delayed Ground Truth**: In many ML applications, the actual outcomes or labels for predictions may only become available after a significant delay (such as loan defaults or customer churn), making it difficult to assess model performance in real-time.
* **Feature Complexity**: Modern ML models often use hundreds or thousands of features with complex interdependencies, making it challenging to monitor and interpret all relevant input dimensions and their relationships to model outputs.
* **Class Imbalance**: Models trained on imbalanced datasets (where some classes are much rarer than others) present special monitoring challenges, as traditional metrics might not detect performance degradation for minority classes.
* **Data Pipeline Dependencies**: ML systems rely on complex data pipelines with multiple sources and transformations, creating numerous potential points of failure that need to be monitored for data integrity issues.
* **Establishing Thresholds**: Determining appropriate alerting thresholds for drift metrics and performance degradation requires balancing sensitivity to real issues against avoiding false alarms that could lead to alert fatigue.
* **Model Opacity**: The black-box nature of complex models like deep neural networks makes understanding the causes of performance issues challenging without specialized explainability techniques.
* **Multiple Environments**: ML models often operate across development, staging, and production environments with different data characteristics, making it difficult to maintain consistent monitoring approaches.
## ML Observability Implementation How-to Guide
1. **Establish Baselines**
* Create a representative baseline dataset from model training data to serve as a reference point for drift detection.
* Define performance benchmarks and acceptable thresholds for key metrics based on business requirements.
2. **Configure Core Monitoring**
* Set up data drift monitoring using appropriate distance metrics (JSD, PSI) for different feature types.
* Implement performance tracking with metrics specific to your model type (classification, regression, etc.).
3. **Implement Data Integrity Checks**
* Configure validation for missing values, type mismatches, and range violations in model inputs.
* Establish alerts for data pipeline issues that could impact model performance.
4. **Define Segments for Analysis**
* Create relevant data segments or cohorts based on business contexts to track performance across different user groups.
* Configure segment-specific monitoring to identify issues that might affect only certain data slices.
5. **Set Up Alerting System**
* Establish appropriate thresholds for alerts based on the criticality of the model and business impact.
* Configure notification routing to ensure the right teams are informed of relevant issues.
6. **Enable Root Cause Analysis**
* Implement explainability tools to understand model decisions and diagnose performance issues.
* Create dashboards that visualize feature impact, drift contributions, and other diagnostic metrics.
## Frequently Asked Questions
**Q: How is ML Observability different from traditional software monitoring?**
ML Observability addresses unique challenges specific to machine learning systems, including data drift, concept drift, and model decay that aren't present in traditional software. While software monitoring focuses on system uptime and resource utilization, ML Observability tracks statistical properties of data, model performance metrics, and the business impact of predictions.
**Q: What metrics should I prioritize for my ML models?**
Priority metrics depend on your use case, but generally include performance metrics (accuracy, precision, recall for classification; MSE, MAE for regression), data drift metrics to detect distribution shifts, data integrity metrics to ensure quality inputs, and business KPIs that connect model outputs to business outcomes.
**Q: How often should I retrain my models based on observability data?**
Retraining frequency should be determined by monitoring data rather than fixed schedules. Models should be retrained when significant drift is detected, when performance metrics drop below acceptable thresholds, or when business requirements change. For some applications, this might be weekly or monthly, while others might maintain performance for longer periods.
**Q: How can I determine appropriate thresholds for drift alerts?**
Start with conservative thresholds based on statistical significance (e.g., drift scores above 0.2 for PSI or JSD metrics) and refine them based on observed correlations between drift metrics and performance degradation in your specific models. Monitor false positives and adjust thresholds to balance sensitivity with alert fatigue.
## Related Terms
* [Data Drift](/glossary/data-drift)
* [Model Performance](/glossary/model-performance)
* [Model Drift](/glossary/model-drift)
* [Metrics](/glossary/metric)
* [Baselines](/glossary/baseline)
## Related Resources
* [ML Monitoring Platform Overview](/index)
* [Data Drift Monitoring](/observability/platform/data-drift-platform)
* [Performance Tracking](/observability/platform/performance-tracking-platform)
* [Ensuring Data Integrity](/observability/platform/data-integrity-platform)
* [Model Segments](/observability/platform/segments)
* [The Leader in ML Observability for MLOps](https://www.fiddler.ai/mlops)
# Model Drift
Source: https://docs.fiddler.ai/glossary/model-drift
Changes in model performance over time due to shifting data patterns, concept evolution, or system degradation. Fiddler detects and diagnoses model drift to maintain AI reliability.
Model drift refers to the degradation in an AI model's performance over time as the relationship between input features and target outputs changes in the real world. Unlike data drift, which focuses solely on shifts in input distributions, model drift encompasses broader changes in model behavior and effectiveness, even when input distributions remain stable.
Model drift can manifest in various ways: prediction patterns may shift, error rates might increase, or the model's underlying assumptions may no longer hold valid. This phenomenon is an inevitable challenge in machine learning deployments, as the real-world environment rarely remains static over extended periods.
In the context of AI observability, model drift represents a critical concern that requires continuous monitoring, timely detection, and proactive remediation to maintain reliable and effective AI systems in production.
## How Fiddler Monitors Model Drift
Fiddler's platform takes a comprehensive approach to model drift detection by monitoring several interconnected dimensions that together provide a complete picture of model health and stability.
Through its performance tracking capabilities, Fiddler continuously evaluates model predictions against ground truth values (when available) to identify degradation in accuracy, precision, recall, or custom business metrics that might indicate model drift. This performance monitoring is complemented by data drift detection, which helps determine whether changes in model behavior are due to shifts in input data patterns.
Fiddler also enables root cause analysis, allowing users to detect model drift that might affect specific segments of the data population differently. By comparing current performance across various cohorts against baseline expectations, Fiddler can identify localized drift that might not be apparent in aggregate metrics.
## Why Monitoring Model Drift Is Important
Monitoring model drift is essential for maintaining reliable and trustworthy AI systems in production environments. Undetected model drift can lead to incorrect predictions, suboptimal decisions, and potentially significant business impact, especially in critical applications like fraud detection, risk assessment, or healthcare diagnostics.
By implementing robust model drift monitoring, organizations can ensure their AI systems continue to perform as expected, identify when retraining or recalibration is necessary, and maintain confidence in automated decision processes. This monitoring capability is particularly crucial as AI deployments scale across the enterprise and become more deeply integrated into business operations.
* **Maintaining Prediction Quality**: Regular monitoring for model drift ensures that prediction quality remains high over time, preventing gradual degradation that might otherwise go unnoticed until it causes significant issues.
* **Timely Model Updates**: Early detection of model drift enables more proactive model maintenance, allowing teams to schedule retraining or updates before performance deteriorates to unacceptable levels.
* **Root Cause Analysis**: Comprehensive model drift monitoring provides insights into why performance is changing, differentiating between data quality issues, concept drift, or other factors affecting model behavior.
* **Business Impact Mitigation**: By detecting model drift early, organizations can prevent potential negative business impacts such as revenue loss, customer dissatisfaction, or compliance violations that might result from degraded model performance.
* **Resource Optimization**: Understanding patterns of model drift helps organizations optimize the frequency of model retraining, avoiding both unnecessary updates (when models remain stable) and delayed responses to genuine performance issues.
* **Regulatory Compliance**: In regulated industries, demonstrating continuous monitoring for model drift is increasingly becoming a requirement for responsible AI governance and compliance with emerging AI regulations.
* **Continuous Improvement**: Tracking model drift over time provides valuable insights that can inform better feature engineering, model architecture decisions, and training practices for future model iterations.
## Types of Model Drift
* **Concept Drift**: Changes in the underlying relationship between input features and target variables, where the statistical properties of the target variable change over time, making previous patterns less predictive.
* **Prediction Drift**: Changes in the statistical distribution of model outputs or predictions over time, which may indicate shifting behavior even if accuracy metrics remain temporarily stable.
* **Accuracy Drift**: Direct degradation in performance metrics like accuracy, precision, recall, or F1-score that indicates the model is becoming less effective at its intended task.
* **Feature Contribution Drift**: Changes in how different features influence model predictions, potentially indicating that the model's internal decision-making process is evolving in unexpected ways.
* **Seasonal Drift**: Cyclical changes in model performance related to time-based patterns such as day/night cycles, weekday/weekend differences, or annual seasonality effects.
* **Population Drift**: Changes in model performance for specific segments or cohorts of the data, even when overall performance metrics remain stable, indicating uneven drift effects across different populations.
* **Business Impact Drift**: Changes in how model predictions affect business outcomes and KPIs, where technical performance metrics might remain stable but the business value of the model's decisions decreases.
## Challenges
While essential for AI system maintenance, effective model drift monitoring comes with several technical and practical challenges that organizations must navigate.
* **Ground Truth Latency**: In many real-world applications, actual outcomes (ground truth) may not be available until significantly after predictions are made, creating delays in detecting accuracy-based drift signals.
* **Distinguishing Causes**: Determining whether observed drift is due to data distribution changes, evolving relationships between variables, or issues with the model itself can be challenging but necessary for appropriate remediation.
* **Setting Appropriate Thresholds**: Defining meaningful thresholds for when drift requires attention is complex, requiring balance between sensitivity to meaningful changes and resistance to false alarms from normal statistical variation.
* **High-Dimensional Monitoring**: Models with many features present challenges for comprehensive drift monitoring, as changes might occur in complex, high-dimensional patterns that are difficult to detect and visualize.
* **Feedback Loops**: When model outputs influence future inputs (as in recommendation systems or pricing models), distinguishing natural system dynamics from problematic drift becomes especially challenging.
* **Resource Constraints**: Comprehensive model drift monitoring can be computationally intensive, particularly for large-scale models or high-volume inference systems, requiring efficient implementation strategies.
* **Interpretability Trade-offs**: More sophisticated drift detection approaches might provide better sensitivity but at the cost of interpretability, making it harder to explain detected issues to stakeholders.
## Model Drift Monitoring Implementation Guide
1. **Define Monitoring Objectives**
* Identify which aspects of model performance are most critical for your specific use case.
* Determine acceptable performance thresholds based on business requirements and risk tolerance.
2. **Establish Comprehensive Baselines**
* Create performance baselines using training or validation data to serve as reference points.
* Consider multiple baselines for different time periods or data segments if appropriate.
3. **Configure Multi-Dimensional Monitoring**
* Set up combined monitoring for both data drift and performance metrics to capture different drift signals.
* Implement segment-specific monitoring for important data cohorts or slices.
4. **Set Up Alert Systems**
* Configure alerts with appropriate thresholds for different drift metrics based on sensitivity needs.
* Establish notification workflows to ensure the right teams are informed when drift is detected.
5. **Implement Regular Review Processes**
* Schedule routine reviews of drift metrics even in the absence of alerts to spot gradual changes.
* Maintain documentation of observed drift patterns and corresponding actions taken.
6. **Develop Remediation Strategies**
* Create predefined response plans for different types and severities of model drift.
* Consider automated model updating pipelines for cases where drift follows predictable patterns.
## Frequently Asked Questions
**Q: How is model drift different from data drift?**
While data drift focuses specifically on changes in the statistical distribution of input features, model drift is a broader concept that encompasses any degradation in model performance over time. Data drift is often a cause of model drift, but model drift can also occur due to other factors such as concept drift (changes in the relationship between inputs and outputs) or model decay (gradual degradation of model parameters or implementation). Fiddler monitors both phenomena to provide comprehensive observability.
**Q: How frequently should I check for model drift?**
The optimal frequency depends on your specific use case, data velocity, and business requirements. For critical applications with high-frequency decisions, daily or even hourly monitoring might be appropriate. For more stable applications with slower-changing environments, weekly or monthly checks might suffice. Fiddler's platform allows for flexible monitoring schedules tailored to your specific needs and can automate this process through continuous monitoring and alerts.
**Q: Should I retrain my model at the first sign of drift?**
Not necessarily. Minor drift might be within normal statistical variation or temporary in nature. Before retraining, it's important to investigate the root cause, assess business impact, and determine if the drift represents a genuine shift requiring model updates. Fiddler's analytics capabilities help you make informed decisions about when retraining is truly warranted versus when other interventions might be more appropriate.
**Q: Can model drift occur even if my data hasn't changed?**
Yes, model drift can occur even with stable input distributions. This might happen due to changes in the underlying relationships between variables (concept drift), subtle implementation issues during deployment, hardware or infrastructure changes affecting numerical precision, or gradual erosion of model effectiveness in dynamic environments. Monitoring both data distributions and model performance is essential for comprehensive drift detection.
**Q: How does Fiddler help distinguish different types of drift?**
Fiddler provides multiple monitoring dimensions that together help diagnose the nature of observed drift. By simultaneously tracking data drift metrics, performance metrics, feature importance, and segment-specific analyses, Fiddler enables users to triangulate the likely causes of detected drift. This multi-faceted approach helps distinguish between data quality issues, concept drift, or other factors, guiding more targeted remediation strategies.
## Related Terms
* [Data Drift](/glossary/data-drift)
* [Model Performance](/glossary/model-performance)
* [Model Drift](/glossary/model-drift)
* [Metrics](/glossary/metric)
* [Baselines](/glossary/baseline)
## Related Resources
* [ML Monitoring Platform Overview](/index)
* [Data Drift Monitoring](/observability/platform/data-drift-platform)
* [Performance Tracking](/observability/platform/performance-tracking-platform)
* [Ensuring Data Integrity](/observability/platform/data-integrity-platform)
* [Model Segments](/observability/platform/segments)
* [Setting Up Alerts](/observability/platform/alerts-platform)
# Model Performance
Source: https://docs.fiddler.ai/glossary/model-performance
Quantitative evaluation of AI model accuracy and effectiveness in production. Fiddler tracks performance metrics over time to detect degradation and identify opportunities for improvement.
[Model Performance](/observability/platform/performance-tracking-platform) refers to the evaluation of how well a machine learning model performs its intended task by comparing its predictions against actual outcomes. It involves measuring the accuracy, reliability, and effectiveness of a model using various metrics specific to the model type (classification, regression, ranking, etc.).
Model performance assessment is a critical component of the machine learning lifecycle, providing insights into a model's strengths, weaknesses, and overall utility. Poor model performance can have significant business implications, affecting decision quality, customer experience, and ultimately business outcomes. Effective performance monitoring helps detect degradation early, enabling timely interventions such as retraining or recalibration.
## How Fiddler Monitors Model Performance
Fiddler's AI Observability platform offers comprehensive model performance monitoring for various model types including binary classification, multi-class classification, regression, and ranking models. The platform provides out-of-the-box performance metrics suited to each model type and visualizes these metrics through charts and dashboards.
For classification models, Fiddler tracks metrics such as accuracy, precision, recall, F1 score, and AUC-ROC. For regression models, it monitors metrics like MSE, MAE, and R-squared. These metrics help users understand how well their models are performing in production, detect performance degradation, and make informed decisions about model maintenance or retraining.
## Why Model Performance Is Important
Model performance monitoring is essential for maintaining reliable and effective AI systems. As models encounter new data in production, their performance can degrade over time due to data drift, concept drift, or other factors. Continuous monitoring of model performance helps organizations identify issues early, understand their root causes, and take appropriate corrective actions.
* **Business Impact Assessment**: Model performance metrics help quantify the business impact of model predictions, enabling stakeholders to understand how well the model supports business objectives and where improvements might be needed.
* **Early Detection of Degradation**: Regular monitoring of performance metrics allows teams to quickly identify when a model's performance starts to deteriorate, enabling proactive intervention before significant business impact occurs.
* **Root Cause Analysis**: Performance metrics, especially when examined alongside other monitoring data like feature distributions and data integrity metrics, help pinpoint the underlying causes of performance issues.
* **Model Comparison**: Performance metrics provide a standardized way to compare different model versions or competing models to select the best performer for a specific use case.
* **Regulatory Compliance**: In regulated industries, monitoring and documenting model performance is often a requirement for demonstrating responsible AI practices and compliance with governance frameworks.
* **Continuous Improvement**: Performance metrics guide the model improvement process by highlighting specific areas where the model underperforms, helping teams focus their enhancement efforts effectively.
## Types of Model Performance Metrics
* **Binary Classification Metrics**: Metrics for evaluating models that predict one of two possible outcomes, including accuracy, precision, recall, F1 score, AUC-ROC, and confusion matrix-based measurements that help understand different aspects of classification performance.
* **Multi-class Classification Metrics**: Metrics for models that predict one of several classes, including accuracy, log loss, and class-specific precision and recall, often calculated using approaches like micro or macro averaging across classes.
* **Regression Metrics**: Metrics for models that predict continuous values, including Mean Squared Error (MSE), Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and R-squared, which measure different aspects of prediction accuracy and model fit.
* **Ranking Metrics**: Metrics for models that rank items by relevance, including Mean Average Precision (MAP) for binary relevance ranking and Normalized Discounted Cumulative Gain (NDCG) for evaluating the quality of ranking results.
* **Time-Series Performance Metrics**: Specialized metrics for time-series forecasting models, often focusing on error measurements across different time horizons and accounting for seasonal patterns in the data.
## Challenges
Monitoring model performance effectively presents several challenges, especially in production environments where models encounter diverse and evolving data.
* **Delayed Ground Truth**: In many applications, the actual outcomes (ground truth) needed to calculate performance metrics become available only after a significant delay, making real-time performance monitoring difficult.
* **Class Imbalance**: When the distribution of classes is heavily skewed, standard performance metrics may provide an overly optimistic view of model performance, requiring specialized metrics or approaches to properly evaluate imbalanced classification.
* **Changing Data Distributions**: As production data distributions shift over time (data drift), performance metrics may degrade, requiring monitoring solutions that can detect and quantify distribution changes along with performance changes.
* **Metric Selection**: Choosing the right metrics for a specific model and use case can be challenging, as different metrics emphasize different aspects of performance and may lead to different conclusions about model quality.
* **Threshold Optimization**: For classification models, performance often depends on the chosen decision threshold, requiring methods to optimize and adjust thresholds based on business requirements and changing data patterns.
* **Resource Constraints**: Computing performance metrics for large-scale models or high-volume data streams can be resource-intensive, requiring efficient implementation and potentially sampling strategies.
* **Interpretability of Metrics**: Some metrics, while mathematically sound, can be difficult for non-technical stakeholders to understand, requiring careful communication and translation to business impacts.
## Model Performance Monitoring How-to Guide
1. **Define Performance Objectives**
* Identify which aspects of model performance are most critical for your specific use case.
* Select appropriate metrics based on your model type and business requirements.
2. **Establish a Baseline**
* Measure and record the model's performance metrics during training/validation.
* Document the expected performance range for each metric to serve as a reference point.
3. **Configure Monitoring**
* Set up regular performance metric calculations on production data.
* Define appropriate time windows and aggregation levels for performance analysis.
4. **Set Up Alerting**
* Establish thresholds for performance metrics that would trigger alerts.
* Configure notification systems to alert relevant team members when performance deteriorates.
5. **Implement Root Cause Analysis**
* When performance issues are detected, investigate potential causes such as data drift or integrity issues.
* Use tools like Fiddler's dashboards to drill down into specific segments or features contributing to performance decline.
6. **Take Corrective Action**
* Based on root cause analysis, implement appropriate interventions such as model retraining, feature engineering, or data pipeline fixes.
* For temporary performance issues, consider adjustments like threshold tuning where appropriate.
## Frequently Asked Questions
**Q: How often should I monitor model performance?**
The optimal monitoring frequency depends on your specific use case, data volume, and business criticality. High-stakes applications might require daily or even real-time monitoring, while less critical models might be monitored weekly or monthly. Also consider the rate of expected data drift and availability of ground truth labels when determining monitoring frequency.
**Q: Which performance metrics should I prioritize?**
The most relevant metrics depend on your model type and business objectives. For classification models with balanced classes, accuracy, precision, recall, and F1 score are common choices. For regression models, MSE, MAE, and R-squared are typically used. Consider the business impact of different types of errors and prioritize metrics that align with your specific goals.
**Q: How do I know if my model's performance is good enough?**
Good performance is context-dependent. Compare your model's performance against relevant benchmarks, including baseline models (e.g., simple heuristics), previous model versions, industry standards, and business requirements. Define acceptable performance thresholds based on the criticality of the use case and the cost of errors.
**Q: What should I do when model performance drops?**
First, verify that the performance drop is statistically significant and not due to random variation. Then, investigate potential causes such as data drift, data quality issues, or changes in the underlying process. Depending on the root cause, solutions might include retraining the model, adjusting features, fixing data pipeline issues, or in some cases, reconsidering the modeling approach entirely.
## Related Terms
* [Data Drift](/glossary/data-drift)
* [Model Performance](/glossary/model-performance)
* [Model Drift](/glossary/model-drift)
* [Baseline](/glossary/baseline)
## Related Resources
* [Performance Tracking Platform Guide](/observability/platform/performance-tracking-platform)
* [Performance Charts Creation](/observability/analytics/performance-charts-creation)
* [Performance Charts Visualization](/observability/analytics/performance-charts-visualization)
* [Data Drift Platform Guide](/observability/platform/data-drift-platform)
* [ML Model Monitoring](https://www.fiddler.ai/ml-model-monitoring)
# Trust Score
Source: https://docs.fiddler.ai/glossary/trust-score
Quantitative scores generated by Fiddler's enrichment processes that measure LLM output quality and safety. These numerical metrics enable monitoring, alerting, and real-time decision-making for AI go
[Trust Scores](/observability/llm/enrichments) are quantitative measurements that result from Fiddler's enrichment processes, providing numerical assessments of LLM output quality, safety, and reliability. These scores translate complex qualitative judgments about generative AI content into measurable metrics that can be monitored over time, used in alerting systems, and leveraged for real-time content filtering decisions.
When [enrichments](/glossary/enrichment) process LLM inputs and outputs through Fiddler's platform, they generate Trust Scores that evaluate dimensions such as safety, toxicity, faithfulness, relevance, and coherence. Each score represents a quantified assessment that enables systematic monitoring and governance of LLM applications at scale.
Trust Scores serve as the critical link between automated evaluation and actionable insights, transforming the output of sophisticated evaluation models into interpretable metrics that teams can use to understand model behavior, set thresholds, configure alerts, and make real-time filtering decisions through Fiddler Guardrails.
## How Fiddler Uses Trust Scores
Trust Scores function as the primary interface between Fiddler's evaluation infrastructure and its monitoring and governance capabilities. Once enrichments generate these scores, they appear as additional data columns alongside original inference data, creating a comprehensive view of both model behavior and output quality.
**Monitoring and Analytics**: Trust Scores populate monitoring dashboards where teams can track quality trends over time, compare performance across different model versions, and identify degradation patterns. The scores enable sophisticated analytics that help organizations understand when and why their LLM applications produce lower-quality outputs.
**Alerting and Notifications**: Organizations configure alert rules based on Trust Score thresholds, enabling proactive notification when scores indicate potential quality or safety issues. These alerts can trigger various responses, from simple notifications to automated escalation procedures.
**Real-time Decision Making**: In Fiddler Guardrails, Trust Scores serve as the evaluation signals that determine whether content should be allowed or filtered. When scores indicate safety violations or quality concerns, Guardrails can automatically block problematic outputs or provide detailed explanations of detected issues.
**Threshold Management**: Trust Scores enable organizations to establish quantitative governance policies by setting acceptable score ranges for different use cases, risk profiles, and compliance requirements.
## Why Trust Scores Are Important
Trust Scores address the fundamental challenge of making subjective content quality assessments scalable and consistent. While human evaluators might assess LLM outputs differently based on context, experience, or interpretation, Trust Scores provide standardized measurements that enable reliable, automated governance at production scale.
**Objective Quality Assessment**: Trust Scores transform qualitative judgments into quantitative metrics, enabling systematic comparison of output quality across time periods, model versions, and different segments of user interactions.
**Operational Scalability**: Manual review of LLM outputs doesn't scale to production volumes. Trust Scores enable automated quality assessment of millions of interactions while maintaining consistent evaluation criteria.
**Risk Management**: By providing numerical thresholds for acceptable content quality and safety, Trust Scores enable proactive risk management that can prevent problematic outputs from reaching users.
**Compliance and Auditability**: Trust Scores create quantitative evidence of content evaluation that supports compliance reporting and provides auditable records of AI governance practices.
**Performance Optimization**: Score patterns reveal insights about model behavior that guide optimization efforts, helping teams identify when prompts, training data, or model configurations need adjustment.
## Types of Trust Scores
### Safety and Content Moderation Scores
**Safety Scores**: Numerical assessments indicating the likelihood that content contains harmful, inappropriate, or policy-violating material across multiple safety dimensions including violence, hate speech, and explicit content.
**Toxicity Scores**: Specialized measurements focusing specifically on offensive, toxic, or harmful language, often broken down into subcategories for more granular content moderation decisions.
**Bias Scores**: Quantitative indicators of potential bias in content, including gender, racial, political, or other forms of bias that might impact fairness and inclusivity.
### Quality and Accuracy Scores
**Faithfulness Scores**: Measurements of factual accuracy and reliability, indicating the likelihood that content contains hallucinations, fabrications, or factual inconsistencies relative to provided context. Fiddler provides two faithfulness approaches: Faithfulness (Centor Model) (powered by the Centor Model for Faithfulness, probability score 0.0–1.0) for real-time guardrails, and RAG Faithfulness (LLM-as-a-Judge, binary Yes/No with reasoning) for diagnostic evaluation.
**Coherence Scores**: Numerical assessments of logical flow, consistency, and readability, identifying content that may be disjointed, confusing, or poorly structured.
**Relevance Scores**: Metrics indicating how well content addresses the specific query or prompt, detecting off-topic or tangential responses. Includes Answer Relevance 2.0 (ordinal: High/Medium/Low) and Context Relevance (ordinal: High/Medium/Low) for RAG-specific evaluation.
### Contextual and Behavioral Scores
**Sentiment Scores**: Measurements of emotional tone and sentiment expressed in content, useful for detecting inappropriately negative or emotional responses in specific contexts.
**Confidence Scores**: Indicators of how certain the evaluation model is about its assessment, helping teams understand when scores may be less reliable.
**Custom Domain Scores**: Application-specific measurements that evaluate content against custom business logic or domain-specific quality criteria.
## Score Interpretation and Thresholds
### Understanding Score Ranges
Most Trust Scores use standardized ranges (typically 0-1 or 0-100) where higher scores generally indicate greater concern or lower quality, though specific interpretation depends on the score type:
* **Safety Scores**: Higher scores indicate greater safety risk
* **Faithfulness Scores**: Higher scores often indicate better factual accuracy (depending on implementation)
* **Quality Scores**: Interpretation varies by specific metric and use case
### Setting Effective Thresholds
**Risk-Based Thresholds**: Organizations should establish score thresholds based on their specific risk tolerance, regulatory requirements, and business context rather than using universal cutoffs.
**Adaptive Thresholds**: Score thresholds may need adjustment over time as models evolve, user expectations change, or business requirements shift.
**Multi-Score Policies**: Complex governance policies often require consideration of multiple Trust Scores simultaneously, using weighted combinations or hierarchical decision trees.
## Challenges in Trust Score Implementation
**Score Reliability**: While Trust Scores provide consistent measurements, their accuracy depends on the underlying evaluation models and may vary across different content types, domains, or edge cases.
**Threshold Calibration**: Determining appropriate score thresholds requires balancing false positives (blocking acceptable content) against false negatives (allowing problematic content), often requiring extensive testing with representative data.
**Score Evolution**: As language usage and societal standards evolve, Trust Score models may need updates to maintain alignment with current expectations, requiring periodic recalibration of thresholds.
**Context Sensitivity**: Content appropriateness often depends on context that may not be fully captured in individual scores, requiring careful consideration of how contextual factors should influence score interpretation.
**Multi-dimensional Analysis**: Monitoring multiple score dimensions simultaneously requires sophisticated dashboard design and alert logic to avoid information overload while ensuring comprehensive coverage.
## Trust Score Implementation Guide
1. **Establish Score Requirements**
* Identify which quality and safety dimensions are most critical for your use case
* Determine required score granularity and update frequency
* Consider regulatory and compliance requirements that may influence score interpretation
2. **Configure Score Generation**
* Enable appropriate enrichments during model onboarding to generate relevant Trust Scores
* Ensure score generation aligns with your monitoring and governance requirements
* Test score generation with representative data samples
3. **Calibrate Thresholds**
* Analyze score distributions across representative datasets to understand normal ranges
* Set initial thresholds based on risk tolerance and business requirements
* Validate threshold effectiveness through controlled testing
4. **Implement Monitoring**
* Create dashboards that effectively visualize score trends and distributions
* Configure alert rules based on score thresholds and trend analysis
* Establish escalation procedures for different types of score anomalies
5. **Optimize and Refine**
* Monitor false positive and false negative rates for threshold adjustments
* Regularly review score patterns to identify opportunities for improvement
* Update thresholds and interpretation guidelines based on operational experience
## Frequently Asked Questions
**Q: How are Trust Scores calculated?**
Trust Scores are generated by Fiddler's enrichment processes using specialized evaluation models. Each enrichment applies domain-specific evaluation logic to assess content quality, safety, or other dimensions, producing numerical scores that quantify the assessment results.
**Q: What's the difference between Trust Scores and enrichments?**
Enrichments are the computational processes that analyze content, while Trust Scores are the numerical outputs that result from those processes. Think of enrichments as the evaluation engines and Trust Scores as the quantified results they produce.
**Q: How reliable are Trust Scores compared to human evaluation?**
Fiddler's Trust Scores are designed to correlate strongly with human judgments while providing consistency and scalability that human evaluation cannot match. While no automated system perfectly replicates human assessment, Trust Scores provide reliable signals that align well with human evaluations across most common use cases.
**Q: Can I customize Trust Score thresholds for my organization?**
Yes, Fiddler allows organizations to set custom thresholds for Trust Scores based on their specific risk tolerance, compliance requirements, and business context. Threshold configuration is separate from score generation, enabling adaptation without changing underlying evaluation logic.
**Q: How do Trust Scores power Fiddler Guardrails?**
Trust Scores serve as the decision signals for Guardrails' real-time content filtering. When scores exceed configured safety thresholds or fall below quality requirements, Guardrails can automatically block content or provide detailed explanations of detected violations.
**Q: Do Trust Scores impact system performance?**
Trust Scores themselves are lightweight numerical values that have minimal impact on system performance. The computational cost comes from the enrichment processes that generate the scores, which are optimized for efficiency in Fiddler's infrastructure.
## Related Terms
* [Enrichments](/observability/llm/enrichments) - The computational processes that generate Trust Scores
* [Fiddler Guardrails](/glossary/guardrails) - Real-time protection using Trust Score evaluation
* [Fiddler Centor Models](/glossary/centor-models) - The purpose-built models powering score generation
* [LLM Observability](/glossary/llm-observability) - The monitoring practice enabled by Trust Scores
## Related Resources
* [LLM Monitoring Overview](/observability/llm)
* [Enrichments Documentation](/observability/llm/enrichments)
* [Selecting Enrichments Guide](/observability/llm/selecting-enrichments)
* [Guardrails for Content Protection](/glossary/guardrails)
* [LLM-based Metrics Guide](/observability/llm/llm-based-metrics)
# Introduction to Fiddler
Source: https://docs.fiddler.ai/index
The only platform delivering enterprise-grade visibility, context, and control across traditional ML models, LLM applications, and autonomous multi-agent systems.
## Get started in under 10 minutes
Choose your path based on your use case:
Monitor AI agents and multi-step workflows
Test and validate LLM outputs
Protect your AI applications
Monitor LLM applications in production
Monitor traditional ML models
***
## Explore Fiddler platform features
Validate LLM applications and models before and after deployment
Monitor production agentic applications, GenAI and LLM applications, and ML models in real time
Ensure AI safety and compliance
# Agentic AI Overview
Source: https://docs.fiddler.ai/integrations/agentic-ai-and-llm-frameworks/agentic-ai
Native SDKs and framework integrations for agentic AI and LLM applications
Monitor and evaluate your agentic AI applications with Fiddler's native SDKs and framework integrations. From auto-instrumented LangGraph agents to Strands agent applications, Fiddler provides comprehensive observability for the next generation of AI systems.
## Why Agentic Observability Matters
Agentic AI systems—autonomous agents that reason, plan, and coordinate—introduce exponential complexity compared to traditional AI applications:
* **[26x more monitoring resources](https://www.capgemini.com/insights/expert-perspectives/ai-lab-the-efficient-use-of-tokens-for-multi-agent-systems/)** required than single-agent systems
* **Non-deterministic behavior** makes traditional debugging approaches inadequate
* **Multi-step workflows** require hierarchical tracing across agents, tools, and LLM calls
* **Cascading failures** demand root cause analysis across distributed agent architectures
Fiddler's agentic observability provides visibility into every stage of the agent lifecycle: Thought → Action → Execution → Reflection → Alignment.
## Native SDKs
Fiddler-built and maintained instrumentation libraries for production-grade agentic observability.
### Fiddler OTel SDK
Core OpenTelemetry instrumentation library for framework-agnostic GenAI observability. The foundation package that all other Fiddler integrations build on.
**Best for:** Custom Python agents with no framework dependency, or any application where you want lightweight, decorator-based instrumentation
**Key Features:**
* `@trace` decorator for zero-boilerplate function instrumentation (sync and async)
* Typed span wrappers: `FiddlerGeneration`, `FiddlerTool`, `FiddlerChain`
* Context isolation — does not interfere with any existing OpenTelemetry setup
* `set_conversation_id()` for multi-turn conversation tracking
* JSONL local capture and console tracing for development
[**Get Started with Fiddler OTel SDK →**](/integrations/agentic-ai/fiddler-otel-sdk)
### Fiddler LangChain SDK
Auto-instrumentation for LangChain V1 agents built with `langchain.agents.create_agent`.
**Best for:** LangChain V1 agents that use the `create_agent` API
**Key Features:**
* One call to `FiddlerLangChainInstrumentor.instrument()` auto-traces all agents
* Clean, flat trace hierarchy: agent → LLM calls → tool calls, no noisy Chain wrappers
* Full async support via `agent.ainvoke()`
* Single-trace multi-agent nesting — sub-agents nest under delegation tool spans automatically
* Retriever-as-tool support
[**Get Started with Fiddler LangChain SDK →**](/integrations/agentic-ai/langchain-sdk)
### Fiddler LangGraph SDK
Auto-instrument LangGraph applications with OpenTelemetry-based tracing.
**Best for:** LangChain LangGraph agent applications with complex multi-agent workflows
**Key Features:**
* Automatic span creation for agent steps, tool calls, and LLM requests
* Hierarchical tracing across Application → Session → Agent → Span levels
* Zero-configuration setup with one environment variable
* Full context preservation for debugging non-deterministic behavior
[**Get Started with LangGraph SDK →**](/integrations/agentic-ai/langgraph-sdk)
### Strands Agents SDK
Native integration for Strands Agents applications.
**Best for:** Teams building agents with the Strands framework
**Key Features:**
* Purpose-built for Strands agent architecture
* Seamless integration with Strands agent runtime
* Multi-agent coordination tracking
* Platform-agnostic deployment (works on AWS, custom infrastructure, etc.)
[**Get Started with Strands Agents SDK →**](/integrations/agentic-ai/strands-sdk)
### Google ADK SDK
Native integration for Google ADK (Agent Development Kit) applications.
**Best for:** Teams building agents with Google's ADK framework and Gemini models
**Key Features:**
* Two-line setup with `FiddlerClient` + `GoogleADKInstrumentor`
* Works with Gemini API keys and Vertex AI authentication
* Automatic capture of agent, LLM, and tool spans
* Session identity propagation across multi-turn conversations
* No monkey-patching -- pure OpenTelemetry SpanProcessor approach
[**Get Started with Google ADK SDK →**](/integrations/agentic-ai/google-adk-sdk)
### Claude Code Integration
Zero-instrumentation integration for teams using [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) — Anthropic's CLI coding agent. Captures LLM calls, tool invocations, user prompts, token usage, and permission decisions via Claude Code's built-in OpenTelemetry tracing.
**Best for:** Teams deploying Claude Code for software engineering and wanting visibility into coding agent sessions, tool usage patterns, and safety compliance
**Key Features:**
* **Zero code changes** — Claude Code emits OTel traces natively; just set environment variables
* Captures user prompts, LLM call metadata (tokens, latency, model), and tool invocations
* Session correlation via `session.id` for multi-turn coding session replay
* Permission decision tracking (accept/deny) for tool invocations
* Compatible with Fiddler enrichment rules (Prompt Safety scoring on agent spans)
Claude Code's OTel tracing is in **beta** and has limitations — LLM response content and tool input/output are not available in traces. See the integration guide for full details.
[**Get Started with Claude Code Integration →**](/integrations/agentic-ai/claude-code-integration)
### Fiddler Evals SDK
LLM experiments framework with pre-built evaluators and custom eval support.
**Best for:** Offline evaluation of LLM applications and agentic workflows
**Key Features:**
* 14+ pre-built evaluators (faithfulness, toxicity, PII, coherence, etc.)
* Custom evaluator framework for domain-specific metrics
* Batch evaluation for datasets
* Integration with the Fiddler platform for tracking and comparison
[**Get Started with Evals SDK →**](/integrations/agentic-ai/evals-sdk)
## AI Gateways
Already routing LLM traffic through a third-party AI gateway? Fiddler plugs in at the proxy layer for observability and guardrail enforcement — **Kong AI Gateway**, **AgentGateway**, and **LiteLLM** — with no application code changes, and independently of the SDKs above. (This is distinct from the [Fiddler LLM Gateway](/reference/administration/llm-gateway), which manages Fiddler's own provider credentials.)
[**AI Gateway Integrations →**](/integrations/agentic-ai/ai-gateways) — category overview, capability comparison, and per-gateway setup links.
## Platform SDKs
Core API access for building custom integrations and monitoring workflows.
### Python Client SDK
Comprehensive Python client for all Fiddler platform capabilities.
**Best for:** Custom integrations, ML model monitoring, programmatic access to Fiddler features
**Key Features:**
* Full API coverage for ML and LLM monitoring
* Dataset uploads, model publishing, event ingestion
* Alert configuration, dashboard management
* Custom metrics and enrichments
[**Python Client Documentation →**](/sdk-api/python-client/connection)
### REST API
Complete HTTP API for language-agnostic platform access.
**Best for:** Non-Python environments, webhook integrations, custom tooling
[**REST API Reference →**](/sdk-api/rest-api)
## Advanced Integrations
### S3 Trace Ingestion
Ingest pre-generated OTLP trace files from Amazon S3 into Fiddler without modifying your application. The S3 connector automatically discovers, parses, and forwards trace files to the Fiddler platform.
**Key Features:**
* Automatic file discovery — no manual trigger required
* Supports both base64 and hex-encoded `traceId`/`spanId` fields
* IAM role-based authentication (cross-account supported)
* Per-file retry logic with status tracking via API
* Compatible with any OTLP JSON producer
[**Get Started with S3 Trace Ingestion →**](/integrations/agentic-ai/s3-trace-ingestion)
### OpenTelemetry Integration
Direct OTLP integration for custom agent frameworks and multi-framework environments.
**Best for:** Multi-framework environments, custom agentic frameworks, advanced users requiring full instrumentation control
**Key Features:**
* Vendor-neutral telemetry using OpenTelemetry standards
* Manual span creation for complete control over instrumentation
* Multi-framework support for custom and emerging agent frameworks
* Compatible with existing OpenTelemetry infrastructure
* Attribute mapping to Fiddler semantic conventions
**When to Use OpenTelemetry vs SDKs**
Use OpenTelemetry integration for advanced use cases requiring manual control. For LangGraph and Strands applications, we recommend using the dedicated SDKs for easier setup and automatic instrumentation.
[**Get Started with OpenTelemetry →**](/integrations/agentic-ai/opentelemetry-integration)
### Exporting OTel Traces to Fiddler
Client-side export path for shipping pre-existing OpenTelemetry traces to Fiddler from your own storage or pipeline — map span attributes to Fiddler's schema and POST to the `v1/traces` endpoint.
**Best for:** Replaying traces from a data warehouse, JSONL files, or a logging pipeline; custom export pipelines; batch backfill of historical trace data
[**Exporting OTel Traces to Fiddler →**](/integrations/agentic-ai/otel-trace-export)
## Framework Support
While Fiddler provides native SDKs for LangGraph and Strands, agentic applications can be monitored regardless of framework:
### Supported Frameworks & Tools
**AI Agent Frameworks:**
* **LangGraph** - Native SDK with auto-instrumentation ✓
* **LangChain V1** (`create_agent`) - Native SDK with auto-instrumentation ✓
* **Custom Python agents** - [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) with `@trace` decorator ✓
* **Other agentic frameworks** - [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) is the recommended path for any custom or unsupported framework
**LLM Provider SDKs:**
* **OpenAI SDK** - Track via Python Client or custom instrumentation
* **Anthropic SDK** - Monitor Claude API calls via Python Client
* **Strands Agents** - Native Strands Agents SDK ✓
* **LiteLLM SDK** - [Built-in OTel integration, no Fiddler package required](/integrations/agentic-ai/litellm-integration) ✓
* **Claude Code** - [Zero-instrumentation coding agent integration](/integrations/agentic-ai/claude-code-integration) ✓ — beta OTel tracing, limited content capture
**Gateways & Proxies:** (see [AI Gateway Integrations](/integrations/agentic-ai/ai-gateways) for the capability comparison)
* **Kong AI Gateway** - [Gateway-layer OTel export and guardrails](/integrations/agentic-ai/kong-integration) ✓
* **AgentGateway** - [Proxy-layer tracing, guardrails, and MCP tool tracing](/integrations/agentic-ai/agentgateway-integration) ✓
* **LiteLLM** - [SDK or proxy tracing and guardrails](/integrations/agentic-ai/litellm-integration) ✓
**Observability Standards:**
* **OpenTelemetry** - [Full OTLP support](/integrations/agentic-ai/opentelemetry-integration) for custom instrumentation
* **Custom Tracing** - Python Client API for framework-agnostic monitoring
## Integration Selector
Not sure which SDK to use? Here's a quick decision guide:
| Your Use Case | Recommended Integration | Why |
| ------------------------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| LangGraph agent application | [**LangGraph SDK**](/integrations/agentic-ai/langgraph-sdk) | Auto-instrumentation, zero config, hierarchical tracing |
| LangChain V1 (`create_agent`) | [**LangChain SDK**](/integrations/agentic-ai/langchain-sdk) | One `instrument()` call, flat clean traces, full async |
| Custom Python agent, no framework | [**Fiddler OTel SDK**](/integrations/agentic-ai/fiddler-otel-sdk) | `@trace` decorator, typed span wrappers, context isolation |
| Strands Agents | **Strands Agents SDK** | Purpose-built for Strands framework |
| LLM experiment workflows | **Evals SDK** | Pre-built evaluators, batch processing, tracking |
| LiteLLM SDK (direct calls) | [**LiteLLM Integration**](/integrations/agentic-ai/litellm-integration) | One-line setup, no Fiddler package required, native OTel support |
| LiteLLM proxy / gateway | [**LiteLLM Integration**](/integrations/agentic-ai/litellm-integration) | Zero-code, auto-detects proxy traces, cost attribution |
| AgentGateway | [**AgentGateway Integration**](/integrations/agentic-ai/agentgateway-integration) | Zero code changes, proxy-layer LLM and MCP tool tracing, session grouping |
| Kong AI Gateway | [**Kong AI Gateway Integration**](/integrations/agentic-ai/kong-integration) | Zero code changes, gateway-layer OTel export, multi-provider |
| Guardrail enforcement at your gateway | [**AI Gateway Guardrails**](/integrations/agentic-ai/ai-gateways) | Block or redact PII and secrets inline (Kong, AgentGateway, LiteLLM) |
| Claude Code (coding agent) | [**Claude Code Integration**](/integrations/agentic-ai/claude-code-integration) | Zero code changes, session replay, prompt safety enrichment (beta, limited content) |
| Multi-framework / raw OTel | [**OpenTelemetry Integration**](/integrations/agentic-ai/opentelemetry-integration) | Standards-based manual tracing, multi-framework environments |
| ECS Fargate / air-gapped / S3 batch | [**S3 Trace Ingestion**](/integrations/agentic-ai/s3-trace-ingestion) | No direct connection needed, file-based async ingestion |
| Pre-existing OTel traces to replay | [**Exporting OTel Traces**](/integrations/agentic-ai/otel-trace-export) | Client-side attribute mapping + batch export to `v1/traces` |
| Traditional ML monitoring | **Python Client** | ML-specific features, drift detection, explainability |
## Getting Started
### Quick Start Paths
1. **Custom Python Agents (Fiddler OTel SDK)**
```bash theme={null}
pip install fiddler-otel
```
```python theme={null}
from fiddler_otel import FiddlerClient, trace
client = FiddlerClient(api_key="...", application_id="...", url="...")
@trace(as_type="generation")
def call_llm(prompt: str) -> str:
...
```
[Full Fiddler OTel SDK Guide →](/integrations/agentic-ai/fiddler-otel-sdk)
2. **LangChain V1 Applications**
```bash theme={null}
pip install fiddler-langchain
```
```python theme={null}
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
import langchain.agents
client = FiddlerClient(api_key="...", application_id="...", url="...")
FiddlerLangChainInstrumentor(client=client).instrument()
# All create_agent() calls are now traced automatically
```
[Full LangChain SDK Guide →](/integrations/agentic-ai/langchain-sdk)
3. **LangGraph Applications**
```bash theme={null}
pip install fiddler-langgraph
```
```python theme={null}
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
client = FiddlerClient(api_key="...", application_id="...", url="...")
LangGraphInstrumentor(client).instrument()
```
[Full LangGraph Quick Start →](/integrations/agentic-ai/langgraph-sdk)
4. **Strands Agents**
```bash theme={null}
pip install fiddler-strands
# Configure for your Strands Agent
```
[Full Strands Agents SDK Quick Start →](/integrations/agentic-ai/strands-sdk)
5. **LLM Experiments**
```bash theme={null}
pip install fiddler-evals
# Run experiments on your dataset
```
[Full Evals Quick Start →](/integrations/agentic-ai/evals-sdk)
6. **LiteLLM SDK**
```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-fiddler-instance.com"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ,fiddler-application-id="
export OTEL_RESOURCE_ATTRIBUTES="application.id="
```
```python theme={null}
import litellm
litellm.callbacks = ["otel"] # Traces flow to Fiddler automatically
```
[Full LiteLLM SDK Quick Start →](/integrations/agentic-ai/litellm-integration#litellm-sdk-integration)
7. **LiteLLM Proxy**
```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-fiddler-instance.com"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ,fiddler-application-id="
export OTEL_RESOURCE_ATTRIBUTES="application.id="
litellm --config config.yaml # Traces flow to Fiddler automatically
```
[Full LiteLLM Proxy Quick Start →](/integrations/agentic-ai/litellm-integration#litellm-proxy-integration)
8. **AI Gateways (Kong, AgentGateway, LiteLLM)**
Route LLM traffic through a gateway you already run instead of instrumenting application code. See [AI Gateway Integrations →](/integrations/agentic-ai/ai-gateways) for setup and a capability comparison.
9. **Raw OpenTelemetry (Advanced)**
```bash theme={null}
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
# Configure OTLP endpoint and instrument your agent
```
[Full OpenTelemetry Quick Start →](/developers/quick-starts/opentelemetry-quick-start)
## What's Next?
* [**Span and Resource Attributes**](/integrations/agentic-ai/attributes) - Understand required fields, value typing, custom attributes, and how they flow into metrics and alerts
* [**Agentic Observability Concepts**](/glossary/agentic-observability) - Understand the agent lifecycle and monitoring approach
* [**Agentic Observability Quick Start**](/getting-started/agentic-monitoring) - Complete setup guide
* [**Centor Models Overview**](/glossary/centor-models) - Learn about the evaluation platform powering Fiddler
# AgentGateway Integration
Source: https://docs.fiddler.ai/integrations/agentic-ai/agentgateway-integration
Integrate AgentGateway with Fiddler for zero-instrumentation LLM observability — no application code changes required.
## Overview
[AgentGateway](https://agentgateway.dev/) (v1.1.0 or later, Apache 2.0) is an open-source Rust proxy that sits between your application and its LLM provider. Fiddler integrates with AgentGateway at the proxy layer, giving you full LLM observability — prompts, responses, token usage, latency — without adding any SDK to your application code.
| Capability | Notes |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Zero instrumentation** | Point your app at AgentGateway instead of `api.openai.com` — no code changes needed |
| **LLM span tracing** | Token counts, model name, latency, and content (with CEL config) |
| **MCP tool call tracing** | Each MCP `tools/call` proxied by AgentGateway is captured as a Fiddler `tool` span with tool name, input, and output (with CEL config) |
| **Session grouping** | All calls sharing a conversation ID are grouped into one Fiddler Session |
| **Direct OTLP export** | AgentGateway exports traces to Fiddler over HTTPS with auth headers injected via `requestHeaderModifier` |
***
## Architecture
```mermaid theme={null}
flowchart TD
A["Your app (any language, standard OpenAI client)"] -->|points base_url at AgentGateway| B["AgentGateway (Rust proxy, port 4000)"]
B -->|proxies to OpenAI / Anthropic / Bedrock / …| C["LLM Provider"]
B -->|OTLP/HTTPS with auth headers| D["Fiddler (enrichment, monitoring, dashboards)"]
```
AgentGateway exposes an OpenAI-compatible API (`/v1/chat/completions`). Your application requires no SDK — it just calls the proxy instead of the provider directly.
***
## Prerequisites
* Fiddler account with a GenAI application already created
* [AgentGateway](https://agentgateway.dev/) **v1.1.0+** for tracing. If you also plan to add [Fiddler guardrails](/protection/agentgateway-guardrails), which require **v1.4.0+**, start on v1.4.0+ from the outset — a 1.1.0-era gateway silently misroutes the guardrail webhook.
* A valid LLM provider API key (e.g. `OPENAI_API_KEY`)
* Your Fiddler API key (found under organizational settings) and application UUID (found under application settings)
***
## Quick Start
### Step 1 — Install AgentGateway
```bash theme={null}
# macOS / Linux (Homebrew)
brew install agentgateway/tap/agentgateway
# Or download from https://github.com/agentgateway/agentgateway/releases
agentgateway --version # must be v1.1.0+
```
### Step 2 — Configure AgentGateway
Create `agentgateway_config.yaml`:
```yaml theme={null}
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
tracing:
# Fiddler instance host and port (e.g., your-instance.fiddler.ai:443).
# $FIDDLER_HOST is expanded from the environment.
host: "$FIDDLER_HOST"
protocol: http
randomSampling: true
policies:
# Inject auth headers on every OTLP export request to Fiddler.
# $FIDDLER_API_KEY and $FIDDLER_APP_ID are expanded from the environment.
requestHeaderModifier:
add:
Authorization: "Bearer $FIDDLER_API_KEY"
fiddler-application-id: "$FIDDLER_APP_ID"
# Enable TLS for the HTTPS connection to Fiddler.
backendTLS: {}
resources:
service.name: '"agentgateway"'
application.id: '"$FIDDLER_APP_ID"'
attributes:
# Prompt and completion content, following the OpenTelemetry GenAI
# convention. Model, provider, token usage, request parameters
# (temperature, max_tokens), and gen_ai.operation.name are emitted by
# AgentGateway automatically — no CEL is required for those.
gen_ai.input.messages: toJson(llm.prompt)
gen_ai.output.messages: toJson(llm.completion)
# Tool definitions offered on the request, if any.
gen_ai.tool.definitions: 'toJson(json(request.body).tools)'
# Classify the span as an LLM call and group it into a Session.
fiddler.span.type: '"llm"'
gen_ai.conversation.id: |
request.headers["x-fiddler-conversation-id"] != "" ? request.headers["x-fiddler-conversation-id"] : ""
span.name: |
"chat " + llm.requestModel
binds:
- port: 4000
listeners:
- routes:
- backends:
- ai:
name: openai
provider:
openAI:
model: gpt-5-mini
policies:
backendAuth:
passthrough: {}
```
The `frontendPolicies.tracing` block captures prompt and response content via CEL expressions and exports spans directly to Fiddler over HTTPS. `$FIDDLER_API_KEY` and `$FIDDLER_APP_ID` are expanded from environment variables at runtime — no credentials are hardcoded in the config file.
The `binds` block above is AgentGateway's low-level backend API. AgentGateway v1.4.0+ deprecates it in favor of the `gateways`/`routes` and `llm.models` shapes (see AgentGateway's *Migrate from binds* guide); it still works, but if you also run [Fiddler guardrails](/protection/agentgateway-guardrails) — which attach via `llm.models[].guardrails` — define your model once in the `llm.models` shape and combine the two, per [Combined Tracing and Guardrails](#combined-tracing-and-guardrails).
### Step 3 — Start AgentGateway
```bash theme={null}
export OPENAI_API_KEY="sk-..."
export FIDDLER_API_KEY="your-fiddler-api-key"
export FIDDLER_APP_ID="your-application-uuid"
export FIDDLER_HOST="your-instance.fiddler.ai:443"
agentgateway -f agentgateway_config.yaml
```
### Step 4 — Point Your Application at AgentGateway
```python theme={null}
import os
import uuid
from openai import OpenAI
# OPENAI_API_KEY is read from the environment automatically.
# AgentGateway's backendAuth passthrough forwards it unchanged to OpenAI,
# so the key must be set in both the AgentGateway environment and here.
client = OpenAI(base_url=os.getenv("AGENTGATEWAY_URL", "http://localhost:4000/v1"))
# Generate one conversation UUID per session and send it on every LLM call
# so Fiddler groups them into a single Session.
session_id = str(uuid.uuid4())
response = client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": "What is the capital of France?"}],
extra_headers={"X-Fiddler-Conversation-Id": session_id},
)
print(response.choices[0].message.content)
```
Traces appear in Fiddler automatically — no SDK import, no callback registration, no changes to your application logic.
To override the default proxy URL, set `AGENTGATEWAY_URL` (defaults to `http://localhost:4000/v1`):
```bash theme={null}
export AGENTGATEWAY_URL="http://my-gateway-host:4000/v1"
```
### Step 5 — Verify Traces Are Arriving
Open the Fiddler UI and navigate to your application's **Explorer**. You should see the trace within a few seconds of making your first completion call.
***
## Span Type Mapping
Fiddler's AgentGateway mapper enriches spans based on `gen_ai.operation.name`, which AgentGateway sets automatically:
| `gen_ai.operation.name` | AgentGateway mapper treatment |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `chat` | Enriched and classified `llm` |
| `completion` | Enriched and classified `llm` |
| `execute_tool` (MCP tool calls) | Not enriched by this mapper; classified `tool` downstream — see [MCP Tool Call Tracing](#mcp-tool-call-tracing) |
| anything else | Not enriched by this mapper |
The mapper sets `fiddler.span.type = "llm"` on LLM spans internally — no CEL config is required for that classification. (The `fiddler.span.type: '"llm"'` line in the CEL config above is a safety net for deployments that process spans without the dedicated mapper.) Spans the mapper does not enrich are still **forwarded, not dropped**: an `execute_tool` span is classified as a `tool` span by Fiddler's downstream span-type mapping (the same mapping the Strands and Google ADK integrations use).
***
## Attribute Mapping
AgentGateway emits the [OpenTelemetry GenAI inference](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md#inference) attributes, and Fiddler's semantic layer maps them automatically:
| AgentGateway attribute | Emitted | Fiddler mapping |
| ---------------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------ |
| `gen_ai.request.model` / `gen_ai.response.model` | Automatic | Model name |
| `gen_ai.provider.name` | Automatic | Provider (also aliased to `gen_ai.system`) |
| `gen_ai.operation.name` | Automatic | Span type (`chat` → `llm`) |
| `gen_ai.usage.input_tokens` / `output_tokens` | Automatic | Token counts |
| `gen_ai.usage.total_tokens` | Automatic (or computed as `input + output`) | Total tokens |
| `gen_ai.request.temperature` / `max_tokens` | Automatic (when set on the request) | Request parameters |
| `gen_ai.input.messages` | CEL (`toJson(llm.prompt)`) | Input content |
| `gen_ai.output.messages` | CEL (`toJson(llm.completion)`) | Output content |
| `gen_ai.conversation.id` | CEL (from `X-Fiddler-Conversation-Id` header) | Session grouping |
| `fiddler.span.type` | CEL (`"llm"`) | Span type |
| `gateway`, `listener`, `route`, `endpoint`, `http.*`, `duration`, `src.addr` | Automatic | Passed through as raw routing metadata |
Only prompt/completion content, `conversation.id`, and `fiddler.span.type` need CEL — everything else is emitted by AgentGateway by default.
***
## Session Grouping
Fiddler groups all LLM calls that share the same `gen_ai.conversation.id` into a single Session. The recommended pattern is to generate one UUID per logical conversation in your application and pass it on every LLM call as the `X-Fiddler-Conversation-Id` HTTP header. The CEL expression in the AgentGateway config (see Step 2) extracts the header and stamps it as the span attribute.
The header transport is preferred over OpenAI's `metadata` request body field because:
* OpenAI's `metadata` parameter requires `store=true`, which persists conversation data on OpenAI's side — a privacy concern for many customers.
* AgentGateway is a passthrough proxy: anything in the request body must be a valid OpenAI parameter or the request fails.
* Headers are visible to AgentGateway and silently stripped by OpenAI.
***
## MCP Tool Call Tracing
AgentGateway can also proxy [MCP](https://modelcontextprotocol.io/) (Model Context Protocol) tool servers. With the tracing config below, each MCP tool call is captured in Fiddler as a **`tool` span** — with the tool name, input arguments, and result.
### Configure an MCP Route
```yaml theme={null}
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
tracing:
host: "$FIDDLER_HOST"
protocol: http
randomSampling: true
policies:
requestHeaderModifier:
add:
Authorization: "Bearer $FIDDLER_API_KEY"
fiddler-application-id: "$FIDDLER_APP_ID"
backendTLS: {}
resources:
service.name: '"agentgateway"'
application.id: '"$FIDDLER_APP_ID"'
attributes:
gen_ai.operation.name: |
mcp.methodName == "tools/call" ? "execute_tool" : ""
gen_ai.tool.name: mcp.tool.name
gen_ai.tool.call.arguments: mcp.tool.arguments
gen_ai.tool.call.result: mcp.tool.result
mcp.tool.error: mcp.tool.error
gen_ai.conversation.id: mcp.sessionId
span.name: |
mcp.methodName == "tools/call" ? "tool " + mcp.tool.name : mcp.methodName
binds:
- port: 3000
listeners:
- routes:
- backends:
- mcp:
targets:
- name: my-tools
stdio:
cmd: npx
args: ["-y", "@modelcontextprotocol/server-everything"]
```
### What Gets Captured
The attributes above follow the OpenTelemetry GenAI [execute-tool span](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md#execute-tool-span) convention. Each MCP `tools/call` is recorded as a `tool` span:
| Attribute | Captured as |
| ---------------------------------------- | ------------------------------------------------------------------- |
| `gen_ai.operation.name` (`execute_tool`) | Classifies the span as a `tool` |
| `gen_ai.tool.name` | Tool name |
| `gen_ai.tool.call.arguments` | Tool input (arguments) |
| `gen_ai.tool.call.result` | Tool output (result) |
| `mcp.tool.error` | Tool-call error payload (present only on a JSON-RPC protocol error) |
AgentGateway additionally emits the [MCP-convention](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md) fields on the span automatically — `mcp.method.name`, `mcp.session.id`, `mcp.resource.type`, and `mcp.target` — so no extra CEL is needed for those.
Tool calls that share an MCP session are grouped into one Fiddler Session via `gen_ai.conversation.id`, set here from the MCP session ID (`mcp.sessionId`).
Fiddler classifies these as `tool` spans from `gen_ai.operation.name = execute_tool` — the same downstream classification used for the Strands and Google ADK integrations. Fiddler's AgentGateway mapper enriches only LLM (`chat`/`completion`) spans, so a tool span carries exactly the attributes your CEL config captures above; nothing is added or normalized by the mapper. There is no AgentGateway-specific mapper test for the tool-span path, so re-verify MCP tracing after an AgentGateway or Fiddler upgrade.
***
## Combined Tracing and Guardrails
Tracing and [Fiddler guardrails](/protection/agentgateway-guardrails) are independent, and both can run from the same AgentGateway config file. AgentGateway's schema declares `frontendPolicies`, `backends`, and `llm` as siblings and validates a file that combines them, so you add the guardrail webhook alongside the tracing block:
* Keep the `frontendPolicies.tracing` block from the [Quick Start](#quick-start).
* Add the `fiddler-guardrail` backend and the `llm.models[].guardrails` webhook from the [AgentGateway Guardrails](/protection/agentgateway-guardrails#quick-start) quick start.
* Define your LLM backend **once**, in the `llm.models` shape (guardrails attach there), rather than in both `binds` and `llm.models`.
This combination is valid against AgentGateway's v1.4.0 configuration schema. Validate the combined file against your AgentGateway version before production use, and make sure the tracing export and the LLM listener do not both claim port 4000.
***
## Troubleshooting
**Traces not appearing in Fiddler**
Verify all three environment variables are set before starting AgentGateway:
```bash theme={null}
echo $FIDDLER_API_KEY
echo $FIDDLER_APP_ID
echo $FIDDLER_HOST
echo $OPENAI_API_KEY
```
Both `application.id` (OTel resource attribute) and `fiddler-application-id` (HTTP header on the export request) are required. If `application.id` is missing, Fiddler skips those spans, logs a warning, and increments a `missing_application_id_failure` counter — so check your Fiddler ingestion logs and metrics if traces are absent. Fiddler checks that the attribute is present; it does not validate the UUID's format at this stage.
**Prompt and response content not showing**
The `frontendPolicies.tracing.attributes` CEL block is required. Verify it is present in `agentgateway_config.yaml` and that AgentGateway is v1.1.0+:
```bash theme={null}
agentgateway --version
```
**Span type showing as Unknown**
`service.name: '"agentgateway"'` in the `resources` block is how Fiddler routes AgentGateway spans to its mapper, and setting it is the documented default on every release — rename it and Fiddler no longer recognizes the spans. From 26.17, Fiddler also recognizes the `agentgateway` instrumentation scope; scope-based routing is evaluated first, so spans route correctly even if a deployment overrides `service.name`. Within the mapper, the span is then classified from `gen_ai.operation.name` (set automatically by AgentGateway). Verify `service.name: '"agentgateway"'` is present, that AgentGateway is v1.1.0+, and that the `frontendPolicies.tracing.attributes` block is configured. As a fallback, `fiddler.span.type: '"llm"'` in the `attributes` block covers deployments that process spans without the dedicated mapper.
**Not all LLM calls are producing traces**
`randomSampling: true` is active. Set it to `false` to capture every span:
```yaml theme={null}
frontendPolicies:
tracing:
randomSampling: false
```
***
## Known Limitations
| Limitation | Details |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Scope** | Captures LLM proxy traffic and MCP tool calls. Fiddler has no A2A-specific handling: AgentGateway's agent-to-agent spans are not dropped by Fiddler, but without CEL attributes classifying them they surface (if at all) as unclassified spans rather than a dedicated A2A type. |
| **Content requires CEL config** | Prompt, response, and tool content are only captured when `frontendPolicies.tracing.attributes` is configured in AgentGateway |
| **Sampling** | `randomSampling: true` means not every call produces a trace; set to `false` for complete capture |
***
## Next Steps
* [AgentGateway Guardrails](/protection/agentgateway-guardrails) — redact or block PII and secrets via AgentGateway's webhook guardrail protocol.
* [Kong AI Gateway Integration](/integrations/agentic-ai/kong-integration) — Fiddler observability through the Kong AI Gateway proxy.
* [LiteLLM Integration](/integrations/agentic-ai/litellm-integration) — Fiddler observability through the LiteLLM proxy.
* [LangGraph SDK](/integrations/agentic-ai/langgraph-sdk) — auto-instrumentation for LangGraph agent applications.
* [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration) — instrument a custom framework with the OTel SDK.
* [OTel Trace Export](/integrations/agentic-ai/otel-trace-export) — export OTLP directly to Fiddler without a proxy.
> **Upstream-link maintenance.** AgentGateway's `latest` standalone docs are live-edited and carry no version-pinned URLs (`/docs/standalone/1.4.x/` returns 404), so links to `agentgateway.dev/docs/standalone/latest/...` can silently retarget when a new AgentGateway version ships. Re-verify AgentGateway behavior cited here against your installed version. Official docs: [agentgateway.dev](https://agentgateway.dev/).
# AI Gateway Integrations
Source: https://docs.fiddler.ai/integrations/agentic-ai/ai-gateways
How Fiddler plugs into a third-party AI gateway (Kong, AgentGateway, LiteLLM) for observability and guardrail enforcement — and how that differs from the Fiddler LLM Gateway.
## What This Is
A **third-party AI gateway** is an inline proxy your application already routes LLM traffic
through. Fiddler plugs in at that layer, so you get observability, guardrail enforcement, or both —
with **no application code changes**. Fiddler documents three: [Kong AI Gateway](/integrations/agentic-ai/kong-integration),
[AgentGateway](/integrations/agentic-ai/agentgateway-integration), and [LiteLLM](/integrations/agentic-ai/litellm-integration).
**Not to be confused with the Fiddler LLM Gateway.** The [Fiddler LLM Gateway](/reference/administration/llm-gateway)
manages *Fiddler's own* provider credentials for evaluators and enrichments — Fiddler calling out to
model providers. A third-party AI gateway proxies *your* traffic to *your* models. This page is about
the latter.
## Two Capabilities, Independent
Each integration offers up to two capabilities, and either can be deployed without the other:
| Capability | What it does | How it runs |
| --------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **Observability (tracing)** | Captures prompts, responses, tokens, and latency as spans | Out-of-band — the gateway exports OTel spans to Fiddler |
| **Guardrail enforcement** | Blocks or redacts PII and secrets | Inline — the gateway calls Fiddler's guardrail adapter on each request/response |
```mermaid theme={null}
flowchart LR
App["Your app"] --> GW["AI gateway"]
GW -->|inline guardrail check| FA["Fiddler guardrail adapter"]
GW --> LLM["LLM provider"]
GW -.->|out-of-band OTel spans| FO["Fiddler observability"]
```
## Which Gateways
Tracing on Kong 3.13+. Guardrails need Kong 3.14+ with an **AI Gateway Enterprise** license. Live today. · [Guardrails](/protection/kong-guardrails)
Apache 2.0. Tracing on v1.1.0+. Guardrails need v1.4.0+ and **Fiddler 26.17+**. · [Guardrails](/protection/agentgateway-guardrails)
Open source. SDK or proxy. Live today. · [Guardrails](/protection/litellm-guardrails)
## Capability Comparison
Each page is authoritative for its own exact behavior and version floors; this table is a starting
point, not a substitute.
| | Kong | AgentGateway | LiteLLM |
| ---------------------------------- | ---------------------------------- | ---------------------------------------- | ---------------------- |
| Enforcement outcome | Blocked | Redacted or blocked | Redacted or blocked |
| Failure control | None (fail-open) | Per-route header + gateway `failureMode` | Per-request |
| Streaming redaction | N/A (blocks only) | No-op on streamed responses | Not on streamed output |
| Session grouping | `pre-function` plugin | CEL from a header | Not supported |
| MCP tool tracing | — | Yes (via downstream classification) | Yes |
| Configurability | None (2 wire fields) | Per-route static headers | Per-request body |
| Edition / license | AI Gateway Enterprise (guardrails) | Apache 2.0 | MIT (open source) |
| Product-visible guardrail decision | Customer-emitted span | None | Mapper-emitted |
| Guardrails availability | Live | Fiddler 26.17+ | Live |
## Which Checks Run
Across all three gateways, Fiddler's guardrail adapters run two checks: **PII** and **secrets**.
Faithfulness — offered elsewhere in [Fiddler Protect](/protection/index) — is **not** available on
the gateway path. See each guardrails page for per-check thresholds, actions, and what can be tuned.
## Choosing a Path
Use the gateway you already run. If you only need visibility, start with tracing; add guardrails when
you need enforcement. Kong blocks on a detection; AgentGateway and LiteLLM can redact in place. If
you need to redact streamed responses, review the streaming limits on the owning guardrails page
first.
## What Every Integration Guarantees
* **No application code changes** — you point at the gateway you already run.
* **The guardrail adapter is stateless** — Fiddler evaluates text in-request and returns a decision; no prompt, response, or detection result is retained (the only durable artifact is content-free operational counters).
* **Each page is the source of truth** for its gateway's exact version floors, wire contract, and configuration surface.
## Next Steps
* [Kong AI Gateway Integration](/integrations/agentic-ai/kong-integration) — tracing setup for Kong.
* [Kong AI Gateway Guardrails](/protection/kong-guardrails) — enforcement on Kong.
* [AgentGateway Integration](/integrations/agentic-ai/agentgateway-integration) — tracing setup for AgentGateway.
* [AgentGateway Guardrails](/protection/agentgateway-guardrails) — enforcement on AgentGateway.
* [LiteLLM Integration](/integrations/agentic-ai/litellm-integration) — tracing setup for LiteLLM.
* [LiteLLM Guardrails](/protection/litellm-guardrails) — enforcement on LiteLLM.
# Span and Resource Attributes
Source: https://docs.fiddler.ai/integrations/agentic-ai/attributes
Understand how Fiddler uses span and resource attributes from OpenTelemetry traces — required vs optional fields, typing, custom attributes, and how attributes flow into metrics, alerts, and dashboards.
## Overview
Use this page as a reference for which attributes Fiddler indexes, how to type their values, and how they map to FQL `attribute()` and `Span::` filters.
Attributes are key-value pairs attached to OpenTelemetry spans and resources. They carry the data that powers Fiddler's dashboards, alerts, evaluators, and custom metrics.
Getting attributes right means:
* **Attributes** are present in OpenTelemetry spans.
* **Values use the correct type** so numeric attributes are aggregatable and alertable (not silently treated as strings).
* **Custom attributes** follow the naming convention so they appear in the Explorer and are queryable via FQL.
This page is the single reference for how attributes work across all Fiddler integrations.
***
## Attribute Levels
Attributes are set at two levels in the OTLP hierarchy:
| Level | Scope | Set on | Example |
| ------------ | ------------------------------------ | --------------------- | ------------------------------------------- |
| **Resource** | Entire trace (all spans share these) | `Resource.attributes` | `application.id` |
| **Span** | Individual operation | `Span.attributes` | `gen_ai.request.model`, `fiddler.span.type` |
***
## Required Attributes
These must be present on every trace sent to Fiddler. Fiddler rejects traces that omit either field.
| Attribute | Level | Description |
| ------------------- | -------- | ----------------------------------------------------- |
| `application.id` | Resource | Your Fiddler application UUID (UUID4 format) |
| `fiddler.span.type` | Span | Type of operation: `llm`, `tool`, `chain`, or `agent` |
***
## System Attributes
System attributes are common cross-framework concepts (tokens, agent and tool names, span type) that Fiddler recognizes automatically. They are registered in the attribute catalog and can be referenced in FQL custom metrics — by `semantic_name` (the recommended, framework-independent form) or by their verbatim key. Some are also available as dedicated span columns for fast filtering (via `Span::` redirects); the rest are filtered via `SpanAttribute::`.
| Attribute | Type | Explorer filter | Description |
| ---------------------------- | ------ | ----------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `gen_ai.agent.name` | string | Agent (`Span::agent_name`) | Name of the AI agent |
| `fiddler.span.type` | string | Type (`Span::span_type`) | Type of operation: `llm`, `tool`, `chain`, or `agent` |
| `gen_ai.tool.name` | string | gen\_ai.tool.name (system) (`SpanAttribute::gen_ai.tool.name`) | Tool or function name |
| `gen_ai.usage.input_tokens` | int | gen\_ai.usage.input\_tokens (system) (`SpanAttribute::gen_ai.usage.input_tokens`) | Input tokens consumed |
| `gen_ai.usage.output_tokens` | int | gen\_ai.usage.output\_tokens (system) (`SpanAttribute::gen_ai.usage.output_tokens`) | Output tokens consumed |
| `gen_ai.usage.total_tokens` | int | gen\_ai.usage.total\_tokens (system) (`SpanAttribute::gen_ai.usage.total_tokens`) | Total tokens consumed |
| `latency` | int | Duration (`Span::duration`) | Span duration (computed automatically from start/end time) |
***
## Identity Attributes
Identity attributes are stored as metadata internally and materialized into dedicated span columns. They appear as **Session ID** and **Agent ID** columns in the Explorer. They are **not** registered in the attribute catalog and cannot be used in FQL custom metrics.
| Attribute | Type | Explorer filter | Description |
| ------------------------ | ------ | ------------------------------- | ------------------------------------------- |
| `gen_ai.conversation.id` | string | Session ID (`Span::session_id`) | Groups spans into a multi-turn conversation |
| `gen_ai.agent.id` | string | Agent ID (`Span::agent_id`) | Unique identifier for the agent |
**Set agent attributes on every span.** If you provide `gen_ai.agent.name` or `gen_ai.agent.id`, set them on **every span within the trace**. Fiddler uses these to attribute spans to the correct agent — spans missing these fields will be unattributed even if other spans in the same trace carry them.
***
## Content Attributes
These attributes are visible in the Explorer but are **not** registered as queryable attributes — they cannot be used in FQL custom metrics or span filters.
### LLM Content
| Attribute | Type | Description |
| ------------------------- | ------ | -------------------------------------------- |
| `gen_ai.request.model` | string | Model name (e.g., `gpt-4o`, `claude-3-opus`) |
| `gen_ai.system` | string | LLM provider (e.g., `openai`, `anthropic`) |
| `gen_ai.llm.input.system` | string | System prompt |
| `gen_ai.llm.input.user` | string | User input (last user message) |
| `gen_ai.llm.output` | string | LLM response text |
| `gen_ai.llm.context` | string | Conversation history (prior messages) |
### Tool Content
| Attribute | Type | Description |
| -------------------- | ------------- | ---------------------------- |
| `gen_ai.tool.input` | string (JSON) | Arguments passed to the tool |
| `gen_ai.tool.output` | string (JSON) | Result returned by the tool |
***
## Custom User Attributes
Attach business-level metadata to spans using two namespaces:
| Namespace | Scope | Description |
| ---------------------------- | ---------------------- | --------------------------------------------- |
| Custom span attributes | Individual span | Metadata for a specific operation |
| `fiddler.session.user.{key}` | All spans in the trace | Session-wide context shared across every span |
Custom span attributes are indexed, appear as filterable columns in the Explorer, and are available for [custom metrics](/observability/agentic/custom-metrics) via the FQL `attribute()` function. Custom session attributes are visible in the trace detail side panel only — they do not appear as columns in the Explorer grid and are not available for custom metrics or filtering.
***
## Attribute Value Types
Under the hood, every attribute value is wrapped in an OTel `AnyValue` protobuf message. **Using the correct typed field is critical** — it determines whether Fiddler classifies the attribute as numeric (enabling aggregation, charting, and alerting) or string.
| Python type | `AnyValue` field | Fiddler storage type |
| ----------- | ---------------- | ------------------------- |
| `str` | `string_value` | String |
| `int` | `int_value` | Numeric (stored as float) |
| `float` | `double_value` | Numeric (stored as float) |
| `bool` | `bool_value` | String |
**Use `int_value` / `double_value` for numeric attributes.** If you wrap a number in `string_value` (e.g., `AnyValue(string_value="150")`), Fiddler treats it as a string column. You lose the ability to compute aggregations (sum, average, percentiles) and set numeric alerts on that attribute.
### SDK usage (high-level)
When using the Fiddler OTel SDK, LangGraph SDK, or Strands SDK, pass native Python types and the SDK handles typing automatically:
```python theme={null}
span.set_attribute("gen_ai.usage.input_tokens", 150) # int → int_value
span.set_attribute("confidence", 0.97) # float → double_value
span.set_attribute("region", "us-west") # str → string_value
span.set_attribute("is_internal", False) # bool → bool_value
```
### Protobuf usage (low-level)
When building protobuf `Span` objects directly (e.g., for [OTel trace export](/integrations/agentic-ai/otel-trace-export)), you must set the correct `AnyValue` field explicitly:
```python theme={null}
from opentelemetry.proto.common.v1.common_pb2 import AnyValue, KeyValue
# Numeric — use int_value / double_value
KeyValue(key="gen_ai.usage.input_tokens", value=AnyValue(int_value=150))
KeyValue(key="confidence", value=AnyValue(double_value=0.97))
# String
KeyValue(key="region", value=AnyValue(string_value="us-west"))
# Boolean
KeyValue(key="is_internal", value=AnyValue(bool_value=False))
```
For a copy-pastable `to_any_value()` helper that handles type dispatch automatically, see the [OTel Trace Export — AnyValue Typing](/integrations/agentic-ai/otel-trace-export#anyvalue-typing) section.
***
## How Attributes Flow into Fiddler
Once ingested, attributes power several Fiddler features:
| Feature | How attributes are used |
| ----------------------- | ---------------------------------------------------------------------------------------------- |
| **Explorer** | Filter, search, and group traces by any attribute |
| **Dashboards & Charts** | Built-in charts use system attributes (tokens, latency); custom charts reference any attribute |
| **Custom Metrics** | FQL expressions use `attribute()` to define computed metrics over span attributes |
| **Alerts** | Set threshold or anomaly alerts on numeric attributes (requires correct typing) |
| **Evaluators** | Enrichment evaluators can read span attributes as input context |
### Querying attributes with FQL
The `attribute()` function is the FQL primitive for referencing span data in custom metrics. Reference an attribute by its **verbatim key** (exactly as your application emits it) or by its **`semantic_name`** (a canonical concept that Fiddler resolves across frameworks). See [Attribute Quick Overview](#attribute-quick-overview) for the full classification.
```
attribute('verbatim_key')
attribute(semantic_name='input_tokens')
attribute('status', value='error')
```
As of release 26.16, you provide exactly one of `name` or `semantic_name`, and the `scope` and `type` arguments are optional — they default to `'span'` and `'user'`. The `value` argument filters to spans where the attribute equals a given string, for categorical breakdowns. See [Custom Metrics for Agentic Applications](/observability/agentic/custom-metrics#the-attribute-function) for the full parameter reference and the [supported semantic concepts](/observability/agentic/custom-metrics#supported-semantic-concepts).
**System attribute examples** — reference recognized concepts by `semantic_name`, so the metric works regardless of which framework emitted the trace:
```
-- Average input tokens per span
average(attribute(semantic_name='input_tokens'))
-- Count spans by agent name
count(attribute(semantic_name='agent_name', value='my-agent'))
-- Ratio of LLM spans
count(attribute(semantic_name='span_type', value='llm'))
/ count(attribute(semantic_name='span_type'))
```
**User attribute examples** — reference custom attributes by their verbatim key:
```
-- P95 of a custom latency attribute
quantile(attribute('response_time_ms'), level=0.95)
-- Ratio of premium-tier spans
count(attribute('tier', value='premium'))
/ count(attribute('tier'))
```
See [Custom Metrics](/observability/agentic/custom-metrics) for the full FQL reference and more examples.
***
## Attribute Quick Overview
All attributes listed below are visible in the Explorer. The **Custom metrics** and **Explorer filter** columns indicate additional availability for FQL and UI filtering respectively.
The Explorer filter column shows both the UI column name (what you see in the DataGrid header) and the API filter field (used in programmatic `filter.rules[].field` requests). The UI automatically maps column filters to the API syntax.
| Attribute | Level | Required | Custom metrics | Explorer filter | Type | `AnyValue` field |
| ---------------------------- | -------- | -------- | -------------- | ----------------------------------------------------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------- |
| `application.id` | Resource | Yes | — | — | UUID string | `string_value` |
| `gen_ai.agent.name` | Span | No | `system` | Agent (`Span::agent_name`) | string | `string_value` |
| `fiddler.span.type` | Span | Yes | `system` | Type (`Span::span_type`) | `llm` \| `tool` \| `chain` \| `agent` | `string_value` |
| `gen_ai.agent.id` | Span | No | — | Agent ID (`Span::agent_id`) | string | `string_value` |
| `gen_ai.conversation.id` | Span | No | — | Session ID (`Span::session_id`) | string | `string_value` |
| `gen_ai.tool.name` | Span | No | `system` | gen\_ai.tool.name (system) (`SpanAttribute::gen_ai.tool.name`) | string | `string_value` |
| `gen_ai.usage.input_tokens` | Span | No | `system` | gen\_ai.usage.input\_tokens (system) (`SpanAttribute::gen_ai.usage.input_tokens`) | int | `int_value` |
| `gen_ai.usage.output_tokens` | Span | No | `system` | gen\_ai.usage.output\_tokens (system) (`SpanAttribute::gen_ai.usage.output_tokens`) | int | `int_value` |
| `gen_ai.usage.total_tokens` | Span | No | `system` | gen\_ai.usage.total\_tokens (system) (`SpanAttribute::gen_ai.usage.total_tokens`) | int | `int_value` |
| `latency` | Span | No | — | Duration (`Span::duration`) | int | — (computed) |
| `gen_ai.request.model` | Span | No | — | — | string | `string_value` |
| `gen_ai.system` | Span | No | — | — | string (provider name) | `string_value` |
| `gen_ai.llm.input.system` | Span | No | — | — | string | `string_value` |
| `gen_ai.llm.input.user` | Span | No | — | — | string | `string_value` |
| `gen_ai.llm.output` | Span | No | — | — | string | `string_value` |
| `gen_ai.llm.context` | Span | No | — | — | string | `string_value` |
| `gen_ai.tool.input` | Span | No | — | — | string (JSON) | `string_value` |
| `gen_ai.tool.output` | Span | No | — | — | string (JSON) | `string_value` |
| custom user attributes | Span | No | `user` | \ (user) (`SpanAttribute::`) | any | Use typed field matching the value (see [Attribute Value Types](#attribute-value-types)) |
**Only `system` and `user` attributes support custom metrics.** Content fields (like `gen_ai.llm.output`, `gen_ai.request.model`) are visible in the Explorer but cannot be used in FQL `attribute()` expressions or aggregations. Identity fields (`gen_ai.agent.id`, `gen_ai.conversation.id`) are filter-only. If you need to compute metrics over a non-queryable field, attach it as a custom user attribute instead.
***
## Deprecated Prefix Conventions
Fiddler uses **semantic mappings** to resolve raw attribute keys to canonical concepts (like `input_tokens`, `model_name`, `input`) in the backend. This enables cross-framework analytics without requiring any specific key naming conventions from your instrumentation. Fiddler ships with 140+ pre-configured mappings covering 13+ frameworks, and you can add custom mappings for unsupported frameworks or proprietary attribute names.
Send attribute keys in their native framework format and rely on semantic mappings for cross-framework resolution.
For more details, see [Semantic Mappings](/concepts/semantic-mappings).
***
## Related Documentation
* [Semantic Mappings](/concepts/semantic-mappings) — how Fiddler maps raw attribute keys to canonical concepts across frameworks
* [OTel Trace Export](/integrations/agentic-ai/otel-trace-export) — protobuf serialization and attribute mapping for client-side export
* [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration) — live agent instrumentation via the OTel SDK
* [Custom Metrics](/observability/agentic/custom-metrics) — define FQL metrics over span attributes
* [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) — `@trace` decorator and `set_attribute()` usage
* [Agentic AI Overview](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) — compare all integration options
# Claude Code Integration
Source: https://docs.fiddler.ai/integrations/agentic-ai/claude-code-integration
Integrate Claude Code with Fiddler for coding agent observability — track LLM calls, tool usage, user prompts, and session behavior via OpenTelemetry traces.
## Overview
[Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) is Anthropic's CLI coding agent. It emits OpenTelemetry traces that Fiddler natively ingests and maps to the Fiddler schema, giving you observability over coding agent sessions — including user prompts, LLM calls, tool invocations, token usage, and permission decisions.
No Fiddler SDK is required. Claude Code's built-in OTel instrumentation sends traces directly to Fiddler's OTLP ingestion endpoint.
**Beta telemetry.** Claude Code's OpenTelemetry tracing requires the `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA` flag and is subject to change between versions. LLM response content is **not available** in trace spans (only via log records). Tool input/output requires `OTEL_LOG_TOOL_CONTENT=1`. See [Known Limitations](#known-claude-code-upstream-limitations) for full details.
| Capability | Support |
| ---------------------------- | ------------------------------------------- |
| User prompt capture | ✅ Requires `OTEL_LOG_USER_PROMPTS=1` |
| LLM response capture | ❌ Not available in traces |
| Tool name tracking | ✅ Always available |
| Tool input/output | ✅ Requires `OTEL_LOG_TOOL_CONTENT=1` |
| Token counts | ✅ Input, output, cache read, cache creation |
| Session correlation | ✅ Via `session.id` |
| Permission decision tracking | ✅ Accept/deny on tool invocations |
| Cost tracking | ❌ Not emitted by Claude Code |
| Enrichment (Prompt Safety) | ✅ Via Fiddler evaluator rules |
***
## Prerequisites
* **Claude Code** v2.1.138 or later
* A **Fiddler application** with a valid `application.id`
***
## Quick Start
### Step 1 — Enable Claude Code telemetry
Set the following environment variables before launching Claude Code:
```bash theme={null}
# Required — enable OTel tracing (beta)
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_TRACES_EXPORTER=otlp
# Required — point at Fiddler's OTLP ingestion endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT="https://"
export OTEL_EXPORTER_OTLP_HEADERS="fiddler-application-id=,Authorization=Bearer "
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
# Required — tag traces with your Fiddler application
export OTEL_RESOURCE_ATTRIBUTES="application.id="
# Recommended — enable content capture
export OTEL_LOG_USER_PROMPTS=1
export OTEL_LOG_TOOL_DETAILS=1
export OTEL_LOG_TOOL_CONTENT=1
```
### Step 2 — Launch Claude Code
```bash theme={null}
claude
```
Use Claude Code normally. Every interaction generates OTel traces that flow to Fiddler.
### Step 3 — Verify traces in Fiddler
Navigate to your application in the Fiddler UI. You should see traces appearing within 30 seconds, with:
* **Agent spans** for each user interaction (prompt → reasoning → tool calls → response)
* **LLM spans** for each model call (Sonnet, Haiku) with token counts and latency
* **Tool spans** for each tool invocation (Bash, Read, Write, MCP tools, WebSearch)
***
## Managed Deployment via Settings File
For enterprise or managed deployments, you can configure telemetry via Claude Code's settings file instead of environment variables. This ensures consistent telemetry across all developers without requiring them to set env vars manually.
Create or update `.claude/settings.json` (project-level) or `~/.claude/settings.json` (user-level):
```json theme={null}
{
"env": {
"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
"CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
"OTEL_TRACES_EXPORTER": "otlp",
"OTEL_EXPORTER_OTLP_ENDPOINT": "https://",
"OTEL_EXPORTER_OTLP_HEADERS": "fiddler-application-id=,Authorization=Bearer ",
"OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
"OTEL_RESOURCE_ATTRIBUTES": "application.id=",
"OTEL_LOG_USER_PROMPTS": "1",
"OTEL_LOG_TOOL_DETAILS": "1",
"OTEL_LOG_TOOL_CONTENT": "1"
}
}
```
**Do not commit API keys to version control.** The `OTEL_EXPORTER_OTLP_HEADERS` value contains your Fiddler API key. If using project-level `.claude/settings.json`, add it to `.gitignore` to prevent accidental exposure. For CI/CD or shared environments, prefer setting credentials via environment variables instead of the settings file.
Settings in `.claude/settings.json` (project-level) take precedence over environment variables. Settings in the user-level `~/.claude/settings.json` apply to all projects unless overridden.
***
## What Gets Captured
### Span Hierarchy
Each user interaction produces a trace with the following span structure:
```
claude_code.interaction (agent — root span, one per user prompt)
├── claude_code.llm_request (llm — LLM API call, e.g., Sonnet or Haiku)
├── claude_code.llm_request (llm — parallel model call, e.g., Haiku for routing)
├── claude_code.tool (tool — tool invocation wrapper)
│ ├── claude_code.tool.blocked_on_user (chain — permission prompt wait)
│ └── claude_code.tool.execution (chain — actual tool execution)
├── claude_code.tool (tool — second tool invocation)
│ └── claude_code.tool.execution (chain)
└── claude_code.llm_request (llm — follow-up LLM call with tool results)
```
### Span Type Mapping
| Claude Code Span | Fiddler Span Type | Description |
| ---------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------- |
| `claude_code.interaction` | `agent` | Root span for a user interaction. Contains the user prompt and interaction-level metadata. |
| `claude_code.llm_request` | `llm` | Individual LLM API call. Contains model name, token counts, latency, and success/error status. |
| `claude_code.tool` | `tool` | Tool invocation wrapper. Contains the tool name and total duration (including permission wait). |
| `claude_code.tool.blocked_on_user` | `chain` | Time spent waiting for the user to accept or deny a tool permission prompt. Contains the user's decision. |
| `claude_code.tool.execution` | `chain` | Actual tool execution after permission is granted. Contains success status and execution duration. |
### Captured Attributes by Span Type
#### Agent Spans (`claude_code.interaction`)
| Attribute | Semantic Name | Requires Env Var | Description |
| ------------------------- | ------------------------- | ------------------------- | ------------------------------------------------------------------------------------- |
| `user_prompt` | `gen_ai.llm.input.user` | `OTEL_LOG_USER_PROMPTS=1` | The user's prompt text. Shows `` when env var is not set. |
| `user_prompt_length` | `user_prompt_length` | No | Character count of the user prompt (always available, even when content is redacted). |
| `interaction.sequence` | `interaction.sequence` | No | Sequence number within the session (1 for first interaction, 2 for second, etc.). |
| `interaction.duration_ms` | `interaction.duration_ms` | No | Total duration of the interaction in milliseconds. |
| `session.id` | `gen_ai.conversation.id` | No | Claude Code session UUID. Used for session correlation. |
#### LLM Spans (`claude_code.llm_request`)
| Attribute | Semantic Name | Description |
| ----------------------- | ---------------------------- | -------------------------------------------------------------------------- |
| `input_tokens` | `gen_ai.usage.input_tokens` | Number of input tokens (excluding cache). |
| `output_tokens` | `gen_ai.usage.output_tokens` | Number of output tokens. |
| `cache_read_tokens` | `cache_read_tokens` | Tokens read from Anthropic's prompt cache. |
| `cache_creation_tokens` | `cache_creation_tokens` | Tokens written to Anthropic's prompt cache. |
| `model` | Passthrough | Model identifier (e.g., `claude-sonnet-4-5@20250929`). |
| `gen_ai.request.model` | Passthrough | Same as `model`, following OTel GenAI semantic conventions. |
| `gen_ai.system` | Passthrough | LLM provider (always `anthropic` for Claude Code). |
| `ttft_ms` | `ttft_ms` | Time to first token in milliseconds. |
| `stop_reason` | `stop_reason` | Why the LLM stopped: `end_turn`, `tool_use`, `max_tokens`. |
| `success` | `success` | Whether the LLM call succeeded (`true`/`false`). |
| `error` | `error` | Error message when `success` is `false`. |
| `attempt` | `attempt` | Retry attempt number (1 for first try). |
| `llm_request.context` | `llm_request.context` | Call context: `interaction` (top-level) or `tool` (nested in a tool span). |
**LLM response content is not available in traces.** Claude Code emits LLM output via OTel log records, not trace span attributes. This mapper processes trace spans only. See [Known Limitations](#known-claude-code-upstream-limitations).
#### Tool Spans (`claude_code.tool`)
| Attribute | Semantic Name | Description |
| ------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tool_name` | `gen_ai.tool.name` | Tool identifier (e.g., `Bash`, `Read`, `Write`, `Edit`, `WebSearch`, `mcp__server__tool`). |
| `duration_ms` | `duration_ms` | Total tool duration including permission wait. |
| Tool output | `gen_ai.tool.output` | Tool output body (up to 60 KB), extracted from the `tool.output` span event. The output attribute key varies by tool (see table below). |
| Tool input | `gen_ai.tool.input` | Tool-specific input arguments (e.g., `file_path` for Read, `bash_command` for Bash), extracted from the `tool.output` span event. Serialized as JSON with native types preserved. |
The output attribute key within the `tool.output` span event varies by tool:
| Tool | Output key | Input key(s) |
| ----------- | -------------------- | ------------------------------ |
| Read | `content` | `file_path`, `offset`, `limit` |
| Bash | `output` | `bash_command` |
| Edit | `diff` | `file_path` |
| Write | *(none)* | `content`, `file_path` |
| Unknown/MCP | `content` (fallback) | all other attributes |
**Tool input/output requires `OTEL_LOG_TOOL_CONTENT=1`.** When enabled, Claude Code emits a `tool.output` span event on tool spans. Fiddler looks up the correct output attribute key based on the tool name and extracts it into `gen_ai.tool.output`. All remaining event attributes become `gen_ai.tool.input` (JSON-serialized with native types preserved).
#### Permission Decision Spans (`claude_code.tool.blocked_on_user`)
| Attribute | Semantic Name | Description |
| ------------- | ------------- | -------------------------------------------------------------------------- |
| `decision` | `decision` | User's permission decision: `accept` or `deny`. |
| `source` | `source` | How the decision was made: `user_temporary`, `user_permanent`, `settings`. |
| `duration_ms` | `duration_ms` | Time spent waiting for the user to respond. |
***
## Session Grouping
Claude Code assigns a `session.id` to each coding session. Fiddler maps this to `gen_ai.conversation.id`, enabling session-level grouping in the UI. All spans within a session share the same `session.id`, allowing you to:
* View the full timeline of a coding session
* Correlate LLM calls with tool invocations
* Track interaction sequences within a session
When `session.id` is absent (rare), Fiddler falls back to using the OTel `trace_id` as the conversation ID.
***
## Enrichment Setup
Fiddler can automatically score Claude Code user prompts using Prompt Safety (powered by the Centor Model for Safety) or other evaluators. To configure:
1. Navigate to your application in the Fiddler UI.
2. Create an evaluator rule with:
* **Span type:** `agent`
* **Text field:** `user_prompt`
* **Evaluator:** Prompt Safety (or any custom evaluator)
3. New `claude_code.interaction` spans will be scored automatically as they arrive.
Enrichment requires the `user_prompt` attribute to be populated. Set `OTEL_LOG_USER_PROMPTS=1` to enable content capture. When disabled, `user_prompt` contains `` and enrichment will score the redaction placeholder instead of the actual prompt.
***
## Configuration Reference
### Environment Variables
#### Required
| Variable | Description |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `CLAUDE_CODE_ENABLE_TELEMETRY` | Set to `1` to enable telemetry. |
| `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA` | Set to `1` to enable OTel trace export (beta). |
| `OTEL_TRACES_EXPORTER` | Set to `otlp` to enable OTLP trace export. |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Fiddler's OTLP ingestion endpoint base URL (e.g., `https://your-fiddler-instance.com`). The OTel SDK appends `/v1/traces` automatically. |
| `OTEL_EXPORTER_OTLP_HEADERS` | Must include `fiddler-application-id=` and `Authorization=Bearer `. |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | Set to `http/protobuf`. |
| `OTEL_RESOURCE_ATTRIBUTES` | Must include `application.id=` matching the header above. |
#### Content Logging
| Variable | Default | Description |
| ----------------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `OTEL_LOG_USER_PROMPTS` | `0` | Set to `1` to include user prompt text in traces. When disabled, `user_prompt` shows ``. |
| `OTEL_LOG_TOOL_DETAILS` | `0` | Set to `1` to include additional tool metadata (`file_path` for Read/Edit/Write, `full_command` for Bash). |
| `OTEL_LOG_TOOL_CONTENT` | `0` | Set to `1` to include tool input/output content via `tool.output` span events. |
#### Optional
| Variable | Default | Description |
| ------------------- | ------------- | -------------------------------------------------------------------------------------------------------- |
| `OTEL_SERVICE_NAME` | `claude-code` | Override the service name. Must remain `claude-code` for Fiddler to detect and map the traces correctly. |
***
## Troubleshooting
### Traces not appearing in Fiddler
1. Verify telemetry is enabled:
```bash theme={null}
echo $CLAUDE_CODE_ENABLE_TELEMETRY # should be 1
echo $CLAUDE_CODE_ENHANCED_TELEMETRY_BETA # should be 1
echo $OTEL_TRACES_EXPORTER # should be otlp
```
2. Verify the OTLP endpoint, protocol, and headers are set:
```bash theme={null}
echo $OTEL_EXPORTER_OTLP_ENDPOINT # should be the base URL (no /v1/traces)
echo $OTEL_EXPORTER_OTLP_PROTOCOL # should be http/protobuf
echo $OTEL_EXPORTER_OTLP_HEADERS # should contain fiddler-application-id
echo $OTEL_RESOURCE_ATTRIBUTES # should contain application.id
```
3. Verify `application.id` in `OTEL_RESOURCE_ATTRIBUTES` is a valid UUID for an existing Fiddler application. Spans with an invalid or missing `application.id` are dropped during ingestion.
4. Verify the endpoint URL is the base URL (the SDK appends `/v1/traces` automatically) and `OTEL_EXPORTER_OTLP_PROTOCOL` is set to `http/protobuf`.
### User prompts showing as ``
Set `OTEL_LOG_USER_PROMPTS=1` in the environment before launching Claude Code. This is disabled by default for privacy.
### Tool details missing from spans
Tool spans only include `tool_name` and `duration_ms` by default. Setting `OTEL_LOG_TOOL_DETAILS=1` adds `file_path` (for Read/Edit/Write tools) and `full_command` (for Bash tools) as span attributes. Setting `OTEL_LOG_TOOL_CONTENT=1` enables `tool.output` span events with full tool input/output content.
### Enrichment worker skipping spans
If you see warnings like `Missing gen_ai.agent.name in span.attributes`, the Fiddler mapper automatically sets this to `"claude-code"` on all spans. Ensure you are running the latest version of the external OTel consumer that includes the Claude Code mapper.
### Spans classified as `chain` instead of expected types
Only three span types have specific classifications: `claude_code.interaction` (agent), `claude_code.llm_request` (llm), and `claude_code.tool` (tool). All other spans — including `tool.blocked_on_user` and `tool.execution` — are classified as `chain`. This is expected behavior.
***
## Known Claude Code Upstream Limitations
Claude Code's OpenTelemetry tracing is currently in **beta** and has several limitations that affect what data is available in Fiddler. These are not Fiddler issues — they are limitations of Claude Code's OTel instrumentation. Fiddler captures and maps every attribute that Claude Code emits.
### LLM response content not available in traces
Claude Code does not emit LLM response text as a trace span attribute. Response content is only available via OTel **log records** (controlled by `OTEL_LOGS_EXPORTER` and `OTEL_LOG_RAW_API_BODIES`), not trace spans. OTel log records now include `trace_id` and `span_id` (resolved upstream), enabling log-to-trace correlation.
**Impact:** Session views in Fiddler show user prompts (input) but not Claude's responses (output). Evaluators that require both input and output (e.g., response faithfulness) cannot run on Claude Code traces. Log-to-trace correlation is now possible upstream but not yet implemented in Fiddler's mapper.
### Tool input/output now available via span events
Claude Code emits `tool.output` span events on tool spans when `OTEL_LOG_TOOL_CONTENT=1` is set. The output attribute key varies by tool (`content` for Read, `output` for Bash, `diff` for Edit; Write has no output). Fiddler uses a per-tool lookup to extract the correct attribute into `gen_ai.tool.output`, with remaining event attributes (e.g., `file_path`, `bash_command`) collected into `gen_ai.tool.input`.
This limitation was resolved upstream. Earlier versions of Claude Code did not emit `tool.output` span events despite the env var being set.
### Tracing is beta and requires opt-in
Claude Code's OTel tracing requires the `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA` flag. The trace schema (span names, attribute names, hierarchy) may change between Claude Code versions without notice.
**Impact:** Fiddler's mapper may need updates when Claude Code's trace schema changes. Pin Claude Code versions in managed deployments to avoid unexpected schema changes.
### No distributed tracing to LLM providers
Claude Code does not send W3C `traceparent` headers when making LLM API calls. If Claude Code is routed through a proxy (e.g., LiteLLM), the proxy creates a new trace root rather than continuing Claude Code's trace. Claude Code traces and proxy traces are separate trace trees.
Claude Code does propagate `TRACEPARENT` to Bash subprocesses (e.g., when running shell commands), enabling trace correlation for subprocess-initiated LLM calls in non-interactive mode.
**Impact:** End-to-end distributed traces (user → coding agent → LLM proxy → LLM provider) require joining on `session.id` or `x-claude-code-session-id` at query time rather than native trace propagation.
### Content redaction is all-or-nothing per category
Content capture is controlled by three separate env vars (`OTEL_LOG_USER_PROMPTS`, `OTEL_LOG_TOOL_DETAILS`, `OTEL_LOG_TOOL_CONTENT`). Each controls an entire category — there is no per-attribute or per-tool-type granularity. You cannot, for example, capture user prompts while keeping tool content redacted within a single category.
### Thinking and reasoning content not available
There is no env var to export Claude's extended thinking or reasoning steps in OTel telemetry. Only metadata (token counts, model, latency) is available for the reasoning process, not the content.
### ScopeName and ScopeVersion empty in collected traces
Despite using instrumentation scope `com.anthropic.claude_code.tracing 1.0.0`, traces may show empty `ScopeName` and `ScopeVersion` fields. This does not affect span routing or processing — `service.name` and span name prefixes are used for detection.
### Summary
| Limitation | Severity | Description |
| ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No LLM response in traces | High | Output content available only via OTel log records. Log records now include `trace_id`/`span_id` (resolved upstream), but log-to-trace ingestion not yet implemented in Fiddler. |
| ~~No tool input/output~~ | ~~Resolved~~ | `tool.output` span events are now emitted when `OTEL_LOG_TOOL_CONTENT=1` is set. Fiddler extracts tool input and output using per-tool output key dispatch (Read→`content`, Bash→`output`, Edit→`diff`, Write→no output). |
| Beta tracing | Medium | Schema may change between versions. Requires `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA`. |
| No traceparent to LLM providers | Medium | Separate trace trees for agent and proxy. Join on session ID. |
| All-or-nothing redaction | Low | Cannot selectively control content capture per attribute. |
| No thinking content | Medium | Reasoning steps not available in any telemetry signal. |
| Empty ScopeName/ScopeVersion | Low | Does not affect functionality. |
***
## Related Documentation
* [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration) — Manual OTel instrumentation for custom frameworks
* [LiteLLM Integration](/integrations/agentic-ai/litellm-integration) — Unified LLM gateway with Fiddler observability
* [Strands Agents SDK](/integrations/agentic-ai/strands-sdk) — Native monitoring for Strands agent applications
* [AgentGateway Integration](/integrations/agentic-ai/agentgateway-integration) — Cisco AgentGateway observability
* [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) — Foundation package for custom OTel instrumentation
# Fiddler Evals SDK
Source: https://docs.fiddler.ai/integrations/agentic-ai/evals-sdk
LLM experiments framework with pre-built evaluators and custom metrics
✓ **GA** | 🏆 **Native SDK**
Evaluate LLM application quality with Fiddler's evaluation framework. Run batch experiments with 13 pre-built evaluators or create custom metrics for domain-specific quality assessment.
## What You'll Need
* Fiddler account
* Python 3.10 or higher
* Fiddler API key and access token
* Dataset for experiments
## Quick Start
```bash theme={null}
# Step 1: Install
pip install fiddler-evals
```
```python theme={null}
# Step 2: Initialize connection
from fiddler_evals import init
init(
url='https://your-org.fiddler.ai',
token='your-access-token'
)
# Step 3: Create project and application
from fiddler_evals import Project, Application, Dataset
project = Project.get_or_create(name='my_eval_project')
application = Application.get_or_create(
name='my_llm_app',
project_id=project.id
)
# Step 4: Create dataset and add test cases
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
dataset = Dataset.create(
name='experiment_dataset',
application_id=application.id,
description='Test cases for LLM experiments'
)
test_cases = [
NewDatasetItem(
inputs={"question": "What is the capital of France?"},
expected_outputs={"answer": "Paris is the capital of France"},
metadata={"type": "Factual", "category": "Geography"}
),
]
dataset.insert(test_cases)
# Step 5: Run evaluation
from fiddler_evals import evaluate
from fiddler_evals.evaluators import AnswerRelevance, Conciseness, Coherence
MODEL = "openai/gpt-4o"
CREDENTIAL = "your-credential-name"
def my_llm_task(inputs, extras, metadata):
"""Your LLM application logic"""
question = inputs.get("question", "")
# Call your LLM here
answer = f"Mock response to: {question}"
return {"answer": answer}
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[
AnswerRelevance(model=MODEL, credential=CREDENTIAL),
Conciseness(model=MODEL, credential=CREDENTIAL),
Coherence(model=MODEL, credential=CREDENTIAL)
],
name_prefix="my_experiment",
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["question"],
"rag_response": "answer",
"response": "answer",
}
)
# Step 6: Analyze results in Fiddler UI
print(f"✅ Evaluated {len(results.results)} test cases")
```
## Pre-Built Evaluators
### Safety & Trust
* **FTLPromptSafety** - Detect prompt injection, jailbreaks, and unsafe prompts (runs on Fiddler Centor Models)
### Quality & Accuracy
* **AnswerRelevance** - Assess how well responses address user queries (High / Medium / Low)
* **ContextRelevance** - Evaluate whether retrieved documents are relevant to the query (High / Medium / Low). Available in Agentic Monitoring and Experiments only
* **RAGFaithfulness** - Check if responses are grounded in retrieved documents (Yes / No)
* **FTLResponseFaithfulness** - Powered by the Centor Model for Faithfulness for low-latency guardrails
* **Coherence** - Measure logical flow and consistency
* **Conciseness** - Evaluate response brevity and efficiency
### Content Analysis
* **Sentiment** - Analyze emotional tone
* **TopicClassification** - Categorize content by topic
* **RegexSearch** / **RegexMatch** - Custom pattern-based evaluation
* **EvalFn** - Wrap any Python function as an evaluator
## Example Usage
### Batch Experiment with Multiple Evaluators
```python theme={null}
from fiddler_evals import init, evaluate, Dataset
from fiddler_evals.evaluators import (
AnswerRelevance,
Conciseness,
FTLResponseFaithfulness
)
MODEL = "openai/gpt-4o"
CREDENTIAL = "your-credential-name"
# Initialize connection
init(url='https://your-org.fiddler.ai', token='your-access-token')
# Get existing dataset
dataset = Dataset.get_by_name(
name='llm_outputs',
application_id=application.id
)
# Define your LLM task
def evaluate_llm(inputs, extras, metadata):
question = inputs['question']
context = extras.get('context', '')
# Your LLM call here
response = my_llm_model.generate(question, context)
return {
"answer": response,
"question": question,
"context": context
}
# Run evaluation with multiple evaluators
results = evaluate(
dataset=dataset,
task=evaluate_llm,
evaluators=[
AnswerRelevance(model=MODEL, credential=CREDENTIAL),
Conciseness(model=MODEL, credential=CREDENTIAL),
FTLResponseFaithfulness() # Centor Models don't require model= parameter
],
name_prefix="llm-eval",
description="Comprehensive LLM experiment",
metadata={"model_version": "v2.1", "environment": "production"},
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["question"],
"rag_response": "answer",
"response": "answer",
"context": lambda x: x["extras"].get("context", "")
},
max_workers=4 # Parallel processing
)
# Access results programmatically
for result in results.results:
item = result.experiment_item
print(f"\nTest Case: {item.dataset_item_id}")
print(f"Status: {item.status}")
print(f"Duration: {item.duration_ms}ms")
for score in result.scores:
print(f" {score.name}: {score.value}")
```
### Custom Evaluators
```python theme={null}
from fiddler_evals.evaluators.base import Evaluator
from fiddler_evals.pydantic_models.score import Score
class LengthEvaluator(Evaluator):
"""Custom evaluator for response length"""
def __init__(self, min_length=10, max_length=200):
super().__init__()
self.min_length = min_length
self.max_length = max_length
def score(self, output: str) -> Score:
length = len(output.strip())
if length < self.min_length:
score_value = 0.0
reasoning = f"Too short ({length} chars, min {self.min_length})"
elif length > self.max_length:
score_value = 0.5
reasoning = f"Too long ({length} chars, max {self.max_length})"
else:
score_value = 1.0
reasoning = f"Appropriate length ({length} chars)"
return Score(
name="length_check",
evaluator_name=self.name,
value=score_value,
reasoning=reasoning
)
# Use custom evaluator alongside built-in ones
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[
AnswerRelevance(model="openai/gpt-4o", credential="your-credential-name"),
Conciseness(model="openai/gpt-4o", credential="your-credential-name"),
LengthEvaluator(min_length=15, max_length=100)
],
score_fn_kwargs_mapping={
"user_query": lambda x: x["inputs"]["question"],
"rag_response": "answer",
"response": "answer",
"output": "answer",
}
)
```
### Importing Test Cases from Files
```python theme={null}
# From CSV file
dataset.insert_from_csv_file(
file_path='test_cases.csv',
input_columns=['question'],
expected_output_columns=['answer'],
metadata_columns=['category', 'difficulty']
)
# From JSONL file
dataset.insert_from_jsonl_file(
file_path='test_cases.jsonl',
input_keys=['question'],
expected_output_keys=['answer'],
metadata_keys=['category']
)
# From pandas DataFrame
import pandas as pd
df = pd.DataFrame({
'question': ['What is AI?', 'Explain ML'],
'expected_answer': ['AI is...', 'ML is...'],
'category': ['definition', 'definition']
})
dataset.insert_from_pandas(
df=df,
input_columns=['question'],
expected_output_columns=['expected_answer'],
metadata_columns=['category']
)
```
## Viewing Results
Results are automatically tracked in the Fiddler UI. Navigate to your application to:
* View experiment results with detailed scores
* Compare experiments side-by-side
* Filter and analyze by metadata
* Export results for further analysis
### Programmatic Analysis
```python theme={null}
from fiddler_evals import ScoreStatus, ExperimentItemStatus
# Analyze individual results
for i, result in enumerate(results.results):
item = result.experiment_item
scores = result.scores
print(f"\n📝 Test Case {i + 1}:")
print(f" Status: {item.status}")
print(f" Duration: {item.duration_ms}ms")
if item.status == ExperimentItemStatus.SUCCESS:
for score in scores:
status_emoji = "✅" if score.status == ScoreStatus.SUCCESS else "❌"
print(f" {status_emoji} {score.name}: {score.value}")
if score.reasoning:
print(f" {score.reasoning}")
# Calculate summary statistics
from collections import defaultdict
evaluator_scores = defaultdict(list)
for result in results.results:
for score in result.scores:
if score.value is not None:
evaluator_scores[score.name].append(score.value)
print("\n🎯 Summary by Evaluator:")
for evaluator_name, values in evaluator_scores.items():
avg_score = sum(values) / len(values) if values else 0
print(f" {evaluator_name}: {avg_score:.3f} (avg)")
```
## Advanced Configuration
### Parallel Processing
```python theme={null}
# Process multiple test cases concurrently
results = evaluate(
dataset=large_dataset,
task=my_llm_task,
evaluators=[
AnswerRelevance(model="openai/gpt-4o", credential="your-credential-name"),
Conciseness(model="openai/gpt-4o", credential="your-credential-name"),
],
max_workers=8, # Use 8 parallel workers
name_prefix="parallel-eval"
)
```
### Experiment Metadata and Organization
```python theme={null}
# Track experiments with custom metadata
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[AnswerRelevance(model="openai/gpt-4o", credential="your-credential-name")],
name_prefix="model-comparison",
description="Comparing GPT-4 vs GPT-3.5",
metadata={
"model_name": "gpt-4",
"temperature": 0.7,
"max_tokens": 1000,
"evaluation_date": "2024-01-15",
"environment": "production",
"version": "v2.1"
}
)
```
### Custom Parameter Mapping
```python theme={null}
# Map evaluator parameters to your task output structure
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[
AnswerRelevance(model="openai/gpt-4o", credential="your-credential-name"), # Needs: user_query, rag_response
Conciseness(model="openai/gpt-4o", credential="your-credential-name"), # Needs: response
],
score_fn_kwargs_mapping={
# Map evaluator parameters to task output keys
"user_query": lambda x: x["inputs"]["question"], # Lambda for nested values
"rag_response": "answer", # Simple key mapping
"response": "answer", # Multiple evaluators can use same output
"context": lambda x: x["extras"].get("context", "") # With defaults
}
)
```
## Troubleshooting
### Connection Issues
**Problem**: Cannot connect to Fiddler instance
**Solution**:
1. Verify your URL is correct (e.g., `https://your-org.fiddler.ai`)
2. Ensure your access token is valid and not expired
3. Check network connectivity: `curl -I https://your-org.fiddler.ai`
4. Regenerate token from Fiddler UI: **Settings** > **Credentials**
### Import Errors
**Problem**: `ModuleNotFoundError: No module named 'fiddler_evals'`
**Solution**:
```bash theme={null}
# Verify installation
pip list | grep fiddler-evals
# Reinstall if needed
pip uninstall fiddler-evals
pip install fiddler-evals
# Check Python version (requires 3.10+)
python --version
```
### Experiment Failures
**Problem**: Evaluators failing with parameter errors
**Solution**:
1. Check `score_fn_kwargs_mapping` matches evaluator requirements
2. Verify task output format matches expected structure
3. Test evaluators individually:
```python theme={null}
from fiddler_evals.evaluators import AnswerRelevance
evaluator = AnswerRelevance(model="openai/gpt-4o", credential="your-credential-name")
score = evaluator.score(
user_query="What is AI?",
rag_response="AI is artificial intelligence"
)
print(f"Score: {score.value}, Reasoning: {score.reasoning}")
```
### Performance Issues
**Problem**: Experiment running slowly
**Solution**:
```python theme={null}
# Use parallel processing
results = evaluate(
dataset=dataset,
task=my_llm_task,
evaluators=[
AnswerRelevance(model="openai/gpt-4o", credential="your-credential-name"),
Conciseness(model="openai/gpt-4o", credential="your-credential-name"),
],
max_workers=4 # Adjust based on your system
)
# Or process in smaller batches
for i in range(0, len(all_test_cases), 100):
batch = all_test_cases[i:i+100]
batch_dataset = Dataset.create(name=f"batch_{i}")
batch_dataset.insert(batch)
results = evaluate(dataset=batch_dataset, task=my_llm_task, evaluators=[...])
```
## Related Integrations
* [**Evals SDK Quick Start**](/evaluate-and-test/evals-sdk-quick-start) - Detailed setup guide with code examples
* [**Evals SDK Reference**](/sdk-api/evals/evaluate) - Complete SDK API documentation
* [**LangGraph SDK**](/integrations/agentic-ai/langgraph-sdk) - Runtime monitoring for LangGraph agents
* [**Strands Agents SDK**](/integrations/agentic-ai/strands-sdk) - Monitor Strands Agents
* [**Python Client SDK**](/sdk-api/python-client/connection) - Full platform API access
## Next Steps
1. [**Quick Start Guide**](/evaluate-and-test/evals-sdk-quick-start) - Complete tutorial with working examples
2. [**Getting Started with Experiments**](/getting-started/experiments) - Understand experiment concepts and best practices
3. [**SDK API Reference**](/sdk-api/evals/evaluate) - Explore all available classes and methods
# Fiddler OTel SDK
Source: https://docs.fiddler.ai/integrations/agentic-ai/fiddler-otel-sdk
Instrument any Python AI agent or LLM application with Fiddler's core OpenTelemetry SDK
[](https://pypi.org/project/fiddler-otel/)
Instrument any Python AI agent or LLM application with OpenTelemetry-based tracing for comprehensive agentic observability. The Fiddler OTel SDK is the foundation package used by all Fiddler framework integrations (`fiddler-langgraph`, `fiddler-langchain`). Use it directly when you have no LangGraph or LangChain dependency, or when you want lightweight decorator-based instrumentation for custom Python agents.
**Migrating from `fiddler-langgraph`?** The core instrumentation functionality (`FiddlerClient`, `@trace`, span wrappers, etc.) has been extracted from `fiddler-langgraph` into this standalone `fiddler-otel` package. If you previously imported these symbols from `fiddler_langgraph`, update your imports to use `fiddler_otel` — the classes and behavior are identical. See the [deprecation notice](/changelog/langgraph-sdk) in the LangGraph SDK changelog for details.
## What you'll need
* Fiddler account (cloud or on-premises)
* Python 3.10, 3.11, 3.12, or 3.13
* Fiddler API key and application ID
## Quick start
Get monitoring in 3 steps:
```bash theme={null}
# Step 1: Install
pip install fiddler-otel
```
```python theme={null}
# Step 2: Initialize the Fiddler client
from fiddler_otel import FiddlerClient, trace
client = FiddlerClient(
application_id='your-app-id', # Must be valid UUID4
api_key='your-api-key',
url='https://your-instance.fiddler.ai'
)
# Step 3: Decorate your functions
@trace(as_type='generation')
def call_llm(prompt: str) -> str:
# Your LLM call here
...
```
That's it! Your agent traces are now flowing to Fiddler.
This Quick Start uses the `@trace` decorator. For context manager and manual instrumentation approaches, see [Instrumentation Methods](#instrumentation-methods) below.
## What gets monitored
The Fiddler OTel SDK captures:
### Trace hierarchy
Spans are automatically nested into parent-child relationships based on the call stack:
```
[Span] supervisor_agent (agent - TYPE=agent)
└── [Span] call_llm (LLM - TYPE=llm)
└── [Span] search_hotels (Tool - TYPE=tool)
└── [Span] call_llm (LLM - TYPE=llm)
```
### Captured data
* Function inputs and return values (auto-serialized to JSON)
* LLM prompts, completions, and token usage
* Tool names, inputs, and outputs
* Agent name, agent ID, and conversation ID
* Execution times and error traces
## Application setup
Before instrumenting your application, you must create an application in Fiddler and obtain your Application ID.
Log in to your Fiddler instance and navigate to **GenAI Applications**, then click **Add Application** and follow the onboarding wizard to create your application.
After creating your application, copy the **Application ID** from the GenAI Applications page. This must be a valid UUID4 format (for example, `550e8400-e29b-41d4-a716-446655440000`). You'll need this for initialization.
Go to **Settings** > **Credentials** and copy your API key. You'll need this for initialization.
## Detailed setup
### Installation
```bash theme={null}
pip install fiddler-otel
```
**Requirements:**
* **Python:** 3.10, 3.11, 3.12, or 3.13
* **OpenTelemetry:** API, SDK, and OTLP exporter `>= 1.27.0` (installed automatically)
* **pydantic:** `>= 2.0` (installed automatically)
### Configuration
#### Direct initialization (Recommended)
```python theme={null}
from fiddler_otel import FiddlerClient
client = FiddlerClient(
application_id='your-app-id', # Required (UUID4 format)
api_key='your-api-key', # Required when otlp_enabled=True (default)
url='https://your-instance.fiddler.ai' # Required when otlp_enabled=True (default)
)
```
#### Using environment variables
```python theme={null}
import os
from fiddler_otel import FiddlerClient
client = FiddlerClient(
application_id=os.getenv('FIDDLER_APPLICATION_ID'),
api_key=os.getenv('FIDDLER_API_KEY'),
url=os.getenv('FIDDLER_URL')
)
```
**Environment Variables Reference:**
| Variable | Description | Example |
| ------------------------ | ------------------------- | -------------------------------------- |
| `FIDDLER_API_KEY` | Your Fiddler API key | `fid_...` |
| `FIDDLER_APPLICATION_ID` | Your application UUID4 | `550e8400-e29b-41d4-a716-446655440000` |
| `FIDDLER_URL` | Your Fiddler instance URL | `https://your-instance.fiddler.ai` |
## Instrumentation methods
The Fiddler OTel SDK provides two instrumentation approaches. Choose the one that fits your application:
| Approach | Best For | Key API |
| --------------------------------------------------- | -------------------------------------------- | ----------------------------------------- |
| [Decorator-Based](#decorator-based-instrumentation) | Custom Python functions, minimal boilerplate | `@trace()`, `get_current_span()` |
| [Manual](#manual-instrumentation) | Fine-grained span lifecycle control | `start_as_current_span()`, `start_span()` |
You can combine both approaches in the same application. For example, use the decorator for most functions and manual spans where you need fine-grained lifecycle control.
### Decorator-based instrumentation
Use the `@trace()` decorator to instrument individual Python functions. Works with both synchronous and asynchronous functions.
```python theme={null}
from openai import OpenAI
from fiddler_otel import FiddlerClient, trace, get_current_span, set_conversation_id
import uuid
client = FiddlerClient(
application_id='your-app-id',
api_key='your-api-key',
url='https://your-instance.fiddler.ai'
)
openai_client = OpenAI()
@trace(
as_type='generation',
capture_input=False, # Disable auto-capture to set attributes manually
capture_output=False,
model='gpt-4o-mini', # Sets gen_ai.request.model on the span
system='openai', # Sets gen_ai.system on the span
)
def call_llm(prompt: str) -> str:
span = get_current_span(as_type='generation')
if span:
span.set_user_prompt(prompt)
response = openai_client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}]
)
result = response.choices[0].message.content
if span:
span.set_completion(result)
span.set_usage(
input_tokens=response.usage.prompt_tokens,
output_tokens=response.usage.completion_tokens,
)
return result
@trace(as_type='tool')
def search_hotels(query: str) -> str:
span = get_current_span(as_type='tool')
if span:
span.set_tool_name('search_hotels')
span.set_tool_input({'query': query})
result = f'Found: Grand Hotel for query "{query}"'
if span:
span.set_tool_output(result)
return result
@trace(as_type='chain')
def run_agent(user_message: str) -> str:
span = get_current_span(as_type='chain')
if span:
span.set_agent_name('travel_agent')
span.set_input(user_message)
response = call_llm(user_message)
hotels = search_hotels(user_message)
result = f'{response}\n\n{hotels}'
if span:
span.set_output(result)
return result
# Set conversation ID to link multiple turns
set_conversation_id(str(uuid.uuid4()))
run_agent('Find me a hotel in Tokyo')
```
**`@trace` decorator parameters:**
| Parameter | Type | Default | Description |
| ---------------- | --------------------------------------------- | ------------- | --------------------------------------------------------- |
| `as_type` | `'span'`, `'generation'`, `'chain'`, `'tool'` | `'span'` | Span type for Fiddler categorization |
| `name` | `str \| None` | function name | Custom span name |
| `capture_input` | `bool` | `True` | Auto-serialize function arguments as span input |
| `capture_output` | `bool` | `True` | Auto-serialize return value as span output |
| `model` | `str \| None` | `None` | Sets `gen_ai.request.model` |
| `system` | `str \| None` | `None` | Sets `gen_ai.system` (LLM provider) |
| `user_id` | `str \| None` | `None` | Sets `user.id` |
| `version` | `str \| None` | `None` | Sets `service.version` |
| `client` | `FiddlerClient \| None` | `None` | Override the client to use (defaults to global singleton) |
**Using `get_current_span()` inside a decorated function:**
Call `get_current_span()` inside any `@trace`-decorated function to access the active span and set additional attributes. Pass `as_type` matching the decorator to get a typed wrapper with semantic helpers:
```python theme={null}
@trace(as_type='generation')
def my_llm_call(prompt: str) -> str:
span = get_current_span(as_type='generation')
if span:
span.set_model('gpt-4o-mini')
span.set_user_prompt(prompt)
span.set_system_prompt('You are a helpful assistant.')
...
```
**When to use `capture_input=False`:**
Set `capture_input=False` when you want to control exactly what gets recorded on the span (for example, to set `set_user_prompt()` instead of the default raw argument dict). This avoids double-recording the same data.
### Manual instrumentation
Use context managers for explicit span lifecycle control. This gives you full control over when spans start and end.
```python theme={null}
from openai import OpenAI
from fiddler_otel import FiddlerClient
client = FiddlerClient(
application_id='your-app-id',
api_key='your-api-key',
url='https://your-instance.fiddler.ai'
)
openai_client = OpenAI()
user_message = 'Find me a hotel in Tokyo'
# Context manager (automatic lifecycle)
with client.start_as_current_span('travel_agent', as_type='chain') as chain:
chain.set_agent_name('travel_agent')
chain.set_input(user_message)
with client.start_as_current_span('gpt-4o-mini', as_type='generation') as gen:
gen.set_model('gpt-4o-mini')
gen.set_system('openai')
gen.set_user_prompt(user_message)
response = openai_client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': user_message}]
)
result = response.choices[0].message.content
gen.set_completion(result)
gen.set_usage(
input_tokens=response.usage.prompt_tokens,
output_tokens=response.usage.completion_tokens,
)
with client.start_as_current_span('search_hotels', as_type='tool') as tool:
tool.set_tool_name('search_hotels')
tool.set_tool_input({'query': user_message})
tool_result = search_hotels(user_message)
tool.set_tool_output(tool_result)
chain.set_output(result)
```
**Manual lifecycle (explicit `end()`):**
Use `start_span()` when you need to manage the span lifecycle manually, for example across asynchronous callbacks:
```python theme={null}
prompt = 'Find me a hotel in Tokyo'
span = client.start_span('my_operation', as_type='generation')
try:
span.set_user_prompt(prompt)
result = call_llm(prompt) # your LLM call
span.set_completion(result)
except Exception as exc:
span.record_exception(exc)
raise
finally:
span.end()
```
## Span types and wrappers
The SDK provides four span types, each with semantic convention helpers:
### `FiddlerSpan` — base (any `as_type='span'`)
| Method | OTel Attribute | Description |
| --------------------------- | ------------------------ | ---------------------------------------------------------- |
| `set_input(data)` | `gen_ai.llm.input.user` | Auto-serializes dicts/lists to JSON |
| `set_output(data)` | `gen_ai.llm.output` | Auto-serializes dicts/lists to JSON |
| `set_attribute(key, value)` | custom | Set any custom attribute |
| `update(**kwargs)` | — | Bulk-set `input`, `output`, and any attributes in one call |
| `set_agent_name(name)` | `gen_ai.agent.name` | Agent label in Fiddler UI |
| `set_agent_id(id)` | `gen_ai.agent.id` | Agent identifier |
| `set_conversation_id(id)` | `gen_ai.conversation.id` | Links spans to a conversation |
| `record_exception(exc)` | — | Records exception and marks span ERROR |
| `end()` | — | End span (for manual lifecycle only) |
### `FiddlerGeneration` — LLM spans (`as_type='generation'`)
Extends `FiddlerSpan` with:
| Method | OTel Attribute | Description |
| ----------------------------------------------------------- | ------------------------- | ------------------------------------- |
| `set_model(name)` | `gen_ai.request.model` | LLM model name |
| `set_system(provider)` | `gen_ai.system` | LLM provider (e.g. `'openai'`) |
| `set_system_prompt(text)` | `gen_ai.llm.input.system` | System prompt text |
| `set_user_prompt(text)` | `gen_ai.llm.input.user` | User prompt (last human message) |
| `set_completion(text)` | `gen_ai.llm.output` | LLM completion text |
| `set_usage(input_tokens, output_tokens, total_tokens=None)` | `gen_ai.usage.*` | Token counts |
| `set_context(text)` | `gen_ai.llm.context` | Additional context |
| `set_messages(messages)` | `gen_ai.input.messages` | Full chat history (JSON) |
| `set_output_messages(messages)` | `gen_ai.output.messages` | Output messages (JSON) |
| `set_tool_definitions(defs)` | `gen_ai.tool.definitions` | Available tools (JSON, OpenAI format) |
`set_messages()` and `set_output_messages()` accept simple OpenAI format and auto-convert to OTel parts format:
```python theme={null}
gen.set_messages([
{'role': 'system', 'content': 'You are a travel agent.'},
{'role': 'user', 'content': 'Book a flight to Tokyo.'}
])
```
### `FiddlerTool` — tool spans (`as_type='tool'`)
Extends `FiddlerSpan` with:
| Method | OTel Attribute | Description |
| ---------------------------- | ------------------------- | ------------------------------------- |
| `set_tool_name(name)` | `gen_ai.tool.name` | Tool/function name |
| `set_tool_input(data)` | `gen_ai.tool.input` | Tool input (auto-serialized to JSON) |
| `set_tool_output(data)` | `gen_ai.tool.output` | Tool result (auto-serialized to JSON) |
| `set_tool_definitions(defs)` | `gen_ai.tool.definitions` | Available tools |
### `FiddlerChain` — pipeline spans (`as_type='chain'`)
Extends `FiddlerSpan` with no additional methods. Use for high-level orchestration spans that group multiple LLM calls and tool calls together.
## Advanced usage
### Multi-turn conversation tracking
Use `set_conversation_id()` to link multiple agent invocations into a single conversation in the Fiddler UI. Set a new UUID at the start of each conversation; all spans created in the current thread or async coroutine after this call will carry the same conversation ID.
```python theme={null}
from fiddler_otel import set_conversation_id
import uuid
# Call once per conversation — persists for the current thread/async task
set_conversation_id(str(uuid.uuid4()))
result1 = run_agent('What flights go to Tokyo?')
result2 = run_agent('Book the cheapest one.') # Same conversation_id
```
### Async agents
The `@trace` decorator automatically detects async functions and wraps them correctly. Use the same patterns as sync functions:
```python theme={null}
import asyncio
from openai import AsyncOpenAI
from fiddler_otel import FiddlerClient, trace, get_current_span
client = FiddlerClient(application_id='...', api_key='...', url='...')
async_openai_client = AsyncOpenAI()
@trace(as_type='generation', model='gpt-4o-mini')
async def call_llm_async(prompt: str) -> str:
span = get_current_span(as_type='generation')
if span:
span.set_user_prompt(prompt)
response = await async_openai_client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}]
)
result = response.choices[0].message.content
if span:
span.set_completion(result)
span.set_usage(
input_tokens=response.usage.prompt_tokens,
output_tokens=response.usage.completion_tokens,
)
return result
@trace(as_type='chain')
async def run_agent_async(message: str) -> str:
result = await call_llm_async(message)
return result
asyncio.run(run_agent_async('Hello'))
```
### Context isolation
`fiddler-otel` uses its own isolated OpenTelemetry context that does not interfere with any existing global tracer in your application. If you already use OpenTelemetry for infrastructure tracing, Fiddler spans will not appear in your infrastructure traces and vice versa.
This means you can safely add `fiddler-otel` to an application that already has OpenTelemetry instrumentation without any conflict.
### Global client singleton
The first `FiddlerClient` created in a process becomes the global singleton, accessible via `get_client()`. The `@trace` decorator uses this singleton automatically when `client=` is not passed:
```python theme={null}
from fiddler_otel import FiddlerClient, get_client, trace
# Created once at startup
client = FiddlerClient(application_id='...', api_key='...', url='...')
# Anywhere in your code — no need to pass client=
@trace(as_type='generation')
def call_llm(prompt: str) -> str:
...
# Or retrieve the singleton explicitly
fdl = get_client()
```
### Session attributes
Use `add_session_attributes()` to attach key-value metadata that is automatically applied to **all spans** created in the current thread or async coroutine. Use this for user-level or environment-level metadata that applies across an entire session — such as `user_id`, `environment`, or feature flags.
Attributes are emitted as `fiddler.session.user.{key}` on every span and propagated from parent to child spans automatically.
```python theme={null}
from fiddler_otel import FiddlerClient, trace, add_session_attributes
client = FiddlerClient(application_id='...', api_key='...', url='...')
# Set once per session — applies to all spans that follow in this context
add_session_attributes(key='user_id', value='user_12345')
add_session_attributes(key='environment', value='production')
add_session_attributes(key='tier', value='premium')
add_session_attributes(key='request_count', value=42) # int (since 1.1.1)
add_session_attributes(key='confidence_score', value=0.87) # float (since 1.1.1)
@trace(as_type='chain')
def run_agent(message: str) -> str:
# All spans inside this call carry user_id, environment, tier
...
```
Unlike `set_conversation_id()` (which links invocations into a conversation), `add_session_attributes` is for descriptive metadata. Both can be used together.
### Custom span attributes
Set any custom attribute on an individual span to add business context for that specific operation:
```python theme={null}
@trace(as_type='tool')
def search_hotels(query: str) -> str:
span = get_current_span(as_type='tool')
if span:
span.set_attribute('department', 'travel')
span.set_attribute('region', 'apac')
span.set_attribute('reward_points', 5.0)
...
```
### Production configuration
**Sampling (reduce volume):**
```python theme={null}
from opentelemetry.sdk.trace import sampling
client = FiddlerClient(
application_id='...',
api_key='...',
url='...',
sampler=sampling.TraceIdRatioBased(0.1), # Sample 10% of traces
)
```
**Span limits (large prompts):**
```python theme={null}
from opentelemetry.sdk.trace import SpanLimits
client = FiddlerClient(
application_id='...',
api_key='...',
url='...',
span_limits=SpanLimits(
max_span_attribute_length=8192, # Allow up to 8 KB per attribute
),
)
```
**Resource attributes (environment metadata):**
```python theme={null}
client = FiddlerClient(application_id='...', api_key='...', url='...')
client.update_resource({'service.version': '2.1.0', 'deployment.environment': 'production'})
# Must be called before get_tracer() is invoked (before the first @trace call)
```
### Flush and shutdown
`FiddlerClient` registers an `atexit` handler to flush and shut down automatically. For short scripts or critical workloads, call `force_flush()` explicitly to ensure all buffered spans are exported before the process exits:
```python theme={null}
# Sync
client.force_flush()
client.shutdown()
# Async
await client.aflush()
await client.ashutdown()
# Context manager (calls shutdown() on exit)
with FiddlerClient(application_id='...', api_key='...', url='...') as client:
run_agent('Hello')
```
### Local debugging
**Console output (print spans to stdout in addition to Fiddler export):**
`console_tracer=True` is **additive** — span data is printed to stdout **and** continues to be exported to Fiddler via OTLP. Setting this to `True` does **not** suppress or disable the OTLP export to Fiddler. Use it to visually confirm spans are being created during development.
```python theme={null}
client = FiddlerClient(
application_id='...',
api_key='...',
url='...',
console_tracer=True, # Prints spans to stdout; OTLP export to Fiddler still active
)
```
**JSONL file capture (save a local copy of spans in addition to Fiddler export):**
`jsonl_capture_enabled=True` is **additive** — spans are saved to a local JSONL file **and** continue to be exported to Fiddler via OTLP. Setting this to `True` does **not** suppress or disable the OTLP export to Fiddler. The JSONL format written here is a custom Fiddler format and is **not** compatible with the Fiddler S3 connector. To write S3-compatible files, use `otlp_json_capture_enabled=True` instead (see [Offline and S3 Routing Mode](#offline-and-s3-routing-mode) below).
```python theme={null}
client = FiddlerClient(
application_id='...',
api_key='...',
url='...',
jsonl_capture_enabled=True, # Saves spans to local file; OTLP export still active
jsonl_file_path='trace_data.jsonl',
)
```
Override the output file path via environment variable:
```bash theme={null}
FIDDLER_JSONL_FILE=trace_data.jsonl python my_agent.py
```
### Offline and S3 Routing Mode
Use this mode when traces must be routed through an intermediate store (such as Amazon S3) before reaching Fiddler, rather than being sent directly. This is the correct approach when your security or network policies require all data to pass through a controlled intermediary.
* **`otlp_enabled=False`** — disables all direct OTLP export to Fiddler. `api_key` and `url` are not required in this mode.
* **`otlp_json_capture_enabled=True`** — writes traces to local `.json` files in standard OTLP JSON format (`ExportTraceServiceRequest` envelope). These files are directly consumable by the Fiddler S3 connector.
* **`application_id` is still required** — even though no data is sent to Fiddler directly, the S3 connector uses the `application_id` embedded in the trace files to route ingested traces to the correct application in Fiddler.
```python theme={null}
from fiddler_otel import FiddlerClient
# No api_key or url needed — traces go to local files only
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID', # UUID4 — required for S3 connector routing
otlp_enabled=False, # Disables direct export to Fiddler
otlp_json_capture_enabled=True, # Writes OTLP JSON files locally
otlp_json_output_dir='./fiddler_traces', # Directory for output files (default: 'fiddler_traces')
)
```
After running your application, upload the generated `.json` files from `otlp_json_output_dir` to your S3 bucket. The Fiddler S3 connector reads them directly.
Each batch of spans is written to a separate timestamped `.json` file in the output directory. The directory is created automatically if it does not exist.
### Running in AWS SageMaker
Run the Fiddler OTel SDK inside an [Amazon SageMaker Partner AI App](/integrations/cloud-platforms-and-deployment/aws-sagemaker) to export agent traces with AWS Signature Version 4 (SigV4) signing. Authentication is **entirely environment-variable driven — no code changes are required.** Because the `fiddler-langchain` and `fiddler-langgraph` integrations export through `FiddlerClient`, SageMaker signing flows through them automatically as well.
Install the `sagemaker` extra (pre-installed in SageMaker-managed runtimes):
```bash theme={null}
pip install "fiddler-otel[sagemaker]"
```
Set the following environment variables. When SageMaker authentication is enabled, `FiddlerClient` attaches a SigV4-signing session to the OTLP exporter, so every trace export is signed and routed through the SageMaker partner-app proxy:
| Variable | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------ |
| `AWS_PARTNER_APP_AUTH` | Set to `true` to enable SageMaker Partner App authentication. |
| `AWS_PARTNER_APP_ARN` | The ARN of your Fiddler SageMaker Partner AI App. |
| `AWS_PARTNER_APP_URL` | The base URL of the partner-app endpoint. This is the same value you pass to `FiddlerClient` as `url`. |
AWS credentials are resolved through the standard boto3 credential chain (IAM role inside a SageMaker runtime, `~/.aws/credentials`, instance profile, or environment variables) — no additional AWS credential variables are needed when running inside the partner-app sandbox.
Your instrumentation code is unchanged from any other environment — `FiddlerClient` reads the SageMaker variables itself:
```python theme={null}
import os
from fiddler_otel import FiddlerClient, trace
# In a SageMaker Partner App, the Fiddler URL is the partner-app endpoint
# (the same value as AWS_PARTNER_APP_URL), not a *.fiddler.ai URL.
client = FiddlerClient(
application_id=os.environ['FIDDLER_APPLICATION_ID'], # UUID4
api_key=os.environ['FIDDLER_API_KEY'],
url=os.environ['AWS_PARTNER_APP_URL'],
)
@trace(as_type='generation')
def call_llm(prompt: str) -> str:
...
```
SageMaker authentication is resolved eagerly when you construct `FiddlerClient`. A misconfigured environment — a missing ARN or URL, missing AWS credentials, or the `sagemaker` extra not installed — raises an error at construction time rather than failing silently on the first trace export. Set [`FIDDLER_OTLP_DEBUG=true`](#local-debugging) to log each signed export while you validate the setup.
**Traces are not yet visible in the Fiddler UI on AWS.** In the current SageMaker environment, traces are signed and exported correctly but are not yet surfaced in the Fiddler UI. This is an environment limitation, not an SDK defect.
## Relationship to other Fiddler SDKs
`fiddler-otel` is the foundation package that all Fiddler SDK integrations build on:
| Package | Framework | Instrumentation Approach |
| ------------------- | ---------------------------------- | ------------------------------------------------------------ |
| `fiddler-otel` | Any Python application | `@trace` decorator, context managers |
| `fiddler-langchain` | LangChain V1 (`create_agent`) | `FiddlerLangChainInstrumentor` (auto-patches `create_agent`) |
| `fiddler-langgraph` | LangGraph (`StateGraph.compile()`) | `LangGraphInstrumentor` (callback handler) |
Both `fiddler-langchain` and `fiddler-langgraph` depend on `fiddler-otel` and re-export its core symbols (`FiddlerClient`, `trace`, `get_current_span`, `set_conversation_id`). If your application uses LangChain V1 (`create_agent` API) or LangGraph, install the framework-specific package — it includes `fiddler-otel` automatically.
## API reference
The `fiddler-otel` SDK provides the same core classes as `fiddler-langgraph` — the codebase is shared, and `fiddler-langgraph` re-exports all `fiddler-otel` symbols unchanged. Until a dedicated `fiddler-otel` API reference is autogenerated, the detailed reference pages are co-located in the shared SDK API reference. The classes, parameters, and behavior are identical regardless of which package you import from — use `fiddler_otel` as the import source when using the core SDK standalone.
| Class / Function | Import | Description |
| --------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------ |
| [FiddlerClient](/sdk-api/langgraph/fiddler-client) | `from fiddler_otel import FiddlerClient` | Configure the OTel tracer and manage the connection to Fiddler |
| [get\_client](/sdk-api/langgraph/get-client) | `from fiddler_otel import get_client` | Retrieve the active `FiddlerClient` instance |
| [trace](/sdk-api/langgraph/trace) | `from fiddler_otel import trace` | Decorator for automatic function instrumentation |
| [get\_current\_span](/sdk-api/langgraph/get-current-span) | `from fiddler_otel import get_current_span` | Access the active Fiddler span within a traced function |
| [FiddlerSpan](/sdk-api/langgraph/fiddler-span) | `from fiddler_otel import FiddlerSpan` | Base span wrapper |
| [FiddlerGeneration](/sdk-api/langgraph/fiddler-generation) | `from fiddler_otel import FiddlerGeneration` | Span wrapper for LLM calls |
| [FiddlerChain](/sdk-api/langgraph/fiddler-chain) | `from fiddler_otel import FiddlerChain` | Span wrapper for agent/chain workflows |
| [FiddlerTool](/sdk-api/langgraph/fiddler-tool) | `from fiddler_otel import FiddlerTool` | Span wrapper for tool calls |
| [add\_session\_attributes](/sdk-api/langgraph/add-session-attributes) | `from fiddler_otel import add_session_attributes` | Add session-level attributes applied to all spans in the current context |
| [set\_conversation\_id](/sdk-api/langgraph/set-conversation-id) | `from fiddler_otel import set_conversation_id` | Link spans into a single conversation |
## Troubleshooting
### No spans appearing in Fiddler
1. **Check your credentials** — verify `api_key`, `application_id` (must be a valid UUID4), and `url` are correct. These are only required when `otlp_enabled=True` (the default).
2. **Force flush before exit** — for short scripts, the `BatchSpanProcessor` may not flush before the process exits. Call `client.force_flush()` or use the context manager (`with FiddlerClient(...) as client:`).
3. **Enable console tracing** — set `console_tracer=True` to also print spans to stdout and confirm they are being created. This is additive; OTLP export to Fiddler continues alongside console output:
```python theme={null}
client = FiddlerClient(
application_id='...',
api_key='...',
url='...',
console_tracer=True, # Adds console output; does NOT disable OTLP export
)
```
4. **Check the application ID** — the `application_id` must match an existing application in your Fiddler instance and must be a valid UUID4. `FiddlerClient` raises `ValueError` on initialization if the format is invalid.
### `RuntimeError: No FiddlerClient initialized`
`get_current_span()` or `get_client()` was called before a `FiddlerClient` was created. Create the client before decorating or calling any instrumented functions:
```python theme={null}
from fiddler_otel import FiddlerClient, trace
client = FiddlerClient(application_id='...', api_key='...', url='...')
@trace(as_type='generation')
def call_llm(prompt: str) -> str:
...
```
### `ValueError: application_id must be a valid UUID4`
The `application_id` passed to `FiddlerClient` is not a valid UUID version 4. Copy the Application ID directly from the **GenAI Applications** page in the Fiddler UI — it should look like `550e8400-e29b-41d4-a716-446655440000`.
### Spans missing from async code
Context variables propagate correctly across `await` in the same async task. If you are spawning new tasks with `asyncio.create_task()`, call `set_conversation_id()` and `add_session_attributes()` inside the task so the context is re-established. Use `client.ashutdown()` instead of `client.shutdown()` to avoid blocking the event loop during teardown.
### Spans interfering with another OpenTelemetry tracer
`FiddlerClient` uses an isolated `Context` that is separate from the global OTel context. Spans created via `@trace` or `start_as_current_span()` will not appear in any other tracer, and spans from other tracers will not appear in Fiddler. This isolation is intentional and requires no configuration.
### Local JSONL file is empty
Ensure `jsonl_capture_enabled=True` is set on `FiddlerClient` and that the process has executed instrumented code. The JSONL file is written synchronously, so spans appear immediately after each span ends. Check the path: the default is `fiddler_trace_data.jsonl` in the current working directory; override with `jsonl_file_path` or the `FIDDLER_JSONL_FILE` environment variable.
**Note:** `jsonl_capture_enabled=True` is additive — it saves a local copy of spans while OTLP export to Fiddler continues. If your goal is to write files for S3 upload and stop sending directly to Fiddler, use `otlp_enabled=False` combined with `otlp_json_capture_enabled=True` instead. See [Offline and S3 Routing Mode](#offline-and-s3-routing-mode).
***
## What's next?
* [**LangChain SDK**](/integrations/agentic-ai/langchain-sdk) — If your application uses LangChain V1 `create_agent`
* [**LangGraph SDK**](/integrations/agentic-ai/langgraph-sdk) — If your application uses LangGraph
* [**Agentic Observability Concepts**](/getting-started/agentic-monitoring) — Understand the agent lifecycle and monitoring approach
# Fiddler Google ADK SDK
Source: https://docs.fiddler.ai/integrations/agentic-ai/google-adk-sdk
Native monitoring for Google ADK agents with Fiddler's purpose-built SDK
[](https://pypi.org/project/fiddler-adk/)
Monitor Google ADK (Agent Development Kit) applications with Fiddler's purpose-built SDK. Get deep visibility into agent reasoning, LLM interactions, and tool execution for ADK-based agent applications.
**Platform Compatibility:** Works with ADK agents using either Google Gemini API keys or Vertex AI authentication.
## What You'll Need
* Fiddler account (cloud or on-premises)
* Google ADK agent application
* Python 3.10 or higher
* Fiddler API key
* Google Gemini API key or Vertex AI credentials
## Quick Start
```bash theme={null}
# Step 1: Install (uv recommended)
uv add fiddler-adk
# or: pip install fiddler-adk
```
```python theme={null}
# Step 2: Set up instrumentation
from fiddler_otel import FiddlerClient
from fiddler_adk import GoogleADKInstrumentor
client = FiddlerClient(
api_key="YOUR_FIDDLER_API_KEY",
application_id="YOUR_APPLICATION_UUID",
url="https://your-fiddler-instance.com",
)
GoogleADKInstrumentor(client).instrument()
# Step 3: Create your ADK agent as usual
from google.adk.agents.llm_agent import Agent
agent = Agent(
model="gemini-2.5-flash",
name="my_agent",
description="A helpful assistant",
instruction="You are a helpful assistant. Be concise.",
)
# Step 4: Agent calls are automatically traced
# (use Runner to execute the agent — see Quick Start Guide for full example)
```
## What Gets Monitored
### ADK Agent Operations
* **Agent Invocations** (`invoke_agent`) - Full agent execution with timing and session tracking
* **LLM Calls** (`call_llm`) - LLM request/response capture with input, output, system instructions, and tool definitions
* **Tool Execution** (`execute_tool`) - Tool call arguments and return values
* **Model Inference** (`generate_content`) - Token usage, model name, and finish reasons
### Captured Attributes
* **LLM Input/Output** - User prompt text, model response text, system instructions
* **Tool I/O** - Tool call arguments (JSON) and tool response payloads
* **Token Usage** - Input tokens, output tokens, reasoning tokens
* **Agent Identity** - Agent name, agent ID, session/conversation ID
* **Finish Reasons** - LLM stop reason per generation
## Configuration Options
### Programmatic Configuration
```python theme={null}
from fiddler_otel import FiddlerClient
from fiddler_adk import GoogleADKInstrumentor
client = FiddlerClient(
api_key="YOUR_FIDDLER_API_KEY",
application_id="YOUR_APPLICATION_UUID", # UUID4 from Fiddler GenAI Applications
url="https://your-fiddler-instance.com",
)
GoogleADKInstrumentor(client).instrument()
# All agents created after this point are automatically instrumented
```
### Google Authentication
ADK supports two authentication methods for accessing Gemini models:
```bash theme={null}
# Option A: Gemini API key
export GOOGLE_API_KEY="your-gemini-api-key"
# Option B: Vertex AI (uses gcloud Application Default Credentials)
export GOOGLE_GENAI_USE_VERTEXAI=1
export GOOGLE_CLOUD_PROJECT="your-gcp-project-id"
export GOOGLE_CLOUD_LOCATION="us-central1"
```
### Content Capture
ADK includes full LLM request/response payloads in span attributes by default. To disable payload capture (e.g., for PII protection):
```bash theme={null}
export ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false
```
## Example Applications
### Document Processing Agent with Tools
```python theme={null}
import asyncio
from fiddler_otel import FiddlerClient
from fiddler_adk import GoogleADKInstrumentor
from google.adk.agents.llm_agent import Agent
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types
# Set up Fiddler instrumentation
client = FiddlerClient(
api_key="YOUR_FIDDLER_API_KEY",
application_id="YOUR_APPLICATION_UUID",
url="https://your-fiddler-instance.com",
)
GoogleADKInstrumentor(client).instrument()
# Define tools
def classify_document(doc_type: str, content: str) -> dict:
"""Classify a document into a known category."""
return {"category": "Credit Memo", "confidence": 0.95}
def extract_fields(document_id: str) -> dict:
"""Extract structured fields from a document."""
return {"supplier": "McKesson", "total": "$12,450.00"}
# Create agent with tools
agent = Agent(
model="gemini-2.5-flash",
name="doc_processor",
description="Document processing assistant",
instruction="You classify and extract fields from medical documents.",
tools=[classify_document, extract_fields],
)
async def main():
session_service = InMemorySessionService()
runner = Runner(agent=agent, app_name="demo", session_service=session_service)
session = await session_service.create_session(app_name="demo", user_id="user1")
message = types.Content(
role="user",
parts=[types.Part(text="Classify this credit memo: supplier McKesson, amount $5000")],
)
async for event in runner.run_async(
user_id="user1", session_id=session.id, new_message=message
):
if hasattr(event, "content") and event.content:
parts = getattr(event.content, "parts", []) or []
text = "".join(p.text for p in parts if getattr(p, "text", None))
if text:
print(f"Agent: {text}")
# Flush traces to Fiddler before exit
client.force_flush(timeout_millis=5000)
asyncio.run(main())
```
[**View complete example -->**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Google_ADK_Integration.ipynb)
## Viewing Your Data
Navigate to Fiddler UI to analyze Google ADK agent performance:
1. **Explorer** - View full span trees: agent > LLM > tool hierarchy
2. **Session Analysis** - Multi-turn conversation flows grouped by session
3. **LLM Performance** - Input/output content, token usage, latency
4. **Tool Metrics** - Tool call arguments, responses, and execution time
5. **Cost Tracking** - Token usage per agent session
### Span Types
| ADK Operation | Fiddler Span Type | Content |
| ------------------ | ----------------- | --------------------------------------------------------------- |
| `call_llm` | LLM | User input, model output, system instructions, tool definitions |
| `invoke_agent` | Agent | Agent name, session ID |
| `execute_tool` | Tool | Tool arguments, tool response |
| `generate_content` | Span | Token counts, model name, finish reason |
| `invocation` | Span | Root span per turn |
## How It Works
Google ADK emits OpenTelemetry spans natively through a tracer named `gcp.vertex.agent`. The `fiddler-adk` SDK:
1. **Sets up an isolated tracing pipeline** via `FiddlerClient` (provider, processor, OTLP exporter) and promotes it to the global tracer provider so ADK's tracer resolves to it.
2. **Propagates session identity** by backfilling `gen_ai.conversation.id` from child spans onto the root `invocation` span, ensuring all spans in a turn share the same session ID.
3. **Delegates content extraction to the Fiddler backend**, which parses ADK's JSON span attributes (`gcp.vertex.agent.llm_request`, etc.) into human-readable fields.
The SDK operates in standalone mode -- it does not interact with customer-configured tracers or providers.
## Troubleshooting
### Traces Not Appearing in Fiddler
**Verify credentials:**
```python theme={null}
from fiddler_otel import FiddlerClient
client = FiddlerClient(
api_key="YOUR_KEY",
application_id="YOUR_APP_UUID",
url="https://your-instance.com",
)
# If credentials are wrong, you'll see 401 errors in logs
```
**Check instrumentation is active:**
```python theme={null}
import logging
logging.basicConfig(level=logging.INFO)
# Look for: "Google ADK instrumentation enabled"
```
**Ensure traces are flushed before exit:**
```python theme={null}
# ADK agents run async — ensure flush before process exits
client.force_flush(timeout_millis=10000)
```
### Missing Content on Spans
Content extraction (LLM input/output, tool I/O) is handled by the Fiddler backend. If you see raw JSON instead of extracted text, ensure your Fiddler instance has the Google ADK backend mapper deployed.
### Orphan Root Spans
If `invocation` root spans appear disconnected from their children, update to the latest `fiddler-adk` version. The `ADKSpanProcessor` backfills `gen_ai.conversation.id` to root spans automatically.
## Related Documentation
* [**Google ADK Quick Start**](/developers/quick-starts/google-adk-quick-start) - Detailed setup guide
* [**Fiddler Evals SDK**](/integrations/agentic-ai/evals-sdk) - Evaluate agent quality
* [**Google ADK SDK Reference**](/sdk-api/adk/google-adk-instrumentor) - Complete class and method documentation
# Kong AI Gateway Integration
Source: https://docs.fiddler.ai/integrations/agentic-ai/kong-integration
Integrate Kong AI Gateway with Fiddler for zero-instrumentation LLM observability — no application code changes required.
## Overview
[Kong AI Gateway](https://konghq.com/products/kong-ai-gateway) (3.13 or later) is an API gateway with built-in AI proxy and OpenTelemetry support. Fiddler integrates with Kong at the gateway layer via Kong's `opentelemetry` plugin, giving you full LLM observability — prompts, responses, token usage, latency — without adding any SDK to your application code.
| Capability | Notes |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Zero instrumentation** | Point your app at Kong instead of `api.openai.com` — no code changes needed |
| **LLM span tracing** | Token counts, model name, latency, and content (with `log_payloads: true`) |
| **Multi-provider support** | OpenAI, Anthropic, Cohere, Azure OpenAI, Google Gemini, and more via the `ai-proxy` plugin |
| **Direct OTLP export** | Kong exports traces directly to Fiddler over HTTPS with auth headers |
| **Guardrails** | Block personally identifiable information (PII) and secrets at the gateway via the `ai-custom-guardrail` plugin (requires Kong 3.14+ with an AI Gateway Enterprise license) — see [Kong Guardrails](/protection/kong-guardrails) |
Kong's Gen AI OpenTelemetry tracing is labelled [Tech Preview](https://developer.konghq.com/ai-gateway/llm-open-telemetry/) by Kong — "currently in Tech Preview and should not be used in a production environment." This page's tracing setup is built on that feature; re-verify after Kong upgrades. The current Kong Gateway release is 3.15.0.3 (as of 2026-08-19); the 3.13+ floor is the minimum that emits Gen AI span attributes.
## Architecture
```mermaid theme={null}
flowchart TD
A["Your app (any language, standard OpenAI client)"] -->|points base_url at Kong :8000| B["Kong AI Gateway (port 8000)"]
B -->|proxies to OpenAI / Anthropic / …| C["LLM Provider"]
B -->|OTLP/HTTPS with auth headers| D["Fiddler (enrichment, monitoring, dashboards)"]
```
Kong exposes an OpenAI-compatible endpoint at `/openai`. Your application requires no SDK — it just calls Kong instead of the provider directly.
## Prerequisites
* Fiddler account with a GenAI application already created
* A running [Kong Gateway](https://konghq.com/install) **3.13+** instance (Gen AI OTel attributes require 3.13+)
* A valid LLM provider API key (e.g. `OPENAI_API_KEY`)
* Your Fiddler API key (found under organizational settings) and application UUID (found under application settings)
## Quick Start
### Step 1: Create the Kong Configuration File
Save the following as `kong_fiddler_config.yaml`. Kong performs no shell-style `${VAR}` substitution in its declarative config, so the `${...}` placeholders below must be handled one of two ways. On fields Kong marks *referenceable* — such as `ai-proxy`'s `auth.header_value` — you can use a [Kong Vault reference](https://developer.konghq.com/gateway/entities/vault/) like `{vault://env/OPENAI_KEY}`, which Kong resolves from an environment variable at runtime (supported in declarative config on OSS and Enterprise alike, at no license cost). On non-referenceable fields you replace the placeholder with the real value in Step 2. The example uses a vault reference for the provider key and literal replacement for the rest.
```yaml theme={null}
_format_version: "3.0"
services:
- name: openai-service
host: api.openai.com
port: 443
protocol: https
routes:
- name: openai-chat-route
paths:
- /openai
strip_path: true
plugins:
- name: ai-proxy
service: openai-service
config:
route_type: llm/v1/chat
auth:
header_name: Authorization
# auth.header_value is Vault-referenceable, so the provider key need not be
# written into the file. A vault reference must be the WHOLE field value —
# export the complete header: export OPENAI_KEY='Bearer sk-...'
header_value: "{vault://env/OPENAI_KEY}"
model:
provider: openai
name: gpt-5-mini
options:
max_tokens: 512
temperature: 0.7
# Set log_payloads: true to include prompt and completion text in OTel spans.
# WARNING: payloads may contain PII. Disable in production unless needed.
logging:
log_statistics: true
log_payloads: true
# Session grouping: read the X-Fiddler-Conversation-Id request header and stamp
# it as gen_ai.conversation.id on the request's OTel spans, so multi-turn calls
# sharing one conversation id group into a single Fiddler Session.
# See the "Session Grouping" section below for how this works.
- name: pre-function
config:
access:
- |
local conv_id = kong.request.get_header("X-Fiddler-Conversation-Id")
if conv_id and conv_id ~= "" then
ngx.ctx.fiddler_conversation_id = conv_id
end
header_filter:
- |
local conv_id = ngx.ctx.fiddler_conversation_id
if conv_id then
local spans = ngx.ctx.KONG_SPANS
if spans then
for _, span in ipairs(spans) do
span:set_attribute("gen_ai.conversation.id", conv_id)
end
end
end
- name: opentelemetry
config:
traces_endpoint: "${FIDDLER_URL}/v1/traces"
# Unlike ai-proxy's auth.header_value, the opentelemetry plugin's config.headers
# are NOT Vault-referenceable, so the Fiddler API key must be written in literally.
# Restrict the file's permissions and inject it at deploy time from a secret manager.
headers:
Authorization: "Bearer ${FIDDLER_API_KEY}"
fiddler-application-id: "${FIDDLER_APP_ID}"
resource_attributes:
service.name: kong
application.id: "${FIDDLER_APP_ID}"
propagation:
default_format: w3c
```
`service.name: kong` in `resource_attributes` is how Fiddler routes Kong spans to its Kong mapper, and setting it is the documented default on every release. From 26.17, Fiddler also recognizes Kong's `kong-internal` instrumentation scope; scope-based routing is evaluated first, so a Kong span is routed correctly even if a deployment overrides `service.name`. `application.id` is also required: a resource with no `application.id` is skipped — Fiddler logs a warning and increments a `missing_application_id_failure` counter, so the drop is visible in ingestion logs and metrics rather than silent.
### Step 2: Replace the Placeholders With Your Values
Supply each value the way its field allows. Kong resolves a `{vault://env/...}` reference only on *referenceable* fields; on all other fields you replace the `${...}` placeholder with the literal value before starting Kong.
| Value | Field | How to supply it |
| ------------------------ | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Provider API key | `ai-proxy` `auth.header_value` — **referenceable** | Keep the `{vault://env/OPENAI_KEY}` reference in the file and export the whole header value: `export OPENAI_KEY='Bearer sk-...'` |
| Fiddler API key | `opentelemetry` `config.headers` — **not referenceable** | Replace `${FIDDLER_API_KEY}` with the literal key |
| Fiddler instance URL | `opentelemetry` `config.traces_endpoint` | Replace `${FIDDLER_URL}` (a URL, not a secret) |
| Fiddler application UUID | `resource_attributes.application.id` and the `fiddler-application-id` header | Replace `${FIDDLER_APP_ID}` |
Because the Fiddler API key must be written in literally, the finished file contains a live secret: restrict its file permissions and inject it at deploy time from your secret manager rather than committing it. If any `${...}` placeholder is left unreplaced, Kong fails to start with an error like `'traces_endpoint': missing host in url`. Apply the config the way you already manage Kong — a DB-less declarative file, [decK](https://docs.konghq.com/deck/), the Admin API, or your Helm chart's config.
If you already have a Kong declarative config, you do not need to replace your existing file. Copy just the three plugin entries (`ai-proxy`, `pre-function`, `opentelemetry`) and add them under your existing `plugins:` block. The `services:` and `routes:` blocks in the example above are only needed if you do not already have an OpenAI route configured.
### Step 3: Enable Tracing on Kong
The `opentelemetry` plugin only emits spans if Kong's tracing is enabled at the process level. These three settings **cannot** be set in the declarative config — set them wherever your Kong reads its configuration (`kong.conf`, `KONG_*` environment variables, or your Helm chart's `env:` values):
| Setting (`kong.conf`) | Environment variable | Value |
| -------------------------- | ------------------------------- | ----- |
| `tracing_instrumentations` | `KONG_TRACING_INSTRUMENTATIONS` | `all` |
| `tracing_sampling_rate` | `KONG_TRACING_SAMPLING_RATE` | `1.0` |
| `untrusted_lua` | `KONG_UNTRUSTED_LUA` | `on` |
`tracing_instrumentations` and `tracing_sampling_rate` are what make Kong produce OTel spans at all — without them no spans are emitted regardless of the `opentelemetry` plugin config. `untrusted_lua = on` is required for the `pre-function` session-grouping plugin to run its inline Lua.
### Step 4: Point Your Application at Kong
```python theme={null}
import os
import uuid
from openai import OpenAI
KONG_URL = os.getenv("KONG_URL", "http://localhost:8000")
# api_key is not forwarded to OpenAI — Kong handles auth via ai-proxy.
client = OpenAI(base_url=f"{KONG_URL}/openai", api_key="kong-managed")
# Generate one conversation UUID per session and send it on every LLM call
# so Fiddler groups them into a single Session.
session_id = str(uuid.uuid4())
response = client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": "What is the capital of France?"}],
extra_headers={"X-Fiddler-Conversation-Id": session_id},
)
print(response.choices[0].message.content)
```
Traces appear in Fiddler automatically — no SDK import, no callback registration, no changes to your application logic.
To override the default Kong URL, set `KONG_URL` (defaults to `http://localhost:8000`):
```bash theme={null}
export KONG_URL="http://my-kong-host:8000"
```
### Step 5: Verify Traces Are Arriving
Open the Fiddler UI and navigate to your application's **Explorer**. You should see the trace within a few seconds of making your first completion call.
## Span Type Mapping
Fiddler classifies Kong spans based on the span name and `gen_ai.operation.name`.
Kong 3.13 names LLM spans `"{operation} {model}"` (e.g. `"chat gpt-5-mini"`). All infrastructure spans start with `"kong"`.
| Kong span | Fiddler treatment |
| --------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `"chat {model}"`, `"text_completion {model}"`, `"generate_content {model}"` | `llm` (forwarded) |
| `fiddler-guardrail` (emitted by the guardrail `check` function) | `guardrail` (forwarded) — see [Kong Guardrails](/protection/kong-guardrails) |
| A Gen AI span with any other operation (for example `embeddings`) | dropped (not a recognized LLM operation) |
| `kong` (root HTTP span) | dropped (infrastructure) |
| `kong.router` | dropped (infrastructure) |
| `kong.access.plugin.ai-proxy` | dropped (infrastructure) |
| `kong.dns`, `kong.balancer` | dropped (infrastructure) |
| Any other span starting with `"kong"` | dropped (infrastructure) |
Kong emits a span hierarchy per request: a root HTTP span (`kong`), routing and plugin spans (all starting with `kong.`), and the LLM Gen AI span (named `"{operation} {model}"`). Fiddler keeps only the spans it classifies — the LLM span, plus a `fiddler-guardrail` span if you run guardrails — and drops every other Kong-resource span. That includes infrastructure spans **and** any Gen AI operation outside `chat`, `text_completion`, and `generate_content`: an embeddings call through the same route is dropped with no separate signal. Kong is the only Fiddler integration that filters its own infrastructure spans this way. The forwarded LLM span carries `gen_ai.conversation.id`, so multi-turn calls sharing one conversation id group under a single Session.
## Attribute Mapping
Kong follows the OTel Gen AI semantic conventions. Fiddler's mapper normalizes these automatically:
| Kong attribute | Fiddler treatment |
| ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `gen_ai.provider.name` | Passed through unchanged (e.g. `"openai"`, `"anthropic"`) |
| `gen_ai.request.model`, `gen_ai.response.model` | Passed through unchanged |
| `gen_ai.usage.input_tokens` | Passed through unchanged |
| `gen_ai.usage.output_tokens` | Passed through unchanged |
| `gen_ai.usage.total_tokens` | Computed as `input + output` when absent |
| `gen_ai.agent.name` | Defaulted to `` when absent |
| `gen_ai.conversation.id` | Defaulted to `trace_id.hex()` when absent |
| `gen_ai.input.messages` | Parsed into `gen_ai.llm.input.system` (first system message) and `gen_ai.llm.input.user` (last user message); the latter surfaces as the **Input** column (requires `log_payloads: true` on ai-proxy) |
| `gen_ai.output.messages` | Parsed into `gen_ai.llm.output` (assistant message); surfaces as the **Output** column (requires `log_payloads: true` on ai-proxy) |
## Session Grouping
Each Kong request is its own trace, so without a shared `gen_ai.conversation.id` Fiddler falls back to `trace_id.hex()` and every LLM call shows up as a separate Session. To group a multi-turn conversation into one Session, the **same** `gen_ai.conversation.id` must be set on each call's LLM span.
Kong 3.13 has **no native conversation-id field** — it only supports W3C `traceparent` for distributed tracing. The Quick Start config above solves this with the built-in **`pre-function`** plugin (this is the approach Kong Support recommends for adding a custom attribute from a request header):
* In the **`access`** phase it reads the `X-Fiddler-Conversation-Id` request header and stashes it in the per-request `ngx.ctx`.
* In the **`header_filter`** phase it stamps that value as `gen_ai.conversation.id` on the request's OTel spans, *before* the `opentelemetry` plugin exports them in the `log` phase.
Because only the LLM span is forwarded to Fiddler (Kong's infrastructure spans are dropped), stamping in `header_filter` is sufficient: the LLM span already exists at that point, so it always receives the conversation id. There are no later-created infrastructure spans to worry about — they are discarded by Fiddler's Kong mapper before reaching the UI.
Your application just sends the same header value on every call in a conversation:
```python theme={null}
import uuid
conversation_id = str(uuid.uuid4()) # one per conversation
client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": "..."}],
extra_headers={"X-Fiddler-Conversation-Id": conversation_id},
)
```
**Why iterate `ngx.ctx.KONG_SPANS` instead of `kong.tracing.active_span()`?** The documented `kong.tracing.active_span()` returns the **root (`kong`) span**, which Fiddler drops as infrastructure. Fiddler keeps only Kong's Gen AI/LLM span, so the attribute must land on that span — iterating every span guarantees it does, regardless of which span is the LLM one. `ngx.ctx.KONG_SPANS` is an internal Kong field rather than a stable public API. This mechanism has **no automated test coverage in Fiddler** — no unit, fixture, or e2e case exercises the header-to-`ngx.ctx`-to-attribute path or the `KONG_UNTRUSTED_LUA` requirement — and Kong has no local Fiddler harness, so treat it as manually verified only and re-verify session grouping after every Kong upgrade. The `pre-function` plugin also requires `KONG_UNTRUSTED_LUA=on` and a sampling rate of `1.0` (so spans always exist when the plugin runs). If you would rather not depend on an internal field, use the application-side alternative below.
**Alternative — application-side root span (no Kong plugin).** If you already instrument your app with the OpenTelemetry SDK, open a parent span per conversation, tag it, and let Kong nest its spans under your trace via the propagated `traceparent`. This is the pattern in Kong's [Voice AI observability cookbook](https://developer.konghq.com/cookbooks/voice-ai-observability/). It avoids the `pre-function` plugin but requires OTel SDK code in your application.
## Guardrails
Beyond tracing, Kong can enforce [Fiddler Guardrails](/protection/kong-guardrails) on prompts before they reach the model — blocking PII and secrets at the gateway via Kong's `ai-custom-guardrail` plugin (which requires Kong 3.14+ with an AI Gateway Enterprise license). The guardrail plugin's own `check` function can also emit a connected Guardrail span in the same trace as the LLM span.
See [Kong Guardrails](/protection/kong-guardrails) for setup, check behavior, and the full configuration.
## Troubleshooting
**Kong fails to start (`missing host in url` or similar)**
This means the config still contains `${...}` placeholders. Kong does not read environment variables — open `kong_fiddler_config.yaml` and replace every `${...}` with your actual value (see [Step 2](#step-2-replace-the-placeholders-with-your-values)).
**Traces not appearing in Fiddler**
Verify `kong_fiddler_config.yaml` has no `${...}` placeholders left (every value filled in):
```bash theme={null}
grep '\${' kong_fiddler_config.yaml # prints nothing when fully filled in
```
Also confirm Kong is emitting spans at all — check your Kong logs for OpenTelemetry export activity (for example, `grep -i otel` over your Kong proxy/error logs).
Both `application.id` (OTel resource attribute) and `fiddler-application-id` (HTTP header on the export request) are required. If `application.id` is missing, Fiddler skips those spans, logs a warning, and increments a `missing_application_id_failure` counter — so check your Fiddler ingestion logs and metrics if traces are absent. Fiddler checks that the attribute is present; it does not validate the UUID's format at this stage.
**Prompt and response content not showing**
`log_payloads: true` is required in the `ai-proxy` plugin `logging` config. Without it, `gen_ai.input.messages` and `gen_ai.output.messages` are not captured by Kong, so Fiddler will show token counts and model info but no text content.
**Spans not being emitted by Kong**
Confirm tracing is enabled at the Kong process level (see [Step 3](#step-3-enable-tracing-on-kong)): `tracing_instrumentations=all` and `tracing_sampling_rate=1.0`, set as `kong.conf` settings or `KONG_*` environment variables. These cannot be configured via the declarative config file.
**Span type showing as Unknown, or unexpected extra spans in the trace**
Fiddler routes Kong spans to its Kong mapper by `service.name == "kong"` (exact string match, case-sensitive) and, from 26.17, also by the `kong-internal` instrumentation scope. Verify the `resource_attributes` block in the `opentelemetry` plugin config has `service.name: kong`. The same routing decision also gates the dropping of Kong's infrastructure spans, so a mismatch has two symptoms at once: LLM spans show as `Unknown`, **and** Kong's infrastructure spans stop being filtered out, appearing as extra orphaned spans in the trace.
For guardrail-specific troubleshooting, see [Kong Guardrails → Troubleshooting](/protection/kong-guardrails#troubleshooting).
## Known Limitations
| Limitation | Details |
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Content requires payload logging** | Prompt and completion text are only captured when `log_payloads: true` is set on the `ai-proxy` plugin. This may expose PII — disable in production if needed. |
| **Session grouping relies on a `pre-function` plugin** | Kong has no native conversation-id field, so the Quick Start config uses a `pre-function` plugin to map the `X-Fiddler-Conversation-Id` header onto spans (requires `KONG_UNTRUSTED_LUA=on`). It reads spans from the internal `ngx.ctx.KONG_SPANS`, which is not a stable public API and has no automated test coverage in Fiddler — re-verify after every Kong upgrade. See [Session Grouping](#session-grouping). |
| **Kong Gateway 3.13+ required** | Gen AI semantic convention attributes (`gen_ai.*`) are only emitted by Kong 3.13+. Earlier versions emit only HTTP-level spans. |
## Deployment Modes
Fiddler's Kong tracing works wherever Kong runs the `opentelemetry` plugin and can export over HTTPS. Two conveniences documented here and on the guardrails page — the `pre-function` session-grouping plugin and the guardrail `check` function — additionally require `untrusted_lua = on`, a process-level setting that some managed Kong offerings do not expose.
| Deployment mode | Tracing (`opentelemetry`) | `untrusted_lua` features (session grouping, guardrail `check`) |
| ----------------------------------------------------------------------- | ------------------------- | ----------------------------------------------------------------------- |
| Self-managed — traditional (DB-backed), DB-less, hybrid, Kubernetes/KIC | Documented | Documented (you control the process settings) |
| Konnect-managed and Kong-managed dedicated cloud data planes | Unverified | Unverified — `untrusted_lua` may not be settable on managed data planes |
`Unverified` means Fiddler has not confirmed the mode — it is not a statement that the mode is unsupported. If your managed gateway does not allow `untrusted_lua = on`, use the [application-side session-grouping alternative](#session-grouping), which needs no Kong plugin.
## Next Steps
* [Kong Guardrails](/protection/kong-guardrails) — block PII and secrets at the Kong gateway.
* [AgentGateway Integration](/integrations/agentic-ai/agentgateway-integration) — add Fiddler observability through the AgentGateway proxy.
* [LiteLLM Integration](/integrations/agentic-ai/litellm-integration) — add Fiddler observability through the LiteLLM proxy.
* [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration) — instrument a custom framework with the OTel SDK.
* [OTel Trace Export](/integrations/agentic-ai/otel-trace-export) — export OTLP directly to Fiddler without a proxy.
* [Kong AI Gateway documentation](https://docs.konghq.com/gateway/latest/ai-gateway/) — Kong's official AI Gateway docs.
# Fiddler LangChain SDK
Source: https://docs.fiddler.ai/integrations/agentic-ai/langchain-sdk
Instrument LangChain V1 agents with Fiddler observability
[](https://pypi.org/project/fiddler-langchain/)
Instrument your LangChain V1 agents built with `langchain.agents.create_agent` for comprehensive agentic observability. The Fiddler LangChain SDK produces a clean, flat trace hierarchy — agent → LLM calls → tool calls — with no noisy Chain wrappers. One call to `FiddlerLangChainInstrumentor.instrument()` auto-traces every agent in your application.
**Using LangChain prior to v1?** The `fiddler-langchain` SDK requires LangChain v1 (`langchain.agents.create_agent` API). For applications using earlier LangChain versions or LangGraph workflows, use the [Fiddler LangGraph SDK](/integrations/agentic-ai/langgraph-sdk) instead — it covers both LangGraph and earlier LangChain-based agents.
## What you'll need
* Fiddler account (cloud or on-premises)
* Python 3.10-3.14
* LangChain V1 application using `langchain.agents.create_agent`
* Fiddler API key and application ID
## Quick start
Get monitoring in 4 steps:
```bash theme={null}
# Step 1: Install
pip install fiddler-langchain
```
```python theme={null}
# Step 2: Initialize the Fiddler client
import langchain.agents
from langchain_openai import ChatOpenAI
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(
application_id='your-app-id', # Must be valid UUID4
api_key='your-api-key',
url='https://your-instance.fiddler.ai'
)
# Step 3: Instrument (patches create_agent globally)
FiddlerLangChainInstrumentor(client=client).instrument()
# Step 4: Create your agent — Fiddler middleware is injected automatically
agent = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[...],
name='my_agent', # Optional — label this agent in traces
)
result = agent.invoke({'messages': [{'role': 'user', 'content': 'Hello!'}]})
```
That's it! Your agent traces are now flowing to Fiddler.
**Important:** Use `langchain.agents.create_agent` (the module attribute), not `from langchain.agents import create_agent`. Call `instrument()` before importing `create_agent` if you use the `from ... import` style so the local name is bound to the patched version.
## What gets monitored
### Trace hierarchy
Each agent invocation produces a clean, flat trace with no noisy Chain wrappers:
```
[Span] my_agent (Agent root - TYPE=agent)
└── [Span] gpt-4o-mini (LLM call - TYPE=llm)
└── [Span] search_docs (Tool call - TYPE=tool)
└── [Span] gpt-4o-mini (LLM call - TYPE=llm)
```
### Captured data
**Agent root span:**
* Agent name and agent ID
* Conversation ID (if set via `set_conversation_id()`)
**LLM spans (per model invocation):**
* Model name and provider
* System prompt and user prompt (last human message)
* Full input message history (`gen_ai.input.messages`)
* LLM completion and output messages (`gen_ai.output.messages`)
* Token usage (input, output, total)
* LLM context (if set via `set_llm_context()`)
* Available tool definitions
**Tool spans (per tool call):**
* Tool name, input arguments, and output
## Application setup
Before instrumenting your application, you must create an application in Fiddler and obtain your Application ID.
### 1. Create your application in Fiddler
Log in to your Fiddler instance and navigate to **GenAI Applications**, then click **Add Application** and follow the onboarding wizard to create your application.
### 2. Copy your Application ID
After creating your application, copy the **Application ID** from the GenAI Applications page. This must be a valid UUID4 format (for example, `550e8400-e29b-41d4-a716-446655440000`). You'll need this for initialization.
### 3. Get your API key
Go to **Settings** > **Credentials** and copy your API key. You'll need this for initialization.
## Detailed setup
### Installation
```bash theme={null}
pip install fiddler-langchain
```
**Framework Compatibility:**
* **LangChain V1:** `>= 1.0.0` — agents built with `langchain.agents.create_agent`
* **Python:** 3.10-3.14
* **fiddler-otel:** `>= 1.0.0` (installed automatically)
* **OpenTelemetry:** API and SDK `>= 1.27.0` (installed automatically)
### Configuration
#### Direct initialization (Recommended)
```python theme={null}
import langchain.agents
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(
application_id='your-app-id', # Required (UUID4 format)
api_key='your-api-key', # Required when otlp_enabled=True (default)
url='https://your-instance.fiddler.ai' # Required when otlp_enabled=True (default)
)
instrumentor = FiddlerLangChainInstrumentor(client=client)
instrumentor.instrument()
```
#### Using environment variables
```python theme={null}
import os
import langchain.agents
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(
application_id=os.getenv('FIDDLER_APPLICATION_ID'),
api_key=os.getenv('FIDDLER_API_KEY'),
url=os.getenv('FIDDLER_URL')
)
FiddlerLangChainInstrumentor(client=client).instrument()
```
**Environment Variables Reference:**
| Variable | Description | Example |
| ------------------------ | ------------------------- | -------------------------------------- |
| `FIDDLER_API_KEY` | Your Fiddler API key | `fid_...` |
| `FIDDLER_APPLICATION_ID` | Your application UUID4 | `550e8400-e29b-41d4-a716-446655440000` |
| `FIDDLER_URL` | Your Fiddler instance URL | `https://your-instance.fiddler.ai` |
## Instrumentation methods
The Fiddler LangChain SDK provides two instrumentation approaches:
| Approach | Best For | Key API |
| --------------------------------------------- | ---------------------------- | ------------------------------ |
| [Auto-Instrumentation](#auto-instrumentation) | All agents in an application | `FiddlerLangChainInstrumentor` |
| [Manual Middleware](#manual-middleware) | Per-agent control | `FiddlerAgentMiddleware` |
### Auto-instrumentation
`FiddlerLangChainInstrumentor.instrument()` monkey-patches `langchain.agents.create_agent` once. Every subsequent call to `create_agent()` automatically receives a `FiddlerAgentMiddleware`. No changes to individual agent creation calls are needed.
```python theme={null}
import langchain.agents
from langchain_openai import ChatOpenAI
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(application_id='...', api_key='...', url='...')
# Instrument once at startup
instrumentor = FiddlerLangChainInstrumentor(client=client)
instrumentor.instrument()
# All agents created after this point are automatically traced
agent1 = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[search_tool],
name='search_agent',
)
agent2 = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[booking_tool],
name='booking_agent',
)
```
**Key behaviors:**
* **Idempotent:** Calling `instrument()` multiple times is safe — it will not create duplicate middleware.
* **Agent naming:** If `name='...'` is passed to `create_agent()`, that name is used for the agent in traces. If omitted, no agent name is set and the agent appears without a label in the UI.
* **Existing middleware preserved:** If you pass a `FiddlerAgentMiddleware` instance manually in `middleware=[...]`, the instrumentor skips injection for that call so your manual configuration is preserved.
**Uninstrumenting:**
```python theme={null}
# Restore original create_agent (agents already created keep their middleware)
instrumentor.uninstrument()
```
### Manual middleware
For per-agent control, pass `FiddlerAgentMiddleware` directly to `create_agent()` without using the instrumentor:
```python theme={null}
import langchain.agents
from langchain_openai import ChatOpenAI
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerAgentMiddleware
client = FiddlerClient(application_id='...', api_key='...', url='...')
agent = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[search_tool],
name='my_agent',
middleware=[FiddlerAgentMiddleware(client=client, agent_name='my_agent')],
)
```
Use manual middleware when you want to trace only specific agents, or when you need different configurations per agent.
## Advanced usage
### Multi-turn conversations
Use `set_conversation_id()` to link multiple agent invocations into a single conversation in the Fiddler UI. All agents in the application that share the same `conversation_id` appear together in conversation-level views.
```python theme={null}
from fiddler_langchain import set_conversation_id
import uuid
# Call once per conversation
set_conversation_id(str(uuid.uuid4()))
# All agent invocations after this carry the same conversation_id
result1 = agent.invoke({'messages': [{'role': 'user', 'content': 'Find hotels in Tokyo.'}]})
result2 = agent.invoke({'messages': [{'role': 'user', 'content': 'Book the cheapest one.'}]})
```
### LLM context
Attach contextual metadata to LLM spans by calling `set_llm_context()` before the agent runs. The instrumentation reads this value from the model's metadata at invocation time and records it as `gen_ai.llm.context` on every LLM span for that model.
```python theme={null}
import langchain.agents
from langchain_openai import ChatOpenAI
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor, set_llm_context
client = FiddlerClient(application_id='...', api_key='...', url='...')
FiddlerLangChainInstrumentor(client=client).instrument()
model = ChatOpenAI(model='gpt-4o-mini')
set_llm_context(model, 'User profile: enterprise customer, prefers concise answers')
agent = langchain.agents.create_agent(model=model, tools=[...], name='my_agent')
```
`set_llm_context()` accepts both plain model instances (`BaseLanguageModel`) and `RunnableBinding` instances (for example, models wrapped with `.with_config()` or `.bind_tools()`).
#### Clearing LLM context for non-RAG steps
In multi-step agent workflows, context set after a RAG retrieval step leaks into subsequent non-RAG LLM calls (tool planning, routing, etc.), causing unintended faithfulness evaluation. Use `clear_llm_context()` to explicitly remove context before non-RAG steps:
```python theme={null}
from fiddler_langchain import set_llm_context, clear_llm_context
# After RAG retrieval — attach context for faithfulness evaluation
set_llm_context(model, retrieved_documents)
response = agent.invoke(rag_prompt) # faithfulness evaluated
# Before non-RAG steps — clear context to skip faithfulness evaluation
clear_llm_context(model)
plan = agent.invoke(planning_prompt) # no faithfulness evaluation
```
`clear_llm_context(model)` is equivalent to `set_llm_context(model, None)`.
### Span-level attributes
Use `add_span_attributes()` to attach custom metadata to a specific LangChain component (model, tool, or retriever). The middleware reads these attributes when creating the span for that component and records them as custom span attributes.
```python theme={null}
import langchain.agents
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor, add_span_attributes
client = FiddlerClient(application_id='...', api_key='...', url='...')
FiddlerLangChainInstrumentor(client=client).instrument()
# Add attributes to a specific model — appear only on that model's LLM spans
model = ChatOpenAI(model='gpt-4o-mini')
add_span_attributes(model, model_version='v2', cost_center='cc-123')
# Add attributes to a specific tool — appear only on that tool's spans
@tool
def search_hotels(query: str) -> str:
"""Search for hotels."""
return f'Hotels for: {query}'
add_span_attributes(search_hotels, category='travel', region='apac')
agent = langchain.agents.create_agent(model=model, tools=[search_hotels], name='my_agent')
```
Unlike `add_session_attributes` (which applies to every span in the context), `add_span_attributes` is scoped to a single component.
### Session attributes
Use `add_session_attributes()` to attach metadata that appears on **every span** created in the current thread or async coroutine. Use this for user-level or environment-level metadata that applies to the whole session.
```python theme={null}
import langchain.agents
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor, add_session_attributes
client = FiddlerClient(application_id='...', api_key='...', url='...')
FiddlerLangChainInstrumentor(client=client).instrument()
# Set once — applies to all spans in this context (emitted as fiddler.session.user.{key})
add_session_attributes('user_id', 'user_12345')
add_session_attributes('environment', 'production')
add_session_attributes('tier', 'premium')
agent = langchain.agents.create_agent(model=ChatOpenAI(model='gpt-4o-mini'), tools=[...], name='my_agent')
agent.invoke({'messages': [{'role': 'user', 'content': 'Hello!'}]})
```
### Retriever instrumentation
The LangChain V1 middleware does not expose a dedicated retriever hook. Following the same convention used in `fiddler-langgraph`, **retrievers are treated as tools**.
Wrap your retriever with `@tool` (or use `create_retriever_tool`) and pass it to `create_agent`. The middleware's tool hook captures the retriever call automatically as a `TYPE=tool` span — with the query as `tool_input` and the retrieved documents as `tool_output`.
```python theme={null}
import langchain.agents
from langchain_core.tools import tool
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(application_id='...', api_key='...', url='...')
FiddlerLangChainInstrumentor(client=client).instrument()
retriever = vector_store.as_retriever()
@tool
def search_docs(query: str) -> str:
"""Search company documents for relevant information."""
return str(retriever.invoke(query))
agent = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[search_docs],
name='rag_agent',
)
```
The resulting trace:
```
[Span] rag_agent (Agent root - TYPE=agent)
└── [Span] gpt-4o-mini (LLM call - TYPE=llm)
└── [Span] search_docs (Retriever as Tool - TYPE=tool)
└── [Span] gpt-4o-mini (LLM call - TYPE=llm)
```
### Multi-agent setup
With the instrumentor, a single `instrument()` call patches `create_agent` so every agent is traced. Pass `name='...'` to each `create_agent()` to label agents in traces.
When a sub-agent is invoked from within a delegation tool, its root Agent span is automatically created as a **child of the tool span** — the entire multi-agent flow appears in a **single trace**. No manual linking is needed: `wrap_tool_call` attaches the active tool span into the OTel context before invoking the handler, and `before_agent` detects that active span and nests under it.
```python theme={null}
import langchain.agents
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(application_id='...', api_key='...', url='...')
FiddlerLangChainInstrumentor(client=client).instrument()
# Sub-agents
flight_agent = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[book_flight],
name='flight_assistant',
)
hotel_agent = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[search_hotel, book_hotel],
name='hotel_assistant',
)
# Delegation tools — the tool span becomes the parent of the sub-agent span automatically
@tool
def delegate_to_flight_assistant(task: str) -> str:
"""Delegate a flight booking task to the flight assistant."""
result = flight_agent.invoke({'messages': [{'role': 'user', 'content': task}]})
return result['messages'][-1].content
@tool
def delegate_to_hotel_assistant(task: str) -> str:
"""Delegate a hotel search and booking task to the hotel assistant."""
result = hotel_agent.invoke({'messages': [{'role': 'user', 'content': task}]})
return result['messages'][-1].content
# Supervisor — invoking this produces one trace containing all agents
supervisor = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[delegate_to_flight_assistant, delegate_to_hotel_assistant],
name='supervisor',
)
supervisor.invoke({'messages': [{'role': 'user', 'content': 'Book a flight and a hotel in Tokyo.'}]})
```
Trace output (all agents appear in a **single trace**):
```
[Span] supervisor (root - TYPE=agent)
└── [Span] gpt-4o-mini (LLM - TYPE=llm)
└── [Span] delegate_to_flight_assistant (Tool - TYPE=tool)
└── [Span] flight_assistant (Agent - TYPE=agent)
└── [Span] gpt-4o-mini (LLM - TYPE=llm)
└── [Span] book_flight (Tool - TYPE=tool)
└── [Span] delegate_to_hotel_assistant (Tool - TYPE=tool)
└── [Span] hotel_assistant (Agent - TYPE=agent)
└── [Span] gpt-4o-mini (LLM - TYPE=llm)
└── [Span] search_hotel (Tool - TYPE=tool)
└── [Span] book_hotel (Tool - TYPE=tool)
```
`set_conversation_id()` is useful for linking **multiple top-level invocations** (e.g., multi-turn conversations) — not for joining sub-agents within a single invocation, since they already share the same trace automatically.
### Async agents
The instrumentation fully supports async agents via the `awrap_model_call` and `awrap_tool_call` hooks. Use `agent.ainvoke()` — no additional configuration needed:
```python theme={null}
import asyncio
import langchain.agents
from langchain_openai import ChatOpenAI
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(application_id='...', api_key='...', url='...')
FiddlerLangChainInstrumentor(client=client).instrument()
agent = langchain.agents.create_agent(
model=ChatOpenAI(model='gpt-4o-mini'),
tools=[...],
name='my_agent',
)
async def main():
result = await agent.ainvoke({'messages': [{'role': 'user', 'content': 'Hello!'}]})
print(result['messages'][-1].content)
asyncio.run(main())
```
The instrumentation automatically uses the async lifecycle hooks when the agent is invoked asynchronously, producing the same span hierarchy as the sync path.
### Error handling
If an LLM call or tool call raises an exception, the instrumentation:
1. Catches the exception and marks the failing span with `StatusCode.ERROR`
2. Re-raises the exception so normal error handling in your application is unaffected
3. Cleanly closes the root agent span — no dangling open spans
```python theme={null}
try:
result = agent.invoke({'messages': [{'role': 'user', 'content': 'Hello!'}]})
except Exception as e:
# The failing LLM or tool span is already marked ERROR in Fiddler
# The root agent span is closed — all spans are recorded
raise
```
This means partial traces are never lost — all spans up to the point of failure are recorded and visible in Fiddler.
### Running in AWS SageMaker
Because `fiddler-langchain` exports traces through `FiddlerClient`, it inherits the Fiddler OTel SDK's [AWS SageMaker Partner App authentication](/integrations/agentic-ai/fiddler-otel-sdk#running-in-aws-sagemaker). Install the `sagemaker` extra (`pip install "fiddler-otel[sagemaker]"`) and set the `AWS_PARTNER_APP_AUTH`, `AWS_PARTNER_APP_ARN`, and `AWS_PARTNER_APP_URL` environment variables — your instrumentation code is unchanged.
### Flush and shutdown
```python theme={null}
# Sync
client.force_flush()
client.shutdown()
# Async
await client.aflush()
await client.ashutdown()
# Context manager (calls shutdown() on exit)
with FiddlerClient(application_id='...', api_key='...', url='...') as client:
FiddlerLangChainInstrumentor(client=client).instrument()
result = agent.invoke({'messages': [...]})
```
### Local debugging
**JSONL file capture (save a local copy of spans in addition to Fiddler export):**
`jsonl_capture_enabled=True` is **additive** — spans are saved to a local JSONL file **and** continue to be exported to Fiddler via OTLP. Setting this to `True` does **not** suppress or disable the OTLP export to Fiddler.
```python theme={null}
client = FiddlerClient(
application_id='...',
api_key='...',
url='...',
jsonl_capture_enabled=True, # Saves spans locally; OTLP export to Fiddler still active
jsonl_file_path='trace_data.jsonl',
)
```
Override the output file path via environment variable:
```bash theme={null}
FIDDLER_JSONL_FILE=trace_data.jsonl python my_agent.py
```
Each line in the output file is a JSON object. Fields:
| Field | Description |
| ------------------------ | ----------------------------------- |
| `trace_id` | Trace identifier |
| `span_id` | Span identifier |
| `parent_span_id` | Parent span identifier |
| `span_type` | Span type (`agent`, `llm`, `tool`) |
| `agent_name` | Name of the agent |
| `conversation_id` | Conversation identifier |
| `model_name` | LLM model name |
| `model_provider` | LLM provider |
| `llm_input_system` | System prompt |
| `llm_input_user` | User prompt |
| `llm_output` | LLM completion text |
| `llm_context` | Context set via `set_llm_context()` |
| `llm_token_count_input` | Input token count |
| `llm_token_count_output` | Output token count |
| `llm_token_count_total` | Total token count |
| `gen_ai_input_messages` | Full input message history (JSON) |
| `gen_ai_output_messages` | Output messages (JSON) |
| `tool_name` | Tool/function name |
| `tool_input` | Tool input arguments |
| `tool_output` | Tool result |
| `tool_definitions` | Available tool schemas (JSON) |
## Relationship to fiddler-langgraph
| Package | Framework | Instrumentation Approach |
| ------------------- | ---------------------------------- | --------------------------------------------------------------------------- |
| `fiddler-langchain` | LangChain V1 (`create_agent`) | `FiddlerLangChainInstrumentor` auto-patches `langchain.agents.create_agent` |
| `fiddler-langgraph` | LangGraph (`StateGraph.compile()`) | `LangGraphInstrumentor` callback handler |
Both packages depend on `fiddler-otel` for the core `FiddlerClient` and span wrappers.
## What's next?
* [**Fiddler OTel SDK**](/integrations/agentic-ai/fiddler-otel-sdk) — For decorator-based instrumentation of custom Python functions
* [**LangGraph SDK**](/integrations/agentic-ai/langgraph-sdk) — If your application uses LangGraph
* [**Agentic Observability Concepts**](/getting-started/agentic-monitoring) — Understand the agent lifecycle and monitoring approach
# Fiddler LangGraph SDK
Source: https://docs.fiddler.ai/integrations/agentic-ai/langgraph-sdk
Instrument LangGraph agents and custom AI applications with Fiddler's native SDK
[](https://pypi.org/project/fiddler-langgraph/)
Instrument your LangGraph agent applications and custom AI workflows with OpenTelemetry-based tracing for comprehensive agentic observability. The Fiddler LangGraph SDK provides three instrumentation approaches — auto-instrumentation for LangGraph workflows, decorator-based tracing for custom functions, and manual span creation for fine-grained control — capturing every step from thought to action to execution.
## What you'll need
* Fiddler account (cloud or on-premises)
* Python 3.10-3.14
* LangGraph or LangChain application
* Fiddler API key and application ID
## Quick start
Get monitoring in 3 steps:
```bash theme={null}
# Step 1: Install
pip install fiddler-langgraph
```
```python theme={null}
# Step 2: Initialize the Fiddler client
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
fdl_client = FiddlerClient(
application_id='your-app-id', # Must be valid UUID4
api_key='your-api-key',
url='https://your-instance.fiddler.ai'
)
# Step 3: Instrument your application
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
# Your existing LangGraph code runs normally
# Traces will automatically be sent to Fiddler
```
That's it! Your agent traces are now flowing to Fiddler.
This Quick Start uses auto-instrumentation for LangGraph applications. For custom functions or fine-grained control, see [Instrumentation Methods](#instrumentation-methods) below.
## What gets monitored
The LangGraph SDK automatically captures:
### Hierarchical tracing
* **Application Level** - Overall system performance and health
* **Session Level** - User interaction and conversation flows
* **Agent Level** - Individual agent behavior and decisions
* **Span Level** - Tool calls, LLM requests, state transitions
### Agent lifecycle stages
Every agent operation is tracked through five observable stages:
1. **Thought** - Data ingestion, context retrieval, information interpretation
2. **Action** - Planning processes, tool selection, decision-making
3. **Execution** - Task performance, API calls, external integrations
4. **Reflection** - Self-evaluation, learning signals, adaptation
5. **Alignment** - Trust validation, safety checks, policy enforcement
### Captured data
* Agent state transitions and decision points
* Tool invocations with inputs and outputs
* LLM API calls with prompts and responses
* Execution times and latency metrics
* Error traces and exception handling
* Custom metadata and tags
## Application setup
Before instrumenting your application, you must create an application in Fiddler and obtain your Application ID:
### 1. Create your application in Fiddler
Log in to your Fiddler instance and navigate to **GenAI Applications**, then click **Add Application** and follow the onboarding wizard to create your application.
### 2. Copy your Application ID
After creating your application, copy the **Application ID** from the GenAI Applications page using the copy icon next to the ID. This must be a valid UUID4 format (for example, `550e8400-e29b-41d4-a716-446655440000`). You'll need this for initialization.
### 3. Get your access token
Go to **Settings** > **Credentials** and copy your access token. You'll need this for initialization.
## Detailed setup
### Installation
```bash theme={null}
pip install fiddler-langgraph
```
**Framework Compatibility:**
* **LangGraph:** >= 0.3.28 and \<= 1.1.0 OR **LangChain:** >= 0.3.28 and \<= 1.1.0
* **Python:** 3.10, 3.11, 3.12, or 3.13
* **OpenTelemetry:** API and SDK >= 1.28.0 and \<= 1.39.1 (installed automatically)
### Configuration
#### Direct initialization (Recommended)
```python theme={null}
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
fdl_client = FiddlerClient(
application_id='your-app-id', # Required (UUID4 format)
api_key='your-api-key', # Required
url='https://your-instance.fiddler.ai' # Required
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
```
#### Using environment variables
You can use environment variables instead of hardcoding credentials:
```python theme={null}
import os
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
fdl_client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL")
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
```
**Environment Variables Reference:**
| Variable | Description | Example |
| ------------------------ | ------------------------- | -------------------------------------- |
| `FIDDLER_API_KEY` | Your Fiddler API key | `fid_...` |
| `FIDDLER_APPLICATION_ID` | Your application UUID4 | `550e8400-e29b-41d4-a716-446655440000` |
| `FIDDLER_URL` | Your Fiddler instance URL | `https://your-instance.fiddler.ai` |
## Instrumentation methods
The Fiddler LangGraph SDK provides three instrumentation approaches. Choose the one that fits your application:
| Approach | Best For | Key API |
| --------------------------------------------------- | ---------------------------------------- | ----------------------------------------- |
| [Auto-Instrumentation](#auto-instrumentation) | LangGraph and LangChain applications | `LangGraphInstrumentor` |
| [Decorator-Based](#decorator-based-instrumentation) | Custom Python functions, mixed workflows | `@trace()`, `get_current_span()` |
| [Manual](#manual-instrumentation) | Fine-grained span lifecycle control | `start_as_current_span()`, `start_span()` |
You can combine all three approaches in the same application. For example, use auto-instrumentation for your LangGraph graph and decorators for custom helper functions that the graph calls.
### Auto-Instrumentation
Auto-instrumentation captures LangGraph and LangChain workflows automatically. Initialize the instrumentor once, and all graph invocations produce traces with no additional code changes.
```python theme={null}
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
instrumentor = LangGraphInstrumentor(client)
instrumentor.instrument()
# Your LangGraph/LangChain code runs normally — traces are captured automatically
```
**When to use:** Your application uses LangGraph `StateGraph` or LangChain runnables and you want comprehensive tracing with zero instrumentation code.
See the [Quick Start](#quick-start) section above for a complete walkthrough, or the [Advanced Usage](#advanced-usage) section for context enrichment and production configuration.
### Decorator-based instrumentation
Use the `@trace()` decorator to instrument individual Python functions. This is the recommended approach for custom functions that are not part of a LangGraph graph, such as standalone LLM calls, tool implementations, or orchestration logic.
```python theme={null}
from openai import OpenAI
from fiddler_langgraph import FiddlerClient, trace, get_current_span
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
openai_client = OpenAI() # Assumes OPENAI_API_KEY is set in environment
# vector_store: a pre-configured vector database client (e.g., Pinecone, Chroma, Weaviate)
@trace(as_type="generation", model="gpt-4o", system="openai")
def call_llm(prompt: str) -> str:
span = get_current_span(as_type="generation")
if span:
span.set_user_prompt(prompt)
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
result = response.choices[0].message.content
if span:
span.set_completion(result)
return result
@trace(as_type="tool", name="search_knowledge_base")
def search_kb(query: str) -> list[str]:
span = get_current_span(as_type="tool")
if span:
span.set_tool_input({"query": query})
results = vector_store.search(query)
if span:
span.set_tool_output(results)
return results
@trace(name="handle_request")
def handle_request(user_input: str) -> str:
context = search_kb(user_input) # Child span (tool)
response = call_llm(user_input) # Child span (generation)
return response # Parent span auto-closes
```
**When to use:** You have custom Python functions — LLM wrappers, tool implementations, or orchestration logic — that you want to trace with full control over span metadata.
#### `@trace()` Arguments
| Argument | Type | Default | Description |
| ---------------- | --------------- | ------------- | ------------------------------------------------------------------------------------- |
| `name` | `str` | Function name | Custom span name |
| `as_type` | `str` | `"span"` | Span type: `"span"`, `"generation"`, `"chain"`, or `"tool"` |
| `capture_input` | `bool` | `True` | Automatically capture function arguments as span input |
| `capture_output` | `bool` | `True` | Automatically capture return value as span output |
| `model` | `str` | `None` | LLM model name (sets `gen_ai.request.model`) |
| `system` | `str` | `None` | LLM provider such as `"openai"` or `"anthropic"` (sets `gen_ai.system`) |
| `user_id` | `str` | `None` | User identifier |
| `version` | `str` | `None` | Service version string |
| `client` | `FiddlerClient` | `None` | Explicit client instance (defaults to the [global singleton](#global-client-pattern)) |
#### Accessing the current span
Inside a decorated function, call `get_current_span()` to access the active span and add metadata:
```python theme={null}
from fiddler_langgraph import get_current_span
@trace(as_type="generation")
def my_llm_call(prompt: str) -> str:
span = get_current_span(as_type="generation")
if span:
span.set_user_prompt(prompt)
span.set_model("gpt-4o")
span.set_system("openai")
# ... make your LLM call ...
if span:
span.set_completion(result)
span.set_usage(input_tokens=50, output_tokens=120)
return result
```
Pass `as_type` to get a type-specific wrapper with semantic helper methods. See [Span Types and Helper Methods](#span-types-and-helper-methods) for the full list.
Always check `if span:` before calling helper methods. `get_current_span()` returns `None` if no Fiddler span is active — for example, during unit tests or when the client is not initialized.
#### Async support
The `@trace()` decorator works with both sync and async functions. No additional configuration is needed:
```python theme={null}
@trace(as_type="generation", model="gpt-4o", system="openai")
async def async_llm_call(prompt: str) -> str:
span = get_current_span(as_type="generation")
if span:
span.set_user_prompt(prompt)
response = await openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
)
result = response.choices[0].message.content
if span:
span.set_completion(result)
return result
```
#### Automatic parent-child relationships
Nested decorated functions create proper span hierarchies automatically. The outer function becomes the parent span, and inner calls become child spans:
```python theme={null}
@trace(name="agent_workflow", as_type="chain")
def run_agent(query: str) -> str:
context = retrieve_docs(query) # Child span
answer = generate_answer(context) # Child span
return answer # Parent span
@trace(as_type="tool")
def retrieve_docs(query: str) -> list[str]:
# Automatically a child of "agent_workflow"
return vector_db.search(query)
@trace(as_type="generation", model="gpt-4o")
def generate_answer(context: list[str]) -> str:
# Also a child of "agent_workflow"
return llm.generate(context)
```
### Manual instrumentation
Create spans manually using context managers or explicit start/end calls. This gives you full control over span lifecycle — useful for dynamic span creation, conditional instrumentation, or code where decorator syntax does not apply.
#### Context manager (automatic lifecycle)
Use `start_as_current_span()` to create a span that ends automatically when the block exits:
```python theme={null}
from openai import OpenAI
from fiddler_langgraph import FiddlerClient
openai_client = OpenAI() # Assumes OPENAI_API_KEY is set in environment
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
with client.start_as_current_span("llm_call", as_type="generation") as span:
span.set_model("gpt-4o")
span.set_system("openai")
span.set_user_prompt("What is the capital of France?")
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What is the capital of France?"}]
)
span.set_completion(response.choices[0].message.content)
span.set_usage(
input_tokens=response.usage.prompt_tokens,
output_tokens=response.usage.completion_tokens
)
# Span ends automatically here
```
#### Explicit span control
Use `start_span()` when you need to manage span lifecycle manually — for example, in callback-driven or event-based code:
```python theme={null}
span = client.start_span("background_task", as_type="tool")
try:
span.set_tool_name("data_processor")
span.set_tool_input({"file": "data.csv"})
result = process_file("data.csv")
span.set_tool_output(result)
finally:
span.end() # Must call end() explicitly
```
Always call `span.end()` when using `start_span()`. Forgetting to end a span causes a resource leak. Prefer `start_as_current_span()` unless you need explicit lifecycle control.
**When to use:** You need explicit control over when spans start and end — for example, in callback-driven code, conditional spans, or complex control flow where decorators do not fit.
### Span types and helper methods
Both decorator and manual instrumentation support four span types. Set the `as_type` parameter to select a type, which determines which semantic helper methods are available on the span wrapper.
| Type | Wrapper Class | Use For |
| -------------- | ------------------- | --------------------------------------------- |
| `"span"` | `FiddlerSpan` | Generic operations, orchestration |
| `"generation"` | `FiddlerGeneration` | LLM calls (prompts, completions, token usage) |
| `"chain"` | `FiddlerChain` | Multi-step workflows, processing chains |
| `"tool"` | `FiddlerTool` | Tool or function calls (name, input, output) |
#### Common methods (all types)
| Method | Description |
| ----------------------------- | --------------------------------------------------------- |
| `set_input(data)` | Set input data (auto-serializes dicts and lists to JSON) |
| `set_output(data)` | Set output data (auto-serializes dicts and lists to JSON) |
| `set_attribute(key, value)` | Set a custom span attribute |
| `set_agent_name(name)` | Set the agent name (`gen_ai.agent.name`) |
| `set_agent_id(id)` | Set the agent ID (`gen_ai.agent.id`) |
| `set_conversation_id(id)` | Set the conversation ID (`gen_ai.conversation.id`) |
| `record_exception(exception)` | Record an error on the span |
#### Generation methods (`FiddlerGeneration`)
| Method | Sets Attribute |
| ------------------------------------------------------ | ------------------------- |
| `set_model(name)` | `gen_ai.request.model` |
| `set_system(provider)` | `gen_ai.system` |
| `set_system_prompt(text)` | `gen_ai.llm.input.system` |
| `set_user_prompt(text)` | `gen_ai.llm.input.user` |
| `set_completion(text)` | `gen_ai.llm.output` |
| `set_usage(input_tokens, output_tokens, total_tokens)` | `gen_ai.usage.*` |
| `set_context(text)` | `gen_ai.llm.context` |
| `set_messages(messages)` | `gen_ai.input.messages` |
| `set_output_messages(messages)` | `gen_ai.output.messages` |
| `set_tool_definitions(definitions)` | `gen_ai.tool.definitions` |
#### Tool methods (`FiddlerTool`)
| Method | Sets Attribute |
| ----------------------------------- | ------------------------- |
| `set_tool_name(name)` | `gen_ai.tool.name` |
| `set_tool_input(data)` | `gen_ai.tool.input` |
| `set_tool_output(data)` | `gen_ai.tool.output` |
| `set_tool_definitions(definitions)` | `gen_ai.tool.definitions` |
For complete API documentation, see the LangGraph SDK API Reference.
## Context isolation
The Fiddler LangGraph SDK maintains its own isolated OpenTelemetry context. Fiddler traces do not interfere with other OpenTelemetry tracers that may be active in your application, and vice versa.
Each `FiddlerClient` creates a private `Context` instance. All span creation, parent-child linking, and context propagation happen within this isolated context. When you use `@trace()`, `start_as_current_span()`, or `start_span()`, the SDK manages context attachment and detachment automatically.
You can verify whether a span belongs to Fiddler using `is_fiddler_span()`:
```python theme={null}
from fiddler_otel.utils import is_fiddler_span
from opentelemetry import trace
otel_span = trace.get_current_span()
if is_fiddler_span(otel_span):
print("This span belongs to Fiddler")
```
This isolation matters if your application uses other OpenTelemetry-based observability tools (such as Datadog, Honeycomb, or custom OTel exporters). Fiddler traces remain completely separate, so you can run multiple tracing systems side by side without conflicts.
## Global client pattern
The Fiddler SDK uses a singleton pattern for `FiddlerClient`. The first client created in your process is automatically registered as the global default. Retrieve it anywhere using `get_client()`:
```python theme={null}
from fiddler_langgraph import FiddlerClient, get_client
# Initialize once (automatically registered as global singleton)
client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
# Retrieve anywhere in your application
def some_utility_function():
fdl_client = get_client() # Returns the same client instance
with fdl_client.start_as_current_span("utility_op") as span:
span.set_attribute("source", "utility")
# ... your logic ...
```
The `@trace()` decorator uses `get_client()` internally, so you do not need to pass a client to each decorated function. As long as a `FiddlerClient` has been created somewhere in your application, all `@trace()` decorators and `get_current_span()` calls work automatically.
There is no `set_current_client()` function. The singleton is set automatically during `FiddlerClient` initialization. If you create multiple clients, only the first one becomes the global default. Pass an explicit `client` argument to `@trace()` to use a different client.
## Advanced usage
### Adding context and metadata
Enrich traces with custom context and conversation tracking:
```python theme={null}
from fiddler_langgraph import set_llm_context, clear_llm_context, set_conversation_id
import uuid
# Set descriptive context for LLM processing
set_llm_context(model, 'Customer support conversation')
# Set conversation ID for tracking multi-turn conversations
conversation_id = str(uuid.uuid4())
set_conversation_id(conversation_id)
```
#### Clearing LLM context for non-RAG steps
In multi-step agent workflows, context set after a RAG retrieval step leaks into subsequent non-RAG LLM calls (tool planning, routing, etc.), causing unintended faithfulness evaluation. Use `clear_llm_context()` to explicitly remove context before non-RAG steps:
```python theme={null}
from fiddler_langgraph import set_llm_context, clear_llm_context
# After RAG retrieval — attach context for faithfulness evaluation
set_llm_context(llm, retrieved_documents)
response = llm.invoke(rag_prompt) # faithfulness evaluated
# Before non-RAG steps — clear context to skip faithfulness evaluation
clear_llm_context(llm)
plan = llm.invoke(planning_prompt) # no faithfulness evaluation
```
`clear_llm_context(llm)` is equivalent to `set_llm_context(llm, None)`.
### Custom span and session attributes
Add custom attributes to individual spans or entire sessions:
```python theme={null}
from fiddler_langgraph import add_span_attributes, add_session_attributes
# Add attributes to specific spans
add_span_attributes({
"user_id": "user_123",
"request_type": "billing_inquiry"
})
# Add attributes that persist across all spans in a session
add_session_attributes({
"session_type": "premium",
"feature_flags": ["new_ui", "advanced_mode"]
})
```
### Sampling configuration
Control trace sampling for high-volume applications:
```python theme={null}
from opentelemetry.sdk.trace import sampling
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
# Sample 10% of traces
sampler = sampling.TraceIdRatioBased(0.1)
fdl_client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
sampler=sampler
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
```
For production deployments, consider these sampling strategies:
* **High-volume applications:** Sample 5-10% (`TraceIdRatioBased(0.05)`)
* **Development/testing:** Sample 100% (default - no sampler specified)
* **Cost optimization:** Sample 1-5% (`TraceIdRatioBased(0.01)`)
### Production configuration
For high-volume production applications, configure span limits and batch processing:
```python theme={null}
import os
from opentelemetry.sdk.trace import SpanLimits, sampling
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
# Configure batch processing BEFORE initializing FiddlerClient
os.environ['OTEL_BSP_MAX_QUEUE_SIZE'] = '500' # Increased from default 100
os.environ['OTEL_BSP_SCHEDULE_DELAY_MILLIS'] = '500' # Faster export than default 1000ms
os.environ['OTEL_BSP_MAX_EXPORT_BATCH_SIZE'] = '50' # Larger batches than default 10
# Increase span limits to capture more data
production_limits = SpanLimits(
max_events=128, # Default: 32
max_span_attributes=128, # Default: 32
max_span_attribute_length=8192, # Default: 2048
)
# Sample 5-10% of traces
production_sampler = sampling.TraceIdRatioBased(0.05)
fdl_client = FiddlerClient(
application_id=os.getenv("FIDDLER_APPLICATION_ID"),
api_key=os.getenv("FIDDLER_API_KEY"),
url=os.getenv("FIDDLER_URL"),
span_limits=production_limits,
sampler=production_sampler,
compression=Compression.Gzip,
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
```
### Offline and S3 Routing Mode
Use this mode when traces must be routed through an intermediate store (such as Amazon S3) before reaching Fiddler, rather than being sent directly. This is the correct approach when your security or network policies require all data to pass through a controlled intermediary.
* **`otlp_enabled=False`** — disables all direct OTLP export to Fiddler. `api_key` and `url` are not required in this mode.
* **`otlp_json_capture_enabled=True`** — writes traces to local `.json` files in standard OTLP JSON format (`ExportTraceServiceRequest` envelope). These files are directly consumable by the Fiddler S3 connector with no reformatting.
* **`application_id` is still required** — even though no data is sent to Fiddler directly, the S3 connector uses the `application_id` embedded in the trace files to route ingested traces to the correct application in Fiddler.
```python theme={null}
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
# No api_key or url needed — traces go to local files only
fdl_client = FiddlerClient(
application_id="YOUR_APPLICATION_ID", # UUID4 — required for S3 connector routing
otlp_enabled=False, # Disables direct export to Fiddler
otlp_json_capture_enabled=True, # Writes OTLP JSON files locally
otlp_json_output_dir="./fiddler_traces", # Directory for output files (default: 'fiddler_traces')
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
# Your LangGraph code runs normally — traces are written to ./fiddler_traces/*.json
```
Upload the generated `.json` files from `otlp_json_output_dir` to your S3 bucket. The Fiddler S3 connector reads them directly.
Each batch of spans is written to a separate timestamped `.json` file in the output directory. The directory is created automatically if it does not exist.
**Do not confuse `console_tracer` or `jsonl_capture_enabled` with this mode.** Both of those flags are **additive** — they add a local output on top of the existing OTLP export to Fiddler but do not disable it. Only `otlp_enabled=False` fully stops direct export to Fiddler.
### Running in AWS SageMaker
Because `fiddler-langgraph` exports traces through `FiddlerClient`, it inherits the Fiddler OTel SDK's [AWS SageMaker Partner App authentication](/integrations/agentic-ai/fiddler-otel-sdk#running-in-aws-sagemaker). Install the `sagemaker` extra (`pip install "fiddler-otel[sagemaker]"`) and set the `AWS_PARTNER_APP_AUTH`, `AWS_PARTNER_APP_ARN`, and `AWS_PARTNER_APP_URL` environment variables — your instrumentation code is unchanged.
### Flush and shutdown handling
The SDK uses OpenTelemetry's batch span processor, which buffers spans in memory and exports them on a schedule. To avoid losing buffered spans when your process exits, use explicit flush and shutdown:
* **Process exit:** The SDK registers an `atexit` handler that flushes and shuts down the tracer when the process exits. For short scripts or environments where `atexit` may not run (e.g. SIGKILL, forked processes), call `force_flush()` and `shutdown()` explicitly—for example in a `try`/`finally` or signal handler.
* **Long-running servers (e.g. FastAPI, uvicorn):** On graceful shutdown (SIGTERM), call the Fiddler client's shutdown so pending spans are exported before the process exits. From async code use `ashutdown()` (or `aflush()` then `ashutdown()`) so the event loop is not blocked; the sync `force_flush()` and `shutdown()` can block for up to the flush timeout (default 30 seconds).
**Sync (scripts or signal handler):**
```python theme={null}
# Ensure spans are sent before exit (e.g. in finally or SIGTERM handler)
fdl_client.shutdown()
```
**Async (e.g. FastAPI/uvicorn lifespan):**
```python theme={null}
# In your app's shutdown/lifespan handler (e.g. on SIGTERM)
await fdl_client.ashutdown()
```
**Context manager (scripts):** Use `with FiddlerClient(...) as client:` so `shutdown()` is called automatically when the block exits.
## Example applications
### Multi-agent travel planner
```python theme={null}
from langgraph.graph import StateGraph, END
from langchain_openai import ChatOpenAI
from typing import TypedDict
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
# Initialize Fiddler monitoring
fdl_client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
# Define your agent graph
class TravelState(TypedDict):
destination: str
budget: float
itinerary: list
# Research agent
def research_agent(state: TravelState):
# Agent logic - automatically traced
return {"research_complete": True}
# Booking agent
def booking_agent(state: TravelState):
# Agent logic - automatically traced
return {"bookings": [...]}
# Build graph
graph = StateGraph(TravelState)
graph.add_node("research", research_agent)
graph.add_node("booking", booking_agent)
graph.add_edge("research", "booking")
graph.add_edge("booking", END)
# Run - traces automatically sent to Fiddler
app = graph.compile()
result = app.invoke({"destination": "Paris", "budget": 5000})
```
[**View the Advanced Observability Notebook →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LangGraph_Advanced_Observability.ipynb) | [**Custom Instrumentation Notebook →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LangGraph_Custom_Instrumentation.ipynb)
### Customer support agent with tools
```python theme={null}
from langchain.tools import Tool
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
# Initialize Fiddler monitoring
fdl_client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
# Define tools - calls automatically traced
tools = [
Tool(name="search_kb", func=search_knowledge_base),
Tool(name="create_ticket", func=create_support_ticket),
Tool(name="escalate", func=escalate_to_human)
]
# Create agent - instrumentation is automatic
model = ChatOpenAI(model="gpt-4o")
agent = create_react_agent(model, tools)
# Run agent - full trace in Fiddler
response = agent.invoke({
"messages": [{"role": "user", "content": "My order is delayed"}]
})
```
## Viewing your data
After running your instrumented application:
1. **Navigate to Fiddler UI** - `https://your-instance.fiddler.ai`
2. **Select "GenAI Applications"** - View your application
3. **Inspect traces** - Drill down from application → session → agent → span
4. **Analyze patterns** - Use analytics to identify bottlenecks and errors
### Key metrics tracked
* **Latency**: P50, P95, P99 response times across agents
* **Error Rate**: Percentage of failed agent executions
* **Token Usage**: LLM token consumption per agent/session
* **Tool Calls**: Frequency and success rate of tool invocations
* **State Transitions**: Agent decision path analysis
## Troubleshooting
### Application not showing as "Active"
**Check your configuration:**
* Ensure your application executes instrumented code
* Verify your Fiddler access token and application ID are correct
* Check network connectivity to your Fiddler instance
**Enable console tracer for debugging:**
```python theme={null}
fdl_client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai",
console_tracer=True # Also prints spans to console; OTLP export to Fiddler continues
)
```
`console_tracer=True` is **additive** — span data is printed to stdout **and** continues to be exported to Fiddler via OTLP. Setting this to `True` does **not** disable or suppress the OTLP export to Fiddler. Use it to visually confirm spans are being created during local development.
### Network connectivity issues
**Verify connectivity to your Fiddler instance:**
```bash theme={null}
curl -I https://your-instance.fiddler.ai
```
**Check firewall settings:**
* Ensure HTTPS traffic on port 443 is allowed
* Verify your Fiddler instance URL is correct
### Import errors
**Problem:** `ModuleNotFoundError: No module named 'fiddler_langgraph'`
**Solution:** Ensure you've installed the correct package:
```bash theme={null}
pip install fiddler-langgraph
```
**Problem:** `ImportError: cannot import name 'LangGraphInstrumentor'`
**Solution:** Ensure you have the correct import path:
```python theme={null}
from fiddler_langgraph import LangGraphInstrumentor
```
### Version compatibility issues
**Verify your versions match requirements:**
```bash theme={null}
python --version # Should be 3.10, 3.11, 3.12, or 3.13
pip show langgraph # Should be >= 0.3.28 and <= 1.1.0
```
**If you have version conflicts:**
```bash theme={null}
pip install langgraph>=0.3.28,<=1.1.0
```
### Invalid application ID
**Problem:** `ValueError: application_id must be a valid UUID4`
**Solution:** Ensure your Application ID is in proper UUID4 format:
```python theme={null}
# ❌ This will fail
fdl_client = FiddlerClient(
application_id="invalid-id", # Not a valid UUID4
api_key="your-access-token",
url="https://instance.fiddler.ai"
)
# ✅ Correct format
fdl_client = FiddlerClient(
application_id="550e8400-e29b-41d4-a716-446655440000", # Valid UUID4
api_key="your-access-token",
url="https://instance.fiddler.ai"
)
```
Copy the Application ID directly from the Fiddler dashboard to avoid formatting issues.
### Agent shows as "UNKNOWN\_AGENT"
**For LangChain applications**, ensure you're setting the agent name in the config parameter:
```python theme={null}
from langchain_core.output_parsers import StrOutputParser
# Define your LangChain runnable
chat_app_chain = prompt | llm | StrOutputParser()
# Run with agent name configuration
response = chat_app_chain.invoke({
"input": user_input,
"history": messages,
}, config={"configurable": {"agent_name": "service_chatbot"}})
```
**Note:** LangGraph applications automatically extract agent names. This manual configuration is only needed for LangChain applications.
## OpenTelemetry compatibility
The LangGraph SDK is built on OpenTelemetry Protocol (OTLP). The SDK uses standard OpenTelemetry components, allowing you to:
* Integrate with existing observability infrastructure
* Export traces to multiple backends (with custom configuration)
* Use custom OTEL collectors and processors
All telemetry data follows OpenTelemetry semantic conventions for AI/ML workloads.
## Related integrations
* [**Fiddler Evals SDK**](/integrations/agentic-ai/evals-sdk) - Evaluate LangGraph agent quality offline
* [**Python Client SDK**](/sdk-api/python-client/connection) - Additional monitoring capabilities
## Migration guides
### From LangSmith
```python theme={null}
# Before (LangSmith)
import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "ls_..."
# After (Fiddler - both can run simultaneously)
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
fdl_client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
# Your LangGraph code remains unchanged
```
### From manual tracing
If you've built custom tracing, migration is straightforward:
```python theme={null}
# Before (manual timing/logging)
import time
start = time.time()
result = my_agent.run()
duration = time.time() - start
log_to_system(duration, result)
# After (Fiddler SDK - automatic instrumentation)
# Initialize Fiddler client and instrumentor (once)
fdl_client = FiddlerClient(
application_id="your-app-id",
api_key="your-api-key",
url="https://your-instance.fiddler.ai"
)
instrumentor = LangGraphInstrumentor(fdl_client)
instrumentor.instrument()
# Remove all manual timing/logging code
result = my_agent.run() # Automatic instrumentation
```
## API reference
Full SDK documentation:
* [**LangGraph SDK Reference**](/sdk-api/langgraph/fiddler-client) - Complete class and method documentation
## Next steps
Now that your application is instrumented:
1. **Explore the data:** Check your Fiddler dashboard for traces, metrics, and performance insights
2. **Learn advanced features:** See our [Advanced Usage Guide](/developers/tutorials/llm-monitoring/langgraph-sdk-advanced) for complex multi-agent scenarios
3. **Review the SDK reference:** Check the [Fiddler LangGraph SDK Reference](/sdk-api/langgraph/fiddler-client) for complete documentation
4. **Optimize for production:** Review configuration options for high-volume applications
# LiteLLM Integration
Source: https://docs.fiddler.ai/integrations/agentic-ai/litellm-integration
Integrate LiteLLM with Fiddler for unified LLM cost tracking, latency monitoring, and LLM call observability — via the LiteLLM SDK or the LiteLLM proxy gateway.
## Overview
[LiteLLM](https://docs.litellm.ai/) provides a unified interface for calling 100+ LLM providers.
This page covers **observability**: routing LiteLLM's OpenTelemetry traces to Fiddler.
Pick your path before you start:
| Goal | Recommended path |
| ---------------------------------------------------- | ----------------------------------------------------------- |
| Trace prompts, responses, tokens, costs, and latency | This page |
| Block or redact PII and secrets inline | [LiteLLM Guardrails](/protection/litellm-guardrails) |
| Observe and protect the same traffic | Configure both — they are independent and share no settings |
Fiddler supports two integration modes:
| Mode | Best for | Required packages |
| ----------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **LiteLLM SDK** | Applications calling LLM providers directly via `litellm.completion()` | `litellm`, `opentelemetry-sdk`, `opentelemetry-exporter-otlp-proto-http` |
| **LiteLLM Proxy** | Teams routing all LLM traffic through a centrally managed LiteLLM proxy gateway | `litellm[proxy]`, `opentelemetry-sdk`, `opentelemetry-exporter-otlp-proto-http` |
No Fiddler-specific package is required in either mode,
but the OpenTelemetry SDK and OTLP/HTTP exporter packages are —
neither `litellm` nor `litellm[proxy]` installs them
(verified at LiteLLM 1.96.2; they ship only in LiteLLM's Docker images).
If they are missing, requests keep succeeding and **traces silently never appear** —
see [Troubleshooting Trace Delivery](#troubleshooting-trace-delivery).
Both modes work by routing OpenTelemetry traces to Fiddler's OTLP ingestion endpoint
using standard environment variables.
This page was verified end-to-end with LiteLLM 1.96.2;
use **LiteLLM 1.85.0 or later** —
earlier versions have known trace-content gaps
(see [Known LiteLLM Upstream Caveats](#known-litellm-upstream-caveats)),
and versions before 1.81.0 do not emit the `gen_ai.operation.name` attribute
Fiddler uses to classify spans.
If the same proxy also runs [Fiddler guardrails](/protection/litellm-guardrails),
that page's higher floor (1.91.0) governs the combined deployment.
***
## LiteLLM SDK Integration
### Overview
LiteLLM includes a built-in OpenTelemetry integration.
When you enable it and point the OTLP exporter at Fiddler,
every LLM call is automatically traced.
Fiddler natively ingests LiteLLM-generated OTel traces and maps them to the Fiddler schema,
giving you observability over prompts, responses, token usage, and cost
across all LLM providers.
The following SDK functions are supported:
| SDK Function | `gen_ai.operation.name` | Fiddler Span Type |
| ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ----------------- |
| `litellm.completion()` / `litellm.acompletion()` | `chat` / `completion` / `acompletion` | `llm` |
| `litellm.text_completion()` / `litellm.atext_completion()` | `text_completion` / `atext_completion` | `llm` |
| `litellm.responses()` / `litellm.aresponses()` | `responses` / `aresponses` | `llm` |
| `litellm.anthropic_interface.messages.create()` / `acreate()` | `anthropic_messages` | `llm` |
| `litellm.generate_content()` / `litellm.agenerate_content()` | `generate_content` / `agenerate_content` | `llm` |
| `litellm.embedding()` / `litellm.aembedding()` | `embedding` / `aembedding` | `embedding` |
| `litellm.rerank()` / `litellm.arerank()` | `rerank` / `arerank` | `reranker` |
| `litellm.image_generation()`, `litellm.speech()`, `litellm.transcription()`, `litellm.moderation()`, `litellm.ocr()` (and async variants) | operation-specific | `unknown` |
Fiddler's LLM observability focuses on text-based generative completions:
image, audio, moderation, and OCR operations are ingested and remain visible in traces,
but carry the `unknown` span type rather than `llm`.
### Architecture
### Prerequisites
* A Fiddler account with a GenAI application already created,
and the application's UUID
(open the application in the Fiddler UI and copy the UUID from the URL or application settings)
* A Fiddler API key, created under **Settings → [Credentials](/reference/administration/settings#credentials)**
* LiteLLM **1.85.0 or later** plus the OpenTelemetry packages:
```bash theme={null}
uv add litellm opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```
* A valid LLM provider API key (e.g. `OPENAI_API_KEY` for OpenAI models)
### Quick Start
#### Step 1: Set Environment Variables
Set these before starting your application:
```bash theme={null}
# Fiddler OTel ingestion
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-fiddler-instance.com"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ,fiddler-application-id="
export OTEL_RESOURCE_ATTRIBUTES="application.id="
# LLM provider key (name varies by provider)
export OPENAI_API_KEY="your-openai-key"
```
Three details that prevent silent failures:
* **The endpoint is your instance base URL.**
LiteLLM appends `/v1/traces` automatically —
the final request is `POST https://your-fiddler-instance.com/v1/traces`
over OTLP/HTTP (the default protocol once an endpoint is set; standard HTTPS port).
* **Header values are parsed literally.**
LiteLLM splits `OTEL_EXPORTER_OTLP_HEADERS` on commas and the first `=` of each part,
with no URL-decoding and no whitespace trimming:
keep the literal space in `Bearer `, never URL-encode it,
and put no space after the comma.
* **The application UUID appears in two places, doing different jobs.**
The `fiddler-application-id` header authenticates the request;
the `application.id` resource attribute routes spans to your application.
Both are required and must match.
#### Step 2: Enable the Built-In OTel Callback
Add one line to your application startup:
```python theme={null}
import litellm
litellm.callbacks = ["otel"]
```
#### Step 3: Make Completions as Normal
No other code changes are required:
```python theme={null}
response = litellm.completion(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
],
)
print(response.choices[0].message.content)
```
Every call is now automatically traced and exported to Fiddler.
#### Step 4: Verify Traces Are Arriving
Open the Fiddler UI and navigate to your application's **Explorer**.
You should see the trace within a few seconds of making your first completion call.
If nothing appears, work through [Troubleshooting Trace Delivery](#troubleshooting-trace-delivery).
### What Gets Captured
**Message Content**
| Fiddler Field | Source |
| ---------------- | ----------------------------------------------------- |
| System prompt | First `role: system` message |
| User input | Last `role: user` message (the most recent user turn) |
| Assistant output | Last `role: assistant` message |
**Token Usage**
| Attribute | Description |
| ---------------------------- | --------------------------- |
| `gen_ai.usage.input_tokens` | Prompt tokens consumed |
| `gen_ai.usage.output_tokens` | Completion tokens generated |
| `gen_ai.usage.total_tokens` | Total tokens |
**Model Information**
`gen_ai.system` and `gen_ai.request.model` are stored at their unprefixed keys,
making them queryable via `SpanAttribute::gen_ai.system` and `SpanAttribute::gen_ai.request.model`.
| Attribute | Description |
| ----------------------- | ------------------------------------- |
| `gen_ai.request.model` | Model requested (e.g. `gpt-4o-mini`) |
| `gen_ai.response.model` | Model actually used |
| `gen_ai.system` | Provider (e.g. `openai`, `anthropic`) |
**Cost Metadata**
LiteLLM emits `gen_ai.cost.input_cost`, `gen_ai.cost.output_cost`, and `gen_ai.cost.total_cost`
whenever it can price the model from its cost map;
models LiteLLM cannot price emit no cost attributes.
Fiddler stores these attributes verbatim and resolves them to its cost concepts at query time.
**Conversation Grouping**
Fiddler groups spans into conversations via `gen_ai.conversation.id`.
By default each request is its own conversation (the OTel trace ID is used);
to group a multi-turn conversation, send an `x-genai-conversation-id` header
with the same value on every request of the conversation (proxy mode).
### Supported Features
| Feature | Support | Notes |
| ------------------------------ | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Chat/text completion tracing | ✅ Full | Prompts, responses, token usage via `completion()`, `text_completion()` |
| Responses API tracing | ✅ On LiteLLM ≥ 1.85.0 | Input/output messages and instructions missing on earlier versions — see [Known LiteLLM Upstream Caveats](#known-litellm-upstream-caveats) |
| Anthropic Messages API tracing | ✅ On current LiteLLM | The `system` prompt is coalesced into `gen_ai.system_instructions` on current versions |
| Token usage | ✅ Full | Input, output, and total tokens |
| Model information | ✅ Full | Requested and actual model |
| Cost tracking | ✅ When priced | Emitted whenever LiteLLM's cost map covers the model |
| Embeddings, rerank | ✅ As `embedding` / `reranker` span types | Captured with token and cost metadata |
| Images, audio, moderation, OCR | ⚠️ As `unknown` span type | Visible in traces, not classified as `llm` |
| Conversation grouping | ⚠️ Header-driven | Requires `x-genai-conversation-id`; otherwise one conversation per request |
***
## LiteLLM Proxy Integration
### Overview
The LiteLLM proxy is an OpenAI-compatible gateway that lets you call 100+ LLM providers
through a single API.
When the proxy emits OpenTelemetry traces, Fiddler automatically detects and ingests them —
no application-side changes required.
This gives you observability over every LLM call routed through your proxy:
prompts, responses, token usage, cost metadata, and latency —
across all models and providers in one place.
### Architecture
### When to Use This Integration
Use the LiteLLM proxy integration when:
* You are already running LiteLLM proxy as your LLM gateway
* You want to monitor all LLM traffic centrally regardless of underlying provider (OpenAI, Anthropic, Bedrock, etc.)
* You want cost attribution and latency tracking without instrumenting individual applications
### Prerequisites
* A Fiddler GenAI application and its UUID, plus a Fiddler API key
(**Settings → [Credentials](/reference/administration/settings#credentials)**)
* LiteLLM proxy **1.85.0 or later** with the OpenTelemetry packages:
```bash theme={null}
uv add "litellm[proxy]" "fastapi<0.140.7" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```
The `"fastapi<0.140.7"` pin works around a LiteLLM 1.96.x dependency conflict
that otherwise breaks proxy startup
(`ImportError: cannot import name 'get_flat_dependant'`);
remove it once upstream ships a fix.
* A `config.yaml` with at least one configured model (`model_list`) and a provider API key
* Network access from the proxy host to your Fiddler instance over HTTPS
### Quick Start
#### Step 1: Configure the Proxy to Emit OpenTelemetry
Enable the OTel callback in `config.yaml` and set the export target in the environment:
```yaml theme={null}
model_list:
- model_name: gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
litellm_settings:
callbacks: ["otel"]
```
```bash theme={null}
export OPENAI_API_KEY=""
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-fiddler-instance.com"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ,fiddler-application-id="
export OTEL_RESOURCE_ATTRIBUTES="application.id="
litellm --config config.yaml
```
The endpoint is your instance base URL — LiteLLM appends `/v1/traces` automatically
and exports over OTLP/HTTP.
The same header-format and two-locations rules from the
[SDK integration](#litellm-sdk-integration) quick start apply verbatim.
#### Step 2: Verify Traces Are Arriving
Make a test request through your proxy:
```bash theme={null}
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
Then open the Fiddler UI and navigate to your application's **Explorer**.
You should see the trace within a few seconds.
If not, see [Troubleshooting Trace Delivery](#troubleshooting-trace-delivery).
### Using with OpenAI Codex CLI
[OpenAI Codex CLI](https://github.com/openai/codex) does not need its own Fiddler integration.
Codex's native OpenTelemetry **traces** carry only token counts —
no prompt, response, or tool content.
(That content exists only in Codex's OpenTelemetry **logs**, which Fiddler does not ingest.)
So the recommended way to observe Codex in Fiddler is to point it at your LiteLLM proxy.
The proxy then emits full `llm` spans (prompt + response + token usage)
that Fiddler ingests through the integration described above, with no Fiddler-side code.
#### Step 1: Use LiteLLM 1.85+ With a Codex-Compatible Model
Codex uses the OpenAI Responses API,
whose content is fully captured in LiteLLM's OTel traces from **1.85.0** onward:
```bash theme={null}
uv add "litellm[proxy]>=1.85.0" "fastapi<0.140.7" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```
In your proxy `config.yaml`, map an alias (the name Codex will request) to a Codex-capable model.
Use a current Codex-family model from your OpenAI account for the upstream `model` —
for example `gpt-5.2-codex`.
General chat models such as `gpt-4o-mini` will **not** work,
because they reject Codex's tool calls.
```yaml theme={null}
model_list:
- model_name: codex-model # alias Codex requests (any name; reused in Step 2)
litellm_params:
model: openai/ # e.g. gpt-5.2-codex — must accept Codex's tool calls
api_key: os.environ/OPENAI_API_KEY
litellm_settings:
callbacks: ["otel"]
drop_params: true
```
#### Step 2: Point Codex at the Proxy
Add a custom model provider to `~/.codex/config.toml`
so Codex sends requests to your LiteLLM proxy instead of OpenAI directly:
```toml theme={null}
model = "codex-model" # must match the model_name alias from Step 1
model_provider = "fiddler-gw"
[model_providers.fiddler-gw]
name = "Fiddler GW"
base_url = "http://localhost:4000/v1" # your LiteLLM proxy URL
wire_api = "responses" # Codex only supports the Responses API
env_key = "GW_KEY" # any non-empty value; LiteLLM holds the real provider key
```
```bash theme={null}
export GW_KEY=any-non-empty-value
```
#### Step 3: Use Codex Normally
```bash theme={null}
codex exec "Plan a 4-day trip to Japan"
```
Each turn appears in your application's **Explorer** as an `llm` span
with the full user prompt, assistant response, and token usage.
**Requirements and limitations**
* **API-key mode only.** Routing through a proxy uses your OpenAI API key; it does not work with ChatGPT-subscription login.
* **The upstream model must accept Codex's tool calls.** Use a current Codex-family model from your OpenAI account (for example `gpt-5.2-codex`); general chat models such as `gpt-4o-mini` reject Codex's tool format.
### What Gets Captured
#### Span Types
Fiddler classifies LiteLLM spans by the `gen_ai.operation.name` attribute
(and by span name for guardrail spans):
| Traffic | `gen_ai.operation.name` | Fiddler Span Type |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------ | ----------------- |
| `/chat/completions`, `/completions` | `chat` / `completion` / `acompletion` / `text_completion` / `atext_completion` | `llm` |
| `/v1/responses`, `/responses` | `responses` / `aresponses` | `llm` |
| `/v1/messages` (Anthropic format) | `anthropic_messages` | `llm` |
| Google Gemini native endpoints | `generate_content` / `agenerate_content` (+ streaming variants) | `llm` |
| `/embeddings` | `embedding` / `aembedding` | `embedding` |
| `/rerank` | `rerank` / `arerank` | `reranker` |
| MCP tool invocations | tool-call operations | `tool` |
| LiteLLM guardrail executions | span name `guardrail` | `guardrail` |
| Images, audio, moderation, OCR | operation-specific | `unknown` |
| Internal infrastructure (`self`, `router`, `proxy_pre_call`) | — | `unknown` |
Each proxy request typically produces up to three spans:
`Received Proxy Server Request` (top-level server span),
`litellm_request` (the primary span carrying all attributes),
and `raw_gen_ai_request` (a child span with the raw provider-level request and response).
#### Captured Attributes
**Message Content** —
LiteLLM writes full conversation history as JSON on the span
(`gen_ai.input.messages` / `gen_ai.output.messages`).
Fiddler extracts the first `role: system` message as the system prompt,
the last `role: user` message as the user input,
and the last `role: assistant` message as the assistant output.
If you have disabled message logging in LiteLLM (`turn_off_message_logging: true`),
the message content fields are absent from traces;
token counts and cost metadata are still captured.
**Token Usage** —
`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.total_tokens`.
**Model Information** —
`gen_ai.request.model` (requested), `gen_ai.response.model` (actually used),
`gen_ai.system` (provider), queryable via `SpanAttribute::` as on the SDK path.
**Cost Metadata** —
`gen_ai.cost.input_cost`, `gen_ai.cost.output_cost`, `gen_ai.cost.total_cost`,
emitted when LiteLLM can price the model.
Fiddler stores customer attribute keys verbatim —
there is no renaming into Fiddler-specific namespaces —
and resolves cost concepts at query time.
**Proxy Metadata** —
LiteLLM proxy emits `metadata.*` attributes containing API key, team, user,
and routing information, all preserved verbatim for auditing and cost attribution.
**Conversation Grouping** —
as on the SDK path:
send `x-genai-conversation-id` with a stable value per conversation to group multi-turn sessions;
without it, each request is its own conversation.
#### LiteLLM OTel v2
The span and attribute names above describe LiteLLM's default (v1) OpenTelemetry integration.
Setting `LITELLM_OTEL_V2=true` switches the proxy to LiteLLM's semconv-aligned v2 output,
which renames some spans and attributes:
the guardrail span becomes `execute_guardrail `, MCP tool calls become `execute_tool`,
and cost, guardrail, team, and metadata attributes move to the
`litellm.cost.*`, `litellm.guardrail.*`, `litellm.team.*`, and `litellm.metadata.*` namespaces.
Fiddler's span mapper ingests both generations automatically —
traces arrive either way; query the namespace that matches the generation your proxy emits.
***
## Troubleshooting Trace Delivery
Two different failure classes exist, and they look different from your side:
**authentication problems are loud** (the exporter logs an error in *your* logs),
while **routing problems are silent** (the endpoint accepts the spans and they never appear).
Work through the checks in order — they apply to both the SDK and proxy paths.
**1. Did the OTel callback load?**
If the OpenTelemetry packages are missing, requests still succeed and nothing is exported.
The only symptom is a log line at startup (proxy) or on the first request (SDK):
```text theme={null}
Error initializing custom logger: No module named 'opentelemetry'
```
Install `opentelemetry-sdk` and `opentelemetry-exporter-otlp-proto-http` and restart.
**2. Is the exporter pointed at the right place?**
With `OTEL_EXPORTER_OTLP_ENDPOINT` set to your instance base URL,
LiteLLM exports over OTLP/HTTP to `/v1/traces`.
An export failure appears in your logs as:
```text theme={null}
Failed to export span batch code: , reason:
```
A `404` usually means a wrong base URL.
A `401`, `403`, or `404` from a correct URL means the **authentication headers** were rejected —
an invalid API key, an application UUID that does not exist,
or a key without permission on that application.
These header problems are always visible as export errors; they are never silent.
**3. Is the header string formatted exactly right?**
`OTEL_EXPORTER_OTLP_HEADERS` is parsed literally:
comma-separated `key=value` pairs, split on the first `=`,
with no URL-decoding and no whitespace trimming.
Keep the literal space in `Bearer `, never URL-encode it
(a `%20` reaches the server encoded and breaks authentication),
and put no space after the comma.
**4. Do both application-ID locations agree?**
The `fiddler-application-id` **header** authenticates;
the `application.id` **resource attribute** (in `OTEL_RESOURCE_ATTRIBUTES`) routes.
If the resource attribute is missing or carries a different UUID,
the endpoint returns success and the spans are dropped or routed elsewhere —
**this is the silent case**, and there is currently no customer-visible server-side
diagnostic for it.
Verify both values are the same UUID, character for character.
**5. Does the application UUID exist?**
Open the application in the Fiddler UI and copy the UUID from the URL or application settings —
do not construct it by hand.
**6. Send a test span and watch your own logs.**
Make one request, wait a few seconds for the batch exporter to flush,
and confirm no `Failed to export` lines appear.
**7. Check Explorer.**
The trace should appear in the expected application within a few seconds.
Two timing caveats: the batch exporter flushes on an interval (allow \~10 seconds),
and a freshly deployed Fiddler instance may drop spans for the first few minutes
while its pipeline warms up.
If all seven checks pass and traces still do not appear,
contact Fiddler support with the application UUID, the OTel trace ID of a test request,
and the request timestamp — never include prompts, responses, or credentials.
**Message content missing from traces?**
LiteLLM's message logging may be disabled — remove `turn_off_message_logging: true`
from `litellm_settings` to re-enable message capture
(token counts and cost are unaffected by this setting).
**Overrode `OTEL_SERVICE_NAME`?**
Fiddler recognizes LiteLLM spans primarily by their instrumentation scope,
with `service.name` values `litellm` and `litellm-proxy` both recognized as fallbacks.
Leave `OTEL_SERVICE_NAME` unset, or set it to one of those two values.
***
## Known LiteLLM Upstream Caveats
Earlier LiteLLM versions had gaps in the OTel callback that affected every downstream
OTel consumer, not just Fiddler.
Status as of LiteLLM 1.96.2:
**Fixed in LiteLLM 1.85.0 — Responses API content.**
Before 1.85.0, `/v1/responses` traces were missing `gen_ai.input.messages`,
`gen_ai.output.messages`, and the `instructions` system prompt
([BerriAI/litellm#25840](https://github.com/BerriAI/litellm/issues/25840), closed as completed).
Current versions also coalesce the per-endpoint system-prompt field names
(`instructions`, `system`, `system_instructions`) into `gen_ai.system_instructions`.
If you run an older proxy and see token counts but no message content on Responses-API
traffic, upgrade to ≥ 1.85.0.
**Still open — provider attribution on non-chat endpoints.**
For endpoint families other than `/chat/completions`,
LiteLLM may emit an empty `gen_ai.system`
and prefix raw provider attributes on the `raw_gen_ai_request` child span with `llm.None.*`
([BerriAI/litellm#25240](https://github.com/BerriAI/litellm/issues/25240),
closed by the stale-bot without a fix;
the proposed fix [PR #25309](https://github.com/BerriAI/litellm/pull/25309) remains open).
Fiddler's span classification is unaffected —
it keys off `gen_ai.operation.name`, not the provider attribution.
***
## Guardrails
Fiddler also implements the LiteLLM Generic Guardrail API spec,
letting you plug Fiddler's real-time guardrails directly into the LiteLLM proxy gateway.
Once configured, every request routed through the proxy is checked for
**secrets and PII** — blocking or redacting them before the request reaches the model.
Guardrails and observability are configured independently and compose freely
on the same proxy:
the `guardrails` and `litellm_settings.callbacks` keys coexist in one `config.yaml`,
and traces capture the sanitized request —
content redacted by a `pre_call` guardrail never appears in the exported spans.
With the OTel callback enabled on the same proxy,
each guardrail execution is also traced as its own span next to the model call,
so every block or redact decision — and its reason — shows up in Explorer as an audit trail.
See [LiteLLM Guardrails](/protection/litellm-guardrails) for the verified quick start,
check behavior, failure controls, and the full API reference.
***
## Next Steps
* [LiteLLM Guardrails](/protection/litellm-guardrails) — block or redact PII and secrets inline on the same proxy
* [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration) — instrument custom frameworks by hand when you outgrow built-in callbacks
* [Strands Agents SDK](/integrations/agentic-ai/strands-sdk) — native monitoring if your agents run on Strands
* [LangGraph SDK](/integrations/agentic-ai/langgraph-sdk) — auto-instrumentation for LangGraph applications
* [LiteLLM OTel documentation](https://docs.litellm.ai/docs/proxy/logging#opentelemetry) — LiteLLM's own OpenTelemetry configuration reference
# Fiddler MCP Server
Source: https://docs.fiddler.ai/integrations/agentic-ai/mcp-server
Connect MCP-compatible AI assistants to Fiddler's GenAI observability platform and query applications, traces, evaluators, metrics, and more using the Fiddler MCP Server.
This capability is currently in [public preview](/reference/feature-maturity-definitions#public-preview). It is available to every customer and Fiddler responds promptly to issues, but it is not yet covered by an SLA and its APIs may still change.
MCP tools run with your full account permissions: anything you can view or edit in Fiddler, a connected assistant can too. Because the external AI client controls tool execution, Fiddler is not responsible for actions taken by connected agents. Only authorize clients you trust and be intentional with your prompts.
Fiddler provides a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server that exposes your GenAI observability data to MCP-compatible AI assistants. Connect any MCP-compatible client and your AI assistant can explore applications, investigate traces, review evaluator results, monitor metrics, and more — without leaving the editor.
The Fiddler MCP Server is dramatically improved in release **26.16**, and we're actively expanding its capabilities. We'd love your feedback: share your thoughts and suggestions with [support@fiddler.ai](mailto:support@fiddler.ai).
## Prerequisites
* A Fiddler account you have signed in to on the Fiddler web application at least once — your Fiddler account is created on your first web sign-in, and the MCP sign-in does not create it. Without this, the MCP connection completes the browser sign-in but every tool call is rejected with an authentication error.
* Your Fiddler instance URL (e.g., `https://your-instance.fiddler.ai`)
* An MCP client that supports remote HTTP MCP with OAuth 2.0, such as Claude Code or OpenCode. For other clients, see [Limitations](#limitations) for the OAuth 2.0 redirect URL requirements.
**Claude Desktop and Claude.ai are not currently supported** due to a known issue in how those clients handle OAuth 2.0 authorization. We're tracking this and will update this page once it's resolved. See [Limitations](#limitations) for client requirements.
## Authentication
The Fiddler MCP Server authenticates with **OAuth 2.0**. On first connection, your client opens your browser to sign in through Fiddler and receives a short-lived token — you never paste a static key into a config file. The client then acts with your own account's permissions, and standard role-based access control (RBAC) applies: a client can see and do only what your account is authorized to.
Those permissions cover write actions too. Depending on your role, the tool catalog includes creating, modifying, and permanently deleting resources such as traces and sessions, and Fiddler does not add a confirmation step of its own before a tool runs. Most MCP clients ask for approval before executing tools; keeping that enabled for write actions is a good default.
## Connect with OAuth 2.0
Fiddler composes the exact configuration for your deployment and shows it in the product. To connect your MCP client:
In the Fiddler UI, go to **Settings > MCP**. The MCP Integration page shows a ready-to-use configuration for each supported client: select **Claude Code**, **OpenCode**, or **Generic**.
Copy the command or config snippet shown for your client. For Claude Code this is a single terminal command; for OpenCode it is a JSON block for your `opencode.jsonc`. The snippet already contains your deployment's OAuth 2.0 client ID, scopes, issuer, and MCP Server URL — see [OAuth 2.0 client setup](#oauth-2-0-client-setup) below for what each client expects.
Start (or restart) your MCP client. On first connection, your browser opens to sign in through Fiddler. After you sign in, the client acts with your own account's permissions and can begin querying your GenAI observability data.
Ask the assistant something simple, such as "Show me my GenAI applications and give me a short summary of what's there." If your client lists the server's tools, seeing just three (`search_tools`, `describe_tools`, `execute_tool`) is expected; the assistant uses them to discover and run Fiddler's full tool catalog.
### OAuth 2.0 Client Setup
Use the tab that matches your client. The Client ID, scopes, issuer, and redirect URLs are specific to your deployment — always copy the live values from **Settings > MCP**. The snippets below show the shape of each client's configuration.
Your client requests three OAuth 2.0 scopes:
* `openid` — standard OpenID Connect scope; signs you in and identifies your account.
* `offline_access` — issues a refresh token so your client can renew its short-lived access token without repeated sign-ins.
* `` — targets the token at Fiddler's MCP resource server so it's accepted. This value is specific to your deployment and is included in the config you copy.
Run the command shown under **Settings > MCP > Claude Code** in your terminal. It registers the Fiddler MCP Server with its OAuth 2.0 details in one step:
```bash theme={null}
claude mcp add-json fiddler-mcp '{
"type": "http",
"url": "https:///v1/mcp/genai",
"oauth": {
"clientId": "",
"scopes": "openid offline_access "
}
}'
```
The first time an assistant uses the server, Claude Code opens your browser to sign in.
Add the following block to the `"mcp"` section of your `opencode.jsonc` file, then restart OpenCode. Copy the exact values from **Settings > MCP > OpenCode**:
```json theme={null}
"fiddler-mcp": {
"enabled": true,
"type": "remote",
"url": "https:///v1/mcp/genai",
"oauth": {
"clientId": "",
"scope": "openid offline_access "
}
}
```
OpenCode opens your browser to sign in on first connection.
For any other OAuth 2.0-capable client, configure the connection manually using the values from **Settings > MCP > Generic**:
| Parameter | Description | Example |
| -------------- | ------------------------------------------------------------------- | ------------------------------------------------ |
| Client ID | The OAuth 2.0 client identifier your MCP client authenticates with. | `` |
| Scopes | Space-separated OAuth 2.0 scopes requested during authentication. | `openid offline_access ` |
| MCP Server URL | The Fiddler MCP endpoint your client connects to. | `https:///v1/mcp/genai` |
| Issuer | The OAuth 2.0 issuer (authorization server) that issues tokens. | `https://` |
Your client must use one of Fiddler's pre-registered OAuth 2.0 redirect URLs; see [Limitations](#limitations) for the supported callback paths.
Refer to your client's documentation for how to configure a remote MCP Server with OAuth 2.0.
### Disconnect a Client
To disconnect a client, remove the Fiddler MCP Server from its configuration:
* **Claude Code** — run `claude mcp remove fiddler-mcp`.
* **OpenCode** — delete the `fiddler-mcp` block from your `opencode.jsonc`.
* **Other clients** — remove the Fiddler MCP Server entry as described in your client's documentation.
Removing the configuration stops that client from connecting to Fiddler again. Credentials the client already obtained remain valid until they expire: by default, access tokens expire within one hour and refresh tokens within 24 hours. A refresh token also expires after 2 hours 24 minutes without use — even if its 24-hour absolute lifetime has not elapsed — so an idle client must sign in again. Deployments with custom token policies may differ.
## Recommended Models
MCP provides tools to an agent, but the results are only as good as the agent and model using them: the model decides which tools to call, how to chain them, and how to interpret what comes back. Smaller and faster model tiers can struggle with the multi-step tool use that observability questions often require, so we recommend the following minimums from each major provider:
| Provider | Minimum recommended model |
| --------- | ----------------------------------- |
| Anthropic | Claude Sonnet (Sonnet 4.5 or newer) |
| OpenAI | GPT-5 |
| Google | Gemini 2.5 Pro |
A few additional considerations:
* For data analysis work, such as asking questions over large volumes of trace and span data, prefer higher-tier models with large context windows.
* Agents can make mistakes when condensing large amounts of information, so spot-check important conclusions against the Fiddler UI.
* For other providers or open-weight models, choose a tier similar or stronger to those listed above, with solid tool-calling support.
## Using MCP
There is no special syntax to learn: ask in plain language and the assistant uses Fiddler's tools to answer your questions. The tools support creating, viewing, updating, and deleting information in Fiddler, so an assistant can both answer questions and make changes on your behalf. Below are the tool domains covered and some example prompts you can use to get started.
"Tell me about my most active applications and what the most active agents we are observing are."
"Tell me about my Customer Support app: its health, recent traffic, and which evaluators are running on it."
"Find a recent session in my Customer Support app and walk me through what happened: what was asked, what the agents did, and how it ended."
"Search my Customer Support app for spans that mention payment failures and tell me which sessions they belong to."
"Which evaluators are running on my Customer Support app?"
"Help me set up sentiment scoring evaluators on my Customer Support app."
"Let's look at results from my sentiment evaluator and help me iterate: test improved prompts against sample responses, then update the evaluator once the scores look right."
"Create a chart tracking p95 latency over the last 7 days for my Customer Support app."
"What does the dashboard for my Customer Support app show right now? Tell me what charts are on it."
"What's the average number of input tokens per LLM call in my Customer Support app?"
"Create a custom metric that tracks average input tokens per LLM call, then chart it over the last 7 days."
"What evals datasets and experiments does my Customer Support app have, and how did the latest experiment score?"
"Show me the five worst rows of the latest experiment by answer similarity, with each row's input and score."
"What alert rules are configured on my Customer Support app?"
"Set up an alert that notifies me if traffic to my Customer Support app drops to zero."
"Which LLM providers and models are configured in my gateway?"
"What model options are available for each provider I've configured?"
"Which LLM models show up in my Customer Support app's traffic?"
"List the chart annotations in my project and tell me if any line up with last week's latency spike."
"Did the historical evaluator run on my Customer Support app finish? Are there any failed tasks I should worry about?"
"Score last month's traffic with my secret detection evaluator: backfill the last 30 days."
## Limitations
* **GenAI and agentic tools only** — ML model monitoring features are not available through MCP.
* **Trace content truncation** — long trace content may be truncated by default to manage context window usage; ask your assistant to fetch a specific span in full if you need to inspect its complete content.
* **No Dynamic Client Registration** — Fiddler does not implement OAuth 2.0 Dynamic Client Registration (DCR), so MCP clients must be pre-registered and use a pre-registered redirect URL for the OAuth 2.0 callback. The supported redirect URLs are:
* `http://127.0.0.1/mcp/oauth/callback`
* `http://127.0.0.1/callback`
Since these use the loopback IP literal (`127.0.0.1`) without a fixed port, a single entry covers any ephemeral local port your client picks. If your client uses a different callback path, contact Fiddler Support to add it. Clients that can only register via DCR — with no way to configure a pre-registered client ID — cannot connect today. The MCP specification has deprecated DCR in favor of Client ID Metadata Documents (CIMD), which Fiddler is evaluating.
* **Claude Desktop and Claude.ai** — not currently supported due to a known issue in how those clients handle OAuth 2.0 authorization; we're tracking this and will update this page once it's resolved.
* **Rate limiting** — requests to the MCP Server are rate-limited at 120 requests per minute per connected client by default. Exceeding the limit returns a 429 response with a `Retry-After` header.
## What's Next?
* [**Agentic AI Overview**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) — Browse all Fiddler integrations for agentic AI
* [**Fiddler OTel SDK**](/integrations/agentic-ai/fiddler-otel-sdk) — Instrument custom Python agents with OpenTelemetry
* [**Fiddler Evals SDK**](/integrations/agentic-ai/evals-sdk) — Run offline evaluations on your LLM applications
* [**Agentic AI Observability**](/getting-started/agentic-monitoring) — Complete setup guide for GenAI monitoring
# Multi-Application Trace Routing
Source: https://docs.fiddler.ai/integrations/agentic-ai/multi-application-trace-routing
Send OpenTelemetry traces for many Fiddler applications through one endpoint and one API key. Fiddler authorizes and routes each span by its application.id, so a single shared pipeline can serve every application.
Multi-application trace routing lets a **single** OTLP pipeline — one endpoint, one API key — carry spans for **many** Fiddler applications at once. You tag each span (or its resource) with an `application.id`; Fiddler authorizes every span against the applications your key is allowed to write to, stores the authorized ones under the right application, and reports the rest back to you.
***
## Overview
In the single-application flow ([Exporting OTel Traces to Fiddler](/integrations/agentic-ai/otel-trace-export)), each request targets one application — you set a `fiddler-application-id` header and every span in that request lands in that one application.
Multi-application routing removes that one-application-per-request limit. A single stream can mix spans for any number of applications, and Fiddler does the routing for you:
* **One endpoint, one API key.** A shared gateway or collector fronting many applications sends everything to the same `/v1/traces` endpoint with a single API key — no per-application pipeline, and no separate key wired in per application.
* **No routing infrastructure to build or run.** You do not stand up and maintain your own collector or configure routing connectors to split traffic by application. You tag each span with its `application.id` — one attribute, set once in your emitter's config — and Fiddler handles the routing server-side.
* **Authorization-aware routing.** Fiddler verifies each span's `application.id` against the applications your API key is permitted to write to, and rejects the rest — so a span can never reach an application your API key cannot write to. This bounds routing to your API key's scope; it does not isolate a shared gateway's callers from each other — see [Trust Model and Caller-Supplied Application IDs](#trust-model-and-caller-supplied-application-ids).
* **Everything in one call.** Authorized and unauthorized spans can coexist in the same batch; the authorized ones are ingested and the unauthorized ones are reported back in the response.
**When to use this**
Reach for multi-application routing when a single component — an AI gateway, a shared OpenTelemetry Collector, or a common egress service — emits telemetry for several Fiddler applications and you want to send it all through one authenticated channel instead of one pipeline per application.
For a single application, the [single-application header flow](/integrations/agentic-ai/otel-trace-export) is all you need and continues to work unchanged.
***
## How It Works
Set `application.id` on each span (or on its resource, when a whole `ResourceSpans` belongs to one application). One batch can contain spans for many different applications.
POST the batch to `/v1/traces` with a single `Authorization: Bearer ` header. No `fiddler-application-id` header is used in this flow — routing is driven by each span's `application.id`.
Fiddler resolves the full set of applications your API key may write to and matches each span's `application.id` against that set — no single application is declared up front.
Authorized spans are ingested under their application. Unauthorized spans are not stored; instead they are counted and named in the response so you can act on them.
The call returns **HTTP 200** with an OTLP `partialSuccess` block whenever any span is rejected — see [The partialSuccess Response](#the-partialsuccess-response). Authorized spans in the same batch are still ingested.
**Why one API key can serve many applications.** In the single-application flow you declare one application up front (via the `fiddler-application-id` header) and Fiddler answers a single yes/no. Multi-application routing instead authorizes against your API key's **full set** of permitted applications and matches each span's `application.id` against it. Nothing is declared up front, so one API key serves every application it is allowed to write to.
***
## Prerequisites
* One or more Fiddler applications created — you will need each **Application UUID**.
* A valid **Fiddler API key** (from **Settings** > **Credentials**) whose user has **write access** to each application you intend to send to. Access is governed by your role assignments — see [Access Control](#access-control).
* Multi-application routing **enabled on your Fiddler deployment**.
Multi-application routing is an opt-in capability. If sending a multi-application batch returns everything as rejected, confirm it is enabled on your deployment and that your API key has write access to the target applications — contact your Fiddler Customer Success Manager if you are unsure.
**A newly created application takes a few minutes to propagate.** Fiddler caches each API key's authorized-application list (the collector's default is 5 minutes), so an application you just created can be rejected until that cache refreshes — wait a few minutes and resend.
***
## How the Application ID Reaches Each Span
This is the question every shared-pipeline setup has to answer, so it is worth stating plainly.
The component that emits telemetry — a gateway, a collector, an egress service — is usually **not** the component that knows which Fiddler application a request belongs to. The **caller** knows that. So the application ID has to travel from the caller, through the emitter, and onto each span as the `application.id` attribute. Where and how you set it depends on what is emitting the spans:
* **One application per process** (an instrumented service) — set `application.id` on the resource, via an environment variable or the SDK. See [One Application per Process](#one-application-per-process).
* **Many applications from one process** (a gateway, or a service that switches application per request) — set `application.id` per span. See [Many Applications from One Process](#many-applications-from-one-process).
* **A batch or replay pipeline** — set `application.id` as you assemble each span; see the protobuf example below.
### Resource or Span by Topology
Fiddler resolves each span's application **resource-preferred, span-fallback**: a resource-level `application.id` decides for every span under it, a span-level `application.id` is used only when the resource sets none, and a span with neither fails closed (rejected as unauthorized). Which level you set depends on your topology — and for the shared-pipeline case this page targets, resource-level is the wrong choice:
* **One application per process** (an instrumented service) — an OpenTelemetry `TracerProvider` carries exactly one `Resource`, so a resource-level `application.id` tags every span the process emits with the same application. That is correct when a process serves one application — the [single-application flow](/integrations/agentic-ai/otel-trace-export).
* **Many applications from one process** (a gateway, collector, or shared exporter — what this page is about) — set `application.id` **per span**, and do **not** set it on the resource.
**On a shared emitter, a resource-level `application.id` silently overrides every per-request value.** The process has one resource and resource beats span, so a resource-level value pins *everything* it sends to that one application and ignores the per-span value — with no error. For multi-application routing, set `application.id` per span and leave the resource unset. (A batch or replay pipeline is the exception: it groups each application's spans under their own `ResourceSpans`, so a resource-level value there is correct — see the protobuf example below.)
***
## Setting application.id
### One Application per Process
When a process serves a single application, set `application.id` on the **resource**. The cheapest form is environment variables — no code — and works with any OpenTelemetry SDK:
```bash theme={null}
OTEL_EXPORTER_OTLP_ENDPOINT="https://your-instance.fiddler.ai"
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer "
OTEL_RESOURCE_ATTRIBUTES="application.id=,service.name=my-agent-service"
```
Or in code, with the OpenTelemetry SDK's `Resource` helper (rather than hand-building protobuf):
```python theme={null}
from opentelemetry.sdk.resources import Resource
resource = Resource.create({
"application.id": "",
"service.name": "my-agent-service",
})
# pass `resource` when you build your TracerProvider
```
This is still multi-application routing — just spread across processes: point many such single-application services at the **same** Fiddler endpoint and API key, and Fiddler routes each service's spans to its own application. No per-service pipeline, and no routing config to build.
### Many Applications from One Process
When one process emits spans for several applications — a gateway, or a service that switches application per request — set `application.id` **per span** and leave it off the resource (see [the trap above](#resource-or-span-by-topology)).
**With the OpenTelemetry SDK**, set it on each span as you create it, and do not put `application.id` on the `Resource`:
```python theme={null}
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
# Configure the exporter once — no application.id on the Resource.
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
endpoint="https://your-instance.fiddler.ai/v1/traces",
headers={"Authorization": "Bearer "},
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
# Per request: tag the span with the caller's application ID — this routes it.
with tracer.start_as_current_span("chat") as span:
span.set_attribute("application.id", "")
span.set_attribute("fiddler.span.type", "llm")
# ... your LLM call and other span attributes
# BatchSpanProcessor exports asynchronously; in a short-lived script, flush
# before exit or the spans never leave the process.
provider.shutdown()
```
**With AgentGateway**, the caller sends the application ID as a request header and a CEL expression copies it onto each span — the same shape AgentGateway already uses for `gen_ai.conversation.id`:
```yaml theme={null}
frontendPolicies:
tracing:
# ... host / protocol / auth as in the AgentGateway integration guide
attributes:
# Per-request application ID, copied from the caller's header onto each span.
application.id: |
request.headers["x-fiddler-application-id"] != "" ? request.headers["x-fiddler-application-id"] : ""
gen_ai.conversation.id: |
request.headers["x-fiddler-conversation-id"] != "" ? request.headers["x-fiddler-conversation-id"] : ""
```
Two changes from the single-application [AgentGateway integration](/integrations/agentic-ai/agentgateway-integration) config are required:
* **Move `application.id` out of `resources` and into `attributes`.** The single-application config sets `resources.application.id` to a fixed value; leaving it there pins every span to one application (resource beats span) and the header is ignored.
* **Drop the `fiddler-application-id` request header** that the single-application config adds via `requestHeaderModifier`. On a multi-application deployment that header is ignored, so it is dead config — removing it is a cleanup, not a correctness fix.
The caller then sends its application ID on every request, e.g. `X-Fiddler-Application-Id: ` (exactly as it already sends `X-Fiddler-Conversation-Id`).
When you hold spans as data rather than running a live tracer — replaying spans from object storage, running an ETL job, or backfilling history — assemble the OTLP payload directly. Group each application's spans under their own `ResourceSpans` with that application's `application.id` on the resource (here a resource-level value is correct, because each `ResourceSpans` belongs to exactly one application), and send the whole batch with one API key.
```python theme={null}
import gzip
import httpx
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
from opentelemetry.proto.common.v1.common_pb2 import AnyValue, InstrumentationScope, KeyValue
from opentelemetry.proto.resource.v1.resource_pb2 import Resource
from opentelemetry.proto.trace.v1.trace_pb2 import ResourceSpans, ScopeSpans, Span, Status
FIDDLER_URL = "https://your-instance.fiddler.ai"
FIDDLER_API_KEY = "your-api-key"
APP_A = "11111111-1111-1111-1111-111111111111"
APP_B = "22222222-2222-2222-2222-222222222222"
def llm_span(name: str) -> Span:
return Span(
trace_id=bytes.fromhex("4bf92f3577b34da6a3ce929d0e0e4736"),
span_id=bytes.fromhex("00f067aa0ba902b7"),
name=name,
kind=Span.SPAN_KIND_INTERNAL,
start_time_unix_nano=1_700_000_000_000_000_000,
end_time_unix_nano=1_700_000_001_000_000_000,
status=Status(code=Status.STATUS_CODE_OK),
attributes=[
KeyValue(key="fiddler.span.type", value=AnyValue(string_value="llm")),
KeyValue(key="gen_ai.request.model", value=AnyValue(string_value="gpt-4o")),
],
)
def app_resource_spans(app_uuid: str, span_name: str) -> ResourceSpans:
return ResourceSpans(
resource=Resource(attributes=[
KeyValue(key="application.id", value=AnyValue(string_value=app_uuid)),
]),
scope_spans=[ScopeSpans(
scope=InstrumentationScope(name="my-tracer", version="1.0.0"),
spans=[llm_span(span_name)],
)],
)
# One batch, two applications
payload = ExportTraceServiceRequest(resource_spans=[
app_resource_spans(APP_A, "chat-app-a"),
app_resource_spans(APP_B, "chat-app-b"),
]).SerializeToString()
response = httpx.post(
f"{FIDDLER_URL}/v1/traces",
content=gzip.compress(payload),
headers={
"Authorization": f"Bearer {FIDDLER_API_KEY}",
"Content-Type": "application/x-protobuf",
"Content-Encoding": "gzip",
},
timeout=30.0,
)
response.raise_for_status() # 200 even when some spans are rejected — inspect partial_success
```
For the full span/attribute schema (span types, LLM attributes, typing) and a value-to-`AnyValue` helper, see [Exporting OTel Traces to Fiddler](/integrations/agentic-ai/otel-trace-export) and [Span and Resource Attributes](/integrations/agentic-ai/attributes).
***
## Producer Support
Any producer that can set `application.id` **per span** can drive multi-application routing. Today that is:
| Producer | Per-request multi-application routing | How |
| ----------------- | ------------------------------------- | --------------------------------------------------------------------- |
| OpenTelemetry SDK | ✅ Supported | `span.set_attribute("application.id", ...)` per span |
| AgentGateway | ✅ Supported | CEL maps a request header to the `application.id` span attribute |
| LiteLLM | ⚠️ Single application per proxy | Set one `application.id` for the proxy via `OTEL_RESOURCE_ATTRIBUTES` |
**LiteLLM does not currently support per-request application routing.** A caller's per-request `application.id` (whether sent as a header or in request metadata) is captured only inside namespaced metadata attributes such as `metadata.requester_custom_headers`, which Fiddler resolves *after* the routing decision has already been made. Routing keys off the bare `application.id` at ingest, and LiteLLM never sets that per request — so a per-request application ID cannot influence which application a span lands in. (This is the same reason per-request `gen_ai.conversation.id` works but `application.id` does not: a conversation ID is a display field resolved late in the pipeline, while an application ID is a routing field needed at ingest — same transport, different timing.) Use LiteLLM for single-application export: set one `application.id` for the proxy via `OTEL_RESOURCE_ATTRIBUTES`. Per-request routing through LiteLLM would require a Fiddler-supplied callback that sets `application.id` at emit time — contact your Fiddler Customer Success Manager if you need it.
***
## Migrating from the Single-Application Header
**Once multi-application routing is enabled, the `fiddler-application-id` header is ignored.** Routing is driven entirely by each span's `application.id` attribute. A client that identified its application **only** through that header — with no `application.id` on the resource or span — will have every span fail closed (rejected as unauthorized).
Fiddler's documented integrations already set `application.id` alongside the header: both the [LiteLLM](/integrations/agentic-ai/litellm-integration) and [AgentGateway](/integrations/agentic-ai/agentgateway-integration) guides set the `application.id` attribute as well, so they keep working. Only a client that was trimmed down to header-only is affected. Before enabling multi-application routing, confirm every producer sets `application.id` on the resource or span.
***
## Trust Model and Caller-Supplied Application IDs
Multi-application routing authorizes the **API key**, not the caller. Fiddler checks each span's `application.id` against the applications your API key is permitted to write to, and rejects the rest. This protects Fiddler's boundary between tenants: a caller cannot route a span to an application your API key has no access to.
It does **not**, on its own, isolate a shared gateway's callers from one another. The `application.id` on a span is caller-supplied input, and an API key that can write to many applications can write to **any** of them — so any caller of that gateway can tag a span as any application the gateway's API key can reach. If a shared gateway must keep its own callers in separate applications, enforce that at the gateway — for example, derive the application ID from an authenticated caller identity rather than trusting a raw client header. Per-span authorization protects Fiddler's tenancy boundary, not the gateway's internal one.
***
## The partialSuccess Response
When every span is authorized, the response is a plain `200` with an empty body. When one or more spans are rejected, Fiddler still returns `200` and populates the standard OTLP **`partialSuccess`** block:
| Field | Meaning |
| --------------- | ------------------------------------------------------------------------------------------------ |
| `rejectedSpans` | How many spans were not stored because the API key was not authorized for their `application.id` |
| `errorMessage` | A human-readable summary naming the unauthorized `application.id` values |
```json theme={null}
{
"partialSuccess": {
"rejectedSpans": 1,
"errorMessage": "1 span(s) rejected: unauthorized application IDs [22222222-2222-2222-2222-222222222222]"
}
}
```
`partialSuccess` is **not** an error — the HTTP status is still `200` and the authorized spans in the batch are ingested. Do not treat a populated `partialSuccess` as a failed request; instead, read `rejectedSpans` and `errorMessage` to see which applications were rejected and why.
The most common cause of a rejection is an API key without write access to the named application. Grant the API key's user the appropriate role on that application (see below) and resend.
***
## Access Control
Authorization is per span: Fiddler compares each span's `application.id` against the set of applications the API key's user is allowed to write to. A span is stored only if its application is in that set; otherwise it is rejected and reported in `partialSuccess`.
To let an API key send to an application, ensure its user has a role that grants write access to that application's project. See [Role-Based Access Control](/reference/access-control/role-based-access) for the available roles, and [API Keys](/reference/administration/settings#credentials) for creating and managing API keys.
***
## Frequently Asked Questions
The caller sends it as a request header and the gateway copies it onto each span as the `application.id` attribute. For AgentGateway this is a one-line CEL expression — see [Many Applications from One Process](#many-applications-from-one-process). The gateway itself does not know which application a request belongs to; the caller does.
No. In the multi-application flow, routing is driven by each span's `application.id` and the header is ignored. The single-application `fiddler-application-id` header still works for one-application-per-request exports on deployments without multi-application routing.
Resource-level takes precedence. If a `ResourceSpans` sets `application.id` on its resource, that value applies to every span under it; the span-level attribute is a fallback used only when the resource does not set one. From a shared emitter, set it per span so one process can route to many applications — see [the trap above](#resource-or-span-by-topology).
They are still ingested. Rejection is per span — authorized and unauthorized spans can share a batch, and only the unauthorized ones are dropped and reported in `partialSuccess`.
It is treated as unauthorized (fail-closed): it is not stored and is counted in `rejectedSpans`.
Both are rejected the same way as any unauthorized application — counted in `rejectedSpans` and named in `errorMessage`. Fiddler matches each span's `application.id` against the set of applications your API key can write to; a well-formed UUID that no application matches, or a value that isn't a valid UUID at all, is simply not in that set (Fiddler does not validate UUID format at ingest). Only a missing or empty `application.id` behaves differently — it fails closed (see the previous question).
Common causes: the application was **just created** and hasn't propagated yet — Fiddler caches each API key's authorized-application list (the collector's default is 5 minutes), so a brand-new application is rejected until that cache refreshes; wait a few minutes and resend. Otherwise: the API key does not have write access to the target applications; multi-application routing is not enabled on your deployment; or a fixed resource-level `application.id` (or header-only setup) is overriding the per-span value. Verify the API key's roles, confirm the capability is enabled with your Fiddler Customer Success Manager, and check that `application.id` is set per span.
***
## Related Documentation
Configure AgentGateway's tracing, including the CEL header-to-attribute mapping.
The single-application export flow, span schema, and attribute mapping.
Live agent instrumentation via the OTel SDK.
Attribute typing, custom attributes, and how they flow into metrics.
Roles that govern which applications an API key can write to.
Create and manage the API keys used for trace ingestion.
# OpenTelemetry Integration
Source: https://docs.fiddler.ai/integrations/agentic-ai/opentelemetry-integration
Connect custom AI agents and multi-framework agentic applications to Fiddler using OpenTelemetry's OTLP protocol for comprehensive observability.
## Overview
Fiddler supports native OpenTelemetry integration through the OTLP (OpenTelemetry Protocol) standard, enabling you to monitor custom AI agents and multi-framework environments with full observability. This integration provides a vendor-neutral approach to collecting telemetry data from your agentic applications, allowing you to instrument agents built with any framework or custom implementation.
The OpenTelemetry integration maps your agent's span attributes to Fiddler's semantic conventions, capturing LLM calls, tool executions, agent chains, and custom business metrics. This approach is ideal for teams running multiple agentic frameworks or building custom agent architectures that require unified monitoring across diverse technologies.
With OpenTelemetry, you maintain complete control over your instrumentation while benefiting from Fiddler's advanced analytics, trace visualization, cost tracking, and performance monitoring capabilities.
## When to Use OpenTelemetry Integration
Use OpenTelemetry integration when:
* **Multi-framework environments**: You're using multiple agent frameworks and need unified observability
* **Custom agent architectures**: Your agent framework doesn't have a dedicated Fiddler SDK
* **Advanced instrumentation control**: You need fine-grained control over trace attributes and sampling
* **Standards-based approach**: You want a vendor-neutral telemetry solution using industry standards
* **Existing OpenTelemetry setup**: You're already using OpenTelemetry and want to route traces to Fiddler
**When to Use Fiddler SDKs Instead**
For specific frameworks with dedicated SDKs, we recommend using the framework-specific integration for easier setup and automatic instrumentation:
* **LangGraph and LangChain** → [Use Fiddler LangGraph SDK](/sdk-api/langgraph/fiddler-client)
* **Strands Agents** → Use [Strands Agents SDK](/sdk-api/strands/strands-agent-instrumentor)
These SDKs provide auto-instrumentation and require significantly less code to set up.
## Quick Links
### Getting Started
* [**OpenTelemetry Quick Start Guide**](/developers/quick-starts/opentelemetry-quick-start) - Step-by-step guide to integrate OpenTelemetry with Fiddler (\~10-15 minutes)
* [**Advanced OpenTelemetry Notebook**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_OpenTelemetry_Advanced.ipynb) - Comprehensive working examples with production patterns ([Open in Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_OpenTelemetry_Advanced.ipynb))
### Related Documentation
* [Getting Started: Agentic Observability](/getting-started/agentic-monitoring) - Overview of agentic observability concepts
* [Agentic AI Integrations](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) - Compare integration options for different frameworks
* [Fiddler LangGraph SDK](/integrations/agentic-ai/langgraph-sdk) - Alternative for LangGraph/LangChain users
* [Strands Agents SDK](/integrations/agentic-ai/strands-sdk) - Alternative for Strands agent users
## Supported Features
OpenTelemetry integration with Fiddler supports:
| Feature | Support | Description |
| ------------------------- | ------- | -------------------------------------------------------- |
| **LLM Tracing** | ✅ Full | Track LLM calls with prompts, responses, and token usage |
| **Tool Execution** | ✅ Full | Monitor tool calls with inputs and outputs |
| **Agent Chains** | ✅ Full | Visualize complex agent workflows and decision flows |
| **Custom Attributes** | ✅ Full | Add business context with user-defined attributes |
| **Conversation Tracking** | ✅ Full | Track multi-turn conversations across sessions |
| **Token Cost Analysis** | ✅ Full | Monitor LLM API costs through token tracking |
| **Performance Metrics** | ✅ Full | Latency, throughput, and error rate tracking |
| **Sampling** | ✅ Full | Configure trace sampling for high-volume applications |
| **Batch Processing** | ✅ Full | Optimize network usage with batched exports |
| **Compression** | ✅ Full | Reduce data transmission with gzip compression |
## Integration Overview
### Architecture
```mermaid theme={null}
graph TB
subgraph app["Your AI Agent Application"]
sdk["OpenTelemetry SDK TracerProvider, span creation, attribute mapping"]
exporter["OTLP Exporter batch processing, compression"]
sdk --> exporter
end
exporter -->|"HTTPS (OTLP/HTTP)"| ingest
subgraph fiddler["Fiddler Platform"]
ingest["OTLP Ingestion Endpoint"]
processing["Trace Processing semantic validation, attribute extraction"]
analytics["Analytics and Visualization dashboards, alerts, cost tracking"]
ingest --> processing --> analytics
end
style sdk fill:#e1f5ff
style exporter fill:#e1f5ff
style ingest fill:#e6ffe6
style processing fill:#e6ffe6
style analytics fill:#e6ffe6
```
### Required Setup
1. **Environment Configuration**
* Set `OTEL_EXPORTER_OTLP_ENDPOINT` to your Fiddler instance URL
* Configure `OTEL_EXPORTER_OTLP_HEADERS` with authentication and application ID
* Set `OTEL_RESOURCE_ATTRIBUTES` with your application UUID
2. **OpenTelemetry Initialization**
* Install OpenTelemetry packages
* Initialize `TracerProvider` with proper configuration
* Configure `OTLPSpanExporter` for Fiddler endpoint
* Add `BatchSpanProcessor` for efficient transmission
3. **Span Instrumentation**
* Create spans for agent operations (chains, LLM calls, tools)
* Map attributes to Fiddler semantic conventions
* Add custom business context as needed
## Attribute Mapping
Fiddler requires specific attributes to properly process and visualize your traces:
### Required Attributes
**Resource Level:**
* `application.id` - Your Fiddler application UUID (UUID4 format)
**Span Level:**
* `fiddler.span.type` - Type of operation: `chain`, `llm`, `tool`, or `agent`
### Optional Attributes
* **Agent**: `gen_ai.agent.name` - Name of your AI agent. If provided, set on **every span in the trace** so all spans are attributed to the correct agent.
* **Agent**: `gen_ai.agent.id` - Unique identifier for the agent. If provided, set on **every span in the trace** alongside `gen_ai.agent.name`.
* **LLM Spans**: Model, system prompt, user input, output, token usage
* **Tool Spans**: Tool name, input JSON, output JSON
* **Conversation**: `gen_ai.conversation.id` for session tracking
* **Custom**: `fiddler.session.user.*` session attributes and custom span attributes for business context
## Getting Started
Ready to integrate OpenTelemetry with Fiddler?
1. [**Follow the Quick Start Guide**](/developers/quick-starts/opentelemetry-quick-start) - Complete setup in 10-15 minutes
2. [**Explore the Advanced Notebook**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_OpenTelemetry_Advanced.ipynb) - Learn production patterns
3. **Review the attribute reference** - Understand required and optional attributes
4. **Test your integration** - Verify traces appear in Fiddler dashboard
## Example: Instrumenting a Custom Agent
While OpenTelemetry integration works with any framework, here's a basic example:
`tracer.start_as_current_span()` can be used as a function decorator (recommended
when a span wraps an entire function) or as a context manager (when you need a
span narrower than a function). Both are pure OpenTelemetry — no Fiddler SDK required.
```python theme={null}
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
# Recommended: decorate the function to span its full execution
@tracer.start_as_current_span("custom_agent")
def run_agent(user_request: str):
span = trace.get_current_span()
span.set_attribute("fiddler.span.type", "chain")
span.set_attribute("gen_ai.agent.name", "my_agent") # optional
span.set_attribute("gen_ai.agent.id", "my_agent_v1") # optional
# Alternative: use a context manager for a span narrower than the
# function — wrapping just one section instead of the whole call.
with tracer.start_as_current_span("db_lookup") as section_span:
section_span.set_attribute("fiddler.span.type", "tool")
# ... a specific section of code
# ... rest of your agent code
```
# Exporting OTel Traces to Fiddler
Source: https://docs.fiddler.ai/integrations/agentic-ai/otel-trace-export
Map pre-existing OpenTelemetry span attributes to Fiddler's schema and export them to the v1/traces protobuf endpoint from your own storage or pipeline.
## Overview
This guide covers the **client-side export** scenario: your application has already generated OpenTelemetry traces, you manage their storage and processing, and you need to ship them to Fiddler. You are responsible for:
1. **Attribute mapping** — translating your OTel span attributes to Fiddler's schema
2. **Protobuf serialization** — building `ResourceSpans → ScopeSpans → Span` structures
3. **Export** — POSTing the compressed payload to Fiddler's `v1/traces` endpoint
This differs from [live instrumentation](/integrations/agentic-ai/opentelemetry-integration), where the OTel SDK exports spans automatically as your agent runs.
**When to use this approach**
Use client-side export when you have:
* Traces stored in a data warehouse, JSONL files, or a logging pipeline that you want to replay into Fiddler
* A custom export pipeline that processes spans before sending (e.g., filtering, enrichment, or ID regeneration)
* Batch backfill of historical trace data
For real-time agent instrumentation, use the [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration) or a framework SDK instead.
***
## Prerequisites
* A Fiddler account with a GenAI application created — you will need its **Application UUID**
* A valid **Fiddler API token** (from **Settings** > **Credentials**)
* Python packages:
```bash theme={null}
pip install opentelemetry-proto httpx
```
***
## The v1/traces Endpoint
Send traces as a gzip-compressed protobuf `ExportTraceServiceRequest` payload:
| Property | Value |
| -------------------------- | ------------------------------------------- |
| **URL** | `https:///v1/traces` |
| **Method** | `POST` |
| **Content-Type** | `application/x-protobuf` |
| **Content-Encoding** | `gzip` |
| **Authorization** | `Bearer ` |
| **fiddler-application-id** | `` |
```python theme={null}
import gzip
import httpx
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
payload = ExportTraceServiceRequest(resource_spans=[...]).SerializeToString()
httpx.post(
"https://your-instance.fiddler.ai/v1/traces",
content=gzip.compress(payload),
headers={
"Authorization": "Bearer ",
"fiddler-application-id": "",
"Content-Type": "application/x-protobuf",
"Content-Encoding": "gzip",
},
)
```
***
## Protobuf Structure
Fiddler expects the standard OTLP hierarchy:
```
ExportTraceServiceRequest
└── ResourceSpans[]
├── Resource
│ └── attributes: [application.id, service.name, ...]
└── ScopeSpans[]
└── Span[]
├── trace_id, span_id, parent_span_id
├── name, kind, status
├── start_time_unix_nano, end_time_unix_nano
└── attributes: [gen_ai.agent.name, fiddler.span.type, ...]
```
**`application.id`** must be set at the **Resource** level (not on individual spans):
```python theme={null}
from opentelemetry.proto.resource.v1.resource_pb2 import Resource
from opentelemetry.proto.common.v1.common_pb2 import AnyValue, KeyValue
resource = Resource(attributes=[
KeyValue(key="application.id", value=AnyValue(string_value="")),
KeyValue(key="service.name", value=AnyValue(string_value="my-agent-service")),
])
```
***
## AnyValue Typing
Each `KeyValue` in the protobuf structure carries its value in an `AnyValue` message. `AnyValue` is a **oneof** — exactly one typed field must be set. Using the correct field ensures Fiddler classifies the data with the right type (e.g., numeric fields enable aggregation, charting, and alerting).
| Python type | `AnyValue` field | Fiddler storage type | Example |
| ----------- | ---------------- | ------------------------- | --------------------------------- |
| `str` | `string_value` | String | `AnyValue(string_value="gpt-4o")` |
| `int` | `int_value` | Numeric (stored as float) | `AnyValue(int_value=150)` |
| `float` | `double_value` | Numeric (stored as float) | `AnyValue(double_value=0.95)` |
| `bool` | `bool_value` | String | `AnyValue(bool_value=True)` |
**Use `int_value` / `double_value` for numeric attributes.** If you wrap a number in `string_value` (e.g., `AnyValue(string_value="150")`), Fiddler will treat it as a string column. This means you lose the ability to compute numerical aggregations (sum, average, percentiles) and set alerts on that attribute.
```python theme={null}
from opentelemetry.proto.common.v1.common_pb2 import AnyValue
def to_any_value(value: str | int | float | bool) -> AnyValue:
"""
Convert a Python value to a correctly typed AnyValue.
:param value: The value to convert.
:returns: An AnyValue with the appropriate typed field set.
:raises TypeError: If *value* is not a supported type (including
``None`` and ``bytes``). Filter ``None`` values upstream — OTel
attributes have no null concept, so omit the attribute instead.
For ``bytes``, decode to ``str`` first — Fiddler stores bytes as
strings internally.
Example::
to_any_value(150) # AnyValue(int_value=150)
to_any_value("us-west") # AnyValue(string_value="us-west")
"""
# bool must be checked before int (bool is a subclass of int in Python)
if isinstance(value, bool):
return AnyValue(bool_value=value)
if isinstance(value, int):
return AnyValue(int_value=value)
if isinstance(value, float):
return AnyValue(double_value=value)
if isinstance(value, str):
return AnyValue(string_value=value)
# For bytes, consider decoding to str to unblock: AnyValue(string_value=value.decode("utf-8"))
# For None, consider Filtering upstream, omitting the attribute instead.
raise TypeError(f"Unsupported type {type(value).__name__} for AnyValue")
```
For the full attribute quick overview (all attributes, types, and `AnyValue` fields), see [Span and Resource Attributes — Quick Overview](/integrations/agentic-ai/attributes#attribute-quick-overview).
***
## Attribute Mapping Reference
### Span Structure Fields
The following fields control the span's structural properties and are **not** sent as span attributes. Map them to the corresponding protobuf `Span` fields instead:
| Your field | Protobuf `Span` field |
| -------------------- | ------------------------------------------ |
| `trace_id` | `trace_id` (bytes, 16 bytes / 32-char hex) |
| `span_id` | `span_id` (bytes, 8 bytes / 16-char hex) |
| `parent_span_id` | `parent_span_id` (bytes, same format) |
| `span_name` / `name` | `name` |
| `span_kind` | `kind` (SpanKind enum) |
| `status_code` | `status.code` (StatusCode enum) |
| `start_time` | `start_time_unix_nano` |
| `end_time` | `end_time_unix_nano` |
### Required Span Attributes
Every span sent to Fiddler must include this attribute:
| Attribute | Description |
| ------------------- | --------------------------------------------- |
| `fiddler.span.type` | Span type: `llm`, `tool`, `chain`, or `agent` |
`application.id` is required at the **Resource** level (see above).
### Span Type: Deriving from gen\_ai.operation.name
If your spans follow the GenAI semantic conventions and carry `gen_ai.operation.name`, map it to `fiddler.span.type` as follows:
| `gen_ai.operation.name` | `fiddler.span.type` |
| ----------------------------- | ------------------- |
| `chat` | `llm` |
| `execute_tool` | `tool` |
| `invoke_agent` | `chain` |
| *(any other value or absent)* | `chain` *(default)* |
### LLM Semantic Convention Mappings
| Your OTel attribute | Fiddler attribute | Notes |
| ---------------------------- | ---------------------------------------------- | ----------------------------------------------------------- |
| `gen_ai.input.messages` | `gen_ai.llm.input.user` + `gen_ai.llm.context` | See [message parsing](#parsing-gen_ai-input-messages) below |
| `gen_ai.output.messages` | `gen_ai.llm.output` | |
| `gen_ai.system_instructions` | `gen_ai.llm.input.system` | |
| `gen_ai.response.model` | `gen_ai.request.model` | Fiddler normalizes to the request-side attribute |
| `gen_ai.usage.input_tokens` | `gen_ai.usage.input_tokens` | Pass through unchanged |
| `gen_ai.usage.output_tokens` | `gen_ai.usage.output_tokens` | Pass through unchanged |
| `gen_ai.usage.total_tokens` | `gen_ai.usage.total_tokens` | Pass through unchanged |
| `gen_ai.system` | `gen_ai.system` | LLM provider identifier (e.g. `openai`) |
| `gen_ai.request.model` | `gen_ai.request.model` | Pass through unchanged |
### Tool Semantic Convention Mappings
| Your OTel attribute | Fiddler attribute |
| ---------------------------- | -------------------- |
| `gen_ai.tool.call.arguments` | `gen_ai.tool.input` |
| `gen_ai.tool.call.result` | `gen_ai.tool.output` |
| `gen_ai.tool.name` | `gen_ai.tool.name` |
### Agent and Conversation Attributes
| Your OTel attribute | Fiddler attribute | Notes |
| ------------------------ | ------------------------ | ---------------------------------------------------- |
| `gen_ai.agent.name` | `gen_ai.agent.name` | Optional |
| `gen_ai.agent.id` | `gen_ai.agent.id` | Optional |
| `gen_ai.conversation.id` | `gen_ai.conversation.id` | Optional — used for multi-turn conversation grouping |
**Set agent attributes on every span.** If `gen_ai.agent.name` or `gen_ai.agent.id` are provided, set both on **every span within the trace**. Fiddler uses these attributes to attribute spans to the correct agent — spans missing these fields will be unattributed even if other spans in the same trace carry them.
### Legacy / Underscore Field Names
If your traces use older underscore-style field names, map them to the Fiddler dotted equivalents before serialization:
| Legacy field | Fiddler attribute |
| ------------------ | ------------------------- |
| `model_name` | `gen_ai.request.model` |
| `model_provider` | `gen_ai.system` |
| `tool_name` | `gen_ai.tool.name` |
| `tool_input` | `gen_ai.tool.input` |
| `tool_output` | `gen_ai.tool.output` |
| `llm_input_system` | `gen_ai.llm.input.system` |
| `llm_input_user` | `gen_ai.llm.input.user` |
| `llm_output` | `gen_ai.llm.output` |
| `llm_context` | `gen_ai.llm.context` |
### Custom User Attributes
To attach business-level metadata to spans, set custom user attributes directly:
```python theme={null}
# Source field: custom_attributes = {"session_type": "onboarding", "region": "us-west"}
# Maps to:
span.attributes["session_type"] = "onboarding"
span.attributes["region"] = "us-west"
```
These attributes are indexed and queryable in Fiddler's [Explorer](/observability/agentic/trace-explorer).
***
## Parsing gen\_ai.input.messages
The `gen_ai.input.messages` attribute is a JSON array of chat-style message objects. Fiddler expects it split into two separate attributes:
| Output attribute | Content |
| ----------------------- | ------------------------------------------------------------------------- |
| `gen_ai.llm.input.user` | Text content of the **last message with `role: user`** |
| `gen_ai.llm.context` | All other messages (conversation history), formatted as `[role]: content` |
**Example input:**
```json theme={null}
[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "Paris."},
{"role": "user", "content": "And Germany?"}
]
```
**Resulting mapping:**
| Attribute | Value |
| ----------------------- | ----------------------------------------------------------------------------------------------------------- |
| `gen_ai.llm.input.user` | `"And Germany?"` |
| `gen_ai.llm.context` | `"[system]: You are a helpful assistant.\n\n[user]: What is the capital of France?\n\n[assistant]: Paris."` |
If no `user` message is found, all messages are placed in `gen_ai.llm.context` and `gen_ai.llm.input.user` is omitted.
Python implementation
```python theme={null}
import json
def parse_input_messages(value: str | list) -> dict[str, str]:
"""
Parse gen_ai.input.messages and split into Fiddler attributes.
Extracts the last user message as ``gen_ai.llm.input.user`` and
formats all other messages as ``gen_ai.llm.context``.
:param value: JSON string or already-parsed list of message dicts.
Each message must have a ``role`` key and either a
``content`` string or a ``parts`` list of text objects.
:returns: Dict with zero, one, or both of:
- ``gen_ai.llm.input.user`` – last user message text
- ``gen_ai.llm.context`` – prior conversation history
"""
# Parse JSON string if needed
if isinstance(value, str):
try:
messages = json.loads(value)
except (json.JSONDecodeError, ValueError):
return {"gen_ai.llm.input.user": value}
else:
messages = value
if not isinstance(messages, list) or not messages:
return {}
def extract_content(message: dict) -> str:
"""Extract plain text from a message object."""
parts = message.get("parts", [])
if isinstance(parts, list):
text_parts = [
part.get("content", "") if isinstance(part, dict) and part.get("type") == "text"
else part
for part in parts
if isinstance(part, str) or (isinstance(part, dict) and part.get("type") == "text")
]
if text_parts:
return "\n".join(text_parts)
elif isinstance(parts, str):
return parts
content = message.get("content", "")
return content if isinstance(content, str) else str(content)
# Find the index of the last user message
last_user_idx = next(
(i for i in range(len(messages) - 1, -1, -1)
if isinstance(messages[i], dict) and messages[i].get("role") == "user"),
None,
)
result: dict[str, str] = {}
if last_user_idx is None:
# No user message — put everything in context
context_parts = [
f"[{m.get('role', 'unknown')}]: {extract_content(m)}"
for m in messages
if isinstance(m, dict)
]
if context_parts:
result["gen_ai.llm.context"] = "\n\n".join(context_parts)
return result
# Last user message → input.user
user_text = extract_content(messages[last_user_idx])
if user_text:
result["gen_ai.llm.input.user"] = user_text
# All other messages → context
context_messages = messages[:last_user_idx] + messages[last_user_idx + 1:]
context_parts = [
f"[{m.get('role', 'unknown')}]: {extract_content(m)}"
for m in context_messages
if isinstance(m, dict) and extract_content(m)
]
if context_parts:
result["gen_ai.llm.context"] = "\n\n".join(context_parts)
return result
# Usage
attributes = parse_input_messages(span_row["gen_ai.input.messages"])
# e.g. {
# "gen_ai.llm.input.user": "And Germany?",
# "gen_ai.llm.context": "[system]: You are a helpful assistant.\n\n[user]: What is the capital of France?\n\n[assistant]: Paris."
# }
```
***
## Span Kind and Status Mappings
**Span kind** (`SpanKind` enum):
| String value | Protobuf value |
| ------------ | ------------------------------- |
| `INTERNAL` | `SpanKind.INTERNAL` *(default)* |
| `SERVER` | `SpanKind.SERVER` |
| `CLIENT` | `SpanKind.CLIENT` |
| `PRODUCER` | `SpanKind.PRODUCER` |
| `CONSUMER` | `SpanKind.CONSUMER` |
**Status code** (`StatusCode` enum):
| String value | Protobuf value |
| ------------ | --------------------------- |
| `OK` | `StatusCode.OK` *(default)* |
| `ERROR` | `StatusCode.ERROR` |
| `UNSET` | `StatusCode.UNSET` |
***
## Minimal End-to-End Example
```python theme={null}
import gzip
import httpx
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
from opentelemetry.proto.common.v1.common_pb2 import AnyValue, InstrumentationScope, KeyValue
from opentelemetry.proto.resource.v1.resource_pb2 import Resource
from opentelemetry.proto.trace.v1.trace_pb2 import ResourceSpans, ScopeSpans, Span, Status
from opentelemetry.trace import SpanKind
from opentelemetry.trace.status import StatusCode
FIDDLER_URL = "https://your-instance.fiddler.ai"
APP_UUID = "550e8400-e29b-41d4-a716-446655440000"
FIDDLER_TOKEN = "your-api-token"
# Build a single LLM span
span = Span(
trace_id=bytes.fromhex("4bf92f3577b34da6a3ce929d0e0e4736"),
span_id=bytes.fromhex("00f067aa0ba902b7"),
name="chat",
kind=SpanKind.INTERNAL.value,
start_time_unix_nano=1_700_000_000_000_000_000,
end_time_unix_nano=1_700_000_001_000_000_000,
status=Status(code=StatusCode.OK.value),
attributes=[
KeyValue(key="gen_ai.agent.name", value=AnyValue(string_value="my-agent")),
KeyValue(key="fiddler.span.type", value=AnyValue(string_value="llm")),
KeyValue(key="gen_ai.request.model", value=AnyValue(string_value="gpt-4o")),
KeyValue(key="gen_ai.llm.input.user", value=AnyValue(string_value="What is the capital of France?")),
KeyValue(key="gen_ai.llm.output", value=AnyValue(string_value="Paris.")),
# Numeric attributes — use int_value so Fiddler registers them as numeric columns
KeyValue(key="gen_ai.usage.input_tokens", value=AnyValue(int_value=12)),
KeyValue(key="gen_ai.usage.output_tokens",value=AnyValue(int_value=3)),
KeyValue(key="gen_ai.usage.total_tokens", value=AnyValue(int_value=15)),
# Custom user attributes — use the AnyValue field matching each value's type
KeyValue(key="confidence", value=AnyValue(double_value=0.97)),
KeyValue(key="region", value=AnyValue(string_value="us-west")),
KeyValue(key="is_internal", value=AnyValue(bool_value=False)),
],
)
# Wrap in ResourceSpans
resource_spans = ResourceSpans(
resource=Resource(attributes=[
KeyValue(key="application.id", value=AnyValue(string_value=APP_UUID)),
KeyValue(key="service.name", value=AnyValue(string_value="my-agent-service")),
]),
scope_spans=[ScopeSpans(
scope=InstrumentationScope(name="my-tracer", version="1.0.0"),
spans=[span],
)],
)
# Serialize and compress
payload = ExportTraceServiceRequest(resource_spans=[resource_spans]).SerializeToString()
# Send
response = httpx.post(
f"{FIDDLER_URL}/v1/traces",
content=gzip.compress(payload),
headers={
"Authorization": f"Bearer {FIDDLER_TOKEN}",
"fiddler-application-id": APP_UUID,
"Content-Type": "application/x-protobuf",
"Content-Encoding": "gzip",
},
timeout=30.0,
)
response.raise_for_status()
```
***
## Related Documentation
* [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration) — live agent instrumentation via the OTel SDK
* [OpenTelemetry Quick Start](/developers/quick-starts/opentelemetry-quick-start) — step-by-step setup guide
* [Agentic AI Overview](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) — compare all integration options
* [Span and Resource Attributes](/integrations/agentic-ai/attributes) — attribute typing, custom attributes, and how they flow into metrics and alerts
# S3 Trace Ingestion
Source: https://docs.fiddler.ai/integrations/agentic-ai/s3-trace-ingestion
Ingest pre-generated OTLP trace files from Amazon S3 into Fiddler without modifying your application code. Ideal for ECS Fargate, air-gapped environments, or any architecture where direct SDK integration is not possible.
## Overview
The Fiddler S3 Connector allows you to ingest OpenTelemetry (OTLP) trace data from Amazon S3 into Fiddler without requiring any live SDK integration. Your application writes OTLP trace files to an S3 bucket (or a compatible object store), and Fiddler's ingestion service automatically discovers, parses, and forwards those traces into the observability platform.
This is the recommended approach for:
* **Air-gapped or high-security deployments** where direct outbound connections from the application to Fiddler are not permitted
* **Batch ingestion pipelines** that transform logs or events into OTLP format and stage them in S3
* **Custom log transformers** that convert raw LangGraph or other framework logs into the Fiddler OTLP format
**When to use the SDK instead**
If your application can make direct outbound HTTPS requests, the [Fiddler LangGraph SDK](/integrations/agentic-ai/langgraph-sdk) or [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) provide zero-config auto-instrumentation with no file staging required. Use S3 ingestion only when direct SDK integration is not possible.
***
## Architecture
***
## Prerequisites
* A Fiddler account with a GenAI **Application** created — you will need its **Application UUID**
* A valid **Fiddler API key** (from **Settings → Credentials**) — this is used to authenticate the worker with the OTEL Collector
* An **Amazon S3 bucket** (or S3-compatible store) that Fiddler's worker can read from
* IAM permissions on the bucket — see [IAM Setup](#iam-setup) below
***
## OTLP File Format
Files placed in S3 must be valid OTLP data in one of the supported formats below. All formats use the standard `ExportTraceServiceRequest` schema.
### Supported file extensions
| Format | Extensions | Description |
| ------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| OTLP JSON | `.json`, `.json.gz`, `.json.zst` | One `ExportTraceServiceRequest` JSON envelope per file. |
| OTLP JSONL | `.jsonl`, `.jsonl.gz`, `.jsonl.zst`, `.ndjson`, `.ndjson.gz`, `.ndjson.zst` | One `ExportTraceServiceRequest` JSON envelope per line, each with a single `resourceSpans` entry. |
| OTLP Protobuf | `.binpb`, `.binpb.gz`, `.binpb.zst` | Binary serialized `ExportTraceServiceRequest` protobuf. |
Compression is transparent — files with `.gz` (gzip) or `.zst` (zstd) suffixes are decompressed automatically before parsing. Configure `file_extensions` on the ingestion source to match the format your pipeline produces.
### Required JSON structure
```json theme={null}
{
"resourceSpans": [
{
"resource": {
"attributes": [
{
"key": "application.id",
"value": { "stringValue": "" }
},
{
"key": "service.name",
"value": { "stringValue": "your-service-name" }
}
]
},
"scopeSpans": [
{
"scope": { "name": "your-tracer-name", "version": "1.0.0" },
"spans": [
{
"traceId": "",
"spanId": "",
"parentSpanId": "",
"name": "agent_invocation",
"kind": 1,
"startTimeUnixNano": "1744200000000000000",
"endTimeUnixNano": "1744200005500000000",
"status": { "code": 1 },
"attributes": [
{
"key": "fiddler.span.type",
"value": { "stringValue": "llm" }
}
]
}
]
}
]
}
]
}
```
### Critical fields
| Field | Description | Notes |
| --------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `resource.attributes["application.id"]` | **Fiddler Application UUID** | **Must match** the application UUID in your Fiddler instance. This routes spans to the correct application in the UI. |
| `startTimeUnixNano` / `endTimeUnixNano` | Span timestamps as nanoseconds since Unix epoch | Must reflect the **actual time** of each event. Wrong timestamps cause spans to appear in the wrong time range in the UI. |
| `traceId` | 16-byte trace identifier | Accepted as **base64** (`S/kvNXezTaajzpKdDg5HNg==`) or **hex** (`4bf92f3577b34da6a3ce929d0e0e4736`). Fiddler auto-normalizes hex to base64. |
| `spanId` | 8-byte span identifier | Same encoding rules as `traceId`. |
| `parentSpanId` | Parent span ID | Set to `""` (empty string) for root spans. |
| `fiddler.span.type` | Span type for Fiddler's UI | Recommended values: `llm`, `tool`. See [Supported span types](#supported-span-types-and-attributes) below. |
**`application.id` must be in the file**
The `application.id` resource attribute inside the OTLP file is the source of truth for routing spans to the correct Fiddler application. The `application_ids` field on the ingestion source (see below) is used for access control only — it does **not** override the `application.id` in the span data. If these do not match, spans will be ingested but will not appear under your application in the UI.
### Supported span types and attributes
| Span Type | Key attributes |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `llm` | `gen_ai.system`, `gen_ai.request.model`, `gen_ai.llm.input.user`, `gen_ai.llm.input.system`, `gen_ai.llm.output`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |
| `tool` | `gen_ai.tool.name`, `gen_ai.tool.input`, `gen_ai.tool.output` |
| `agent` | `gen_ai.agent.name`, `gen_ai.agent.id` |
| `chain` *(legacy — LangChain)* | `gen_ai.agent.name`, `gen_ai.agent.id`, `gen_ai.conversation.id` |
For the full attribute reference including all supported keys and their types, see [Span and Resource Attributes](/integrations/agentic-ai/attributes).
Custom span attributes can be added directly as key-value pairs:
```json theme={null}
{ "key": "risk_rating", "value": { "stringValue": "Moderate-High" } }
```
***
**Platform enablement required**
The S3 connector must be enabled for your Fiddler environment before use. Contact your **Fiddler account team or platform admin** to request enablement. Once confirmed, proceed with the steps below to set up your ingestion source.
## Setting Up the Ingestion Source
Create an ingestion source via the Fiddler REST API to tell the connector where to look in S3.
### API endpoint
```
POST /v3/ingestion-sources
```
### Request body
```json theme={null}
{
"name": "my-agent-traces",
"description": "Production LangGraph agent traces from ECS Fargate",
"provider": "s3",
"region": "us-west-2",
"bucket": "my-company-traces",
"prefix": "fiddler/prod/",
"scan_interval_seconds": 60,
"file_extensions": [".json"],
"credential_type": "iam_role",
"role_arn": "arn:aws:iam::123456789012:role/my-ingestion-role",
"application_ids": [""]
}
```
### Request fields
| Field | Required | Description |
| ----------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `name` | ✅ | Unique name for this ingestion source (1–255 characters) |
| `bucket` | ✅ | S3 bucket name (without `s3://` prefix) |
| `application_ids` | ✅ | List containing exactly one Fiddler application UUID (v1 enforces 1:1) |
| `provider` | ❌ | Object store provider. Only `"s3"` is supported today (Amazon S3 and S3-compatible stores such as MinIO). Default: `"s3"` |
| `region` | ❌ | AWS region of the bucket (e.g. `"us-west-2"`). Optional |
| `prefix` | ❌ | S3 key prefix to scan (e.g. `"traces/prod/"`). Default: `""` (scan entire bucket) |
| `file_extensions` | ❌ | List of file extensions to process. Default: `[".json"]`. Supported values: `.json`, `.json.gz`, `.json.zst`, `.jsonl`, `.jsonl.gz`, `.jsonl.zst`, `.ndjson`, `.ndjson.gz`, `.ndjson.zst`, `.binpb`, `.binpb.gz`, `.binpb.zst` |
| `credential_type` | ❌ | `"iam_role"` (default) to use an IAM role, or `"access_key"` for static access key/secret |
| `access_key_id` | ❌ | Required when `credential_type` is `"access_key"`. Write-only, never returned in responses |
| `secret_access_key` | ❌ | Required when `credential_type` is `"access_key"`. Write-only, never returned in responses |
| `role_arn` | ❌ | IAM role ARN to assume via STS (cross-account). Omit to use the worker's node IAM role |
| `external_id` | ❌ | STS external ID for cross-account role assumption. Write-only, never returned in responses |
| `endpoint_url` | ❌ | Custom endpoint URL for S3-compatible stores (e.g. MinIO). Default: `null` |
| `scan_interval_seconds` | ❌ | How often the manager scans for new files. Default: `60`. Minimum: `10` |
| `description` | ❌ | Optional description (max 1000 characters) |
### Example using Python
```python theme={null}
import requests
FIDDLER_URL = "https://your-instance.fiddler.ai"
API_KEY = "your-api-key"
APPLICATION_ID = "your-application-uuid"
response = requests.post(
f"{FIDDLER_URL}/v3/ingestion-sources",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
},
json={
"name": "my-agent-traces",
"description": "Agent traces from S3",
"provider": "s3",
"region": "us-west-2",
"bucket": "my-traces-bucket",
"prefix": "traces/",
"scan_interval_seconds": 60,
"file_extensions": [".json"],
"credential_type": "iam_role",
"application_ids": [APPLICATION_ID],
},
)
print(response.json())
```
***
## IAM Setup
The Fiddler worker needs read access to your S3 bucket. The recommended approach is an IAM role.
### Minimum required IAM policy
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "FiddlerS3ReadAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-traces-bucket",
"arn:aws:s3:::my-traces-bucket/traces/*"
]
}
]
}
```
### Encrypted buckets (SSE-KMS)
If your bucket is encrypted with a **customer-managed KMS key (CMK)**, the worker needs decrypt permission on that key in addition to the S3 actions above. Without it, `s3:GetObject` succeeds at the S3 authorization step but the request fails at the decryption step, and the file is marked `failed` with an error like:
```
AccessDenied ... not authorized to perform: kms:Decrypt on resource: arn:aws:kms:...:key/...
because no identity-based policy allows the kms:Decrypt action
```
Add the following statement to the role's IAM policy, alongside the `FiddlerS3ReadAccess` statement:
```json theme={null}
{
"Sid": "FiddlerKmsDecrypt",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:us-west-2:123456789012:key/"
}
```
`kms:Decrypt` is the only permission strictly required for the read path; `kms:DescribeKey` is optional but commonly included for diagnostics.
KMS access is gated by **two policies evaluated together** — the role's IAM policy **and** the KMS key policy. If the key policy keeps the default `arn:aws:iam::123456789012:root` / `kms:*` delegation, granting `kms:Decrypt` in the IAM policy above is sufficient. If that delegation was removed or restricted, you must also add the worker role as an allowed principal in the **KMS key policy** with `kms:Decrypt` and `kms:DescribeKey`.
Buckets encrypted with **SSE-S3 (AES-256)** or the **AWS-managed `aws/s3` key** do not require any extra KMS permissions.
### Options
| Option | How to configure | When to use |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ |
| **EKS Node IAM Role** | Grant the Fiddler worker node's IAM role access to your bucket. Omit `role_arn` in the API request. | Same AWS account, simple setup |
| **Cross-account IAM Role** | Create a role in your account with a trust policy allowing Fiddler's worker role to assume it. Set `role_arn` in the API request. | Cross-account or tighter security boundary |
***
## Monitoring File Processing Status
### Aggregate stats
Get a summary count of files by status and total spans ingested:
```
GET /v3/ingestion-sources/{source_id}/stats
```
```bash theme={null}
curl -H "Authorization: Bearer " \
"https://your-instance.fiddler.ai/v3/ingestion-sources//stats"
```
Example response:
```json theme={null}
{
"data": {
"source_id": "",
"total_files": 10,
"pending": 2,
"processing": 1,
"completed": 6,
"failed": 1,
"total_spans_ingested": 342,
"last_file_completed_at": "2026-04-10T09:38:22Z",
"last_file_failed_at": null
}
}
```
### Per-file list
List individual files with their status:
```
GET /v3/ingestion-sources/{source_id}/files
```
```bash theme={null}
curl -H "Authorization: Bearer " \
"https://your-instance.fiddler.ai/v3/ingestion-sources//files"
```
### File statuses
| Status | Meaning |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pending` | Discovered by the manager, queued for processing. Also the status a file returns to after a transient error when retries remain (`retries` counter increments each time). |
| `processing` | Worker is currently downloading and parsing the file |
| `completed` | Successfully ingested. `spans_count` shows how many spans were sent |
| `failed` | Permanently failed. Either a non-retryable error (e.g. malformed OTLP data, auth error) or all retries exhausted. Check `error_message` for details. Use the retry API to re-queue. |
### Example response
```json theme={null}
{
"data": {
"items": [
{
"id": 1,
"s3_key": "traces/sample_trace.json",
"file_size_bytes": 10097,
"status": "completed",
"error_message": null,
"spans_count": 4,
"retries": 0,
"max_retries": 3,
"processed_at": "2026-04-10T09:38:22Z",
"created_at": "2026-04-10T09:38:18Z"
}
]
}
}
```
### Retry a failed file
Retry a single failed file:
```
POST /v3/ingestion-sources/{source_id}/files/{file_id}/retry
```
Bulk retry all failed files for a source at once:
```
POST /v3/ingestion-sources/{source_id}/retry-failed
```
***
## Testing the Connection
Before uploading production traces, verify that the ingestion source can reach your bucket:
```
POST /v3/ingestion-sources/{source_id}/test
```
```bash theme={null}
curl -X POST \
-H "Authorization: Bearer " \
"https://your-instance.fiddler.ai/v3/ingestion-sources//test"
```
A successful response returns `{ "status": "success", "files_found": N }` where `files_found` is the number of files discovered under the configured prefix. A failure returns `{ "status": "error", "message": "" }` (e.g. `AccessDenied`, `NoSuchBucket`).
***
## Generating OTLP Files with the Fiddler SDK
If your application can write files locally (but cannot send traces directly to Fiddler), you can use the Fiddler LangGraph SDK's built-in OTLP file capture and upload the files to S3 separately.
```python theme={null}
from fiddler_langgraph import FiddlerClient
client = FiddlerClient(
application_id="your-application-uuid",
otlp_enabled=False, # Do not send traces directly to Fiddler
otlp_json_capture_enabled=True,
otlp_json_output_dir="./traces", # Directory to write OTLP JSON files
)
```
Each LangGraph invocation writes one OTLP JSON file to `./traces/`. Upload these files to your S3 bucket using the AWS CLI or any S3 client:
```bash theme={null}
aws s3 sync ./traces/ s3://my-traces-bucket/traces/
```
For environments where local file writes are also not possible (e.g. ECS Fargate with read-only filesystems), generate the OTLP JSON in memory and stream it directly to S3 using `boto3.client('s3').put_object()` without writing to disk.
***
## Troubleshooting
### Spans not appearing in the UI
| Symptom | Likely cause | Fix |
| ---------------------------------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| File `status: completed` but no spans visible | Time range filter in UI doesn't cover the span timestamps | Change the UI date range to match `startTimeUnixNano` in your files |
| File `status: completed` but spans under wrong application | `application.id` in the file doesn't match the Fiddler application UUID | Update your transformer to embed the correct `application.id` as a resource attribute |
| File `status: failed` with `invalid TraceID length` | `traceId` / `spanId` encoded incorrectly | Use 32-char hex or standard base64. Fiddler auto-normalizes hex; avoid other encodings |
| File `status: failed` with `AccessDenied` | Worker IAM role lacks `s3:GetObject` on your bucket | Update the IAM policy — see [IAM Setup](#iam-setup) |
| File `status: failed` with `kms:Decrypt` `AccessDenied` | Bucket uses SSE-KMS and the worker role lacks `kms:Decrypt` on the CMK | Add `kms:Decrypt` + `kms:DescribeKey` to the role's IAM policy, and ensure the KMS key policy allows the role — see [Encrypted buckets (SSE-KMS)](#encrypted-buckets-sse-kms) |
| File `status: failed` with `CredentialError` | `OTEL_COLLECTOR_AUTH_TOKEN` not configured on the worker | Contact your Fiddler admin to configure the collector auth token |
| Files never appear (stuck in `pending`) | Manager cannot list the bucket | Check `s3:ListBucket` permission and that `prefix` is correct |
### Verifying file format locally
Use the following Python snippets to validate your OTLP files before uploading:
**JSON (`.json`):**
```python theme={null}
import json
from google.protobuf.json_format import ParseDict
from opentelemetry.proto.trace.v1.trace_pb2 import ResourceSpans
with open("my_trace.json") as f:
data = json.load(f)
for rs_dict in data["resourceSpans"]:
try:
rs = ParseDict(rs_dict, ResourceSpans())
print(f"OK: {sum(len(ss.spans) for ss in rs.scope_spans)} spans")
except Exception as e:
print(f"ERROR: {e}")
```
**JSONL (`.jsonl`):**
```python theme={null}
import json
from google.protobuf.json_format import ParseDict
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
with open("my_trace.jsonl") as f:
for i, line in enumerate(f, 1):
line = line.strip()
if not line:
continue
try:
data = json.loads(line)
req = ParseDict(data, ExportTraceServiceRequest())
spans = sum(len(ss.spans) for rs in req.resource_spans for ss in rs.scope_spans)
print(f"Line {i} OK: {spans} spans")
except Exception as e:
print(f"Line {i} ERROR: {e}")
```
**Protobuf (`.binpb`):**
```python theme={null}
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
with open("my_trace.binpb", "rb") as f:
data = f.read()
try:
req = ExportTraceServiceRequest()
req.ParseFromString(data)
spans = sum(len(ss.spans) for rs in req.resource_spans for ss in rs.scope_spans)
print(f"OK: {spans} spans across {len(req.resource_spans)} resourceSpans")
except Exception as e:
print(f"ERROR: {e}")
```
***
## File Naming and Organization
The S3 connector processes every file under the configured `prefix` that matches the configured `file_extensions`. Supported extensions include `.json`, `.jsonl`, `.binpb`, and their compressed variants (`.gz`, `.zst`). Once a file is processed (whether `completed` or `failed`), it is not reprocessed unless you call the retry API.
**Recommended S3 key structure:**
```
traces/
2026/04/10/
agent-run-abc123.json
agent-run-def456.json
2026/04/11/
agent-run-ghi789.json
```
This date-partitioned layout makes it easy to manage retention policies and audit ingestion history.
***
## Related Documentation
* [Exporting OTel Traces to Fiddler](/integrations/agentic-ai/otel-trace-export) — client-side protobuf export for custom pipelines
* [Fiddler LangGraph SDK](/integrations/agentic-ai/langgraph-sdk) — direct SDK integration for LangGraph applications
* [OpenTelemetry Integration](/integrations/agentic-ai/opentelemetry-integration) — live OTLP export for custom agent frameworks
* [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk) — decorator-based tracing for custom Python agents
# Fiddler Strands SDK
Source: https://docs.fiddler.ai/integrations/agentic-ai/strands-sdk
Native monitoring for Strands Agents with Strands Agents SDK
[](https://pypi.org/project/fiddler-strands/)
Monitor Strands Agent applications with Fiddler's purpose-built SDK. The Strands Agents SDK provides deep visibility into agent reasoning, tool execution, and multi-agent coordination for Strands-based agent applications.
**Platform Compatibility:** Works with Strands agents deployed on any platform, including AWS Bedrock, custom infrastructure, or other cloud providers.
## What You'll Need
* Fiddler account (cloud or on-premises)
* Strands agent application
* Python 3.10 or higher
* Fiddler API key
## Quick Start
```bash theme={null}
# Step 1: Install (uv recommended)
uv add fiddler-strands
# or: pip install fiddler-strands
```
```python theme={null}
# Step 2: Set up telemetry and instrumentation
import os
from strands.telemetry import StrandsTelemetry
from fiddler_strandsagents import StrandsAgentInstrumentor
strands_telemetry = StrandsTelemetry()
strands_telemetry.setup_otlp_exporter() # Sends to Fiddler
StrandsAgentInstrumentor(strands_telemetry).instrument()
# Step 3: Create your Strands agent as usual
from strands import Agent
from strands.models.openai import OpenAIModel
model = OpenAIModel(api_key=os.getenv("OPENAI_API_KEY"))
agent = Agent(model=model, system_prompt="You are a helpful assistant")
# Step 4: Agent calls are automatically traced
response = agent("Hello, how are you?")
```
**Prerequisites:** Configure OpenTelemetry environment variables for Fiddler integration:
```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-fiddler-instance.com"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ,fiddler-application-id="
```
## What Gets Monitored
### Strands Agent Operations
* **Agent Invocations** - Full request/response capture with timing
* **Tool Execution** - Tool and API call tracking
* **Knowledge Base Queries** - RAG retrieval and context usage
* **Prompt Orchestration** - Prompt templates and LLM interactions
* **Session Management** - Multi-turn conversation tracking
### Strands-Specific Metrics
* **Reasoning Traces** - Agent thought process and decision-making
* **Tool Execution** - Success rates, latency, error patterns
* **Knowledge Retrieval** - Relevance scores, source attribution
* **Multi-Agent Coordination** - Cross-agent communication patterns
* **Infrastructure Metrics** - Platform-specific infrastructure calls
## Configuration Options
### Environment Variables (OpenTelemetry Standard)
The SDK uses standard OpenTelemetry environment variables for configuration:
```bash theme={null}
# Required - Fiddler OTLP endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-fiddler-instance.com"
# Required - Fiddler authentication and application ID
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ,fiddler-application-id="
# Optional - Application metadata
export OTEL_RESOURCE_ATTRIBUTES="application.id=,service.name=my-agent,deployment.environment=production"
# Required - OpenAI API key (for running agents)
export OPENAI_API_KEY="sk-..."
```
See the [Quick Start Guide](/developers/quick-starts/strands-agent-quick-start) for detailed configuration steps.
### Programmatic Configuration
```python theme={null}
from strands.telemetry import StrandsTelemetry
from fiddler_strandsagents import StrandsAgentInstrumentor
# Basic setup
strands_telemetry = StrandsTelemetry()
strands_telemetry.setup_console_exporter() # Optional: for debugging
strands_telemetry.setup_otlp_exporter() # Sends to Fiddler
StrandsAgentInstrumentor(strands_telemetry).instrument()
# All agents created after this point are automatically instrumented
```
## Example Applications
### Customer Service Agent with Tools
```python theme={null}
import os
from strands import Agent, tool
from strands.models.openai import OpenAIModel
from strands.telemetry import StrandsTelemetry
from fiddler_strandsagents import StrandsAgentInstrumentor, set_conversation_id
# Set up instrumentation
strands_telemetry = StrandsTelemetry()
strands_telemetry.setup_otlp_exporter()
StrandsAgentInstrumentor(strands_telemetry).instrument()
# Define tools
@tool
def lookup_order(order_id: str) -> str:
"""Look up customer order details."""
return f"Order {order_id}: Shipped, arriving Tuesday"
@tool
def check_inventory(product: str) -> str:
"""Check product inventory."""
return f"{product}: 42 units in stock"
# Create agent with tools
model = OpenAIModel(api_key=os.getenv("OPENAI_API_KEY"))
agent = Agent(
model=model,
system_prompt="You are a helpful customer service agent.",
tools=[lookup_order, check_inventory]
)
# Track conversation
set_conversation_id(agent, "customer-12345")
# Use agent - all interactions automatically traced to Fiddler:
# - Agent reasoning steps
# - Tool calls (lookup_order, check_inventory)
# - Response generation
response = agent("What's the status of order #789?")
```
### Multi-Agent System
```python theme={null}
from strands import Agent
from strands.models.openai import OpenAIModel
# All agents are automatically instrumented after setup
model = OpenAIModel(api_key=os.getenv("OPENAI_API_KEY"))
# Create specialized agents with unique IDs
verification_agent = Agent(
model=model,
system_prompt="You verify user identity",
agent_id="verification-agent"
)
account_agent = Agent(
model=model,
system_prompt="You create user accounts",
agent_id="account-agent"
)
# Each agent appears separately in Fiddler with full trace visibility
```
[**View complete example →**](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Strands_Agent_Integration.ipynb)
## Viewing Your Data
Navigate to Fiddler UI to analyze Strands Agent performance:
1. **Agent Overview** - Overall agent performance metrics
2. **Session Analysis** - Multi-turn conversation flows
3. **Action Group Metrics** - Tool usage patterns and success rates
4. **Knowledge Base Performance** - Retrieval quality and relevance
5. **Cost Tracking** - Token usage and AWS costs per agent
### Key Metrics
* **Agent Latency**: P50/P95/P99 response times
* **Tool Success Rate**: Percentage of successful action group executions
* **Retrieval Quality**: Knowledge base query relevance scores
* **Token Usage**: LLM tokens consumed per session
* **Error Rates**: Failed invocations by error type
## Advanced Features
### Custom Metadata with Helper Functions
The SDK provides helper functions to enrich your traces with custom business context:
#### Conversation Tracking
```python theme={null}
from fiddler_strandsagents import set_conversation_id, get_conversation_id
# Set conversation ID for multi-turn tracking
set_conversation_id(agent, 'session_1234567890')
# Retrieve it later
conversation_id = get_conversation_id(agent)
```
#### Session-Level Attributes
```python theme={null}
from fiddler_strandsagents import set_session_attributes, get_session_attributes
# Add business context to all spans in this session
set_session_attributes(agent,
customer_tier="premium",
region="us-west",
campaign="summer-2024",
cost_center="support_team"
)
# Retrieve session attributes
attributes = get_session_attributes(agent)
```
#### Span-Level Attributes
```python theme={null}
from fiddler_strandsagents import set_span_attributes, get_span_attributes
# Add attributes to specific components
set_span_attributes(model,
model_version="gpt-4o-mini",
temperature=0.7,
max_tokens=1000
)
set_span_attributes(search_tool,
department="search",
version="2.0",
environment="production"
)
# Retrieve span attributes
model_attrs = get_span_attributes(model)
```
#### LLM Context
```python theme={null}
from fiddler_strandsagents import set_llm_context, get_llm_context
# Set additional context for LLM interactions
# This context will be added to telemetry spans as 'gen_ai.llm.context'
set_llm_context(model, 'Available hotels: Hilton, Marriott, Hyatt...')
# Retrieve LLM context
context = get_llm_context(model)
```
## Troubleshooting
### Traces Not Appearing in Fiddler
**Verify environment variables:**
```bash theme={null}
echo $OTEL_EXPORTER_OTLP_ENDPOINT
echo $OTEL_EXPORTER_OTLP_HEADERS
```
**Check instrumentation is enabled:**
```python theme={null}
from fiddler_strandsagents import StrandsAgentInstrumentor
instrumentor = StrandsAgentInstrumentor(strands_telemetry)
print(f"Instrumented: {instrumentor.is_instrumented_by_opentelemetry}")
```
**Test with console exporter:**
```python theme={null}
# Add console exporter to see traces locally
strands_telemetry.setup_console_exporter()
```
### Missing Agent Attributes on Child Spans
**Verify SDK instrumentation:**
```python theme={null}
# Ensure StrandsAgentInstrumentor is called
StrandsAgentInstrumentor(strands_telemetry).instrument()
```
**Add custom attributes:**
```python theme={null}
from fiddler_strandsagents import set_span_attributes
set_span_attributes(agent, custom_attr="value")
set_span_attributes(model, environment="production")
```
### Performance Optimization
The SDK uses batch span processing by default for minimal overhead. For additional optimization:
**Disable console exporter in production:**
```python theme={null}
# Only use OTLP exporter in production
strands_telemetry.setup_otlp_exporter()
# Don't call setup_console_exporter()
```
**Adjust batch processor settings:**
```python theme={null}
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# Custom batch settings for high-throughput scenarios
# (See Advanced Configuration in Quick Start Guide)
```
## Related Documentation
* [**Strands Agents SDK Quick Start** ](/developers/quick-starts/strands-agent-quick-start)- Detailed setup guide
* [**Fiddler Evals SDK**](/integrations/agentic-ai/evals-sdk) - Evaluate Strands Agent quality
* [**Strands Agents SDK Reference** ](/sdk-api/strands/strands-agent-instrumentor)- Complete class and method documentation
# Admin Guide
Source: https://docs.fiddler.ai/integrations/aws-sagemaker/partner-ai-app-admin-guide
Admin guide to deploy the Fiddler Partner AI App on AWS SageMaker. Configure IAM roles, permissions, and subscriptions for secure AI systems observability.
### Introduction
This Admin Guide supplements the official [Amazon SageMaker Partner AI Apps documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps.html) with Fiddler-specific guidance and practical examples. While Amazon's documentation provides the foundation for Partner AI Apps, specific implementation details can benefit from additional vendor-specific context. Based on our direct experience deploying the Fiddler Partner AI App in SageMaker environments, this guide offers proven configuration patterns, troubleshooting tips, and concrete examples to help you integrate Fiddler's ML monitoring capabilities into your SageMaker infrastructure.
### SageMaker Partner AI Apps Subscription Prerequisites and Permissions
Before proceeding with procurement, the AWS Admin must have some things in place to enable subscription to and deployment of Amazon SageMaker Partner AI Apps as noted in the [Set up Partner AI Apps](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-app-onboard.html) section.
#### Prerequisites
While Amazon lists both items as optional, having both will help facilitate the deployment and validation processes in the SageMaker console.
* A SageMaker AI domain
* AWS CLI (latest version)
#### Administrative Permissions
Configuring the appropriate permissions is an important step. Every organization will have its own security policies and configuration, so the setup will not be identical across organizations. Some of these grants may already be available to admins in a more permissive environment.
The key steps are:
* Grant permissions to your admin role to complete AWS Marketplace subscriptions
* Grant permissions for SageMaker AI to run other AWS operations on the admin's behalf
* Set up an execution role for the Partner AI App (note this role will be needed when deploying the Fiddler Partner AI App)
* Create an AWS License Manager service-linked role or confirm one exists
* Grant permissions for the Fiddler Partner AI App to access AWS License Manager
* Grant Amazon S3 permissions to the execution role
Attach the `AWSMarketplaceManageSubscriptions` managed policy to the IAM role you'll use to access the SageMaker AI console and subscribe to Fiddler:
```bash theme={null}
aws iam attach-role-policy
--role-name YourAdminRole
--policy-arn arn:aws:iam::aws:policy:AWSMarketplaceManageSubscriptions
```
Or add the policy through the AWS IAM console.
Create and attach the following policy to your administrator role:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sagemaker:CreatePartnerApp",
"sagemaker:DeletePartnerApp",
"sagemaker:UpdatePartnerApp",
"sagemaker:DescribePartnerApp",
"sagemaker:ListPartnerApps",
"sagemaker:CreatePartnerAppPresignedUrl",
"sagemaker:AddTags",
"sagemaker:ListTags",
"sagemaker:DeleteTags"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "arn:aws:iam::*:role/*",
"Condition": {
"StringEquals": {
"iam:PassedToService": "sagemaker.amazonaws.com"
}
}
}
]
}
```
This role allows Fiddler to access the necessary AWS resources:
```bash theme={null}
aws iam create-role
--role-name PartnerAiAppExecutionRole
--assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"sagemaker.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}'
```
```sh theme={null}
# License Manager service-linked roles are typically created automatically
# when you first use License Manager features. You can verify if it exists:
aws iam get-role --role-name AWSServiceRoleForAWSLicenseManagerRole 2>/dev/null ||
echo "License Manager service-linked role will be created automatically when first used"
# If you need to create it manually (rarely required), use:
# aws iam create-service-linked-role --aws-service-name license-manager.amazonaws
```
Comments
```bash theme={null}
aws iam put-role-policy
--role-name PartnerAiAppExecutionRole
--policy-name LicenseManagerPolicy
--policy-document '{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": [
"license-manager:CheckoutLicense",
"license-manager:CheckInLicense",
"license-manager:ExtendLicenseConsumption",
"license-manager:GetLicense",
"license-manager:GetLicenseUsage"
],
"Resource": "*"
}
}'
```
To allow access to the required S3 bucket, add the appropriate S3 permissions to the execution role.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::SageMaker"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::SageMaker/*"
]
}
]
}
```
#### User Permissions
Non-admin users require the following permissions to use the Fiddler Partner AI App:
* Grant permissions for SageMaker AI to run other AWS operations on the user's behalf
* Add the *sts:TagSession* trust policy to the role used to launch Studio and/or Fiddler
* Grant API permission to the IAM role used with the notebook or code editor to enable access to the Fiddler Python client SDK
Create and attach this policy to the execution role of the user profile:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sagemaker:DescribePartnerApp",
"sagemaker:ListPartnerApps",
"sagemaker:CreatePartnerAppPresignedUrl"
],
"Resource": "arn:aws:sagemaker:*:*:partner-app/app-*"
}
]
}
```
Add the `sts:TagSession` trust policy to the role used to launch Studio or the Fiddler Partner AI App directly:
```json theme={null}
{
"Effect": "Allow",
"Principal": {
"Service": "sagemaker.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
```
Enable user access to Fiddler's SDK functionality by adding this permission to the Studio execution role or the IAM role used with notebooks outside Studio:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Action": [
"sagemaker:CallPartnerAppApi"
],
"Resource": [
"arn:aws:sagemaker:region:account:partner-app/app-*"
]
}
]
}
```
### Authorization and Authentication
Fiddler and Partner AI Apps require user identity propagation to function correctly. For detailed configuration requirements, refer to the [Manage user authorization and authentication](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-app-onboard.html#partner-app-onboard-auth) section in the AWS documentation.
**Critical Configuration Points**:
* The username format in your identity source must **exactly match** the admin usernames you'll enter during Fiddler deployment
* For IAM Identity Center: usernames are typically email addresses
* For IAM: usernames match the IAM user or role name
* Enable STS identity propagation during Fiddler configuration (this is mandatory)
### Fiddler Partner AI App SageMaker Subscription Process
Amazon SageMaker Partner AI Apps are only available through the Amazon Marketplace. Currently, Fiddler is available on a subscription basis or as a 30-day free trial.
Partner AI App subscriptions are for software license costs only. The infrastructure necessary to run Partner AI Apps is an additional cost billed to the subscriber under their existing agreement with Amazon.
#### Fiddler 30-day Free Trial
The 30-day free trial is a fully functional Fiddler Partner AI App deployment with no functional limitations. The number of models allowed for monitoring is capped at five. As the Partner AI App free trials are not intended for production use cases, AWS provides no migration path for upgrading trial deployments to fully subscribed deployments.
The 30-day free trial does not include infrastructure costs, which are billed like a full subscription directly to your AWS account per your existing agreement with Amazon.
You MUST delete your Fiddler Partner AI App deployment at the end of the 30 days or whenever you wish to stop incurring AWS infrastructure costs.
### Deploying the Fiddler Partner AI App
#### Admin Management
For the Fiddler Partner AI App, you must enter at least one and up to five root admin users. The usernames entered here must match the identity principal forwarded to the Fiddler UI when launched.
* The root users will not be editable once the Fiddler AI App has been deployed
* This list of root users will persist in SageMaker AI even if the Fiddler app is deleted and then later redeployed
* Admin users created within the Fiddler UI will not persist after the Fiddler Partner AI App deployment is deleted
#### Infrastructure Options
Amazon SageMaker offers Partner AI Apps in small, medium, and large tier sizes. The subscription tier selection screen will have the most current t-shirt-size use cases and estimated AWS costs to help guide your selection.
### Monitor and Manage Your Fiddler Partner AI App Deployment
#### **View Deployment Status**
Navigate to the "My Apps" tab in the Partner AI Apps section to view the status of your Fiddler deployment:
* **Deployed**: Application is ready for use
* **Error**: Issue with deployment; troubleshoot and reconfigure
* **Not deployed**: Subscribed but not yet deployed
#### **Access Fiddler**
Users can access Fiddler in two ways:
* From SageMaker Studio: Navigate to Partner AI Apps and select Fiddler
* Via pre-signed URL: Generate using the `CreatePartnerAppPresignedUrl` with the [Amazon SageMaker API](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreatePartnerAppPresignedUrl.html) or the [AWS CLI](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sagemaker/create-partner-app-presigned-url.html)
#### **Update Configuration**
To update configuration settings:
* Navigate to "My Apps" in Partner AI Apps
* Select the Fiddler application
* Choose "Update" to modify configuration parameters
#### **Delete Deployment**
To delete the Fiddler deployment:
* Navigate to "My Apps" in Partner AI Apps
* Select the Fiddler application
* Choose "Delete"
* Note: This does not cancel your subscription, but it does delete the associated AWS infrastructure
### Troubleshooting
#### **Deployment Failure**
If deployment fails:
* Check IAM roles and permissions
* Verify that the execution role has all required permissions
* Check CloudWatch logs for detailed error information
* Open a support ticket with your AWS support team
#### **Access Issues**
If users cannot access Fiddler:
* Verify that the user IAM permissions include the necessary Partner AI App access
* Confirm identity propagation is correctly configured
* Check that usernames match between your identity source and Fiddler
#### **Python Client Connection Issues**
If users cannot connect using the Fiddler Python client:
* Verify the `sagemaker:CallPartnerAppApi` permission is configured
* Confirm environment variables are correctly set:
```python theme={null}
os.environ['AWS_PARTNER_APP_AUTH'] = 'true'
os.environ['AWS_PARTNER_APP_ARN'] = ''
os.environ['AWS_PARTNER_APP_URL'] = ''
```
The same three environment variables also authenticate trace export from the [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk#running-in-aws-sagemaker) (`fiddler-otel` 1.3.0+) when users instrument GenAI or agentic applications. For OTel tracing, users additionally install the `sagemaker` extra with `pip install "fiddler-otel[sagemaker]"`.
### Additional Resources
* [Amazon SageMaker AI Domain Overview](https://docs.aws.amazon.com/sagemaker/latest/dg/gs-studio-onboard.html)
* [Partner AI Apps Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps.html)
* [AWS Identity and Access Management (IAM) Documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html)
# Quick Setup Script
Source: https://docs.fiddler.ai/integrations/aws-sagemaker/partner-ai-app-quick-setup-script
Quickly set up Fiddler's Partner AI App on SageMaker with our script. Automates IAM roles, permissions, and configuration for fast deployment.
### Introduction
This quick setup script automates the complex IAM configuration required to deploy and use the Fiddler Partner AI App on Amazon SageMaker. Setting up Partner AI Apps involves creating multiple IAM roles, configuring over 20 individual permissions across different AWS services, and establishing proper identity propagation—a process that can take hours to complete manually. This script reduces the setup time to minutes, ensuring all general security best practices are followed.
#### What This Script Does
The script automates the complete setup process, including:
* Configuring your existing AWS admin role with Partner AI App permissions
* Creating all required IAM roles (execution, user access, and domain roles)
* Setting up a SageMaker domain with shared collaboration spaces (if needed)
* Generating helper scripts for user access and SDK configuration
* Validating each step to ensure successful completion
#### Prerequisites
Before running this script, ensure you have:
* AWS CLI installed and configured with appropriate credentials
* An AWS account with permissions to create IAM roles and SageMaker domains
* Your AWS account ID and preferred region
* An existing VPC and subnet IDs if using VPC mode
#### Important Security Considerations
* The script creates roles with specific permissions following AWS's principle of least privilege
* All credentials are handled securely and cleaned up after use
* External IDs are used for role assumption to prevent confused deputy attacks
* Review all created IAM policies to ensure they meet your organization's security requirements
#### Time Estimate
* **With an existing SageMaker domain**: \~5 minutes
* **Creating new SageMaker domain**: \~15-20 minutes (includes domain provisioning time)
#### Post-Script Steps
After the script completes successfully, you'll still need to:
1. Subscribe to Fiddler in the AWS Marketplace
2. Deploy the Fiddler Partner AI App through the SageMaker console
3. Configure admin users during the deployment process
The script will provide detailed instructions for these remaining manual steps upon completion.
```bash theme={null}
#!/bin/bash
# =============================================================================
# COMPLETE FIDDLER PARTNER AI APP & SAGEMAKER SETUP FROM SCRATCH
# =============================================================================
# This script follows AWS documentation to:
# 1. Set up admin permissions for Fiddler Partner AI App
# 2. Create Partner AI App execution role
# 3. Create SageMaker Domain with shared space support
# 4. Configure shared space for collaboration
# 5. Create shared user profile for all AWS users
# 6. Set up non-admin user access to Fiddler Partner AI App and SDK
#
# References:
# - https://docs.aws.amazon.com/sagemaker/latest/dg/partner-app-onboard.html
# - https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps-sdk.html
# =============================================================================
# CONFIGURATION - UPDATE THESE VALUES BEFORE RUNNING
export AWS_ACCOUNT_ID=""
export AWS_REGION=""
# SageMaker Domain Configuration: Choose your own naming convention
export DOMAIN_NAME="fiddler-paa-domain"
export SHARED_SPACE_NAME="fiddler-collaboration-space"
export SHARED_USER_PROFILE_NAME="fiddler-shared-profile"
export VPC_ID="vpc-xxxxxxxxxxxxxxxxxx" # UPDATE: Your VPC ID and Subnet IDs
export SUBNET_IDS=(
"subnet-xxxxxxxxxxxxxxxxx"
"subnet-xxxxxxxxxxxxxxxxx"
"subnet-xxxxxxxxxxxxxxxxx"
)
export EXISTING_ADMIN_ROLE_NAME="" # Existing AWS admin role
export PARTNER_APP_EXECUTION_ROLE_NAME="FiddlerPartnerAppExecutionRole" # Choose your own naming convention
export USER_EXECUTION_ROLE_NAME="FiddlerUserExecutionRole" # Choose your own naming convention
export USER_ACCESS_ROLE_NAME="FiddlerUserAccessRole" # Choose your own naming convention
echo "=== Fiddler Partner AI App & SageMaker Complete Setup ==="
echo ""
echo "This script will create:"
echo " • Partner AI Apps permissions for existing admin role: $EXISTING_ADMIN_ROLE_NAME"
echo " • Fiddler Partner AI App execution role"
echo " • SageMaker Domain (optionally): $DOMAIN_NAME"
echo " • Shared Space: $SHARED_SPACE_NAME"
echo " • Shared User Profile: $SHARED_USER_PROFILE_NAME"
echo " • User access roles and permissions"
echo ""
read -p "Continue with setup? (y/n): " CONFIRM
if [[ "$CONFIRM" != "y" && "$CONFIRM" != "Y" ]]; then
echo "Setup cancelled."
exit 0
fi
# =============================================================================
# STEP 1: ADD PARTNER AI APPS PERMISSIONS TO EXISTING ADMIN ROLE
# =============================================================================
echo ""
echo "=== STEP 1: Adding Partner AI Apps Permissions to Existing Admin Role ==="
# Verify existing admin role exists
echo "Checking existing admin role: $EXISTING_ADMIN_ROLE_NAME"
aws iam get-role --role-name "$EXISTING_ADMIN_ROLE_NAME" --query 'Role.RoleName' --output text >/dev/null 2>&1
if [[ $? -ne 0 ]]; then
echo "❌ Admin role not found: $EXISTING_ADMIN_ROLE_NAME"
echo ""
echo "Please update EXISTING_ADMIN_ROLE_NAME variable with your actual admin role name."
echo "To list your roles, run:"
echo "aws iam list-roles --query 'Roles[*].RoleName' --output table"
exit 1
fi
echo "✅ Found existing admin role: $EXISTING_ADMIN_ROLE_NAME"
# Check if AWS Marketplace permissions are already attached
echo "Checking AWS Marketplace permissions..."
MARKETPLACE_ATTACHED=$(aws iam list-attached-role-policies --role-name "$EXISTING_ADMIN_ROLE_NAME" --query 'AttachedPolicies[?PolicyArn==`arn:aws:iam::aws:policy/AWSMarketplaceManageSubscriptions`].PolicyName' --output text 2>/dev/null)
if [[ -n "$MARKETPLACE_ATTACHED" ]]; then
echo "✅ AWS Marketplace permissions already attached"
else
echo "Adding AWS Marketplace permissions..."
aws iam attach-role-policy
--role-name "$EXISTING_ADMIN_ROLE_NAME"
--policy-arn "arn:aws:iam::aws:policy/AWSMarketplaceManageSubscriptions"
echo "✅ AWS Marketplace permissions attached"
fi
Create Partner AI Apps admin permissions policy
echo "Adding Partner AI Apps admin permissions..."
cat > partner-ai-admin-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sagemaker:CreatePartnerApp",
"sagemaker:DeletePartnerApp",
"sagemaker:UpdatePartnerApp",
"sagemaker:DescribePartnerApp",
"sagemaker:ListPartnerApps",
"sagemaker:CreatePartnerAppPresignedUrl",
"sagemaker:AddTags",
"sagemaker:ListTags",
"sagemaker:DeleteTags"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "arn:aws:iam::*:role/*",
"Condition": {
"StringEquals": {
"iam:PassedToService": "sagemaker.amazonaws.com"
}
}
}
]
}
EOF
# Check if Partner AI Apps policy already exists
EXISTING_POLICY=$(aws iam get-role-policy --role-name "$EXISTING_ADMIN_ROLE_NAME" --policy-name "PartnerAIAppsAdminPolicy" --query 'PolicyName' --output text 2>/dev/null)
if [[ "$EXISTING_POLICY" == "PartnerAIAppsAdminPolicy" ]]; then
echo "Partner AI Apps policy already exists, updating..."
aws iam put-role-policy
--role-name "$EXISTING_ADMIN_ROLE_NAME"
--policy-name "PartnerAIAppsAdminPolicy"
--policy-document file://partner-ai-admin-policy.json
echo "✅ Partner AI Apps policy updated"
else
echo "Adding new Partner AI Apps policy..."
aws iam put-role-policy
--role-name "$EXISTING_ADMIN_ROLE_NAME"
--policy-name "PartnerAIAppsAdminPolicy"
--policy-document file://partner-ai-admin-policy.json
echo "✅ Partner AI Apps policy added"
fi
echo "✅ Existing admin role configured with Partner AI Apps permissions"
# =============================================================================
# STEP 2: CREATE FIDDLER PARTNER AI APP EXECUTION ROLE
# =============================================================================
echo ""
echo "=== STEP 2: Creating Fiddler Partner AI App Execution Role ==="
# Create Partner AI App execution role trust policy
cat > partner-app-execution-trust-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sagemaker.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
]
}
EOF
# Create Partner AI App execution role
echo "Creating Partner AI App execution role..."
aws iam create-role
--role-name "$PARTNER_APP_EXECUTION_ROLE_NAME"
--assume-role-policy-document file://partner-app-execution-trust-policy.json
--description "Execution role for Fiddler Partner AI App"
# Create AWS License Manager permissions policy (required for Fiddler)
echo "Adding AWS License Manager permissions..."
cat > license-manager-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": [
"license-manager:CheckoutLicense",
"license-manager:CheckInLicense",
"license-manager:ExtendLicenseConsumption",
"license-manager:GetLicense",
"license-manager:GetLicenseUsage"
],
"Resource": "*"
}
}
EOF
aws iam put-role-policy
--role-name "$PARTNER_APP_EXECUTION_ROLE_NAME"
--policy-name "LicenseManagerPolicy"
--policy-document file://license-manager-policy.json
# Add S3 permissions for Fiddler data access
echo "Adding S3 permissions for Fiddler..."
cat > fiddler-s3-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "FiddlerS3Access",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::sagemaker-*",
"arn:aws:s3:::sagemaker-*/*"
]
},
{
"Sid": "FiddlerCloudWatchLogs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
],
"Resource": "arn:aws:logs:*:*:log-group:/aws/sagemaker/*"
}
]
}
EOF
aws iam put-role-policy
--role-name "$PARTNER_APP_EXECUTION_ROLE_NAME"
--policy-name "FiddlerS3Policy"
--policy-document file://fiddler-s3-policy.json
# Attach SageMaker execution policy
echo "Attaching SageMaker execution permissions..."
aws iam attach-role-policy
--role-name "$PARTNER_APP_EXECUTION_ROLE_NAME"
--policy-arn "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess"
echo "✅ Fiddler Partner AI App execution role created with all required permissions"
# =============================================================================
# STEP 3: CREATE USER EXECUTION ROLE
# =============================================================================
echo ""
echo "=== STEP 3: Creating User Execution Role ==="
# Create user execution role trust policy
cat > user-execution-trust-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sagemaker.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
]
}
EOF
# Create user execution role
echo "Creating user execution role..."
aws iam create-role
--role-name "$USER_EXECUTION_ROLE_NAME"
--assume-role-policy-document file://user-execution-trust-policy.json
--description "Execution role for SageMaker domain users"
# Create user permissions policy
cat > user-execution-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PartnerAIAppsUserAccess",
"Effect": "Allow",
"Action": [
"sagemaker:DescribePartnerApp",
"sagemaker:ListPartnerApps",
"sagemaker:CreatePartnerAppPresignedUrl",
"sagemaker:CallPartnerAppApi"
],
"Resource": "*"
},
{
"Sid": "SageMakerStudioAccess",
"Effect": "Allow",
"Action": [
"sagemaker:CreateApp",
"sagemaker:DeleteApp",
"sagemaker:DescribeApp",
"sagemaker:ListApps",
"sagemaker:CreateSpace",
"sagemaker:DeleteSpace",
"sagemaker:DescribeSpace",
"sagemaker:ListSpaces",
"sagemaker:UpdateSpace",
"sagemaker:DescribeDomain",
"sagemaker:DescribeUserProfile",
"sagemaker:CreatePresignedDomainUrl"
],
"Resource": "*"
},
{
"Sid": "S3AccessForML",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::sagemaker-*",
"arn:aws:s3:::sagemaker-*/*"
]
}
]
}
EOF
aws iam put-role-policy
--role-name "$USER_EXECUTION_ROLE_NAME"
--policy-name "UserExecutionPolicy"
--policy-document file://user-execution-policy.json
# Attach basic SageMaker policy
aws iam attach-role-policy
--role-name "$USER_EXECUTION_ROLE_NAME"
--policy-arn "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess"
echo "✅ User execution role created"
# =============================================================================
# STEP 4: CREATE USER ACCESS ROLE
# =============================================================================
echo ""
echo "=== STEP 4: Creating User Access Role ==="
# Create user access role trust policy
cat > user-access-trust-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${AWS_ACCOUNT_ID}:root"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
],
"Condition": {
"StringEquals": {
"sts:ExternalId": "fiddler-user-access"
}
}
}
]
}
EOF
# Create user access role
echo "Creating user access role..."
aws iam create-role
--role-name "$USER_ACCESS_ROLE_NAME"
--assume-role-policy-document file://user-access-trust-policy.json
--description "Access role for users to assume for SageMaker and Fiddler access"
# Create user access permissions policy
cat > user-access-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sagemaker:CreatePresignedDomainUrl",
"sagemaker:DescribeDomain",
"sagemaker:DescribeUserProfile",
"sagemaker:DescribeSpace",
"sagemaker:ListSpaces",
"sagemaker:CreatePartnerAppPresignedUrl",
"sagemaker:DescribePartnerApp",
"sagemaker:ListPartnerApps"
],
"Resource": "*"
}
]
}
EOF
aws iam put-role-policy
--role-name "$USER_ACCESS_ROLE_NAME"
--policy-name "UserAccessPolicy"
--policy-document file://user-access-policy.json
echo "✅ User access role created"
# =============================================================================
# STEP 5: CREATE SAGEMAKER DOMAIN (Optional)
# =============================================================================
echo ""
echo "=== STEP 5: Creating SageMaker Domain ==="
# Determine network configuration
NETWORK_CONFIG="PublicInternetOnly" # or "VpcOnly"
VPC_CONFIG="--vpc-id $VPC_ID --subnet-ids ${SUBNET_IDS[@]}"
echo "Using network configuration: $NETWORK_CONFIG with VPC $VPC_ID"
# Create the domain
echo "Creating SageMaker domain..."
DOMAIN_ARN=$(aws sagemaker create-domain
--domain-name "$DOMAIN_NAME"
--auth-mode "IAM"
--default-user-settings '{
"ExecutionRole": "arn:aws:iam::'$AWS_ACCOUNT_ID':role/'$USER_EXECUTION_ROLE_NAME'",
"SharingSettings": {
"NotebookOutputOption": "Allowed"
},
"JupyterServerAppSettings": {
"DefaultResourceSpec": {
"InstanceType": "system"
}
},
"JupyterLabAppSettings": {
"DefaultResourceSpec": {
"InstanceType": "ml.t3.medium"
}
},
"DefaultLandingUri": "studio::",
"StudioWebPortal": "ENABLED"
}'
--default-space-settings '{
"ExecutionRole": "arn:aws:iam::'$AWS_ACCOUNT_ID':role/'$USER_EXECUTION_ROLE_NAME'",
"JupyterServerAppSettings": {
"DefaultResourceSpec": {
"InstanceType": "system"
}
},
"JupyterLabAppSettings": {
"DefaultResourceSpec": {
"InstanceType": "ml.t3.medium"
}
},
"SpaceStorageSettings": {
"DefaultEbsStorageSettings": {
"DefaultEbsVolumeSizeInGb": 20,
"MaximumEbsVolumeSizeInGb": 100
}
}
}'
--domain-settings '{
"ExecutionRoleIdentityConfig": "USER_PROFILE_NAME"
}'
--app-network-access-type "$NETWORK_CONFIG"
$VPC_CONFIG
--region "$AWS_REGION"
--query 'DomainArn'
--output text)
# Extract the Domain ID from the full ARN. This is more portable than sed.
DOMAIN_ID=${DOMAIN_ARN##*/}
if [[ -z "$DOMAIN_ID" ]]; then
echo "❌ Failed to create domain"
exit 1
fi
echo "Domain created with ID: $DOMAIN_ID"
# Wait for domain to be in service
echo "Waiting for domain to be ready..."
while true; do
DOMAIN_STATUS=$(aws sagemaker describe-domain
--domain-id "$DOMAIN_ID"
--region "$AWS_REGION"
--query 'Status' --output text)
if [[ "$DOMAIN_STATUS" == "InService" ]]; then
echo "✅ Domain is ready"
break
elif [[ "$DOMAIN_STATUS" == "Failed" ]]; then
echo "❌ Domain creation failed"
exit 1
else
echo "Domain status: $DOMAIN_STATUS, waiting..."
sleep 30
fi
done
# Check if domain has DefaultSpaceSettings
echo "Verifying domain has shared space settings..."
DOMAIN_SPACE_SETTINGS=$(aws sagemaker describe-domain
--domain-id "$DOMAIN_ID"
--region "$AWS_REGION"
--query 'DefaultSpaceSettings' --output text 2>/dev/null)
if [[ "$DOMAIN_SPACE_SETTINGS" == "None" || -z "$DOMAIN_SPACE_SETTINGS" ]]; then
echo "Domain missing shared space settings, updating domain..."
aws sagemaker update-domain
--domain-id "$DOMAIN_ID"
--default-space-settings '{
"ExecutionRole": "arn:aws:iam::'$AWS_ACCOUNT_ID':role/'$USER_EXECUTION_ROLE_NAME'",
"JupyterServerAppSettings": {
"DefaultResourceSpec": {
"InstanceType": "system"
}
},
"JupyterLabAppSettings": {
"DefaultResourceSpec": {
"InstanceType": "ml.t3.medium"
}
},
"SpaceStorageSettings": {
"DefaultEbsStorageSettings": {
"DefaultEbsVolumeSizeInGb": 20,
"MaximumEbsVolumeSizeInGb": 100
}
}
}'
--region "$AWS_REGION"
echo "Waiting for domain update to complete..."
while true; do
DOMAIN_STATUS=$(aws sagemaker describe-domain
--domain-id "$DOMAIN_ID"
--region "$AWS_REGION"
--query 'Status' --output text)
if [[ "$DOMAIN_STATUS" == "InService" ]]; then
echo "✅ Domain updated with shared space settings"
break
elif [[ "$DOMAIN_STATUS" == "Failed" ]]; then
echo "❌ Domain update failed"
exit 1
else
echo "Domain update status: $DOMAIN_STATUS, waiting..."
sleep 15
fi
done
else
echo "✅ Domain already has shared space settings configured"
fi
# =============================================================================
# STEP 6: CREATE SHARED USER PROFILE
# =============================================================================
echo ""
echo "=== STEP 6: Creating Shared User Profile ==="
echo "Creating shared user profile..."
aws sagemaker create-user-profile
--domain-id "$DOMAIN_ID"
--user-profile-name "$SHARED_USER_PROFILE_NAME"
--user-settings '{
"ExecutionRole": "arn:aws:iam::'$AWS_ACCOUNT_ID':role/'$USER_EXECUTION_ROLE_NAME'",
"SharingSettings": {
"NotebookOutputOption": "Allowed"
},
"JupyterServerAppSettings": {
"DefaultResourceSpec": {
"InstanceType": "system"
}
},
"JupyterLabAppSettings": {
"DefaultResourceSpec": {
"InstanceType": "ml.t3.medium"
}
}
}'
--region "$AWS_REGION"
# Wait for user profile to be ready
echo "Waiting for user profile to be ready..."
while true; do
PROFILE_STATUS=$(aws sagemaker describe-user-profile
--domain-id "$DOMAIN_ID"
--user-profile-name "$SHARED_USER_PROFILE_NAME"
--region "$AWS_REGION"
--query 'Status' --output text)
if [[ "$PROFILE_STATUS" == "InService" ]]; then
echo "✅ User profile is ready"
break
elif [[ "$PROFILE_STATUS" == "Failed" ]]; then
echo "❌ User profile creation failed"
exit 1
else
echo "Profile status: $PROFILE_STATUS, waiting..."
sleep 15
fi
done
# =============================================================================
# STEP 7: CREATE SHARED SPACE
# =============================================================================
echo ""
echo "=== STEP 7: Creating Shared Space ==="
echo "Creating shared space for collaboration..."
aws sagemaker create-space
--domain-id "$DOMAIN_ID"
--space-name "$SHARED_SPACE_NAME"
--region "$AWS_REGION"
# Wait for shared space to be ready
echo "Waiting for shared space to be ready..."
while true; do
SPACE_STATUS=$(aws sagemaker describe-space
--domain-id "$DOMAIN_ID"
--space-name "$SHARED_SPACE_NAME"
--region "$AWS_REGION"
--query 'Status' --output text)
if [[ "$SPACE_STATUS" == "InService" ]]; then
echo "✅ Shared space is ready"
break
elif [[ "$SPACE_STATUS" == "Failed" ]]; then
echo "❌ Shared space creation failed"
exit 1
else
echo "Space status: $SPACE_STATUS, waiting..."
sleep 15
fi
done
# =============================================================================
# STEP 8: CREATE USER ACCESS SCRIPTS
# =============================================================================
echo ""
echo "=== STEP 8: Creating User Access Scripts ==="
# Create user access script
cat > fiddler-user-access.sh << 'EOF'
#!/bin/bash
# =============================================================================
# FIDDLER USER ACCESS SCRIPT
# =============================================================================
# Configuration - UPDATE THESE VALUES
export AWS_ACCOUNT_ID=""
export AWS_REGION=""
export DOMAIN_ID="PLACEHOLDER_DOMAIN_ID"
export USER_PROFILE_NAME="fiddler-shared-profile"
export USER_ACCESS_ROLE_NAME="FiddlerUserAccessRole"
# Get username for identity propagation
CALLER_ARN=$(aws sts get-caller-identity --query 'Arn' --output text)
USERNAME=${CALLER_ARN##*/}
echo "=== Fiddler Partner AI App & SageMaker Access ==="
echo "Setting up access for user: $USERNAME"
# Assume role with session tags for identity propagation
echo "1. Assuming user access role with identity tags..."
TEMP_CREDENTIALS=$(aws sts assume-role
--role-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:role/${USER_ACCESS_ROLE_NAME}"
--role-session-name "${USERNAME}-fiddler-session"
--external-id "fiddler-user-access"
--tags Key=SageMakerPartnerAppUser,Value="$USERNAME"
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
--output text)
if [[ $? -ne 0 ]]; then
echo "❌ Failed to assume role. Please check your permissions."
exit 1
fi
# Extract credentials
export AWS_ACCESS_KEY_ID=$(echo $TEMP_CREDENTIALS | cut -d' ' -f1)
export AWS_SECRET_ACCESS_KEY=$(echo $TEMP_CREDENTIALS | cut -d' ' -f2)
export AWS_SESSION_TOKEN=$(echo $TEMP_CREDENTIALS | cut -d' ' -f3)
echo "✅ Successfully assumed role with user identity: $USERNAME"
# Generate Studio access URL
echo "2. Generating SageMaker Studio access URL..."
STUDIO_URL=$(aws sagemaker create-presigned-domain-url
--domain-id "$DOMAIN_ID"
--user-profile-name "$USER_PROFILE_NAME"
--session-expiration-duration-in-seconds 3600
--region "$AWS_REGION"
--query 'AuthorizedUrl' --output text)
if [[ $? -ne 0 ]]; then
echo "❌ Failed to generate Studio URL"
exit 1
fi
echo ""
echo "=== ACCESS INFORMATION ==="
echo "🔗 SageMaker Studio URL (valid for 1 hour):"
echo "$STUDIO_URL"
echo ""
echo "👤 User Identity: $USERNAME"
echo "🏠 Domain: $DOMAIN_ID"
echo "👥 Shared Profile: $USER_PROFILE_NAME"
echo ""
echo "=== WHAT YOU CAN ACCESS ==="
echo "✅ SageMaker Studio - Full development environment"
echo "✅ Shared Space - 'fiddler-collaboration-space' for team collaboration"
echo "✅ Fiddler Partner AI Apps - AI observability platform"
echo "✅ Jupyter notebooks with Fiddler SDK support"
echo ""
echo "=== INSTRUCTIONS ==="
echo "1. Copy and paste the Studio URL above into your browser"
echo "2. In Studio, access:"
echo " • Partner AI Apps from the Studio interface"
echo " • Shared space: 'fiddler-collaboration-space'"
echo " • Personal spaces for individual work"
echo "3. Your identity ($USERNAME) is properly configured for all services"
# Clean up credentials
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN
echo ""
echo "✅ Session credentials cleaned up for security"
EOF
# Update the user access script with the actual domain ID, using OS-specific sed syntax
if [[ "$(uname -s)" == "Darwin" ]]; then
# macOS (BSD) sed requires an empty string argument for in-place editing
sed -i '' "s/PLACEHOLDER_DOMAIN_ID/$DOMAIN_ID/g" fiddler-user-access.sh
else
# Linux (GNU) sed does not require an argument for in-place editing
sed -i "s/PLACEHOLDER_DOMAIN_ID/$DOMAIN_ID/g" fiddler-user-access.sh
fi
chmod +x fiddler-user-access.sh
# Create Fiddler SDK setup script
cat > setup-fiddler-sdk.sh << 'EOF'
#!/bin/bash
# =============================================================================
# FIDDLER SDK SETUP SCRIPT
# =============================================================================
echo "=== Setting up Fiddler SDK Environment ==="
# Check if running in SageMaker Studio
if [[ -n "$SAGEMAKER_APP_TYPE" ]]; then
echo "✅ Running in SageMaker Studio environment"
else
echo "⚠️ Not running in SageMaker Studio - some features may not work"
fi
# Install compatible SageMaker Python SDK
echo "1. Installing compatible SageMaker Python SDK..."
pip install "sagemaker>=2.237.0" --upgrade
# Install Fiddler Python Client
echo "2. Installing Fiddler Python Client..."
pip install fiddler-client --upgrade
# Set up environment variables for Fiddler SDK
echo "3. Setting up Fiddler environment variables..."
# Create environment setup
cat > setup_fiddler_env.py << 'PYTHON_EOF'
import os
import boto3
def setup_fiddler_environment():
"""Set up environment variables for Fiddler SDK integration with SageMaker Partner AI Apps"""
# Get SageMaker session information
try:
# These environment variables are automatically set in SageMaker Studio
domain_id = os.environ.get('SAGEMAKER_DOMAIN_ID')
region = os.environ.get('AWS_DEFAULT_REGION', 'us-east-1')
print(f"SageMaker Domain ID: {domain_id}")
print(f"AWS Region: {region}")
# Set up Fiddler-specific environment variables
# These will be used by the Fiddler Python Client to connect to the Partner AI App
# Note: The actual Fiddler API endpoint will be provided by the Partner AI App
# when it's deployed. Users will need to get this from the Fiddler UI.
print("\n✅ Environment setup complete!")
print("\n📝 Next steps:")
print("1. Deploy Fiddler Partner AI App from SageMaker Studio")
print("2. Get API key from Fiddler Partner AI App UI")
print("3. Set FIDDLER_API_KEY environment variable:")
print(" os.environ['FIDDLER_API_KEY'] = 'your-api-key'")
print("4. Get Fiddler URL from Partner AI App and set:")
print(" os.environ['FIDDLER_URL'] = 'your-fiddler-app-url'")
return True
except Exception as e:
print(f"❌ Error setting up environment: {e}")
return False
if __name__ == "__main__":
setup_fiddler_environment()
PYTHON_EOF
# Run the environment setup
python setup_fiddler_env.py
echo ""
echo "✅ Fiddler SDK setup complete!"
echo ""
echo "📋 SDK Usage Instructions:"
echo "1. Run this script in a SageMaker Studio Jupyter notebook"
echo "2. Deploy Fiddler Partner AI App from Studio interface"
echo "3. Get API credentials from Fiddler Partner AI App"
echo "4. Use Fiddler Python Client to connect to your models"
echo ""
echo "📖 For detailed Fiddler SDK documentation, visit:"
echo " https://docs.fiddler.ai/"
EOF
chmod +x setup-fiddler-sdk.sh
echo "✅ User access scripts created"
# =============================================================================
# STEP 9: CLEAN UP TEMPORARY FILES
# =============================================================================
echo ""
echo "=== STEP 9: Cleaning up temporary files ==="
rm -f partner-ai-admin-policy.json
rm -f partner-app-execution-trust-policy.json
rm -f license-manager-policy.json
rm -f fiddler-s3-policy.json
rm -f user-execution-trust-policy.json
rm -f user-execution-policy.json
rm -f user-access-trust-policy.json
rm -f user-access-policy.json
echo "✅ Temporary files cleaned up"
# =============================================================================
# SETUP COMPLETE - SUMMARY
# =============================================================================
echo ""
echo "🎉 FIDDLER PARTNER AI APP & SAGEMAKER SETUP COMPLETE!"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo ""
echo "📋 CREATED RESOURCES:"
echo "✅ Existing Admin Role Enhanced: $EXISTING_ADMIN_ROLE_NAME"
echo "✅ Partner AI App Execution Role: $PARTNER_APP_EXECUTION_ROLE_NAME"
echo "✅ User Execution Role: $USER_EXECUTION_ROLE_NAME"
echo "✅ User Access Role: $USER_ACCESS_ROLE_NAME"
echo "✅ SageMaker Domain: $DOMAIN_NAME (ID: $DOMAIN_ID)"
echo "✅ Shared User Profile: $SHARED_USER_PROFILE_NAME"
echo "✅ Shared Space: $SHARED_SPACE_NAME"
echo "✅ User Access Script: fiddler-user-access.sh"
echo "✅ Fiddler SDK Setup Script: setup-fiddler-sdk.sh"
echo ""
echo "🚀 NEXT STEPS:"
echo ""
echo "1. SUBSCRIBE TO FIDDLER IN AWS MARKETPLACE:"
echo " • Use your existing admin role: $EXISTING_ADMIN_ROLE_NAME"
echo " • Go to AWS Marketplace and subscribe to Fiddler AI Observability"
echo " • Follow the subscription process"
echo ""
echo "2. DEPLOY FIDDLER PARTNER AI APP:"
echo " • Use SageMaker Console or AWS CLI to create Partner AI App"
echo " • Specify root admin IAM username for: $EXISTING_ADMIN_ROLE_NAME"
echo " • Use execution role: $PARTNER_APP_EXECUTION_ROLE_NAME"
echo ""
echo "📖 DOCUMENTATION REFERENCES:"
echo "• Partner AI Apps: https://docs.aws.amazon.com/sagemaker/latest/dg/partner-app-onboard.html"
echo "• Fiddler SDK: https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps-sdk.html"
echo "• SageMaker Domains: https://docs.aws.amazon.com/sagemaker/latest/dg/gs-studio-onboard.html"
echo ""
echo "✨ Your team can now collaborate using Fiddler for AI observability in SageMaker!"
```
# User Guide
Source: https://docs.fiddler.ai/integrations/aws-sagemaker/partner-ai-app-user-guide
User guide for Fiddler on AWS SageMaker. Learn to use the Fiddler UI and Python SDK to monitor, explain, and analyze your models and GenAI apps.
### Introduction
The Fiddler Partner AI App for Amazon SageMaker brings enterprise-grade AI observability directly into your SageMaker environment. As a fully integrated Partner AI App, Fiddler enables data scientists and ML engineers to monitor, explain, and analyze their AI applications without leaving the SageMaker ecosystem. This seamless integration allows you to leverage Fiddler's powerful monitoring capabilities—including drift detection, performance tracking, and explainability features—while maintaining your existing SageMaker workflows and security boundaries.
### Prerequisites
Before using the Fiddler Partner AI App, ensure you have:
* Access to an AWS account with the Fiddler Partner AI App deployed by your administrator
* A SageMaker Studio user profile with appropriate permissions configured by your administrator, OR a similar role to assume when creating direct links to the Fiddler UI
* Basic familiarity with SageMaker Studio navigation and notebooks
### Using Fiddler in SageMaker Studio
#### Accessing the Fiddler Partner AI App
The Fiddler Partner AI App integrates seamlessly with SageMaker Studio, providing multiple access points for your model monitoring workflows.
**From SageMaker Studio**
1. Open SageMaker Studio and navigate to the **Applications** section in the left navigation panel
2. Under **Partner AI apps**, locate and select **Fiddler**
3. Click **Open app** to launch the Fiddler interface in a new tab
4. The application will automatically authenticate using your SageMaker credentials
**Direct Link Using the AWS CLI**
* AWS provides a CLI function to generate the Fiddler UI URL: `create-partner-app-presigned-url`
* Along with the appropriate role and permissions, you also need:
* The Fiddler Partner AI App ARN
* The AWS Region
```bash theme={null}
AWS_REGION="us-west-2"
# Use this command to get the partner app ARN:
aws sagemaker list-partner-apps --region "$AWS_REGION" --output table
# Create the URL
aws sagemaker create-partner-app-presigned-url
--arn "$FIDDLER_PAA_ARN"
--expires-in-seconds 300
--session-expiration-duration-in-seconds 1800
--region "$AWS_REGION"
--query 'Url'
--output text
```
#### Navigating the Fiddler Interface
For users new to the Fiddler AI platform, our [Getting Started with ML Model Observability](/getting-started/ml-observability) guide provides a thorough introduction to the model or application onboarding process, event publishing, and Fiddler features.
### Using the Fiddler Python Client SDK
The Fiddler Python client enables programmatic access to Fiddler's capabilities directly from your SageMaker notebooks and code editors.
* Getting started with the [Fiddler Python Client SDK](/sdk-api/python-client/connection)
#### Prerequisites
* The AWS SageMaker Partner App ARN. This is the unique identifier of your SageMaker Fiddler Partner AI App
* The AWS SageMaker Partner App URL. This is the fully qualified domain of the Fiddler UI launched from Studio
* The Fiddler user authentication token, which you can find on the [Credentials](/reference/administration/settings#credentials) tab in the Fiddler UI
If you are using SageMaker Studio to access Fiddler, the SageMaker Partner App ARN and URL can be found by launching Studio and clicking on the Fiddler icon in the Applications list as shown below:
#### Setting Up the SDK Connection
To connect to Fiddler from a SageMaker notebook, code editor, or similar:
1. **Install the latest version of the Fiddler client** (if not pre-installed) and at least 2.237.0 of the SageMaker Python SDK:
```sh theme={null}
pip install fiddler-client "sagemaker>=2.237.0"
```
2. **Configure the connection using SageMaker's built-in authentication**:
```python theme={null}
import os
import fiddler as fdl
# Edit the values in this section!
APP_URL = '' # the SageMaker App URL
APP_ARN = '' # the SageMaker App ARN
TOKEN = '' # the Fiddler User API Key
# Enable Partner AI App authentication
os.environ['AWS_PARTNER_APP_AUTH'] = 'true'
os.environ['AWS_PARTNER_APP_ARN'] = APP_ARN
os.environ['AWS_PARTNER_APP_URL'] = APP_URL
```
3. **Initialize the Fiddler client**:
```python theme={null}
# Initialize the Fiddler client
fdl.init(url=APP_URL, token=TOKEN)
```
#### Common SDK Operations
The [Python Client Guides](/developers/python-client-guides/installation-and-setup) section of Fiddler's documentation contains many examples of everyday use cases and tasks.
### Tracing GenAI Agents from SageMaker
To trace LLM and agentic applications running in your SageMaker Partner AI App, use the [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk). Starting in `fiddler-otel` 1.3.0, the SDK signs trace exports with the same Partner App authentication used by the Python client — set `AWS_PARTNER_APP_AUTH`, `AWS_PARTNER_APP_ARN`, and `AWS_PARTNER_APP_URL`, install the `sagemaker` extra (`pip install "fiddler-otel[sagemaker]"`), and your instrumentation runs unchanged. See [Running in AWS SageMaker](/integrations/agentic-ai/fiddler-otel-sdk#running-in-aws-sagemaker) for the full walkthrough.
In the current SageMaker environment, agent traces are signed and exported correctly but are not yet visible in the Fiddler UI. This is an environment limitation, not an SDK defect.
#### Best Practices for SDK Usage
* **Notebook Organization**: Keep your Fiddler monitoring code in dedicated sections or separate notebooks for clarity
* **Environment Variables**: Store the Partner App environment variables at the beginning of your notebooks for consistent access
* **Batch Operations**: Use batch methods when publishing large volumes of data to optimize performance
#### Troubleshooting SDK Connections
If you encounter connection issues:
1. **Verify that environment variables** are correctly set
2. **Check your IAM permissions** include `sagemaker:CallPartnerAppApi`
3. **Ensure the Fiddler app is deployed** and in "Deployed" status
# AWS SageMaker Partner AI App
Source: https://docs.fiddler.ai/integrations/cloud-platforms-and-deployment/aws-sagemaker
Run Fiddler AI Observability platform within Amazon SageMaker as a Partner AI App. Procure, provision, and securely operate Fiddler seamlessly all within your existing AWS account.
## Introduction
This guide helps AWS administrators configure, deploy, and manage [Fiddler AI ](https://fiddler.ai/)as an Amazon SageMaker Partner AI App. Fiddler AI is an enterprise-grade AI Observability platform that helps monitor, explain, and analyze ML models, LLM applications, GenAI systems, and AI agents directly within your AWS environment.
## What is a Partner AI App?
Amazon SageMaker Partner AI Apps are fully managed AI/ML applications from AWS partners that run natively within your SageMaker environment. Think of them as pre-built, enterprise-ready applications that integrate seamlessly with your existing AWS infrastructure—no complex deployments or external accounts required.
**Key Benefits**:
* **Secure by Design**: Applications run entirely within your AWS account, maintaining your existing security boundaries and compliance requirements
* **Unified Experience**: Access partner tools directly from SageMaker Studio alongside your notebooks, models, and datasets
* **Simplified Procurement**: Subscribe through AWS Marketplace with consolidated billing on your AWS invoice
* **Managed Infrastructure**: AWS handles the underlying infrastructure while you focus on using the application
## How the Fiddler Partner AI App Works
The Fiddler Partner AI App is a fully integrated component within your SageMaker environment. The following sections will explain how this integration works.
### Architecture Overview
1. **Deployment Model**: Fiddler runs as a fully managed, containerized application within your AWS account, requiring no operational support from your team. AWS handles all infrastructure management, updates, and maintenance while isolating your environment and securing access to your data and models.
2. **Data Flow**:
* Your data never leaves your AWS environment
* Fiddler processes model predictions and monitoring data within your existing security boundaries, typically avoiding the need for additional security reviews
* All computations and storage remain under your AWS account's control
3. **Integration Points**:
* **SageMaker Studio UI**: Launch Fiddler's web interface directly from Studio or via pre-signed URLs
* **Python SDK**: Connect to Fiddler from notebooks using authenticated API calls
* **Model Monitoring**: Monitor any model, including those deployed on SageMaker endpoints
* **S3 Storage**: Fiddler uses designated S3 buckets within your account for data persistence
4. **Authentication & Authorization**:
* Leverages SageMaker's identity propagation to maintain user context
* Requires specific IAM permissions configured by your administrator
* No separate Fiddler-specific credentials needed for Studio access
* API access uses Fiddler tokens combined with AWS authentication
## Prerequisites for Subscribing to the Fiddler Partner AI App
Before deploying Fiddler AI, ensure you have:
* An active AWS account with administrator privileges
* An existing SageMaker AI domain if launching Fiddler from Studio is required
* Appropriate IAM permissions to manage AWS Marketplace subscriptions
* Basic understanding of AWS IAM roles and policies
## Configure Required Permissions
The official [Amazon SageMaker Partner AI Apps documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps.html) is the primary reference for preparing, subscribing to, and deploying the Fiddler Partner AI App. This guide assumes you have completed the documentation's Set up Partner AI Apps section, which covers permissions, authentication, and authorization requirements.
### Important Considerations
Before proceeding, be aware of these standard requirements and potential friction points:
**Identity Configuration**:
* Partner AI Apps require identity propagation to be correctly configured
* Your SageMaker domain must use either IAM or IAM Identity Center authentication
* The username format must match between your identity provider and the Fiddler admin configuration
**Permission Requirements**:
* Multiple IAM roles are required (admin role, execution role, and user roles)
* AWS License Manager permissions are mandatory for Fiddler
* The setup involves numerous individual IAM permissions across different services
**Time and Complexity**:
* Initial permission setup typically takes 30-60 minutes
* Consider using our Quick Setup Script to automate this process
* Some organizations may need security team approval for the required IAM policies
**Common Gotchas**:
* The execution role name cannot be changed after Fiddler deployment
* Admin usernames entered during deployment cannot be modified later
* Free trial deployments cannot be upgraded to paid subscriptions (requires redeployment)
## Fiddler Subscription Options
### Pricing Structure
Fiddler Partner AI App pricing consists of two components:
1. **Software License Costs**: Billed through AWS Marketplace
* **30-Day Free Trial**: Full functionality for up to 5 models with no software costs (infrastructure charges still apply)
* **Monthly Subscription**: Pay-as-you-go monthly pricing
* **Annual Subscription**: Discounted pricing with a 12-month commitment
2. **Infrastructure Costs**: Billed separately by AWS based on usage
* Compute instances (EC2)
* Storage (EBS volumes)
* Data transfer
* Managed services (RDS, AMQ, etc.)
* These costs vary based on the tier size you select (Small, Medium, or Large) and your contract discounts
### Subscription Tiers
Tier
Recommended Use Case
Estimated Monthly Infrastructure\* Cost
Small
Development, POCs, teams \<10 users
$500 - $1,000
Medium
Production workloads, teams 10-50 users
$1,500 - $3,000
Large
Enterprise deployments, teams 50+ users
$3,000 - $6,000
\*Infrastructure costs are estimates and vary based on actual usage and AWS region
### Trial vs. Paid Subscription
**30-Day Free Trial**:
* Full Fiddler functionality with no functionality restrictions on up to five models
* No software license charges
* Infrastructure costs still apply
* Cannot be upgraded to paid (requires redeployment)
* Ideal for evaluation and testing
**Paid Subscription**:
* Continuous access with no time limits
* Includes AWS support for Partner AI Apps
* Can be upgraded/downgraded between tiers
* Suitable for production use cases
## Subscribe to the Fiddler Partner AI App
Below is a brief example of using the Amazon SageMaker console to subscribe to the Fiddler Partner AI App.
1. With the admin account configured per the prerequisites, open your Amazon SageMaker AI Console and click the new *Partner AI Apps* link in the left-hand navigation to view the Partner AI Apps.
2. Click the View details link to review the Fiddler details on contract pricing and estimated infrastructure costs.
3. Click the *Go to Marketplace to subscribe* link to purchase or select the free trial. Then click on the *View purchase options* or *Try for free* links to open the subscription page.
4. This page lets you select your desired contract length, license options, and costs. When you are ready to confirm your subscription, click the Subscribe button.
## Provision and Deploy the Fiddler Partner AI App
The documentation's [Partner AI App Provisioning](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps-provision.html) section lists the general instructions for provisioning Partner AI Apps. The guide below is an example walkthrough of provisioning the Fiddler Partner AI App.
1. The configuration and deployment process has six steps. Begin by choosing a name for your Fiddler SageMaker Partner AI App and selecting your desired maintenance time window.
2. Add at least one and up to five users with Org Admin privileges within Fiddler. The username entered here must match the IAM username passed to Fiddler on launch.
3. Select the execution role used by the Fiddler Partner AI App to run AWS resources and authenticate subscription licenses. This role is created following the [Set up Partner AI Apps](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-app-onboard.html) section of the AWS documentation.
4. Ensure that the STS identity propagation section has *Enable STS* selected.
5. Choose the tier size appropriate for your use cases, click *Next* to review, and then click the ***Deploy*** button to deploy the application. Allow time for this process to complete. It should be ready in under an hour.
6. Please return to the Partner AI Apps > My Apps tab in the SageMaker console to check the status. The application can take 1-2 hours to complete the deployment process, at which point the status will be *Deployed*.
## Next Steps
Once your Fiddler Partner AI App is deployed:
* **For Administrators**:
* Leverage the [Fiddler Partner AI App Quick Setup Script](/integrations/aws-sagemaker/partner-ai-app-quick-setup-script) to get started with all the required roles and policies for running Fiddler in Studio
* Get detailed information on IAM requirements, deployment tips, and troubleshooting with the [Admin Guide](/integrations/aws-sagemaker/partner-ai-app-admin-guide)
* **For Users**: Learn how to access and use Fiddler with the [User Guide](/integrations/aws-sagemaker/partner-ai-app-user-guide)
* **For GenAI and agentic applications**: Trace LLM and agent workloads from SageMaker with the [Fiddler OTel SDK](/integrations/agentic-ai/fiddler-otel-sdk#running-in-aws-sagemaker), which signs trace exports using the same Partner App authentication
# Cloud Platforms Overview
Source: https://docs.fiddler.ai/integrations/cloud-platforms-and-deployment/cloud-platforms
Deploy and operate Fiddler natively on leading cloud platforms
Run Fiddler AI Observability in your preferred cloud environment with native platform integrations. Deploy as a fully managed Partner AI App on AWS SageMaker, run in your Kubernetes clusters, or leverage cloud-native services for ML model monitoring at scale.
## Why Cloud Platform Integrations Matter
Modern AI systems are built on cloud infrastructure. Fiddler's cloud platform integrations ensure you can:
* **Maintain Data Sovereignty** - Keep all your data and models within your existing cloud security boundaries
* **Simplify Procurement** - Subscribe through cloud marketplaces with consolidated billing
* **Leverage Existing Infrastructure** - Use your current IAM, networking, and security configurations
* **Scale Seamlessly** - Cloud-native deployment automatically scales with your ML workloads
* **Reduce Operational Overhead** - Managed platform integrations eliminate manual infrastructure management
## AWS Deployments
### AWS SageMaker Partner AI App
Run Fiddler as a fully managed Partner AI App within Amazon SageMaker.
**Best for:** Enterprise AWS customers wanting seamless SageMaker integration with zero external infrastructure
**Key Features:**
* Fully managed infrastructure within your AWS account
* Native SageMaker Studio integration
* No external accounts or data transfer required
* Automatic updates and maintenance handled by AWS
* Consolidated AWS billing through Marketplace
**Deployment Options:**
* **30-Day Free Trial** - Full functionality for up to 5 models (infrastructure costs apply)
* **Tiered Subscriptions** - Small, Medium, and Large configurations for different team sizes
* **AWS Marketplace** - Monthly or annual subscriptions with flexible pricing
**Status:** ✓ **GA** - Production-ready
[**Get Started with AWS SageMaker Partner AI App →**](/integrations/cloud-platforms-and-deployment/aws-sagemaker)
**Related AWS Integrations:**
* [Amazon S3](/integrations/data-platforms/integration-with-s3) - Connect Fiddler to S3 data sources
* [SageMaker Pipelines](/integrations/data-platforms/sagemaker-integration) - Monitor SageMaker ML pipelines
* [Strands SDK](/integrations/agentic-ai/strands-sdk) - Monitor Strands Agents
## Multi-Cloud & Kubernetes
While AWS SageMaker Partner AI App is our featured cloud platform integration, Fiddler supports deployment across multiple cloud providers:
### Supported Deployment Patterns
**Cloud Platforms:**
* **AWS** - Native SageMaker Partner AI App (recommended), EC2, EKS
* **Azure** - Azure ML integration (contact sales), AKS deployment
* **Google Cloud** - GKE deployment, Vertex AI connectivity
* **Private Cloud** - On-premises Kubernetes with cloud connectivity
**Kubernetes Deployments:**
* **Helm Charts** - Production-grade Kubernetes deployment templates
* **Operator Pattern** - Automated lifecycle management for Fiddler clusters
* **Multi-Cluster Support** - Monitor ML models across distributed Kubernetes environments
* **Cloud-Agnostic** - Run on EKS, AKS, GKE, or on-premises Kubernetes
**Container Orchestration:**
* **Docker Compose** - Development and testing environments
* **Docker Swarm** - Small-scale production deployments
* **Kubernetes** - Enterprise production deployments
## Deployment Architecture Patterns
### Pattern 1: Fully Managed (Recommended for AWS)
**Use AWS SageMaker Partner AI App** for zero-infrastructure management:
```
AWS Account (Your VPC)
├── SageMaker Studio (UI Access)
├── Fiddler Partner AI App (Managed)
│ ├── Compute (Managed by AWS)
│ ├── Storage (Your S3 buckets)
│ └── Database (Managed RDS)
└── Your ML Models (SageMaker Endpoints)
```
**Advantages:**
* No operational overhead
* Automatic scaling and updates
* AWS handles infrastructure security
* Seamless Studio integration
### Pattern 2: Self-Managed Kubernetes
**Use Helm Charts** for full control over infrastructure:
```
Cloud Provider (Your Kubernetes Cluster)
├── Fiddler Namespace
│ ├── API Server Pods
│ ├── Worker Pods
│ ├── Database (StatefulSet)
│ └── Storage (PersistentVolumes)
└── Ingress/Load Balancer
```
**Advantages:**
* Full infrastructure control
* Cloud-agnostic deployment
* Custom security configurations
* On-premises compatibility
### Pattern 3: Hybrid Multi-Cloud
**Deploy Fiddler in one cloud, monitor models across all:**
```
Primary Cloud (Fiddler Installation)
├── Fiddler Platform
└── Centralized Monitoring Dashboard
Connected Clouds
├── AWS (SageMaker models)
├── Azure (Azure ML models)
├── GCP (Vertex AI models)
└── On-Premises (Legacy models)
```
**Advantages:**
* Unified observability across clouds
* Centralized governance and compliance
* Flexible hybrid architecture
## Getting Started
### For AWS Customers
**Quick Start Path:**
1. **Subscribe** - Get Fiddler from AWS Marketplace
2. **Deploy** - Use SageMaker Partner AI Apps one-click deployment
3. **Configure** - Run the [Quick Setup Script](/integrations/aws-sagemaker/partner-ai-app-quick-setup-script) for IAM roles
4. **Monitor** - Connect your SageMaker models and LLM applications
[**Full AWS Deployment Guide →**](/integrations/cloud-platforms-and-deployment/aws-sagemaker)
**Need Help?**
* [Quick Setup Script](/integrations/aws-sagemaker/partner-ai-app-quick-setup-script) - Automated IAM configuration
* [Admin Guide](/integrations/aws-sagemaker/partner-ai-app-admin-guide) - Detailed deployment and management
* [User Guide](/integrations/aws-sagemaker/partner-ai-app-user-guide) - Accessing and using Fiddler
### For Kubernetes Deployments
**Prerequisites:**
* Kubernetes 1.21+ cluster
* Helm 3.8+ installed
* Storage provisioner (for persistent volumes)
* Ingress controller (for external access)
**Installation:**
```bash theme={null}
# Add Fiddler Helm repository
helm repo add fiddler https://helm.fiddler.ai
helm repo update
# Install Fiddler
helm install fiddler fiddler/fiddler \
--namespace fiddler \
--create-namespace \
--set license.key= \
--set ingress.enabled=true \
--set ingress.hostname=fiddler.your-domain.com
```
[**Contact Sales for Kubernetes Deployment →**](mailto:sales@fiddler.ai)
### For Other Cloud Providers
**Azure Customers:**
* Azure ML integration available
* AKS deployment supported
* [Contact us for deployment assistance →](mailto:support@fiddler.ai)
**Google Cloud Customers:**
* GKE deployment supported
* Vertex AI connectivity available
* [Contact us for deployment assistance →](mailto:support@fiddler.ai)
## Migration & Upgrade Paths
### From Other Monitoring Platforms
**Migrating from Competitor Platform:**
1. **Parallel Deployment** - Run Fiddler alongside existing monitoring
2. **Data Migration** - Import historical metrics and model metadata
3. **Gradual Cutover** - Model-by-model transition with zero downtime
4. **Training & Onboarding** - Dedicated support during migration
**Common Migration Sources:**
* **Arize** - Direct migration tools available
* **Weights & Biases** - Model metadata import supported
* **DataRobot** - Custom migration scripts available
* **Custom Solutions** - API-based migration assistance
### Upgrading Within Fiddler
**AWS SageMaker Partner AI App Upgrades:**
* **Tier Upgrades** - Scale from Small → Medium → Large as needed
* **Version Updates** - Automatic updates during maintenance windows
* **Trial to Production** - Requires redeployment (preserve data with migration scripts)
**Self-Managed Upgrades:**
* **Helm Upgrades** - Standard `helm upgrade` process
* **Rolling Updates** - Zero-downtime deployments
* **Backup & Restore** - Automated backup before each upgrade
## Security & Compliance
### Data Residency
**AWS SageMaker Partner AI App:**
* All data stays within your AWS account and region
* No external data transfer to Fiddler's infrastructure
* Choose your preferred AWS region during deployment
**Self-Managed Deployments:**
* Full control over data location
* Support for air-gapped environments
* On-premises deployment options
### Compliance Certifications
* **SOC 2 Type II** - Fiddler platform certified
* **GDPR** - Data processing agreements available
* **HIPAA** - Compliant deployment options for healthcare
* **FedRAMP** - In progress (contact for timeline)
### Security Features
* **Encryption at Rest** - All stored data encrypted (AWS KMS, customer-managed keys)
* **Encryption in Transit** - TLS 1.3 for all network communication
* **SSO Integration** - SAML 2.0, OIDC support (Azure AD, Okta, AWS SSO)
* **RBAC** - Role-based access control with fine-grained permissions
* **Audit Logging** - Complete audit trail of all platform activities
## Cost Optimization
### AWS SageMaker Partner AI App Pricing
**Total Cost of Ownership (TCO):**
* **Software License** - Billed through AWS Marketplace
* **Infrastructure** - AWS resource costs (EC2, RDS, S3, etc.)
* **Data Transfer** - Minimal costs (data stays in your VPC)
**Tier Selection Guidelines:**
| Team Size | Models Monitored | Recommended Tier | Est. Monthly Cost\* |
| ----------- | ---------------- | ---------------- | ------------------- |
| \< 10 users | \< 20 models | Small | $500 - $1,000 |
| 10-50 users | 20-100 models | Medium | $1,500 - $3,000 |
| 50+ users | 100+ models | Large | $3,000 - $6,000 |
\*Infrastructure costs only (software license additional)
**Cost Optimization Tips:**
* Start with **Small tier** for POCs and development
* Use **Reserved Instances** for production workloads (20-40% savings)
* **Right-size tier** based on actual usage metrics
* **Scheduled scaling** for non-production environments
### Self-Managed Cost Optimization
* **Spot Instances** - Use for worker nodes (50-70% savings)
* **Auto-Scaling** - Scale compute based on monitoring load
* **Storage Tiering** - Move historical data to cheaper storage classes
* **Multi-Tenancy** - Share Fiddler instance across multiple teams
## Monitoring Infrastructure Health
### Platform Metrics
**AWS SageMaker Partner AI App:**
* View infrastructure health in SageMaker Console
* CloudWatch metrics for Fiddler components
* Automatic alerting for platform issues
* AWS Support for troubleshooting
**Self-Managed:**
* Prometheus metrics exported by default
* Grafana dashboards for infrastructure monitoring
* Health check endpoints for uptime monitoring
* Integration with existing observability stack
## Support & Resources
### AWS SageMaker Partner AI App
* **AWS Support** - Leverage your existing AWS support plan
* **Fiddler Support** - Direct support channel for platform issues
* **Documentation** - [Complete SageMaker deployment guides →](/integrations/cloud-platforms-and-deployment/aws-sagemaker)
* **AWS Marketplace** - [Subscribe and get started →](https://aws.amazon.com/marketplace/pp/prodview-caia5ckldtyhs)
### Self-Managed Deployments
* **Enterprise Support** - 24/7 support with SLA guarantees
* **Deployment Assistance** - Professional services for setup
* **Training** - Platform administrator training programs
* **Community** - [Join our Slack community](/integrations/cloud-platforms-and-deployment/cloud-platforms)
## Related Integrations
* [**Data Platforms**](/integrations/data-platforms-and-pipelines/data-platforms) - Connect Fiddler to S3, Snowflake, BigQuery
* [**ML Platforms**](/integrations/ml-platforms-and-tools/ml-platforms) - Integrate with Databricks, MLflow
* [**Agentic AI**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) - Monitor LangGraph and Strands Agents
* [**Monitoring & Alerting**](/integrations/monitoring-and-alerting/monitoring-alerting) - Send alerts to Datadog, PagerDuty
## Need Help Choosing?
Not sure which deployment option is right for you? We're here to help:
* **AWS SageMaker Customers** - [Get started with Partner AI App →](/integrations/cloud-platforms-and-deployment/aws-sagemaker)
* **Multi-Cloud Requirements** - [Contact sales for architecture consultation →](mailto:sales@fiddler.ai)
* **On-Premises Deployment** - [Discuss private cloud options →](mailto:sales@fiddler.ai)
* **Migration Assistance** - [Talk to our solutions team →](mailto:solutions@fiddler.ai)
***
**Featured Integration:** The AWS SageMaker Partner AI App is our recommended deployment for AWS customers, offering the fastest time-to-value with zero operational overhead. [Learn more →](/integrations/cloud-platforms-and-deployment/aws-sagemaker)
# Data Platforms Overview
Source: https://docs.fiddler.ai/integrations/data-platforms-and-pipelines/data-platforms
Connect Fiddler to data warehouses, streaming platforms, and ML pipelines
Integrate Fiddler with your existing data infrastructure to seamlessly ingest training data, production events, and ground truth labels. From cloud data warehouses to real-time streaming platforms, Fiddler connects to the data sources you already use.
## Why Data Integration Matters
AI observability requires continuous data flow from your ML pipelines and applications. Fiddler's data platform integrations enable:
* **Automated Data Ingestion** - Pull training datasets and production events without manual uploads
* **Real-Time Monitoring** - Stream prediction events for immediate drift and performance detection
* **Unified Data Pipeline** - Single integration point for all your ML data sources
* **Ground Truth Enrichment** - Automatically join production predictions with delayed labels
* **Historical Analysis** - Query data warehouse for model performance over time
## Integration Categories
### 🏢 Data Warehouses
Connect Fiddler to your cloud data warehouse for batch data ingestion and historical analysis.
**Supported Platforms:**
* [**Snowflake**](/integrations/data-platforms/snowflake-integration) - Cloud data warehouse with zero-copy data sharing ✓ **GA**
* [**Google BigQuery**](/integrations/data-platforms/bigquery-integration) - Serverless data warehouse with SQL analytics ✓ **GA**
**Common Use Cases:**
* Import training datasets from data warehouse tables
* Query historical model predictions for performance analysis
* Join production events with delayed ground truth labels
* Export Fiddler metrics back to warehouse for BI tools
### 📊 Data Streaming
Stream real-time prediction events and feedback directly to Fiddler for immediate observability.
**Supported Platforms:**
* [**Apache Kafka**](/integrations/data-platforms/kafka-integration) - Distributed event streaming platform ✓ **GA**
* [**Amazon S3**](/integrations/data-platforms/integration-with-s3) - Object storage with event notifications ✓ **GA**
**Common Use Cases:**
* Stream model predictions in real-time from production services
* Monitor agentic AI interactions as they occur
* Trigger alerts on data quality issues within seconds
* Capture ground truth feedback from user interactions
### 🔄 Orchestration & Pipelines
Integrate Fiddler into your ML workflow orchestration for automated monitoring at every pipeline stage.
**Supported Platforms:**
* [**Apache Airflow**](/integrations/data-platforms/airflow-integration) - Workflow orchestration platform ✓ **GA**
* [**AWS SageMaker Pipelines**](/integrations/data-platforms/sagemaker-integration) - Managed ML pipeline service ✓ **GA**
**Common Use Cases:**
* Automatically upload datasets when training pipelines complete
* Trigger model evaluation as part of CI/CD workflows
* Schedule periodic drift checks and performance reports
* Orchestrate ground truth label collection and enrichment
## Data Warehouse Integrations
### Snowflake
**Why Snowflake + Fiddler:**
* **Zero-Copy Data Sharing** - No data duplication, direct queries to Snowflake
* **Secure Data Access** - OAuth 2.0 and key-pair authentication
* **Scalable Analytics** - Leverage Snowflake's compute for large datasets
* **Cost-Effective** - Pay only for queries executed, no data transfer fees
**Quick Start:**
```sql theme={null}
-- Create Fiddler integration user
CREATE USER fiddler_user PASSWORD='...';
CREATE ROLE fiddler_role;
GRANT ROLE fiddler_role TO USER fiddler_user;
-- Grant permissions to your ML data
GRANT USAGE ON DATABASE ml_data TO ROLE fiddler_role;
GRANT USAGE ON SCHEMA ml_data.predictions TO ROLE fiddler_role;
GRANT SELECT ON ALL TABLES IN SCHEMA ml_data.predictions TO ROLE fiddler_role;
```
[**Full Snowflake Setup Guide →**](/integrations/data-platforms/snowflake-integration)
### Google BigQuery
**Why BigQuery + Fiddler:**
* **Serverless Architecture** - No infrastructure management
* **SQL-Based Queries** - Familiar interface for data teams
* **Federated Queries** - Join Fiddler data with other GCP sources
* **Machine Learning** - BigQuery ML model monitoring integration
**Quick Start:**
```python theme={null}
from fiddler import FiddlerClient
# Connect Fiddler to BigQuery
client = FiddlerClient(api_key="fid_...")
client.add_bigquery_connection(
project_id="your-gcp-project",
dataset_id="ml_predictions",
credentials_path="service-account-key.json"
)
# Import dataset from BigQuery table
client.upload_dataset_from_bigquery(
project="your-project",
dataset="ml_predictions",
table="credit_risk_predictions"
)
```
[**Full BigQuery Setup Guide →**](/integrations/data-platforms/bigquery-integration)
## Streaming Integrations
### Apache Kafka
**Why Kafka + Fiddler:**
* **Real-Time Monitoring** - Sub-second latency from prediction to observability
* **High Throughput** - Handle millions of events per second
* **Event Replay** - Replay historical events for testing and validation
* **Exactly-Once Semantics** - Guaranteed delivery for critical predictions
**Architecture Pattern:**
```
Prediction Service
↓ (publish)
Kafka Topic: model-predictions
↓ (consume)
Fiddler Kafka Consumer
↓ (ingest)
Fiddler Platform
```
**Quick Start:**
```python theme={null}
from fiddler import FiddlerClient
# Configure Kafka consumer for Fiddler
client = FiddlerClient(api_key="fid_...")
client.add_kafka_source(
bootstrap_servers="kafka-broker:9092",
topic="model-predictions",
group_id="fiddler-consumer",
# Message format
value_deserializer="json",
schema={
"prediction_id": "string",
"model_id": "string",
"prediction": "float",
"features": "object"
}
)
```
[**Full Kafka Setup Guide →**](/integrations/data-platforms/kafka-integration)
### Amazon S3
**Why S3 + Fiddler:**
* **Batch Processing** - Ingest large datasets efficiently
* **Event Notifications** - Automatic processing when new files arrive
* **Data Lake Integration** - Monitor models trained on S3 data lakes
* **Cost-Effective Storage** - Archive historical predictions in S3
**Integration Patterns:**
**Pattern 1: Scheduled Batch Upload**
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Upload dataset from S3
client.upload_dataset_from_s3(
project="fraud-detection",
dataset="production-predictions",
s3_uri="s3://my-bucket/predictions/2024-11-10.parquet",
format="parquet"
)
```
**Pattern 2: Event-Driven Upload**
```python theme={null}
# AWS Lambda triggered by S3 event
import json
from fiddler import FiddlerClient
def lambda_handler(event, context):
# Extract S3 bucket and key from event
bucket = event['Records'][0]['s3']['bucket']['name']
key = event['Records'][0]['s3']['object']['key']
# Upload to Fiddler
client = FiddlerClient(api_key=os.environ['FIDDLER_API_KEY'])
client.upload_dataset_from_s3(
project="my-project",
dataset="streaming-data",
s3_uri=f"s3://{bucket}/{key}"
)
```
[**Full S3 Setup Guide →**](/integrations/data-platforms/integration-with-s3)
## Orchestration & Pipeline Integrations
### Apache Airflow
**Why Airflow + Fiddler:**
* **Automated Workflows** - Trigger Fiddler operations as DAG tasks
* **Dependency Management** - Ensure data quality before model training
* **Scheduling** - Periodic drift checks and model evaluations
* **Observability** - Monitor ML pipelines and models in one platform
**Example DAG:**
```python theme={null}
from airflow import DAG
from airflow.operators.python import PythonOperator
from fiddler_airflow import FiddlerDatasetUploadOperator, FiddlerModelEvaluationOperator
from datetime import datetime
dag = DAG(
'ml_pipeline_with_monitoring',
start_date=datetime(2024, 1, 1),
schedule_interval='@daily'
)
# Train model
train_task = PythonOperator(
task_id='train_model',
python_callable=train_model_func,
dag=dag
)
# Upload training data to Fiddler
upload_task = FiddlerDatasetUploadOperator(
task_id='upload_to_fiddler',
project='credit-risk',
dataset='training_data',
data_source='s3://my-bucket/training-data.csv',
dag=dag
)
# Evaluate model in Fiddler
eval_task = FiddlerModelEvaluationOperator(
task_id='evaluate_model',
project='credit-risk',
model='risk_model_v2',
baseline_dataset='training_data',
dag=dag
)
train_task >> upload_task >> eval_task
```
[**Full Airflow Setup Guide →**](/integrations/data-platforms/airflow-integration)
### AWS SageMaker Pipelines
**Why SageMaker Pipelines + Fiddler:**
* **Native AWS Integration** - Seamless with SageMaker Partner AI App
* **End-to-End ML Workflows** - From data prep to model monitoring
* **Model Registry Integration** - Automatic monitoring setup for registered models
* **Cost Optimization** - Leverage existing SageMaker infrastructure
**Example Pipeline:**
```python theme={null}
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep, TrainingStep
from sagemaker.workflow.functions import Join
from fiddler_sagemaker import FiddlerMonitoringStep
# Define SageMaker pipeline steps
processing_step = ProcessingStep(...)
training_step = TrainingStep(...)
# Add Fiddler monitoring step
fiddler_step = FiddlerMonitoringStep(
name="SetupFiddlerMonitoring",
model_name=training_step.properties.ModelName,
baseline_dataset_s3_uri=processing_step.properties.ProcessingOutputConfig.Outputs["baseline"].S3Output.S3Uri,
fiddler_project="sagemaker-models",
depends_on=[training_step]
)
# Create pipeline
pipeline = Pipeline(
name="MLPipelineWithFiddler",
steps=[processing_step, training_step, fiddler_step]
)
```
[**Full SageMaker Pipelines Guide →**](/integrations/data-platforms/sagemaker-integration)
## Integration Selector
Not sure which data integration to use? Here's a quick decision guide:
| Your Data Source | Recommended Integration | Why |
| ---------------------------- | ----------------------- | -------------------------------------- |
| Snowflake data warehouse | **Snowflake connector** | Zero-copy sharing, direct queries |
| BigQuery tables | **BigQuery connector** | Serverless, SQL-based, GCP-native |
| Real-time prediction streams | **Kafka integration** | Sub-second latency, high throughput |
| S3 data lake | **S3 integration** | Batch processing, event-driven uploads |
| Airflow ML pipelines | **Airflow operators** | Automated workflows, task dependencies |
| SageMaker workflows | **SageMaker Pipelines** | Native AWS integration, model registry |
## Getting Started
### Prerequisites
Before setting up data integrations, ensure you have:
* **Fiddler Account** - Cloud or on-premises instance
* **API Key** - Generate from Fiddler UI Settings
* **Data Source Access** - Credentials with read permissions
* **Network Connectivity** - Firewall rules allowing Fiddler → Data Source
### General Setup Pattern
All data integrations follow this pattern:
**1. Configure Connection**
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(
api_key="fid_...",
url="https://app.fiddler.ai"
)
# Add data source
client.add_data_source(
name="my-data-warehouse",
type="snowflake", # or bigquery, kafka, s3
connection_params={...}
)
```
**2. Test Connection**
```python theme={null}
# Verify connectivity
status = client.test_data_source("my-data-warehouse")
print(f"Connection status: {status}")
```
**3. Ingest Data**
```python theme={null}
# Upload dataset from data source
client.upload_dataset_from_source(
project="my-project",
dataset="training-data",
source="my-data-warehouse",
query="SELECT * FROM predictions WHERE date > '2024-01-01'"
)
```
## Advanced Patterns
### Pattern 1: Multi-Source Data Enrichment
Combine data from multiple sources for comprehensive monitoring:
```python theme={null}
# Production predictions from Kafka
client.add_kafka_source(
topic="production-predictions",
# ... additional connection settings
)
# Ground truth labels from Snowflake
client.add_snowflake_source(
database="ml_data",
schema="labels",
# ... additional connection settings
)
# Join predictions with labels
client.configure_ground_truth_enrichment(
project="fraud-detection",
prediction_source="production-predictions",
label_source="ml_data.labels.fraud_labels",
join_key="transaction_id",
time_window_hours=24
)
```
### Pattern 2: Data Quality Validation
Validate data quality before ingestion:
```python theme={null}
from fiddler import DataValidator
validator = DataValidator()
validator.add_rule("age", min_value=18, max_value=100)
validator.add_rule("income", not_null=True)
validator.add_rule("credit_score", pattern=r'^\d{3}$')
# Validate before upload
is_valid = client.upload_dataset_from_source(
...,
validator=validator,
on_validation_failure="abort" # or "warn" or "ignore"
)
```
### Pattern 3: Incremental Updates
Efficiently update datasets with only new data:
```python theme={null}
# Initial full load
client.upload_dataset_from_source(
project="my-project",
dataset="user-events",
source="my-bigquery",
query="SELECT * FROM events"
)
# Daily incremental updates (scheduled via Airflow)
client.update_dataset_from_source(
project="my-project",
dataset="user-events",
source="my-bigquery",
query="SELECT * FROM events WHERE date = CURRENT_DATE()",
mode="append" # or "upsert"
)
```
## Data Format Requirements
### Baseline/Training Data
Must include:
* **Features** - All model input features
* **Predictions** - Model outputs (for validation)
* **Metadata** (optional) - Additional context fields
**Example Schema:**
```json theme={null}
{
"transaction_id": "string",
"amount": "float",
"merchant_category": "string",
"prediction": "float",
"timestamp": "datetime"
}
```
### Production Event Data
Must include:
* **Event ID** - Unique identifier
* **Timestamp** - Event time
* **Features** - Model inputs
* **Predictions** - Model outputs
* **Model Version** (optional) - For multi-model monitoring
**Example Kafka Message:**
```json theme={null}
{
"event_id": "evt_12345",
"timestamp": "2024-11-10T14:32:01Z",
"model": "fraud_detector_v3",
"features": {
"amount": 127.50,
"merchant": "AMZN"
},
"prediction": 0.23
}
```
## Security & Compliance
### Authentication Methods
**Snowflake:**
* Username/Password
* Key Pair Authentication (recommended for production)
* OAuth 2.0
**BigQuery:**
* Service Account JSON key
* Application Default Credentials
* Workload Identity (GKE)
**Kafka:**
* SASL/PLAIN
* SASL/SCRAM
* mTLS
**S3:**
* IAM Role (recommended for AWS deployments)
* Access Key / Secret Key
* Cross-account access via IAM role assumption
### Data Privacy
* **Encryption in Transit** - TLS 1.3 for all data transfers
* **Encryption at Rest** - Data encrypted in Fiddler storage
* **PII Redaction** - Automatically detect and redact sensitive fields
* **Data Retention** - Configurable retention policies per dataset
### Network Security
**Firewall Rules:**
```
Source: Fiddler Platform IP ranges
Destination: Your Data Source
Ports:
- Snowflake: 443 (HTTPS)
- BigQuery: 443 (HTTPS)
- Kafka: 9092/9093 (SASL/SSL)
- S3: 443 (HTTPS)
```
**Private Connectivity:**
* **AWS PrivateLink** - For SageMaker Partner AI App
* **VPC Peering** - Direct connection to data sources
* **VPN Tunnels** - Secure connectivity for on-premises sources
## Monitoring Data Pipeline Health
### Connection Health Checks
```python theme={null}
# Check all data source connections
sources = client.list_data_sources()
for source in sources:
status = client.test_data_source(source.name)
print(f"{source.name}: {status.status} (latency: {status.latency_ms}ms)")
```
### Data Ingestion Metrics
Monitor data pipeline performance:
* **Ingestion Latency** - Time from source to Fiddler
* **Throughput** - Events per second processed
* **Error Rate** - Failed ingestion attempts
* **Data Freshness** - Time since last successful update
### Alerts on Pipeline Failures
```python theme={null}
# Configure alert for data pipeline issues
client.create_alert(
name="Data Pipeline Failure",
trigger="data_ingestion_failed",
source="kafka-production-predictions",
notification_channels=["email", "pagerduty"]
)
```
## Troubleshooting
### Common Issues
**Connection Timeouts:**
* Check network connectivity and firewall rules
* Verify credentials are current and have proper permissions
* Ensure data source is reachable from Fiddler's network
**Schema Mismatches:**
* Validate data types match Fiddler's expected schema
* Check for null values in required fields
* Ensure timestamp fields use supported formats (ISO 8601)
**High Latency:**
* For Kafka: Check consumer lag and partition count
* For Data Warehouses: Optimize queries, add indexes
* For S3: Use Parquet or ORC instead of CSV
**Data Quality Issues:**
* Enable data validation rules before ingestion
* Set up alerts for out-of-range values
* Configure automatic PII redaction
## Related Integrations
* [**Cloud Platforms**](/integrations/cloud-platforms-and-deployment/cloud-platforms) - Deploy Fiddler on AWS, Azure, GCP
* [**ML Platforms**](/integrations/ml-platforms-and-tools/ml-platforms) - Integrate with Databricks, MLflow
* [**Agentic AI**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) - Monitor LangGraph and Strands Agents
* [**Monitoring & Alerting**](/integrations/monitoring-and-alerting/monitoring-alerting) - Send alerts to incident management tools
***
# Apache Airflow
Source: https://docs.fiddler.ai/integrations/data-platforms/airflow-integration
Discover how to integrate Fiddler with an Airflow DAG for your ML pipeline, enabling you to train, manage, onboard models, and monitor performance seamlessly.
Apache Airflow is an open source ETL platform to manage complex
workflows. Companies are increasingly integrating their ML models pipeline into Airflow DAGs to manage and monitor all the components of their ML model system.
By integrating Fiddler into an existing Airflow DAG, you will be able to train, manage, and onboard your models while actively monitoring performance, data quality, and troubleshooting degradations across your models.
Fiddler can be easily integrated into your existing airflow DAG for ML model pipeline. A notebook which is used for publishing events can be orchestrated to run as a part of your airflow DAG using a ‘Papermill Operator’.
### Steps for the walkthrough
1. Setup airflow on your local or docker, these steps can be followed. [Link](https://airflow.apache.org/docs/apache-airflow/stable/start/index.html)
2. Add your jupyter notebook containing the code for publishing to your airflow home directory. In this example we will use the 2 different notebooks -
a. [Notebook to onboard ML model to Fiddler platform](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/integration-examples/airflow/notebooks/Fiddler_Churn_Add_Model.ipynb)
b. [Notebook to push production events to Fiddler platform](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/integration-examples/airflow/notebooks/Fiddler_Churn_Event_Publishing.ipynb)
3. Add an orchestration code to your airflow directory, airflow will pick up the orchestration code and construct a DAG as defined. The orchestration code contains the ‘papermill operator’ to orchestrate the jupyter notebooks which will be used to onboard models and publish events to Fiddler. Please refer to our [orchestration code](https://github.com/fiddler-labs/fiddler-examples/tree/main/integration-examples/airflow/DAGs).
4. The run interval can be set up in orchestration code as ‘schedule\_interval’ in the DAG class. This interval can be based on the frequency of training and inference of your ML model.
5. Once the DAGs are set up it can be monitored on the UI. Below we can see dummy DAGs have been set up with placeholder nodes for ‘data preparation ETL’ and ‘model training/inference’. We have two DAGs -
a. To set up Fiddler model registration after preparing baseline data (training pipeline)
b. To publish events to Fiddler after data preparation and ML model inference (inference pipeline)
### Label Update
An important business use case is integrating Fiddler’s ‘Label Update’ as a part of your ML workflow using Airflow. Label update can be used to update the ground truth feature in your data. This can be done using the ‘publish\_event’ api, passing the event, event\_id parameters, and making the update\_event parameter as ‘True’.
The code to update label can be found in the [notebook](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/integration-examples/airflow/notebooks/Fiddler_Churn_Label_Update.ipynb)
This notebook can be integrated to run as a part of your airflow DAG using the [sample code](https://github.com/fiddler-labs/fiddler-examples/blob/main/integration-examples/airflow/DAGs/fiddler_event_update.py)
### Papermill Operator
```
operator_var = PapermillOperator(
task_id="task_name",
input_nb="input_jupyter_notebook",
output_nb="output_jupyter_notebook",
parameters={"variable_1": "{{ value }}"},
)
```
### Airflow DAG
Below is an example of Model Registration Airflow DAG run history
Model Registration Airflow DAG flow
# BigQuery
Source: https://docs.fiddler.ai/integrations/data-platforms/bigquery-integration
Discover BigQuery integration with Fiddler. Learn how to load ML data from BigQuery tables and use it for tasks like publishing production data to Fiddler.
### Integrating Fiddler with BigQuery
Learn how to connect Fiddler's ML monitoring platform with your BigQuery data to enable comprehensive model tracking and analysis. This guide covers:
* Using BigQuery data to onboard models in Fiddler
* Loading baseline datasets from BigQuery tables
* Monitoring production data by connecting BigQuery to Fiddler
#### Prerequisites:
* A Google Cloud account with BigQuery access
* Fiddler account [credentials](/reference/administration/settings#credentials)
* [Python Client SDK](/sdk-api/python-client/connection) installed
### Configure BigQuery Access
Before importing data, you'll need to set up BigQuery API access and authentication:
1. In the GCP platform, Go to the navigation menu -> click APIs & Services. Once you are there, click + Enable APIs and Services (Highlighted below). In the search bar, enter BigQuery API and click Enable.
2. In order to make a request to the API enabled in Step #1, you need to create a service account and get an authentication file for your Jupyter Notebook. To do so, navigate to the Credentials tab under APIs and Services console and click Create Credentials tab, and then Service account under dropdown.
3. Enter the Service account name and description and click Done. You can use the BigQuery Admin role under Grant this service account access to the project. You can now see the new service account under the Credentials screen. Click the pencil icon beside the new service account you have created and click Add Key to add auth key. Please choose JSON and click CREATE. It will download the JSON file with auth key info. (Download path will be used to authenticate)
### Connect to BigQuery Data
Set up your Python environment and connect to BigQuery:
1. Install Required Packages
```bash theme={null}
pip install google-cloud google-cloud-bigquery[pandas] google-cloud-storage
```
2. Configure Authentication
```python theme={null}
# Set environment variables for your notebook
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path/to/your/auth-key.json'
```
3. Initialize BigQuery Client
```python theme={null}
# Imports google cloud client library and initiates BQ service
from google.cloud import bigquery
bigquery_client = bigquery.Client()
```
4. Query Your Data
```python theme={null}
# Example query to fetch baseline data
query = """
SELECT * FROM `fiddler-bq.fiddler_test.churn_prediction_baseline`
"""
# Execute query and load into pandas DataFrame
baseline_df = bigquery_client.query(query).to_dataframe()
```
#### Next Steps
Now that you've connected BigQuery to Fiddler, you can:
1. [Onboard a model](/developers/python-client-guides/model-onboarding/create-a-project-and-model) using the baseline dataset for the model schema inference sample.
2. [Upload a Baseline dataset](/developers/python-client-guides/publishing-production-data/creating-a-baseline-dataset), which is optional but recommended for monitoring comparisons.
3. [Publish production events](/developers/client-library-reference/publishing-production-data) for continuous monitoring.
# Amazon S3
Source: https://docs.fiddler.ai/integrations/data-platforms/integration-with-s3
Effortlessly extract AWS S3 data for model onboarding and inference publishing to Fiddler for monitoring.
This guide explains how to integrate AWS S3 with Fiddler to retrieve baseline or production data for model monitoring. You'll learn how to:
* Extract data from S3 buckets using different authentication methods
* Load data efficiently based on your needs
* Connect the extracted data with Fiddler's monitoring capabilities
### How to Integrate Fiddler with AWS S3
#### Prerequisites
Before getting started, ensure you have:
* An AWS account with access to the required S3 bucket
* Required Python packages installed: boto3, pandas, and fiddler-client
* Appropriate AWS credentials or profile configuration
* Basic familiarity with Python and AWS S3 concepts
### AWS Authentication Methods
#### Method 1: Using AWS Access Keys
If you're using AWS access keys for authentication, use this approach:
```python theme={null}
import boto3
import pandas as pd
# AWS Configuration
S3_BUCKET = 'your_bucket_name'
S3_FILENAME = 'path/to/your/file.csv'
AWS_ACCESS_KEY_ID = 'your_access_key'
AWS_SECRET_ACCESS_KEY = 'your_secret_key'
AWS_REGION = 'your_region'
# Create AWS session
session = boto3.session.Session(
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
region_name=AWS_REGION,
)
# Initialize S3 client
s3 = session.client('s3')
# Read data into pandas DataFrame
s3_data = s3.get_object(Bucket=S3_BUCKET, Key=S3_FILENAME)['Body']
df = pd.read_csv(s3_data)
```
#### Method 2: Using AWS Profiles (Recommended)
For enhanced security, we recommend using AWS profiles instead of hardcoding credentials:
```python theme={null}
import boto3
import pandas as pd
# Configuration
S3_BUCKET = 'your_bucket_name'
S3_FILENAME = 'path/to/your/file.csv'
AWS_PROFILE = 'your_profile_name'
# Create session using profile
session = boto3.session.Session(profile_name=AWS_PROFILE)
s3 = session.client('s3')
# Read data
s3_data = s3.get_object(Bucket=S3_BUCKET, Key=S3_FILENAME)['Body']
df = pd.read_csv(s3_data)
```
### Data Loading Options
#### Option 1: Direct Memory Loading
For smaller datasets that fit in memory, load directly into a pandas DataFrame as shown in the examples above.
#### Option 2: File System Loading
For larger datasets or when memory constraints exist, save to disk first:
```python theme={null}
import boto3
# AWS Configuration
S3_BUCKET = 'your_bucket_name'
S3_FILENAME = 'path/to/your/file.csv'
OUTPUT_PATH = 'local/path/to/output.csv'
# Initialize S3 client (using either authentication method)
session = boto3.session.Session(profile_name='your_profile_name')
s3 = session.client('s3')
# Download file
s3.download_file(
Bucket=S3_BUCKET,
Key=S3_FILENAME,
Filename=OUTPUT_PATH
)
```
### Using AWS S3 Data with Fiddler
#### For Baseline Datasets
After loading your data, you can use it to create a baseline dataset in Fiddler. See the [Creating a Baseline Dataset](/developers/python-client-guides/publishing-production-data/creating-a-baseline-dataset) guide for more details.
```python theme={null}
import fiddler as fdl
# Assumes an initialized Python client session and instantiated Model
job = model.publish_batch(
source=s3_data_df,
environment=fdl.EnvType.PRE_PRODUCTION,
dataset_name='your_baseline_name',
)
print(
f'Initiated pre-production dataset upload with Job ID = {job.id}'
)
```
#### For Production Traffic
To publish production data for monitoring. Refer to the [batch publishing guide](/developers/python-client-guides/publishing-production-data/publishing-batches-of-events) for more details. For more publishing options, see the additional publishing guides located [here](/developers/client-library-reference/publishing-production-data).
```python theme={null}
import fiddler as fdl
# Assumes an initialized Python client session and instantiated Model
job = model.publish_batch(
source=s3_data_df,
environment=fdl.EnvType.PRODUCTION,
)
print(
f'Initiated Production dataset upload with Job ID = {job.id}'
)
```
#### Best Practices
* Always use AWS profiles instead of hardcoded credentials in production environments
* Implement proper error handling around S3 operations
* Consider data size when choosing between memory and file system loading
* Use appropriate AWS IAM roles and permissions
* Monitor memory usage when working with large datasets
# Apache Kafka
Source: https://docs.fiddler.ai/integrations/data-platforms/kafka-integration
Dive into Fiddler’s Kafka connector services. Learn about prerequisites, installation, and limitations to manage production events and publish them to Fiddler.
Fiddler Kafka connector is an optional Fiddler service that connects to a [Kafka topic](https://kafka.apache.org/documentation/#intro_concepts_and_terms) containing production events for a model, and publishes the events to Fiddler.
### Kafka Integration Pre-requisites
We assume that the user has an account with Fiddler, has already created a project, and onboarded a model.
#### Installation
For Fiddler on-premises installations, the Kafka connector runs on Kubernetes in your environment. It is packaged as a Helm chart for quick installation:
```shell theme={null}
helm repo add fiddler https://helm.fiddler.ai/stable/
helm repo update
kubectl -n kafka create secret generic fiddler-credentials --from-literal=auth=
helm install fiddler-kafka fiddler/fiddler-kafka
--devel
--namespace kafka
--set fiddler.url=https://
--set fiddler.org=
--set fiddler.project_id=
--set fiddler.model_id=
--set fiddler.ts_field=timestamp
--set fiddler.ts_format=INFER
--set kafka.host=kafka
--set kafka.port=9092
--set kafka.topic=
--set kafka.security_protocol=SSL
--set kafka.ssl_cafile=cafile
--set kafka.ssl_certfile=certfile
--set kafka.ssl_keyfile=keyfile
--set-string kafka.ssl_check_hostname=False
```
This creates a deployment that reads event data from the Kafka topic and publishes it to the configured Fiddler model. The deployment can be scaled as needed; however, note that if the Kafka topic is not partitioned, scaling will not result in any gains.
#### Limitations
1. The connector assumes that there is a single dedicated topic containing production events for a given model. Multiple deployments can be created, one for each model, and scaled independently.
2. The connector assumes that events are published as JSON-serialized dictionaries of key-value pairs. Support for other formats can be added on request. As an example, a Kafka message should look like the following:
```json theme={null}
{
“feature_1”: 20.7,
“feature_2”: 45000,
“feature_3”: true,
“output_column”: 0.79,
“target_column”: 1,
“ts”: 1637344470000,
}
```
# SageMaker Pipelines
Source: https://docs.fiddler.ai/integrations/data-platforms/sagemaker-integration
Learn how integrating SageMaker with Fiddler simplifies model monitoring. Explore our guide on using AWS Lambda with the Fiddler Python client.
## Amazon SageMaker Integration
### Introduction
Integrate Amazon SageMaker with Fiddler to monitor your deployed models effectively. This guide shows you how to create an AWS Lambda function that uses the Fiddler Python client to process SageMaker inference logs from Amazon S3 and send them to your Fiddler instance. This integration provides real-time monitoring capabilities and valuable insights into your model's performance and behavior.
Fiddler AI Observability Platform is now available within Amazon SageMaker AI in SageMaker Unified Studio. This native integration lets SageMaker customers monitor ML models privately and securely without leaving the SageMaker environment.
Learn more about the Amazon SageMaker AI with Fiddler native integration [here](https://www.fiddler.ai/blog/fiddler-delivers-native-enterprise-grade-ai-observability-to-amazon-sagemaker-ai-customers).
### Prerequisites
Before you begin, ensure you have:
1. An active SageMaker model with:
* Data capture enabled
* Inference logs saved to S3 in JSONL format
2. Access to a Fiddler environment
3. Your SageMaker model is onboarded to Fiddler (See the [ML Monitoring Quick Start Guide](/developers/quick-starts/simple-ml-monitoring))
4. Latest Fiddler Python client version
### Implementation Steps
#### 1. Configure SageMaker Data Capture
Ensure your SageMaker endpoint has data capture properly configured:
1. Open the SageMaker console
2. Navigate to your model endpoint
3. Verify data capture is enabled and configured to save to your S3 bucket
4. Confirm captured data is in JSONL format
#### 2. Create an AWS Lambda Function
1. Open the AWS Lambda console
2. Click "Create function"
3. Configure the basic settings:
* Name your function (for example, "fiddler-sagemaker-integration")
* Select Python 3.10 or later as the runtime
* Choose execution permissions that allow S3 access
#### 3. Set Up Environment Variables
Configure these environment variables in your Lambda function:
| Variable | Description | Example |
| -------------------- | ------------------------------------------ | ------------------------------------ |
| `FIDDLER_URL` | Your Fiddler environment URL | `https://your_company.fiddler.ai` |
| `FIDDLER_TOKEN` | Your Fiddler authorization token | *(secure token value)* |
| `FIDDLER_MODEL_UUID` | Your model's unique identifier in Fiddler | 8a86cc43-71c1-49e7-a01b-d98ae91975bb |
| `MODEL_COLUMNS` | Comma-separated list of input column names | `feature1,feature2,feature3` |
| `MODEL_OUTPUT` | Name of the model output column | `prediction` |
| `MODEL_TIMESTAMP` | Name of the timestamp column (optional) | `event_time` |
If you provisioned Fiddler via the [SageMaker AI marketplace](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps.html), add these additional variables:
* `AWS_PARTNER_APP_AUTH`: Set to `True`
* `AWS_PARTNER_APP_ARN`: The ARN of your SageMaker AI Fiddler instance
* `AWS_PARTNER_APP_URL`: The URL of your SageMaker AI Fiddler instance
#### 4. Configure S3 Trigger
Set up your Lambda to run automatically when new data arrives:
1. In the Lambda console, select your function
2. Choose the "Add trigger" option
3. Select "S3" as the trigger type
4. Configure these settings:
* Bucket: Select your SageMaker inference logs bucket
* Event type: "All object create events"
* Prefix: (Optional) Specify a path prefix if needed
* Suffix: `.jsonl` (to only process JSON Lines files)
#### 5. Add Lambda Function Code
Copy this code into your Lambda function editor:
```python theme={null}
import os
import json
import uuid
import boto3
import logging
from typing import Dict, List, Any
import fiddler as fdl
# Configure logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Load environment variables, customize to model and use case
url = os.getenv('FIDDLER_URL')
token = os.getenv('FIDDLER_TOKEN')
model_uuid = os.getenv('FIDDLER_MODEL_UUID')
model_columns = os.getenv('MODEL_COLUMNS')
model_output_column = os.getenv('MODEL_OUTPUT')
timestamp_column = os.getenv('MODEL_TIMESTAMP')
# Initialize AWS clients
s3_client = boto3.client('s3')
# Initialize Fiddler connection and Fiddler Model to receive events
fdl.init(url=url, token=token)
fiddler_model = fdl.Model.get(id_=model_uuid)
def get_all_columns():
# The types of columns needed when publishing depend on use case. Typically,
# you would expect to pass at least your model inputs and output(s) and often
# metadata such as IDs, dates, data segments, etc.
return model_columns.split(',') + [timestamp_column] + [model_output_column]
def process_jsonl_content(event_data: str) -> Dict[str, Any]:
input_data = event_data['captureData']['endpointInput']['data']
input_values = input_data.split(',') # Split the CSV string into a list
# Extract the model prediction from 'captureData/endpointOutput/data'
model_prediction = event_data['captureData']['endpointOutput']['data']
# Optionally, you can set your own timestamp value on the inference occurrence time,
# or let Fiddler default it to the time of publish.
timestamp_value = event_data['eventMetadata']['inferenceTime']
# Combine inputs and any metadata values with the output into a single row
all_values = input_values + [timestamp_value] + [model_prediction]
# Create dictionary using zip to pair column names with their values
return dict(zip(get_all_columns(), all_values))
def parse_sagemaker_log(log_file_path: str) -> List[Dict[str, Any]]:
try:
# Collect all events in a List, 1 per JSON-line in the file
event_rows = []
with open(log_file_path, 'r') as file:
for line in file:
event = json.loads(line.strip())
row = process_jsonl_content(event)
event_rows.append(row)
return {
'status': 'success',
'record_count': len(event_rows),
'data': event_rows
}
except json.JSONDecodeError as e:
logger.error(f'Error parsing JSONL content: {str(e)}')
raise
def publish_to_fiddler(inferences: List[Dict[str, Any]], model: fdl.Model):
# There are multiple options for publishing data to Fiddler, check
# the online documentation for batch, streaming, and REST API options.
# publish_stream() sends events synchronously; for large datasets
# consider publish_batch() with a file path or DataFrame instead.
event_ids = model.publish_stream(
events=inferences,
environment=fdl.EnvType.PRODUCTION
)
return event_ids
def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
# Process each record in the event, streaming to Fiddler in batches
for record in event['Records']:
# Extract bucket and key information
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
logger.info(f'Processing new file: {key} from bucket: {bucket}')
# Persist log file to a temporary location
tmp_key = key.replace('/', '')
download_path = f'/tmp/{uuid.uuid4()}{tmp_key}'
s3_client.download_file(bucket, key, download_path)
# Retrieve the inference event(s) from the log file
results = parse_sagemaker_log(download_path)
# Check if the log file was processed successfully
if results['status'] != 'success':
logger.error(f'Error processing log file: {key}')
return {
'statusCode': 500,
'body': {'message': 'Error processing log file', 'results': results},
}
# Push the inference events to Fiddler
event_ids = publish_to_fiddler(results["data"], fiddler_model)
logger.info(f'Published events to Fiddler with ID(s): {event_ids}')
return {
'statusCode': 200,
'body': {'message': 'Successfully processed events', 'results': results},
}
```
# Snowflake
Source: https://docs.fiddler.ai/integrations/data-platforms/snowflake-integration
Learn how to extract baseline or production data from Snowflake for model onboarding and publishing production data to Fiddler for ML and LLM monitoring.
In this article, we will be looking at loading data from Snowflake tables and using the data for the following tasks:
1. Onboarding a model to Fiddler
2. Uploading baseline data to Fiddler
3. Publishing production data to Fiddler
## Import data from Snowflake
In order to import data from Snowflake to a Jupyter notebook, we will use the snowflake library which can be installed using the following command in your Python environment.
```bash theme={null}
pip install snowflake-connector-python
```
The following information is required in order to establish a connection to Snowflake:
* Snowflake Warehouse
* Snowflake Role
* Snowflake Account
* Snowflake User
* Snowflake Password
These values can be obtained from your Snowflake account under the ‘Admin’ option in the Menu as shown below or by running the queries below:
* Warehouse - select CURRENT\_WAREHOUSE()
* Role - select CURRENT\_ROLE()
* Account - select CURRENT\_ACCOUNT()
'User' and 'Password' are the same that you use when logging in to your Snowflake account.
Once you have this information, you can set up a Snowflake connector using the following code:
```python theme={null}
# establish Snowflake connection
connection = connector.connect(
user=snowflake_username,
password=snowflake_password,
account=snowflake_account,
role=snowflake_role,
warehouse=snowflake_warehouse
)
```
You can then write a custom SQL query and import the data to a pandas dataframe.
```python theme={null}
# sample SQL query
sql_query = 'select * from FIDDLER.FIDDLER_SCHEMA.CHURN_BASELINE LIMIT 100'
# create cursor object
cursor = connection.cursor()
# execute SQL query inside Snowflake
cursor.execute(sql_query)
baseline_df = cursor.fetch_pandas_all()
```
## Publish Production Events
Now that we have data imported from Snowflake to a dataframe, we can refer to the following pages to:
1. [Onboard a model](/developers/python-client-guides/model-onboarding/create-a-project-and-model) using the baseline dataset for the model schema inference sample.
2. [Upload a Baseline dataset](/developers/python-client-guides/publishing-production-data/creating-a-baseline-dataset), which is optional but recommended for monitoring comparisons.
3. [Publish production events](/developers/client-library-reference/publishing-production-data) for continuous monitoring.
# Integrations
Source: https://docs.fiddler.ai/integrations/index
Connect Fiddler to your AI stack with native SDKs, cloud platform integrations, data pipeline connectors, and framework support. From agentic workflows to traditional ML models, Fiddler integrates with the tools you already use.
## Quick Start
Choose your integration path based on what you're building:
* [**Monitor LangGraph Agents →**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) Auto-instrument agentic applications with the LangGraph SDK
* [**Deploy on AWS →**](/integrations/cloud-platforms-and-deployment/cloud-platforms) Run Fiddler as a Partner AI App in SageMaker
* [**Connect Your Data →**](/integrations/data-platforms-and-pipelines/data-platforms) Integrate with Snowflake, BigQuery, S3, and more
## Integration Categories
### 🎯 Agentic AI & LLM Frameworks
Native instrumentation for agentic workflows and LLM applications—Fiddler's competitive differentiator.
**Native SDKs:**
* [Fiddler LangGraph SDK](/integrations/agentic-ai/langgraph-sdk) - Auto-instrument LangGraph agents with OpenTelemetry
* [Fiddler Strands SDK](/integrations/agentic-ai/strands-sdk) - Monitor Strands Agents
* [Fiddler Evals SDK](/integrations/agentic-ai/evals-sdk) - LLM experiments framework with pre-built evaluators
**Gateways & Proxies:**
* [AI Gateway Integrations](/integrations/agentic-ai/ai-gateways) - Overview of using Kong, AgentGateway, or LiteLLM as a Fiddler observability and guardrail proxy, with a capability comparison
* [LiteLLM Integration](/integrations/agentic-ai/litellm-integration) - Trace LLM calls, cost, and latency across 100+ providers via the LiteLLM SDK or proxy
* [Kong AI Gateway](/integrations/agentic-ai/kong-integration) - Monitor LLM traffic routed through Kong
* [AgentGateway](/integrations/agentic-ai/agentgateway-integration) - Observe agent traffic through AgentGateway
**Platform Access:**
* Python Client SDK - Core SDK client for ML/LLM monitoring
* [REST API](/sdk-api/rest-api/alert-rules) - Complete HTTP API for all platform features
[**View all agentic AI integrations →**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai)
### ☁️ Cloud Platforms & Deployment
Deploy Fiddler on your preferred cloud infrastructure.
**Featured:**
* [AWS SageMaker Partner AI App](/integrations/cloud-platforms-and-deployment/aws-sagemaker) - Fully managed deployment in SageMaker
**Data Storage:**
* [Amazon S3](/integrations/data-platforms/integration-with-s3) - S3 bucket integration for data ingestion
* [Google BigQuery](/integrations/data-platforms/bigquery-integration) - BigQuery connector for ML pipelines
[**View all cloud integrations →**](/integrations/cloud-platforms-and-deployment/cloud-platforms)
### 🗄️ Data Platforms & Pipelines
Connect Fiddler to your data infrastructure for seamless model monitoring.
**Data Warehouses:**
* [Snowflake](/integrations/data-platforms/snowflake-integration) - Snowflake connector for production data
* [BigQuery](/integrations/data-platforms/bigquery-integration) - Google BigQuery integration
**Data Streaming:**
* [Apache Kafka](/integrations/data-platforms/kafka-integration) - Real-time streaming data integration
**Orchestration:**
* [Apache Airflow](/integrations/data-platforms/airflow-integration) - Workflow automation for model deployments
[**View all data integrations →**](/integrations/data-platforms-and-pipelines/data-platforms)
### 🤖 ML Platforms & Tools
Integrate with your existing ML infrastructure and experiment tracking tools.
**MLOps Platforms:**
* [Databricks](/integrations/ml-platforms/databricks-integration) - Databricks ML integration via MLflow and Spark
* [MLflow](/integrations/ml-platforms/ml-flow-integration) - Model registry and experiment tracking
[**View all ML platform integrations →**](/integrations/ml-platforms-and-tools/ml-platforms)
### 📡 Monitoring & Alerting
Route alerts and metrics to your observability stack.
**Observability Platforms:**
* [Datadog](/integrations/monitoring-alerting/datadog-integration) - Export metrics to Datadog for centralized monitoring
**Incident Management:**
* [PagerDuty](/integrations/monitoring-alerting/pagerduty) - Alert routing and on-call management
**Notifications:**
* Webhooks - Generic webhook support for custom integrations
* Email Alerts - Built-in email notification system
[**View all monitoring integrations →**](/integrations/monitoring-and-alerting/monitoring-alerting)
### 🔐 Authentication & Access
Enterprise SSO and authentication providers for secure access control.
**Enterprise SSO:**
* [Okta Integration](/reference/access-control/okta-integration) - Enterprise single sign-on
* [Google Workspace](/reference/access-control/google-integration) - Google SSO integration
**API Authentication:**
* API Keys - Token-based API authentication
* OAuth - OAuth 2.0 flow support
[**View all authentication options →**](/reference/access-control)
## Integration Types
* **🏆 Native SDK** - Built and maintained by Fiddler; included with platform
* **🤝 Partner Integration** - Built in collaboration with technology partner
* **🔌 Connector** - Pre-built component for data source integration
* **📚 Framework Guide** - Code examples and patterns for specific frameworks
## What's Next?
* **Getting Started:** Follow our [quick start guides](/developers) to set up your first integration
* **SDK Documentation:** Dive into detailed [SDK API reference](/sdk-api) documentation
***
**Ready to integrate?** Choose a category above to explore specific integrations and get started in minutes.
# ML Platforms Overview
Source: https://docs.fiddler.ai/integrations/ml-platforms-and-tools/ml-platforms
Integrate Fiddler with MLOps platforms, experiment tracking tools, and ML frameworks
Integrate Fiddler into your MLOps workflow to monitor models across the entire machine learning lifecycle. From experiment tracking to production deployment, Fiddler works with the ML platforms you already use.
## Why ML Platform Integrations Matter
Modern ML teams use sophisticated platforms for experimentation, training, and deployment. Fiddler's integrations ensure you can:
* **Unified Model Governance** - Track models from experiment to production in one platform
* **Automated Monitoring Setup** - Auto-configure monitoring when models are registered
* **Seamless Workflow Integration** - Add observability without changing existing processes
* **Bi-Directional Sync** - Share metrics between Fiddler and your ML platform
* **Experiment Comparison** - Compare production performance against training experiments
## MLOps Platform Integrations
### Databricks
Integrate Fiddler with Databricks for unified ML development and monitoring.
**Why Databricks + Fiddler:**
* **Lakehouse Architecture** - Monitor models trained on Delta Lake data
* **MLflow Integration** - Automatic sync of registered models to Fiddler
* **Notebook Integration** - Use Fiddler SDK directly in Databricks notebooks
* **Production Monitoring** - Monitor models served via Databricks Model Serving
**Key Features:**
* **Automatic Model Registration** - Models registered in Databricks MLflow automatically appear in Fiddler
* **Feature Store Integration** - Monitor drift using Databricks Feature Store definitions
* **Collaborative Debugging** - Share Fiddler insights in Databricks notebooks
* **Unified Data Access** - Use Delta Lake as data source for baselines and production data
[**Get Started with Databricks →**](/integrations/ml-platforms/databricks-integration)
**Quick Start:**
```python theme={null}
from databricks import mlflow as dbx_mlflow
from fiddler import FiddlerClient
# Register model in Databricks MLflow
model_uri = "models:/credit_risk_model/Production"
model_version = dbx_mlflow.register_model(model_uri, "credit_risk_model")
# Automatically sync to Fiddler
client = FiddlerClient(api_key="fid_...")
client.sync_from_databricks(
model_name="credit_risk_model",
version=model_version,
enable_monitoring=True
)
```
### MLflow
Connect Fiddler to MLflow for experiment tracking and model registry integration.
**Why MLflow + Fiddler:**
* **Open-Source Standard** - Works with any MLflow deployment (Databricks, AWS, GCP, self-hosted)
* **Model Registry Sync** - Automatically monitor models when they transition to "Production"
* **Experiment Tracking** - Compare production metrics with training experiment metrics
* **Model Versioning** - Track performance across model versions
**Key Features:**
* **Automatic Model Onboarding** - Models in MLflow registry auto-configure in Fiddler
* **Metric Synchronization** - Export Fiddler metrics back to MLflow for unified view
* **Artifact Integration** - Link model artifacts between MLflow and Fiddler
* **Stage-Based Monitoring** - Different monitoring configs for Staging vs Production
[**Get Started with MLflow →**](/integrations/ml-platforms/ml-flow-integration)
**Quick Start:**
```python theme={null}
import mlflow
from fiddler import FiddlerClient
# Set MLflow tracking URI
mlflow.set_tracking_uri("https://mlflow.example.com")
# Configure Fiddler to sync with MLflow
client = FiddlerClient(api_key="fid_...")
client.add_mlflow_integration(
tracking_uri="https://mlflow.example.com",
auto_sync_on_stage_transition=True,
stages=["Production", "Staging"]
)
# Models transitioning to "Production" will automatically be monitored in Fiddler
```
## Experiment Tracking & Model Registry
### Unified Model Lifecycle
Track models from experimentation through production:
```
Experiment → Training → Registration → Staging → Production → Monitoring
↓ ↓ ↓ ↓ ↓ ↓
MLflow MLflow MLflow MLflow MLflow Fiddler
Runs Runs Registry Registry Registry + MLflow
```
**Integration Benefits:**
* **Single Source of Truth** - MLflow registry as canonical model inventory
* **Automated Workflows** - Monitoring setup triggered by model registration
* **Version Comparison** - Compare production metrics across model versions
* **Rollback Readiness** - Quick rollback with historical performance data
### Experiment-to-Production Comparison
Compare production model performance against training experiments:
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Get production metrics
prod_metrics = client.get_metrics(
project="fraud-detection",
model="fraud_model_v3",
start_time="2024-11-01",
end_time="2024-11-10"
)
# Compare with training experiment (from MLflow)
experiment_metrics = client.get_experiment_metrics(
mlflow_experiment_id="exp_12345",
mlflow_run_id="run_67890"
)
# Generate comparison report
report = client.compare_metrics(
production=prod_metrics,
experiment=experiment_metrics,
metrics=["accuracy", "precision", "recall", "auc"]
)
```
## ML Framework Support
While Fiddler is framework-agnostic, we provide enhanced support for popular ML frameworks:
### Supported ML Frameworks
**Classical ML:**
* **Scikit-Learn** - Full support for all estimators
* **XGBoost** - Native explainability for tree models
* **LightGBM** - Fast SHAP explanations
* **CatBoost** - Categorical feature support
**Deep Learning:**
* **TensorFlow/Keras** - Model analysis and monitoring
* **PyTorch** - Dynamic graph model support
* **JAX** - High-performance model monitoring
* **ONNX** - Framework-agnostic model format
**AutoML:**
* **H2O.ai** - AutoML model monitoring
* **AutoGluon** - Tabular model support
* **TPOT** - Pipeline optimization monitoring
### Framework-Specific Features
**Tree-Based Models (XGBoost, LightGBM, CatBoost):**
* Fast SHAP explanations using native implementations
* Feature importance tracking over time
* Tree structure analysis for debugging
**Deep Learning (TensorFlow, PyTorch):**
* Layer-wise activation monitoring
* Embedding drift detection
* Custom metric support for complex architectures
**Example - XGBoost Monitoring:**
```python theme={null}
import xgboost as xgb
from fiddler import FiddlerClient
# Train XGBoost model
model = xgb.XGBClassifier()
model.fit(X_train, y_train)
# Upload to Fiddler with automatic feature importance
client = FiddlerClient(api_key="fid_...")
client.upload_model(
project="credit-risk",
model_name="xgb_risk_model",
model=model,
task="binary_classification",
enable_shap=True # Native XGBoost SHAP support
)
```
## Integration Architecture Patterns
### Pattern 1: MLflow-Centric Workflow
Use MLflow as the central hub for all ML operations:
```
Data Preparation
↓
Experimentation (MLflow Tracking)
↓
Model Registry (MLflow)
↓ (webhook on stage transition)
Fiddler Auto-Onboarding
↓
Production Monitoring (Fiddler + MLflow metrics export)
```
**Configuration:**
```python theme={null}
# One-time setup: Configure MLflow webhook
client = FiddlerClient(api_key="fid_...")
client.configure_mlflow_webhook(
mlflow_tracking_uri="https://mlflow.example.com",
webhook_secret="webhook_secret_key",
on_stage_transition={
"Production": "enable_full_monitoring",
"Staging": "enable_basic_monitoring",
"Archived": "disable_monitoring"
}
)
```
### Pattern 2: Databricks Unity Catalog Integration
Leverage Databricks Unity Catalog for governance and Fiddler for monitoring:
```
Unity Catalog (Model Registry)
↓
Databricks Model Serving
↓
Production Traffic
↓ (streaming predictions)
Fiddler Monitoring
↓
Alerts → Databricks Workflow Jobs (retraining)
```
**Configuration:**
```python theme={null}
# Connect Fiddler to Unity Catalog
client = FiddlerClient(api_key="fid_...")
client.add_unity_catalog_integration(
workspace_url="https://dbc-xxxxx.cloud.databricks.com",
catalog="ml_models",
schema="production",
access_token=dbutils.secrets.get("fiddler", "databricks_token")
)
```
### Pattern 3: Multi-Platform Model Tracking
Monitor models across multiple ML platforms:
```
Training Platform Mix:
├── Databricks (Lakehouse models)
├── SageMaker (AWS-native models)
├── Vertex AI (GCP models)
└── On-Premises (Legacy models)
↓ (all models sync to)
Fiddler (Unified Monitoring)
```
## Getting Started
### Prerequisites
* **Fiddler Account** - Cloud or on-premises deployment
* **ML Platform Access** - Databricks workspace or MLflow server
* **Credentials** - Fiddler access token + ML platform credentials
* **Network Connectivity** - Firewall rules for integration
### General Setup Steps
**1. Configure ML Platform Connection**
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(
api_key="fid_...",
url="https://app.fiddler.ai"
)
# Add ML platform integration
client.add_integration(
type="databricks", # or "mlflow"
config={
"workspace_url": "https://dbc-xxxxx.cloud.databricks.com",
"access_token": "dapi...",
"auto_sync": True
}
)
```
**2. Sync Existing Models (Optional)**
```python theme={null}
# One-time sync of existing models
models = client.sync_models_from_platform(
platform="databricks",
filter={"stage": "Production"}
)
print(f"Synced {len(models)} models to Fiddler")
```
**3. Enable Auto-Monitoring**
```python theme={null}
# Future model registrations will automatically be monitored
client.configure_auto_monitoring(
platform="databricks",
enabled=True,
default_config={
"enable_drift_detection": True,
"enable_performance_tracking": True,
"enable_explainability": True
}
)
```
## Advanced Integration Features
### Feature Store Integration
Monitor models using features from Databricks Feature Store:
```python theme={null}
from databricks.feature_store import FeatureStoreClient
fs = FeatureStoreClient()
# Create feature spec
feature_spec = fs.create_feature_spec(
table_name="ml_features.user_features",
primary_keys=["user_id"]
)
# Monitor model with feature store schema
client.upload_model(
project="recommendations",
model="user_model",
feature_spec=feature_spec, # Auto-generates schema from Feature Store
enable_drift_detection=True
)
```
### Automated Retraining Triggers
Trigger retraining workflows when drift is detected:
```python theme={null}
# Configure alert to trigger Databricks job
client.create_alert(
name="High Drift - Retrain Model",
trigger_type="drift",
threshold=0.15,
model="credit_risk_model",
actions=[{
"type": "databricks_job",
"job_id": "12345",
"parameters": {
"model_name": "credit_risk_model",
"reason": "drift_detected"
}
}]
)
```
### Model Lineage Tracking
Track complete model lineage from data to deployment:
```python theme={null}
# Capture full model lineage
lineage = {
"data_source": "s3://bucket/training-data-v2.parquet",
"feature_transformations": "feature_pipeline_v1",
"training_framework": "xgboost==1.7.0",
"mlflow_run_id": "run_67890",
"parent_model": "credit_risk_model_v1"
}
client.update_model_metadata(
project="credit-risk",
model="credit_risk_model_v2",
lineage=lineage
)
```
## Integration Selector
Choose the right ML platform integration for your workflow:
| Your ML Platform | Recommended Integration | Why |
| -------------------- | -------------------------- | ------------------------------------------- |
| Databricks Lakehouse | **Databricks integration** | Native MLflow, Unity Catalog, Feature Store |
| Self-hosted MLflow | **MLflow integration** | Open-source, cloud-agnostic |
| AWS SageMaker | **SageMaker Pipelines** | AWS-native, Partner AI App compatible |
| Azure ML | **MLflow integration** | Azure ML uses MLflow under the hood |
| Vertex AI (GCP) | **MLflow integration** | Vertex AI supports MLflow |
| Multiple platforms | **MLflow integration** | Universal compatibility |
## Bi-Directional Metric Sync
Share metrics between Fiddler and your ML platform:
### Export Fiddler Metrics to MLflow
```python theme={null}
# Log Fiddler metrics to MLflow experiments
client.export_metrics_to_mlflow(
fiddler_project="fraud-detection",
fiddler_model="fraud_model_v3",
mlflow_experiment_name="production_monitoring",
metrics=["drift_score", "accuracy", "f1_score"],
time_range="last_7_days"
)
```
### Import MLflow Metrics to Fiddler
```python theme={null}
# Import custom metrics from MLflow
client.import_metrics_from_mlflow(
mlflow_run_id="run_67890",
fiddler_project="fraud-detection",
fiddler_model="fraud_model_v3",
metrics=["custom_business_metric", "validation_loss"]
)
```
## Security & Access Control
### Authentication Methods
**Databricks:**
* Personal Access Tokens (development)
* Service Principal OAuth (production)
* Azure AD Integration (enterprise)
**MLflow:**
* HTTP Basic Authentication
* Token-Based Authentication
* Custom Auth Plugins
### Permission Requirements
**Databricks Permissions:**
* `CAN_MANAGE` on registered models
* `CAN_READ` on Feature Store tables
* `CAN_USE` on clusters (for SHAP computation)
**MLflow Permissions:**
* Read access to Model Registry
* Read access to Experiment Tracking
* Write access for metric export (optional)
## Monitoring MLOps Pipeline Health
### Track Integration Health
```python theme={null}
# Check integration status
status = client.get_integration_status("databricks")
print(f"Status: {status.connected}")
print(f"Last sync: {status.last_sync_time}")
print(f"Models synced: {status.models_count}")
```
### Alerts for Sync Failures
```python theme={null}
# Alert on integration failures
client.create_alert(
name="MLflow Sync Failure",
trigger_type="integration_error",
integration="mlflow",
notification_channels=["email", "slack"]
)
```
## Troubleshooting
### Common Issues
**Models Not Syncing:**
* Verify MLflow/Databricks credentials are valid
* Check network connectivity from Fiddler to ML platform
* Ensure models are in the correct stage (e.g., "Production")
* Validate webhook endpoint is reachable (for event-driven sync)
**Schema Mismatches:**
* Ensure feature names match between training and production
* Verify data types are consistent
* Check for missing features in production data
**Performance Issues:**
* For large models, use SHAP sampling instead of full computation
* Enable lazy loading for model artifacts
* Use incremental sync for model registry (don't sync all historical versions)
## Related Integrations
* [**Data Platforms**](/integrations/data-platforms-and-pipelines/data-platforms) - Connect to Snowflake, BigQuery for training data
* [**Cloud Platforms**](/integrations/cloud-platforms-and-deployment/cloud-platforms) - Deploy Fiddler on AWS, Azure, GCP
* [**Agentic AI**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) - Monitor LangGraph and LLM applications
* [**Monitoring & Alerting**](/integrations/monitoring-and-alerting/monitoring-alerting) - Alert on model issues
***
# Databricks
Source: https://docs.fiddler.ai/integrations/ml-platforms/databricks-integration
Discover how Fiddler helps monitor, explain, and analyze models in Databricks Workspace. Integrate with MLFlow and Spark to manage, validate, and monitor models.
Fiddler allows your team to monitor, explain and analyze your models developed and deployed in [Databricks Workspace](https://docs.databricks.com/introduction/index.html) by integrating with [MLflow](https://docs.databricks.com/mlflow/index.html) for model asset management and utilizing Databricks Spark environment for data management.
To validate and monitor models built on Databricks using Fiddler, you can follow these steps:
1. [Create a Fiddler model](/developers/python-client-guides/model-onboarding/create-a-project-and-model) using sample data or model information from MLflow
2. [Publish production data](/developers/client-library-reference/publishing-production-data) streaming live or in batches
#### Prerequisites
This guide assumes you have:
* A Databricks account and valid credentials
* A Fiddler environment with an account and valid credentials
* Know how to [connect and use ](/developers/python-client-guides/installation-and-setup)the [Fiddler Python Client SDK](/sdk-api/python-client/connection)
#### Begin with a Databricks Notebook
Launch a [Databricks notebook](https://docs.databricks.com/notebooks/index.html) from your workspace and run the following code:
```bash theme={null}
pip install -q fiddler-client
```
```python theme={null}
import fiddler as fdl
```
Now that you have the Fiddler library installed, you can connect to your Fiddler environment. You will need your authentication token from the [Credentials](/reference/administration/settings#credentials) tab in Application Settings.
```python theme={null}
URL = ""
AUTH_TOKEN = ""
fdl.init(url=URL, token=AUTH_TOKEN)
```
Finally, you can set up a new [project](/sdk-api/python-client/connection) using:
```python theme={null}
# The project.id is required when creating models
project = fdl.Project(name='YOUR_PROJECT_NAME')
project.create()
```
#### Creating the Fiddler Model
**Quickest Option: Let Fiddler Automate Model Creation**
The quickest way to onboard a Fiddler model is to get a sample of data from which Fiddler can infer model schema and metadata. Ideally you will have baseline, testing, or training data that is representative of your model schema. Fiddler can infer your model schema from this sample dataset. You can download baseline or training data from a [delta table](https://docs.databricks.com/getting-started/dataframes-python.html) and share it with Fiddler as a baseline dataset:
```python theme={null}
sample_dataset = spark.read.table("YOUR_DATASET").select("*").toPandas()
```
Now that you have sample data, you can easily create a Fiddler model, as demonstrated in our [Simple Monitoring Quick Start Guide](/developers/quick-starts/simple-ml-monitoring). A rough outline of the steps follows:
```py theme={null}
# Define a ModelSpec that tells Fiddler what role each column
# in your model schema serves.
model_spec = fdl.ModelSpec(
inputs=['feature_input_column', ...],
outputs=['output_column'],
targets=['label_column'],
metadata=['id_column', 'data_segment_column', ...],
)
# Identify the task your ML model performs as Fiddler will use this
# to generate the performance metrics appropriate to the task.
# ModelTask.NOT_SET is also an option if performance metrics are not needed.
model_task = fdl.ModelTask.BINARY_CLASSIFICATION
task_params = fdl.ModelTaskParams(target_class_order=['no', 'yes'])
# Use Model.from_data() to define your model's ModelSchema automatically by
# passing the sample_dataset in the source parameter for schema inference.
model = fdl.Model.from_data(
name='name_for_display_in_Fiddler',
project_id=project.id,
source=sample_dataset,
spec=model_spec,
task=model_task,
task_params=task_params,
event_id_col='your_unique_event_id_column',
event_ts_col='event_timestamp_column'
)
# Create the model in Fiddler
model.create()
```
**Option: Using the MLflow Model Registry**
Another option is to manually construct your model's schema from the details contained in the MLflow registry. Using the MLflow API, you can query the model registry and get the model signature, which describes the inputs and outputs as a dictionary. You can use this dictionary to build the Model, ModelSchema, and ModelSpec objects that define the tabular schema of your model.
```python theme={null}
import mlflow
from mlflow.tracking import MlflowClient
# Initiate MLFlow Client
client = MlflowClient()
# Get the model URI
model_version_info = client.get_model_version(model_name, model_version)
model_uri = client.get_model_version_download_uri(model_name, model_version_info)
#Get the Model Signature
mlflow_model_info = mlflow.models.get_model_info(model_uri)
model_inputs_schema = mlflow_model_info.signature.inputs.to_dict()
model_inputs = [ sub['name'] for sub in model_inputs_schema ]
```
Refer to this [example notebook](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/examples/create_model_from_constructor.ipynb) in GitHub, which demonstrates manually defining your Fiddler model's schema.
#### Publishing Events
Now you can publish all the events from your models. You can do this in two ways:
**Batch Models**
If your models run batch processes with your models or your aggregate model outputs over a time frame, then you can use the table change feed from Databricks to select only the new events and send them to Fiddler:
```python theme={null}
import fiddler as fdl
from pyspark.sql import SparkSession
# Get the active Spark session
spark = SparkSession.builder.getOrCreate()
changes_df = (
spark.read.format("delta")
.option("readChangeFeed", "true")
.option("startingVersion", last_version)
.option("endingVersion", new_version)
.table("inferences")
.toPandas()
)
# Assumes an initialized Python client session and instantiated Model
job = model.publish_batch(
source=changes_df,
environment=fdl.EnvType.PRODUCTION,
)
print(f'Initiated Production dataset upload with Job ID = {job.id}')
```
**Live Models**
For models with live predictions or real-time applications, you can add the following code snippet to your prediction pipeline and send every event to Fiddler in real-time:
```python theme={null}
# Turn your model's output in a pandas dataframe
example_event = model_output.toJSON().map(lambda x: json.loads(x)).collect()
# Assumes an initialized Python client session and instantiated Model
event_id = model.publish_stream(
events=example_event,
environment=fdl.EnvType.PRODUCTION,
)
print(f'Published {event_id_list}')
```
# MLflow
Source: https://docs.fiddler.ai/integrations/ml-platforms/ml-flow-integration
Explore how Fiddler helps your team onboard, monitor, explain, and analyze models with MLFlow. Learn to ingest model metadata and artifacts for observability.
Fiddler allows your team to onboard, monitor, explain, and analyze your models developed with [MLflow](https://mlflow.org/).
This guide shows you how to ingest the model metadata and artifacts stored in your MLflow model registry and use them to set up model observability in the Fiddler Platform:
1. Exporting Model Metadata from MLflow to Fiddler
2. Uploading Model Artifacts to Fiddler for XAI
### Onboarding a Model
Refer to this [section](/integrations/ml-platforms/databricks-integration#creating-the-fiddler-model) of the Databricks integration guide for onboarding your model to Fiddler using model information from MLflow.
### Uploading Model Artifacts
Using the [**MLflow API**](https://mlflow.org/docs/latest/python_api/mlflow.html) you can query the model registry and get the **model signature** which describes the inputs and outputs as a dictionary.
#### Uploading Model Files
Sharing your model artifacts helps Fiddler explain your models. By leveraging the MLflow API you can download these model files:
```python theme={null}
import os
import mlflow
from mlflow.store.artifact.models_artifact_repo import ModelsArtifactRepository
model_name = "example-model-name"
model_stage = "Staging" # Should be either 'Staging' or 'Production'
mlflow.set_tracking_uri("databricks")
os.makedirs("model", exist_ok=True)
local_path = ModelsArtifactRepository(
f'models:/{model_name}/{model_stage}'
).download_artifacts("", dst_path="model")
print(f'{model_stage} Model {model_name} is downloaded at {local_path}')
```
Once you have the model file, you can create a package.py file in this model directory that describes how to access this model.
Finally, you can upload all the model artifacts to Fiddler:
```python theme={null}
# Assumes an initialized Python client session and instantiated Model
job = model.add_artifact(
model_dir=MODEL_ARTIFACTS_DIR,
)
job.wait()
```
Alternatively, you can skip uploading your model and use Fiddler to generate a surrogate model to get low-fidelity explanations for your model.
Please refer to the Explainability guide for detailed information on model artifacts, packages, and surrogate models.
# Datadog
Source: https://docs.fiddler.ai/integrations/monitoring-alerting/datadog-integration
Learn about Fiddler’s Datadog integration to bring AI Observability metrics into your dashboards. Follow steps to centralize ML model and application monitoring.
Fiddler offers an integration with Datadog which allows Fiddler and Datadog customers to bring their AI Observability metrics from Fiddler into their centralized Datadog dashboards. Additionally, a Fiddler license can now be procured through the [Datadog Marketplace](https://www.datadoghq.com/blog/tag/datadog-marketplace/). This [integration](https://www.datadoghq.com/blog/monitor-machine-learning-models-fiddler/) enables you to centralize your monitoring of ML models and the applications that utilize them within one unified platform.
### Integrating Fiddler with Datadog
Instructions for integrating Fiddler with Datadog can be found on the "Integrations" section of your Datadog console. Simply search for "Fiddler" and follow the installation instructions provided on the "Configure" tab. Please reach out to \<[support@fiddler.ai](mailto:support@fiddler.ai)> with any issues or questions.
# PagerDuty
Source: https://docs.fiddler.ai/integrations/monitoring-alerting/pagerduty
Discover how customer churn prediction works with Fiddler’s AI observability platform. Follow our example to detect and diagnose issues step by step.
Fiddler offers powerful alerting tools for monitoring models. By integrating with
PagerDuty services, you gain the ability to trigger PagerDuty events within your monitoring
workflow.
> 📘 If your organization has already integrated with PagerDuty, then you may skip to the [Setup: In Fiddler](#setup-in-fiddler) section to learn more about setting up PagerDuty within Fiddler.
### Setup: In PagerDuty
1. Within your PagerDuty Team, navigate to **Services** → **Service Directory**.
2. Within the Service Directory:
* If you are creating a new service for integration, select **+New Service** and follow the prompts to create your service.
* Click the **name of the service** you want to integrate with.
3. Navigate to **Integrations** within your service, and select **Add a new integration to this service**.
4. Enter an **Integration Name**, and under **Integration Type** select the option **Use our API directly**. Then, select the **Add Integration** button to save your new integration. You will be redirected to the Integrations page for your service.
5. Copy the **Integration Key** for your new integration.
### Setup: In Fiddler
1. Within **Fiddler**, navigate to the **Settings** page, and then to the **PagerDuty Integration** menu. If your organization **already has a PagerDuty service integrated with Fiddler**, you will be able to find it in the list of services.
2. If you are looking to integrate with a new service, select the **`+`** box on the top right. Then, enter the name of your service, as well as the Integration Key copied from the end of the [Setup: In PagerDuty](#setup-in-pagerduty) section above. After creation, confirm that your new entry is now in the list of available services.
> 🚧 Creating, editing, and deleting these services is an **ADMINISTRATOR**-only privilege. Please contact an **ADMINISTRATOR** within your organization to set up any new PagerDuty services
### PagerDuty Alerts in Fiddler
1. Within the **Projects** page, select the model you wish to use with PagerDuty.
2. Select **Monitor** → **Alerts** → **Add Alert**.
3. Enter the condition you would like to alert on, and under **PagerDuty Services**, select all services you would like the alert to trigger for. Additionally, select the **Severity** of this alert, and hit **Save**.
4. After creation, the alert will now trigger for the specified PagerDuty services.
> 📘 Info
>
> Check out the [alerts documentation](/developers/python-client-guides/alerts-with-fiddler-client) for more information on setting up alerts.
### FAQ
**Can Fiddler integrate with multiple PagerDuty services?**
* Yes. So long as the service is present within **Settings** → **PagerDuty Services**, anyone within your organization can select that service to be a recipient for an alert.
# Monitoring & Alerting Overview
Source: https://docs.fiddler.ai/integrations/monitoring-and-alerting/monitoring-alerting
Connect Fiddler alerts to incident management, observability, and communication tools
Route Fiddler AI observability alerts to your existing incident management and communication tools. Integrate with observability platforms, on-call systems, and team collaboration tools to ensure AI issues are detected, triaged, and resolved quickly.
## Why Alert Integration Matters
AI models fail in unique ways—drift, data quality issues, performance degradation, safety violations. Fiddler's alert integrations ensure your team responds immediately:
* **Unified Incident Management** - AI alerts flow into the same systems as infrastructure alerts
* **Faster Response Times** - On-call engineers notified via existing escalation policies
* **Context-Rich Alerts** - Model context, affected predictions, and root cause analysis included
* **Reduced Alert Fatigue** - Intelligent grouping and deduplication across tools
* **Automated Remediation** - Trigger workflows to rollback models or scale resources
## Integration Categories
### 📊 Observability Platforms
Send Fiddler metrics and alerts to enterprise observability platforms for unified monitoring.
**Supported Platforms:**
* [**Datadog**](/integrations/monitoring-alerting/datadog-integration) - Application performance monitoring and infrastructure observability ✓ **GA**
**Common Use Cases:**
* Correlate AI model issues with infrastructure metrics
* Build unified dashboards combining Fiddler + infrastructure data
* Use Datadog's anomaly detection on Fiddler metrics
* Alert on compound conditions (model drift + high latency)
### 🚨 Incident Management
Connect alerts to on-call systems for immediate engineer notification.
**Supported Platforms:**
* [**PagerDuty**](/integrations/monitoring-alerting/pagerduty) - Incident management and on-call scheduling ✓ **GA**
**Common Use Cases:**
* Page on-call ML engineers for critical model failures
* Escalate unresolved AI incidents automatically
* Track MTTR (Mean Time To Resolution) for model issues
* Integrate with incident runbooks and response workflows
### 💬 Team Collaboration
Send alerts to team communication tools for visibility and collaboration.
**Supported Platforms:**
* **Slack** - Team messaging and collaboration ✓ **GA** *(Coming Soon)*
* **Microsoft Teams** - Enterprise communication platform ✓ **GA** *(Coming Soon)*
**Common Use Cases:**
* Notify ML team channel when drift is detected
* Alert data science team on data quality issues
* Share model performance reports automatically
* Collaborative incident triage in team channels
## Observability Platform Integrations
### Datadog
Integrate Fiddler with Datadog for unified application and AI monitoring.
**Why Datadog + Fiddler:**
* **Unified Dashboards** - Combine infrastructure, application, and AI model metrics
* **Correlated Alerts** - Alert on compound conditions (e.g., "high model drift + high API latency")
* **Service Map Integration** - See model health in Datadog service dependency graphs
* **Anomaly Detection** - Leverage Datadog's ML-based alerting on Fiddler metrics
**Key Features:**
* **Metric Export** - Send Fiddler drift, performance, and data quality metrics to Datadog
* **Event Streaming** - Stream model events (predictions, drift detections) as Datadog events
* **Alert Forwarding** - Route Fiddler alerts to Datadog for unified incident management
* **Tag Propagation** - Maintain consistent tagging across platforms (model, environment, team)
**Status:** ✓ **GA** - Production-ready
[**Get Started with Datadog →**](/integrations/monitoring-alerting/datadog-integration)
**Quick Start:**
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Configure Datadog integration
client.add_datadog_integration(
api_key="datadog_api_key",
app_key="datadog_app_key",
site="datadoghq.com", # or datadoghq.eu
# Metric export configuration
export_metrics=True,
metric_prefix="fiddler.model.",
tags=["env:production", "team:ml-platform"],
# Event export configuration
export_events=True,
event_priority="normal" # or "low"
)
```
**Example Datadog Dashboard:**
```yaml theme={null}
# Datadog dashboard combining infrastructure + AI metrics
widgets:
- title: "Model Latency vs API Latency"
type: timeseries
queries:
- metric: fiddler.model.latency.p95
scope: model:fraud_detector
- metric: trace.flask.request.duration.p95
scope: service:fraud-api
- title: "Model Drift Detection"
type: query_value
queries:
- metric: fiddler.model.drift.score
aggregation: max
conditional_formats:
- comparator: ">"
value: 0.1
palette: "red"
```
## Incident Management Integrations
### PagerDuty
Route critical AI alerts to on-call engineers via PagerDuty.
**Why PagerDuty + Fiddler:**
* **On-Call Escalation** - Page the right ML engineer based on escalation policies
* **Incident Deduplication** - Prevent alert storms from related model issues
* **Incident Timeline** - Track when AI issues were detected, acknowledged, resolved
* **Postmortem Integration** - Include model context in incident reports
**Key Features:**
* **Severity Mapping** - Map Fiddler alert criticality to PagerDuty severity levels
* **Service Integration** - Associate alerts with PagerDuty services (e.g., "Fraud Detection Service")
* **Custom Payloads** - Include model metadata, drift scores, affected predictions
* **Bidirectional Updates** - Acknowledge/resolve incidents in PagerDuty or Fiddler
**Status:** ✓ **GA** - Production-ready
[**Get Started with PagerDuty →**](/integrations/monitoring-alerting/pagerduty)
**Quick Start:**
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Configure PagerDuty integration
client.add_pagerduty_integration(
integration_key="pagerduty_integration_key",
service_name="ML Models - Production",
# Alert routing rules
severity_mapping={
"critical": "critical", # Fiddler → PagerDuty severity
"high": "error",
"medium": "warning",
"low": "info"
}
)
# Create alert with PagerDuty notification
client.create_alert(
name="Critical Model Drift",
project="fraud-detection",
model="fraud_detector_v3",
metric="drift_score",
threshold=0.15,
severity="critical",
notification_channels=["pagerduty"]
)
```
**Example PagerDuty Incident:**
```json theme={null}
{
"incident_key": "fiddler_drift_fraud_detector_v3",
"type": "trigger",
"description": "High drift detected on fraud_detector_v3",
"details": {
"model": "fraud_detector_v3",
"project": "fraud-detection",
"drift_score": 0.23,
"affected_features": ["transaction_amount", "merchant_category"],
"time_window": "2024-11-10 14:00 - 14:30 UTC",
"fiddler_url": "https://app.fiddler.ai/projects/fraud-detection/models/fraud_detector_v3/drift"
},
"client": "Fiddler AI Observability",
"client_url": "https://app.fiddler.ai"
}
```
## Team Collaboration Integrations
### Slack *(Coming Soon)*
**Planned Features:**
* **Channel Notifications** - Post alerts to team Slack channels
* **Interactive Messages** - Acknowledge, snooze, or resolve alerts from Slack
* **Scheduled Reports** - Daily/weekly model performance summaries
* **Threaded Discussions** - Collaborate on incident resolution in threads
**Example Configuration:**
```python theme={null}
# Future Slack integration API
client.add_slack_integration(
webhook_url="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXX",
channel="#ml-alerts",
username="Fiddler",
icon_emoji=":robot_face:"
)
```
### Microsoft Teams *(Coming Soon)*
**Planned Features:**
* **Adaptive Cards** - Rich, interactive alert notifications
* **Team Channels** - Route alerts to relevant team channels
* **Bot Commands** - Query model status from Teams chat
* **Integration with Workflows** - Trigger Teams workflows on alerts
## Alert Routing Patterns
### Pattern 1: Severity-Based Routing
Route alerts to different channels based on severity:
```python theme={null}
# Critical alerts → PagerDuty (page on-call)
client.create_alert(
name="Model Offline",
severity="critical",
notification_channels=["pagerduty"]
)
# High alerts → Slack (notify team)
client.create_alert(
name="High Drift Detected",
severity="high",
notification_channels=["slack"]
)
# Medium alerts → Datadog (track as events)
client.create_alert(
name="Minor Data Quality Issue",
severity="medium",
notification_channels=["datadog"]
)
```
### Pattern 2: Team-Based Routing
Different teams get different alerts:
```python theme={null}
# ML Engineering team
client.create_alert(
name="Model Performance Degradation",
project="fraud-detection",
notification_channels=["slack"],
slack_channel="#ml-engineering",
tags=["team:ml-engineering"]
)
# Data Engineering team
client.create_alert(
name="Data Pipeline Failure",
project="fraud-detection",
notification_channels=["slack"],
slack_channel="#data-engineering",
tags=["team:data-engineering"]
)
# On-Call team (escalation)
client.create_alert(
name="Critical Model Failure",
project="fraud-detection",
notification_channels=["pagerduty"],
pagerduty_service="ml-models-production",
tags=["team:on-call"]
)
```
### Pattern 3: Composite Alerting
Alert on compound conditions across multiple platforms:
```python theme={null}
# Example: Alert if BOTH model drift is high AND API latency is high
# (Indicates model update needed + production impact)
# Configure in Datadog (using Fiddler + Datadog metrics)
datadog_composite_alert = {
"name": "Model Drift + High Latency",
"query": """
avg(last_5m):fiddler.model.drift.score{model:fraud_detector} > 0.1
AND
avg(last_5m):trace.flask.request.duration.p95{service:fraud-api} > 500
""",
"message": "@pagerduty-ml-oncall Model drift detected with production latency impact",
"tags": ["model:fraud_detector", "alert:composite"]
}
```
## Metric Export Patterns
### Export Fiddler Metrics to Datadog
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Configure metric export
client.configure_metric_export(
destination="datadog",
metrics=[
"drift_score",
"data_quality_score",
"prediction_latency_p95",
"prediction_count",
"error_rate"
],
export_interval=60, # seconds
tags=["source:fiddler", "env:production"]
)
```
**Exported Metrics:**
* `fiddler.model.drift.score` - Overall drift score (0-1)
* `fiddler.model.drift.feature.<feature_name>` - Per-feature drift
* `fiddler.model.performance.<metric>` - Model performance metrics
* `fiddler.model.data_quality.score` - Data quality score
* `fiddler.model.predictions.count` - Prediction volume
* `fiddler.model.predictions.latency` - Prediction latency percentiles
### Query Fiddler Metrics in Datadog
```python theme={null}
# Datadog Metrics Query Language (MQL)
queries = {
"high_drift_models": """
avg:fiddler.model.drift.score{*} by {model} > 0.1
""",
"low_performance_models": """
avg:fiddler.model.performance.accuracy{*} by {model} < 0.85
""",
"high_volume_models": """
sum:fiddler.model.predictions.count{*} by {model}.as_count()
"""
}
```
## Alert Lifecycle Management
### Alert States
Fiddler alerts transition through these states:
```
Triggered → Acknowledged → Investigating → Resolved → Closed
↓ ↑
Escalated --------------------------------┘
```
**Synchronization with External Tools:**
* **PagerDuty**: Bidirectional state sync (acknowledge, resolve)
* **Datadog**: Event-based updates
* **Slack**: Interactive message updates
### Alert Deduplication
Prevent alert storms with intelligent deduplication:
```python theme={null}
# Configure deduplication rules
client.configure_alert_deduplication(
project="fraud-detection",
model="fraud_detector_v3",
# Group related alerts
grouping_window=300, # seconds
grouping_keys=["model", "metric"],
# Suppress similar alerts
suppression_window=3600, # seconds
max_alerts_per_window=3
)
```
## Custom Webhook Integrations
For platforms not natively supported, use generic webhooks:
```python theme={null}
# Send alerts to any webhook endpoint
client.add_webhook_integration(
name="custom-alerting-system",
url="https://your-system.com/webhook/fiddler",
method="POST",
headers={
"Authorization": "Bearer your-token",
"Content-Type": "application/json"
},
payload_template={
"alert_name": "{{alert.name}}",
"model": "{{model.name}}",
"severity": "{{alert.severity}}",
"timestamp": "{{alert.timestamp}}",
"details": "{{alert.details}}"
}
)
# Use webhook in alerts
client.create_alert(
name="Custom Alert",
notification_channels=["webhook:custom-alerting-system"]
)
```
**Webhook Payload Example:**
```json theme={null}
{
"alert_name": "High Drift Detected",
"model": "fraud_detector_v3",
"project": "fraud-detection",
"severity": "high",
"timestamp": "2024-11-10T14:32:01Z",
"metric": "drift_score",
"current_value": 0.23,
"threshold": 0.15,
"fiddler_url": "https://app.fiddler.ai/projects/fraud-detection/models/fraud_detector_v3",
"affected_features": ["transaction_amount", "merchant_category"],
"recommended_actions": [
"Investigate feature distribution changes",
"Consider model retraining",
"Review data pipeline for issues"
]
}
```
## Monitoring Integration Health
### Track Integration Status
```python theme={null}
# Check integration health
integrations = client.list_integrations()
for integration in integrations:
status = client.get_integration_health(integration.name)
print(f"{integration.name}: {status.status}")
print(f" Last successful send: {status.last_success}")
print(f" Failed alerts (24h): {status.failed_count}")
```
### Alerts on Integration Failures
```python theme={null}
# Alert if alert delivery fails
client.create_meta_alert(
name="Alert Delivery Failure",
trigger="integration_failure",
integration="pagerduty",
threshold=3, # failures
time_window=3600, # seconds
notification_channels=["email"] # use different channel!
)
```
## Best Practices
### Alert Fatigue Prevention
**1. Use Appropriate Severity Levels:**
```python theme={null}
# Reserve "critical" for page-worthy issues
severity_guidelines = {
"critical": "Model offline, safety violations, production outages",
"high": "Significant drift, performance degradation",
"medium": "Minor data quality issues, gradual drift",
"low": "Informational, trend observations"
}
```
**2. Implement Alert Throttling:**
```python theme={null}
client.create_alert(
name="Drift Detection",
threshold=0.1,
evaluation_window=300, # 5 minutes
cooldown_period=3600, # Don't re-alert for 1 hour
max_alerts_per_day=5 # Cap daily alerts
)
```
**3. Use Alert Grouping:**
```python theme={null}
# Group related alerts into single notification
client.configure_alert_grouping(
group_name="Feature Drift Alerts",
alerts=["drift_feature_1", "drift_feature_2", "drift_feature_3"],
send_as_digest=True,
digest_interval=1800 # 30 minutes
)
```
### Incident Response Runbooks
Include runbook links in alert payloads:
```python theme={null}
client.create_alert(
name="High Model Drift",
metadata={
"runbook_url": "https://wiki.company.com/ml/runbooks/drift-response",
"on_call_team": "ml-platform",
"escalation_policy": "ml-models-production",
"sla": "30 minutes to acknowledge, 2 hours to investigate"
}
)
```
## Security & Compliance
### Secure Credential Management
**Never hardcode credentials:**
```python theme={null}
import os
# Use environment variables
client.add_pagerduty_integration(
integration_key=os.environ['PAGERDUTY_INTEGRATION_KEY']
)
# Or use secret management systems
from secret_manager import get_secret
client.add_datadog_integration(
api_key=get_secret('datadog-api-key'),
app_key=get_secret('datadog-app-key')
)
```
### Alert Data Privacy
**PII Redaction in Alerts:**
```python theme={null}
client.configure_alert_privacy(
redact_pii=True,
pii_fields=["email", "ssn", "phone_number"],
redaction_string="[REDACTED]"
)
```
### Audit Logging
**Track alert delivery:**
```python theme={null}
# Query alert delivery audit log
audit_log = client.get_alert_audit_log(
start_time="2024-11-01",
end_time="2024-11-10",
include_payload=True
)
for entry in audit_log:
print(f"Alert: {entry.alert_name}")
print(f"Delivered to: {entry.channel}")
print(f"Status: {entry.status}")
print(f"Timestamp: {entry.timestamp}")
```
## Troubleshooting
### Common Issues
**Alerts Not Delivered:**
* Verify integration credentials are valid and not expired
* Check network connectivity from Fiddler to external platform
* Ensure webhook endpoints are reachable (not blocked by firewall)
* Validate alert thresholds are actually being triggered
**Duplicate Alerts:**
* Enable alert deduplication with appropriate time windows
* Check if multiple notification channels are configured
* Verify integration isn't configured twice
**Missing Alert Context:**
* Ensure `include_context=True` in alert configuration
* Check payload template includes necessary fields
* Verify external platform supports rich payloads (some SMS gateways don't)
## Integration Selector
Choose the right integration for your use case:
| Your Need | Recommended Integration | Why |
| -------------------------- | ------------------------- | ------------------------------------------ |
| On-call engineer paging | **PagerDuty** | Escalation policies, incident management |
| Infrastructure correlation | **Datadog** | Unified metrics, correlated dashboards |
| Team notifications | **Slack** *(Coming Soon)* | Channel-based, collaborative triage |
| Custom internal tools | **Generic Webhooks** | Flexible, integrate with any HTTP endpoint |
| Multi-tool strategy | **Datadog + PagerDuty** | Metrics + incidents in one workflow |
## Related Integrations
* [**Cloud Platforms**](/integrations/cloud-platforms-and-deployment/cloud-platforms) - Deploy Fiddler on AWS, Azure, GCP
* [**Data Platforms**](/integrations/data-platforms-and-pipelines/data-platforms) - Ingest data from Snowflake, Kafka
* [**ML Platforms**](/integrations/ml-platforms-and-tools/ml-platforms) - Integrate with Databricks, MLflow
* [**Agentic AI**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) - Monitor LangGraph and Strands Agents
***
# Annotations
Source: https://docs.fiddler.ai/observability/agentic/annotations
Add human evaluation scores to individual spans in the Explorer to review and assess LLM application outputs alongside automated evaluators.
This capability is currently in [public preview](/reference/feature-maturity-definitions#public-preview). It is available to every customer and Fiddler responds promptly to issues, but it is not yet covered by an SLA and its APIs may still change.
## Overview
Annotations let you attach human evaluation scores directly to individual spans in the [Explorer](/observability/agentic/trace-explorer). Use them to review LLM outputs, flag quality issues, and build a human-labeled dataset alongside your automated evaluators.
Each annotation records a **name**, a **score type** (numeric, boolean, categorical, or text), the score itself (in the type-appropriate field — see [Score Types](#score-types)), and an optional **Reason** field (`reasoning` in the API) explaining the assessment. Annotations are stored as scores with `source=human` and are accessible through both the UI and the `/v3/scores` REST API.
## Score Types
| Type | Value | API field | Example use case |
| --------------- | ---------------------------------------- | ----------------------------- | ----------------------------------------- |
| **Numeric** | Any number | `value` | Relevance score (0.0 to 1.0) |
| **Boolean** | True or False | `value` — send `1.0` or `0.0` | "Is this response correct?" |
| **Categorical** | A text label | `label` | Sentiment: positive, neutral, negative |
| **Text** | Free-form text (up to 65,536 characters) | `text` | Detailed feedback or a corrected response |
Every annotation also accepts an optional **Reason** field for free-text explanation of why the score was given.
***
## Using the UI
### Annotating a Span
Navigate to any GenAI Application Details page and select the **Explorer** tab.
Click the **View Trace** icon on a span row to open the side drawer. The **Annotate** button appears in the top-right corner of the span detail panel.
Click **Annotate** to open the **Add Annotation** dialog. Enter a **Name** (used as the score identifier across spans), select an **Annotation Type**, provide the score **Value**, and optionally add a **Reason**.
Use the same name on multiple spans to group scores — for example, annotating every span with a "correctness" score lets you compare correctness across your application.
Click **Add Annotation**. The annotation appears immediately in the **Annotations** section of the span detail panel.
### Viewing Annotations
Existing annotations appear in the **Annotations** accordion section at the bottom of the span detail panel. Each annotation card displays the score name, value, type, annotator, and timestamp.
### Editing and Deleting Annotations
* Click the **edit** (pencil) icon on an annotation card to update its value or reasoning. The score name and type cannot be changed — create a new annotation instead.
* Click the **delete** (trash) icon to remove an annotation. A confirmation dialog appears before deletion.
***
## API Access
Annotations are stored as scores with `source=human` via the `/v3/scores` API. You can create, list, update, and delete annotations programmatically.
### Quick Example
```bash theme={null}
curl -X POST https://your-fiddler-instance.com/v3/scores \
-H "Authorization: Bearer $FIDDLER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"application_id": "your-app-uuid",
"span_id": "target-span-id",
"name": "correctness",
"score_type": "numeric",
"source": "human",
"value": 0.9,
"reasoning": "Response accurately addressed the user question"
}'
```
### Endpoints
| Operation | Method | Endpoint | Reference |
| ----------- | -------- | ----------------------- | ----------------------------------------------------------------- |
| Create | `POST` | `/v3/scores` | [Create a score](/sdk-api/rest-api/scores/create-score) |
| List | `GET` | `/v3/scores` | [List scores](/sdk-api/rest-api/scores/list-scores) |
| Get | `GET` | `/v3/scores/{score_id}` | [Get a score](/sdk-api/rest-api/scores/get-score) |
| Update | `PATCH` | `/v3/scores/{score_id}` | [Update a score](/sdk-api/rest-api/scores/update-score) |
| Delete | `DELETE` | `/v3/scores/{score_id}` | [Delete a score](/sdk-api/rest-api/scores/delete-score) |
| Bulk create | `POST` | `/v3/scores/bulk` | [Bulk create scores](/sdk-api/rest-api/scores/bulk-create-scores) |
Filter by `source=human` to retrieve only human-created annotations. Evaluator-generated scores use `source=evaluator` and are read-only through this API.
***
## Related Resources
* [Explorer](/observability/agentic/trace-explorer) — the span exploration surface where annotations are created
* [Scores REST API](/sdk-api/rest-api/scores) — full API reference for creating and managing scores
* [Custom Metrics for Agentic Applications](/observability/agentic/custom-metrics) — define custom metrics over the same span data using FQL
* [Feature Maturity Definitions](/reference/feature-maturity-definitions#public-preview) — what public preview means
# Custom Metrics for Agentic Applications
Source: https://docs.fiddler.ai/observability/agentic/custom-metrics
Define custom metrics for your agentic and GenAI applications using FQL and span attributes to track business KPIs, quality scores, and operational signals beyond built-in metrics.
### Overview
Custom metrics let you define measurements that align precisely with your agentic application's requirements. Whether tracking business KPIs, aggregating quality scores from enrichments, or computing cost and latency signals, custom metrics let you tailor observability to your specific needs. Once defined, they are available in charts and dashboards.
Custom metrics for agentic applications are defined using [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language) and reference **span attributes** captured from your application's OpenTelemetry traces. This differs from ML custom metrics, which reference model schema columns.
Custom metrics can be **organization-level** (visible across all projects) or **project-scoped** (visible only within a specific project). See [Metric Visibility](#metric-visibility-organization-vs-project) below.
### Metric Visibility: Organization vs Project
When creating a custom metric, you choose whether it is **organization-level** (global) or **project-scoped**. This determines who can see it and who can delete it, and cannot be changed after creation.
#### Organization-level metrics
Created without selecting a project. Visible across all projects in the organization. The metric name is reserved org-wide — no other metric in the organization can share the same name.
| Action | Roles |
| ------ | --------------------- |
| Create | Org Admin, Org Member |
| View | Org Admin, Org Member |
| Delete | Org Admin |
#### Project-scoped metrics
Created with a specific project selected. Visible only to users who have access to that project. The same metric name can be reused in a different project as long as the projects don't overlap.
| Action | Roles |
| ------ | --------------------------------------------------------------- |
| Create | Project Admin, Project Writer (on that project) |
| View | Project Admin, Project Writer, Project Viewer (on that project) |
| Delete | Project Admin (on that project) |
Metric visibility is **immutable** — an organization-level metric cannot be changed to project-scoped or vice versa after creation.
### The `attribute()` Function
The `attribute()` function is the GenAI-specific FQL primitive for referencing span data. It replaces the column references used in ML custom metrics.
#### Syntax
Reference an attribute in one of two ways — by **name**, its verbatim key exactly as your application sends it, or by **semantic concept**, which resolves consistently across instrumentation frameworks.
```text theme={null}
# All forms below require 26.16+
attribute('my_attribute')
attribute(semantic_name='input_tokens')
attribute(semantic_name='finish_reason', value='stop')
```
| Parameter | Required | Description |
| --------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | Exactly one of `semantic_name` or `name` | The verbatim attribute key as it appears in your trace data (e.g., `tier`, `cost`). |
| `semantic_name` | Exactly one of `semantic_name` or `name` | A canonical semantic concept (e.g., `input_tokens`, `model_name`). Fiddler resolves it to whichever raw attribute key your framework emitted. See [Supported semantic concepts](#supported-semantic-concepts). |
| `value` | No | Filters to spans where the attribute equals this string; returns the value if matched, `null` otherwise. Used for categorical attributes (e.g., `attribute('status', value='error')`). When set, the attribute is always treated as a string. |
| `scope` | No | Optional in 26.16. When omitted, defaults to `'span'`. |
| `type` | No | Optional in 26.16. When omitted, defaults to `'user'`. |
`semantic_name` requires semantic mappings, which are enabled by default from release 26.16. On earlier releases, `attribute()` requires `name`, `scope`, and `type`, and rejects `semantic_name` — the shorthand `attribute('my_attribute')` shown above also requires 26.16 or later. Existing metrics written in the older form continue to work unchanged.
#### Supported semantic concepts
`semantic_name` accepts the following concepts. Attributes your application defines that fall outside this list are referenced by `name` instead.
| Category | Concepts |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| Tokens | `input_tokens`, `output_tokens`, `total_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`, `reasoning_tokens` |
| Cost | `input_cost`, `output_cost`, `total_cost` |
| Latency | `ttft` |
| Model and provider | `model_name`, `provider_name` |
| Response metadata | `finish_reason` |
| Agent and tool | `agent_name`, `tool_name`, `tool_type` |
| Span identity | `span_type` |
`ttft` returns the value exactly as your instrumentation emits it, so its unit depends on the source framework — some frameworks emit seconds, others milliseconds. To track overall span latency, use the built-in latency chart; system-computed span duration is not currently queryable through `attribute(semantic_name='latency')` in a custom metric.
See [Semantic Mappings](/concepts/semantic-mappings) for how raw attribute keys resolve to these concepts, and how to add mappings for a framework Fiddler doesn't ship out of the box.
The legacy system attribute, custom user attribute, and content attribute prefix conventions are deprecated as of release 26.16. Reference these attributes by `semantic_name` instead — existing metrics using the older keys continue to resolve through backward-compatible default mappings.
#### Type inference
Fiddler infers the attribute type from context — you do not need to declare it explicitly:
* When used inside a numeric aggregate like `sum()` or `average()`, the attribute is treated as a number.
* When used with string functions like `length()` or `match()`, the attribute is treated as a string.
* When the `value` keyword is provided, the attribute is always treated as a string.
### Adding a Custom Metric
1. Navigate to the **Custom Metrics** section in the Fiddler UI.
2. Click **Add Custom Metric**.
3. Enter a **Metric name**, an optional **Description**, and the **Metric definition**.
4. Optionally select a **project** to scope the metric to. If no project is selected, the metric is created as an **organization-level** metric visible across all projects.
5. Click **Create Metric**.
### Using Custom Metrics in Charts
After saving a custom metric, you can use it in chart definitions:
1. Open or create a chart in the Fiddler UI.
2. Set **Metric Type** to **Custom Metric**.
3. Select your custom metric from the list.
### Deleting Custom Metrics
To delete a custom metric, click the trash icon next to the metric in the **Custom Metrics** tab. Deletion runs as a background job that automatically:
* Removes the metric from any charts that reference it
* Deletes charts that have no remaining metrics after cleanup
* Updates dashboard layouts to remove deleted charts
* Deletes dashboards that become empty as a result
### Examples
Custom metrics must return either an aggregate (produced by aggregate functions) or a combination of aggregates. See the [FQL reference](/observability/platform/fiddler-query-language) for the full list of supported operators and functions.
#### Average input token usage
Track the mean number of input tokens consumed per span to monitor LLM cost drivers over time. Because this references a semantic concept, the same metric works whether your traces come from OpenInference, the Vercel AI SDK, LiteLLM, or any other supported framework.
```text theme={null}
average(attribute(semantic_name='input_tokens'))
```
→ Returns a `Number` (e.g., `312.4`)
#### Premium user ratio
Measure the fraction of spans attributed to premium-tier users by filtering on a categorical attribute.
```text theme={null}
count(attribute('tier', value='premium')) / count(attribute('tier'))
```
→ Returns a `Number` between 0 and 1 (e.g., `0.34`)
#### P95 total token usage
Use the `quantile()` function to track the 95th-percentile of a per-span value. A percentile gives a stable high-end bound, unlike an average that a few extreme spans can distort.
```text theme={null}
quantile(attribute(semantic_name='total_tokens'), level=0.95)
```
→ Returns a `Number` representing the 95th-percentile total tokens per span (e.g., `2048`)
To track a latency value your own instrumentation emits, reference it by its verbatim key — for example, `quantile(attribute('response_time_ms'), level=0.95)` — where the returned value is in whatever unit your application sends.
#### Conditional cost (weighted by outcome)
Apply different weights to successful and failed spans to surface the true cost impact of errors. The `if(condition, true_value, false_value)` function evaluates the condition per span and returns one of two values.
```text theme={null}
average(if(attribute('status') == 'error', attribute('cost') * 2, attribute('cost')))
```
→ Returns a `Number` (e.g., `0.0042`)
#### Token usage range
Track the spread of a per-span value across a time window. A widening range can point to inconsistent request sizes or a few unusually large requests — here, the gap between the largest and smallest total token counts.
```text theme={null}
max(attribute(semantic_name='total_tokens')) - min(attribute(semantic_name='total_tokens'))
```
→ Returns a `Number` representing the spread between the largest and smallest total token counts (e.g., `3072`)
#### Minimum token usage
Find the smallest input token count across all spans in a window. Useful for detecting unusually short requests that may indicate truncated inputs or misconfigured clients.
Reference the attribute by its verbatim key, which matches only the frameworks that emit that exact key:
```text theme={null}
min(attribute('gen_ai.usage.input_tokens'))
```
Or by semantic concept, which matches whichever key your framework emitted:
```text theme={null}
min(attribute(semantic_name='input_tokens'))
```
→ Returns a `Number` (e.g., `12`)
#### Null-safe cost with markup
Apply a price adjustment to spans that have a cost attribute, while preserving `null` for spans where cost data is absent — avoiding accidental zero-inflation of the average.
```text theme={null}
average(if(is_null(attribute('cost')), null, attribute('cost') * 1.15))
```
→ Returns a `Number` representing the average marked-up cost, excluding null-cost spans (e.g., `0.0048`)
Use `is_null()` to test whether an attribute is absent. The `null` keyword is for use as a **return value** in expressions (e.g., `if(condition, value, null)`) to propagate missing data explicitly.
### Related Resources
* [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language) — full reference for operators, aggregate functions, and expression syntax
* [Custom Metrics for ML Models](/observability/platform/custom-metrics) — custom metrics using model schema columns instead of span attributes
* [Agentic Observability](/observability/agentic/index) — overview of dashboards, metrics, and integrations for agentic applications
* [Custom Metrics glossary entry](/glossary/custom-metrics)
* [Data Retention Policy](/developers/client-library-reference/publishing-production-data#data-retention-policy) — includes planned support for retaining a chart's history in Mission Control beyond raw trace retention
# Agentic Observability
Source: https://docs.fiddler.ai/observability/agentic/index
Monitor AI agents and multi-step workflows with specialized dashboards, metrics, and trace visualization
Agentic observability provides specialized observability for AI agents and multi-step workflows through dedicated dashboards, metrics, and trace visualization. Unlike traditional monitoring, which tracks single-shot inferences, agentic observability captures the complete lifecycle of autonomous-agent behavior—from initial reasoning through tool execution and final response.
## Dashboards & Visualization
Agentic observability uses a dedicated UI with Projects → Applications (instead of the legacy Projects → Models structure) designed specifically for observing agent workflows.
### Agentic Dashboards
Access pre-built dashboards optimized for agentic workflows:
* **Agent Performance Overview** - Monitor success rates, latency, and throughput across all agents
* **Workflow Execution Traces** - Visualize complete multi-step reasoning chains from start to finish
* **Tool Usage Analytics** - Track which external tools and APIs your agents are calling
* **Error & Exception Tracking** - Identify where agent workflows fail and why
### Trace Visualization
Every agent interaction is captured as a hierarchical trace showing:
* **Agent Steps** - Each decision point in the agent's reasoning process
* **LLM Calls** - All language model interactions with inputs, outputs, and metadata
* **Tool Invocations** - External function calls, API requests, and data retrievals
* **Timing Information** - Duration of each step to identify performance bottlenecks
* **Parent-Child Relationships** - How multi-agent systems coordinate and delegate tasks
**Viewing Traces:**
1. Navigate to your Application in the Fiddler UI
2. Select a conversation or workflow from the list
3. Click on any trace to expand the full execution tree
4. Drill down into individual spans to see inputs, outputs, and metadata
### Custom Dashboards
Create custom dashboards to monitor specific agent behaviors:
* Combine multiple charts to track KPIs relevant to your use case
* Filter by agent type, user segments, or time periods
* Share dashboards with team members
* Set up alerts based on dashboard metrics
## Metrics & Analytics
Agentic observability provides specialized metrics that go beyond traditional model monitoring:
### Agent-Specific Metrics
* **Agent Success Rate** - Percentage of workflows that complete successfully
* **Tool Call Distribution** - Which tools agents use most frequently
* **Reasoning Chain Length** - Average number of steps per workflow
* **Agent Handoffs** - How often agents delegate to other agents
* **Retry & Recovery Rate** - How often agents recover from errors
### Performance Metrics
* **End-to-End Latency** - Total time from user request to final response
* **Per-Step Latency** - Duration of individual reasoning steps, LLM calls, and tool invocations
* **Token Usage** - Track LLM consumption across all agent interactions
* **API Call Volume** - Monitor external tool and API usage
### Quality Metrics
* **Response Accuracy** - Validate agent outputs against expected results (requires ground truth)
* **Hallucination Detection** - Identify when agents generate unsupported claims
* **Safety & Guardrails** - Track safety violations and guardrail activations
* **User Satisfaction** - Capture feedback signals from end users
### Analyzing Metrics
All metrics are available in:
* **Real-time Dashboards** - Monitor live agent performance
* **Historical Trends** - Analyze patterns over days, weeks, or months
* **Comparative Analysis** - Compare different agent versions or configurations
* **Custom Queries** - Use Fiddler Query Language (FQL) for advanced analysis
## Integration Options
Agentic observability ingests OpenTelemetry spans and traces from your agent applications. Choose the integration method that fits your framework:
### LangGraph Applications
**Best for:** Applications built with LangGraph or LangChain
The Fiddler LangGraph SDK provides automatic instrumentation with zero code changes required.
```bash theme={null}
pip install fiddler-langgraph
```
→ [**LangGraph SDK Quick Start**](/developers/quick-starts/langgraph-sdk-quick-start) - Get started in 10 minutes
→ [**LangGraph SDK Documentation**](/sdk-api/langgraph/fiddler-client) - Complete integration guide
### Strands Agent Framework
**Best for:** Applications built with Strands Agents
The Fiddler Strands SDK integrates directly with the Strands framework for seamless monitoring.
```bash theme={null}
pip install fiddler-strands
```
→ [**Strands Agent SDK Quick Start**](/developers/quick-starts/strands-agent-quick-start) - Get started in 15 minutes
→ [**Strands SDK Documentation**](/sdk-api/strands/strands-agent-instrumentor) - Complete integration guide
### Custom Instrumentation (OpenTelemetry)
**Best for:** Custom agent frameworks or non-Python applications
Use OpenTelemetry directly for maximum flexibility and control.
```bash theme={null}
pip install opentelemetry-api opentelemetry-sdk
```
→ [**OpenTelemetry Quick Start**](/developers/quick-starts/opentelemetry-quick-start) - Get started in 20 minutes
→ [**OpenTelemetry Documentation**](/integrations/agentic-ai/opentelemetry-integration) - Complete integration guide
### LiteLLM (SDK & Proxy)
**Best for:** Applications or gateways routing LLM calls through LiteLLM
LiteLLM's built-in OpenTelemetry integration sends every LLM call to Fiddler — no Fiddler-specific package required.
→ [**LiteLLM Integration**](/integrations/agentic-ai/litellm-integration) - Complete integration guide
### AI Gateways (Kong, AgentGateway, LiteLLM)
**Best for:** Traffic already routed through a third-party AI gateway.
Fiddler plugs into Kong, AgentGateway, or LiteLLM at the proxy layer for observability and guardrail enforcement, with no application code changes.
→ [**AI Gateway Integrations**](/integrations/agentic-ai/ai-gateways) - Capability comparison and per-gateway setup links
### Integration Comparison
| Feature | LangGraph SDK | Strands SDK | OpenTelemetry |
| ----------------------------- | -------------------- | -------------------- | -------------------------- |
| **Automatic Instrumentation** | ✅ Zero code changes | ✅ Native integration | ❌ Manual setup |
| **Framework Support** | LangGraph, LangChain | Strands Agents | Any framework |
| **Language Support** | Python | Python | Python, Java, Go, JS, etc. |
| **Setup Time** | \~10 minutes | \~10 minutes | \~15 minutes |
| **Customization** | Medium | Medium | High |
## Next Steps
* **Getting Started:** Learn the concepts behind agentic observability in our [Agentic Observability Getting Started Guide](/getting-started/agentic-monitoring)
* **Choose Integration:** Pick your framework and follow the quick start guide above
* **Explore Features:** Set up dashboards, configure alerts, and analyze agent performance
* **Advanced Topics:** Learn about [custom metrics](/observability/platform/custom-metrics), [alerts](/observability/platform/alerts-platform), and [segmentation](/observability/platform/segments)
# Explorer
Source: https://docs.fiddler.ai/observability/agentic/trace-explorer
Explore, filter, and search every span ingested into your GenAI application with the Explorer DataGrid.
This capability is currently in [public preview](/reference/feature-maturity-definitions#public-preview). It is available to every customer and Fiddler responds promptly to issues, but it is not yet covered by an SLA and its APIs may still change.
## Overview
The Explorer is a spans-first exploration surface for GenAI applications. It gives you a paginated, filterable, searchable view of every span ingested into a single application — useful for debugging individual requests, investigating evaluator failures, drilling into latency outliers, and validating instrumentation.
Open the **Explorer** tab on any GenAI Application Details page.
## Concepts
| Term | Meaning |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Span** | A single unit of work in your application — an LLM call, a tool invocation, an agent step, or a chain step. The Explorer grid is one row per span. |
| **Trace** | A collection of spans that share a `trace_id` and represent one end-to-end request through your application. |
| **Session** | A higher-level grouping (typically a multi-turn conversation) identified by a `session_id`. A session may contain multiple traces. |
| **Time range** | Mandatory. The Explorer always filters by a time window. The UI defaults to the last 7 calendar days (ending at end-of-day UTC). The time range is enforced server-side to keep queries fast against the underlying ClickHouse partitioning. |
***
## Using the UI
### Where it lives
Open any GenAI Application Details page, then select the **Explorer** tab.
### The DataGrid
Each row in the grid is a span. The grid is server-side paginated and sorted, with rich column controls.
#### Columns visible by default
| Column | Notes |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Type | Span type — typically one of `llm`, `tool`, `agent`, `chain` |
| Name | Span operation name |
| Input | LLM or tool input content. Query via the search bar with the **Input** scope selector (see [Searching in the UI](#searching-in-the-ui)). |
| Output | LLM or tool output content. Query via the search bar with the **Output** scope selector (see [Searching in the UI](#searching-in-the-ui)). |
| Timestamp | When the span started |
| Duration | Span duration (displayed in human-friendly units) |
| Status | OpenTelemetry status code (`Ok`, `Error`, `Unset`) |
| **Actions** | Pinned to the right of the grid; always visible. Contains the **View Trace** icon. |
#### Columns hidden by default
These columns are available via the column picker but are not shown out of the box:
* Session ID
* Trace ID
* Span ID
* Agent
* Agent ID
#### Dynamic columns
Explorer adds columns automatically based on your application's schema:
* **Evaluator output columns** — one column per output of every evaluator rule configured for the application.
* **Annotation columns** — one column per annotation score name. See [Annotations](/observability/agentic/annotations).
* **User-defined span attribute columns** — one column per attribute you've instrumented (for example, `user_id`, `model_version`, `environment`).
* **Token count columns** — `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.total_tokens`.
See [Span and Resource Attributes](/integrations/agentic-ai/attributes) for the full attribute catalog and naming conventions.
### Filtering in the UI
Open the filter panel from any filterable column header, or from the toolbar. Filters are applied server-side. By default, filter rules are combined with **AND**; the MUI filter panel also lets you switch to **OR** within a single column.
* **ID columns** (`Session ID`, `Trace ID`, `Span ID`) — `equals` only in the UI (enable the column via the column picker first; these are hidden by default).
* **String columns** (Name, Agent, span attributes) — `is`, `is not`, `is any of` (autocomplete) or `equals` / `does not equal` (free text), depending on whether the field has a known value set.
* **Numeric columns** (Duration, token counts, numeric attributes) — `=`, `!=`, `>`, `>=`, `<`, `<=`.
* **Enum columns** (Type, Status) — single- or multi-select.
* **Boolean evaluator outputs** — `is` or `is not` with values `true` or `false`.
### Searching in the UI
Use the search bar at the top of the grid to run a case-insensitive substring match against prompt and response content captured on your spans.
| Setting | Detail |
| ---------- | ----------------------------------------------------------------------------------------------- |
| **Length** | 3 to 64 characters |
| **Match** | Case-insensitive substring |
| **Index** | Backed by a dedicated content table with n-gram and token bloom-filter indexes for fast lookups |
The **scope selector** next to the search bar restricts which content fields are searched:
| Scope | Searches |
| ----------------- | ---------------------------------------------- |
| **All** (default) | LLM input, LLM output, tool input, tool output |
| **Input** | LLM input + tool input |
| **Output** | LLM output + tool output |
Search automatically covers every attribute mapped to the `input`, `output`, `tool_input`, or `tool_output` [semantic concepts](/concepts/semantic-mappings#content). If your framework uses custom attribute keys, mapping them to one of these concepts makes them searchable as well.
Changes to the searchable attribute set apply to newly ingested spans only. Existing spans are not re-indexed. Removing a semantic mapping does not remove the attribute from the search index — once indexed, the attribute remains searchable.
Search runs in addition to any structured filters — both must match for a span to appear in results.
### Sorting in the UI
The grid is sorted by **Timestamp** (most recent first) by default. Click the **Timestamp** or **Duration** column header to change the sort. Other columns are not sortable in the UI in this release.
### Opening the trace drawer
To inspect the full trace context for a span, click the **View Trace** (eye) icon in the rightmost **Actions** column. This opens a side drawer that loads every span in the row's session and renders them as a tree.
* The drawer is **session-scoped**, not trace-scoped — if a session contains multiple traces, all of them are visible in the drawer.
* The span detail panel includes an **Annotate** button for adding human evaluation scores. See [Annotations](/observability/agentic/annotations).
* The View Trace icon is **disabled** for spans without a `session_id`, with the tooltip *"No trace found"*.
***
## Related Resources
* [Annotations](/observability/agentic/annotations) — add human evaluation scores to individual spans
* [Custom Metrics for Agentic Applications](/observability/agentic/custom-metrics) — define custom metrics over the same span data using FQL
* [Span and Resource Attributes](/integrations/agentic-ai/attributes) — full attribute catalog and naming conventions
* [Agentic Observability](/observability/agentic/index) — overview of dashboards, metrics, and integrations
* [OpenTelemetry Trace Export](/integrations/agentic-ai/otel-trace-export) — export pre-existing OpenTelemetry traces to Fiddler from your own storage or pipeline
***
## Limitations & Roadmap (Public Preview)
The following are not available in this release and are tracked for future work:
* **Session-attribute filters.** You can filter by `session_id`, but filters over session-level attributes are not yet supported.
* **Multi-pattern structural filters.** Constructs such as "trace contains tool X *then* LLM Y" or "trace where root has positive feedback" — sometimes written with `->` (temporal) or `=>` (parent / child) operators in other tools — are not yet supported.
* **Saved filter sets.** Filters live for the current session; they cannot be named, saved, or shared yet.
* **CSV / notebook export.** Bulk export of filtered results is not yet available.
We're prioritizing these based on customer feedback during public preview — please reach out to your Customer Success Manager or file a ticket if any of the above are blocking your workflow.
# Events Table in RCA
Source: https://docs.fiddler.ai/observability/analytics/data-table-in-rca
Learn how to use Fiddler's root cause analysis features to quickly hone in on the data issues adversely impacting your ML models and LLM applications.
Allow visualizing events corresponding to a certain bin in a monitoring chart. Note that this view is to provide example rows used for the computation. The maximum number of rows that can viewed is 1000.
### Analyzing a Sample of Events
* Navigate to the `Charts` tab in your Fiddler AI instance
* Click on the `Add Chart` button on the top right
* In the modal, select a project
* Select **Monitoring**
* Create a Monitoring chart and click on a time range
* This will display the RCA (Root Cause Analysis) tab
* In RCA, select the `Events Table` tab
### Support
This visualization is supported for any model and data type.
### Represented data
The displayed events are production events coming from the selected model and bin, and segment if it was selected in the monitoring chart.
### Available Controls
* **Column selection**: On the top right side of the table, select the columns to be displayed. By default all non-vector columns are displayed.
* **Vector columns**: By default the vector columns are not fetched for latency reasons. Toggle on if vectors need to be fetched.
* **Download**: Download the sample events to `CSV` or `PARQUET` format.
# Feature Analytics
Source: https://docs.fiddler.ai/observability/analytics/feature-analytics-chart
Dive into our guide on creating feature analytics charts and visualizations for important features in your ML Models and LLM applications.
### Creating a Feature Analytics Chart
To create a Feature Analytics chart, follow these steps:
* Navigate to the `Charts` tab in your Fiddler AI instance
* Click on the `Add Chart` button on the top right
* In the modal, select a project
* Select **Feature Analytics**
### Support
Feature analytics is supported for any model task type, but does not support time stamps or columns of type `string` or `vector`.
### Available Right-Side Controls
| Parameter | Value |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model | List of models in the project |
| Version | List of versions for the selected model |
| Environment | `Production` or `Pre-Production` |
| Dataset | Displayed only if `Pre-Production` is selected. List of pre-production env uploaded for the model version. |
| Visual | List of possible visualizations, Feature Distribution or Feature Correlation. |
| Segment |
- Selecting a saved segment - Defining an applied (on the fly) segment. This applied segment isn’t saved (unless specifically required by the user) and is applied for analysis purposes.
|
| Feature |
- Feature Distribution: Select a single column of a supported data type to view the distribution of its values - Feature Correlation: Select two columns to visualize their correlation - Correlation Matrix: Select up to eight columns to visualize their pairwise correlations
|
### Available In-Chart Controls
| Control | Value |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Time range selection | Selecting start time and end time or time label for production data. Default is last 30 days |
| Display | Select how to display the chart, Histogram or Kernel Density Estimate, depending on the data type of the feature selected. |
### Displays for Feature Distribution
#### Kernel Density Estimate
For numeric column types, visualize feature distribution as a Kernel Density Estimate chart.
#### Histogram
Categorical and boolean value may only be displayed as Histograms, but other column types may also be displayed as such.
### Displays for Feature Correlation
#### Line Plot
#### Scatter Plot
#### Stacked Bar Plot
#### Box Plot
### Correlation Matrix Interactions
Clicking a cell in the correlation matrix opens the Feature Correlation chart, enabling a more detailed analysis of the relationship between the selected features.
# Analytics
Source: https://docs.fiddler.ai/observability/analytics/index
Explore our UI guide to Fiddler analytics. Learn about interfaces for various analytics charts and root cause analysis to better understand your models.
## Analytics Charts
There are three supported analytics chart types:
1. ***Performance Analytics*** — Visualize various performance metrics tailored to different task types, such as confusion matrices, prediction scatterplots, and more.
2. ***Feature Analytics*** — Analyze the behavior and relationships between features, helping to identify patterns and insights within the data, using feature distribution, feature correlation, and correlation matrix charts.
3. ***Metric Card*** — Display a single numeric representation of a metric, offering a concise overview of performance, data quality, data integrity, or custom metrics on a card.
## Root Cause Analysis
The Root Cause Analysis (RCA) experience consists of four parts, all based on the FQL segment and time range provided in the monitoring chart:
1. ***Events*** — Browse a sample of 1,000 events.
2. ***Data Drift*** — View a breakdown of drift for your features, along with prediction drift impact.
3. ***Data Integrity*** — Review a summary of data integrity violations, including counts across range, type, and missing value issues.
4. ***Analyze*** — View performance and feature analytics charts.
# Metric Card
Source: https://docs.fiddler.ai/observability/analytics/metric-card
Dive into our guide for metric card creation. Follow step-by-step instructions to create metric cards, use custom metrics, right-side controls, and save charts.
### Creating a Metric Card Chart
To create a Metric Card, follow these steps:
* Navigate to the `Charts` tab in your Fiddler AI instance
* Click on the `Add Chart` button on the top right
* In the modal, select a project
* Select **Metric Card**
### Support
Metric card is supported for any model task type and for both production and pre-production data. Metric card allows displaying of Custom Metric, Data Drift, Data Integrity, Performance, Traffic, and Statistic metrics.
### Available Right-Side Controls
| Parameter | Value |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model | List of models in the project |
| Version | List of versions for the selected model |
| Environment | `Production` or `Pre-Production` |
| Dataset | Displayed only if `Pre-Production` is selected. List of pre-production env uploaded for the model version. |
| Metric | Selecting a metric across Custom Metrics, Data Drift, Data Integrity, Performance, Traffic, and Statistic to display as an aggregated number. Select up to 4 metrics to display on the chart. |
| Segment |
- Selecting a saved segment - Defining an applied (on the fly) segment. This applied segment isn’t saved (unless specifically required by the user) and is applied for analysis purposes.
|
### Available In-Chart Controls
| Control | Model Task | Value |
| ------------------------ | --------------------- | -------------------------------------------------------------------------------------------- |
| Time range selection | All | Selecting start time and end time or time label for production data. Default is last 30 days |
| Positive class threshold | Binary classification | For performance metrics, select a threshold to be applied for computation. Default is 0.5 |
| Top-K | Ranking | For performance metrics, select a top-k value to be applied for computation. Default is 20. |
### Saving the Chart
Once you're satisfied with your metric card Chart, you can save it which allows it to be added to Dashboards and viewed individually from the Charts page.
# Performance Charts Creation
Source: https://docs.fiddler.ai/observability/analytics/performance-charts-creation
Discover our guide to creating performance charts. Learn key steps to select charts, use right-side and in-chart controls, and save your customized charts.
### Creating a Performance Chart
To create a Performance chart, follow these steps:
* Navigate to the `Charts` tab in your Fiddler AI instance
* Click on the `Add Chart` button on the top right
* In the modal, Select the project that has a model with Custom features
* Select **Performance Analytics**
### Available Performance charts
| Model task | Available Chart(s) |
| -------------------------- | ------------------------------------------------------------------------------------------- |
| Binary classification |
|
| Ranking / LLM / Not Set | *Not available* |
### Available right side controls
| Parameter | Value |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Model | List of models in the project |
| Version | List of versions for the selected model |
| Environment | `Production` or `Pre-Production` |
| Dataset | Displayed only if `Pre-Production` is selected. List of pre-production env uploaded for the model version. |
| Visual | List of possible [performance visualization](/observability/analytics/performance-charts-visualization) depending on the model task. |
| Segment |
- Selecting a saved segment - Defining an applied (on the fly) segment. This applied segment isn’t saved (unless specifically required by the user) and is applied for analysis purposes.
|
| Max Sample size |
Integer representing the maximum number of rows used for computing the chart, up to 500,000. If the data selected has fewer rows, we will use all the available rows with non-null target and output(s). Fiddler selects the first n rows from the selected slice.
Note: Clickhouse is configured using multiple shards, which means slightly different results can be observed if data is only selected on a specific shard (usually when little observation are queried).
|
### Available in-chart controls
| Control | Model Task | Value |
| ------------------------ | -------------------------- | -------------------------------------------------------------------------------------------------- |
| Time range selection | All | Selecting start time and end time or time label for production data. Default to last 30 days |
| Positive class threshold | Binary classification | Selecting threshold applied for computation / visualization. Default to 0.5 |
| Displayed labels | Multi-class Classification | Selecting the labels to display on the confusion matrix (up to 12). Default to the 12 first labels |
### Saving the Chart
Once you're satisfied with your visualization, you can save the chart. This chart can then be added to a Dashboard. This allows you to revisit the Performance visualization at any time easily either directly going to the Chart or to the dashboard.
# Performance Charts Visualization
Source: https://docs.fiddler.ai/observability/analytics/performance-charts-visualization
Dive into our guide on performance charts and visualizations used to monitor the behavior and performance of your ML models.
### Performance Charts Visualizations for ML and LLM Models
List of possible performance visualization depending on the model task. To see how to create a Performance chart, visit [this page](/observability/analytics/performance-charts-creation).
### Binary Classification
#### Confusion Matrix
A 2x2 table that shows how many predicted and actual values exist for positive and negative classes. Also referred to as an error matrix. The percentage is computed per row.
#### Receiver Operating Characteristic (ROC) Curve
A graph showing the performance of a classification model at different classification thresholds. Plots the true positive rate (TPR), also known as recall, against the false positive rate (FPR).
#### Precision-Recall Curve
A graph that plots the precision against the recall for different classification thresholds.
#### Calibration Plot
A graph that tell us how well the model is calibrated. The plot is obtained by dividing the predictions into 10 quantile buckets (0-10th percentile, 10-20th percentile, etc.). The average predicted probability is plotted against the true observed probability for that set of points.
### Multi-class Classification
#### Confusion Matrix
A table that shows how many predicted and actual values exist for different classes. Also referred to as an error matrix. The percentage is computed per row (using all classes). In the full view, up to 15 classes can be displayed. In the chart creation mode, up to 12 classes can be displayed. The displayed labels can be controlled in the chart.
### Regression
#### Prediction Scatterplot
Plots the predicted values against the actual values. The more closely the plot hugs the `y=x line`, the better the fit of the model.
#### Error Distribution
A histogram showing the distribution of errors (differences between model predictions and actuals). The closer to 0 the errors are, the better the fit of the model.
# Dashboard Interactions
Source: https://docs.fiddler.ai/observability/dashboards/dashboard-interactions
Explore our guide to dashboard interactions. Learn to remove, edit, zoom into charts, switch between bar and line views, and undo toolbar changes.
### Remove a Chart
If you want to remove a chart from your dashboard, simply click on the "X" located at the top right of the chart. This will remove the chart from the dashboard, but it will still be available in the saved charts list for future use. If you change your mind and decide to add the chart back to the dashboard, you can simply find it in the saved charts list and add it back to the dashboard at any time.
### Edit a Saved Chart
To edit a saved chart, simply click on the chart title within your dashboard. This will open the chart studio in a new tab, where you can make any necessary changes. Once you have made your changes, be sure to select the `Default` time range and then use the Dashboard refresh button to see your updated chart.
### Zoom Into Details
To zoom into a chart within your dashboard, you have two different utilities at your disposal. The first one is located on the top right of the chart component, in the toolbar. After clicking on the **zoom icon**, you can drag your cursor over the data points you wish to zoom into.
You can also use the **horizontal zoom bar** located at the base of the chart to zoom in. Once you've identified the time range you want to focus on, you can use the zoom bar to drag the range across time. For instance, if you want to analyze your data weekly over the past six months, you can use the toolbar or horizontal zoom bar to zoom in on the desired time range and then click and drag the selected range using the base horizontal zoom bar.
### Bar & Line Charts
You can switch between visualizing your chart as a line or a bar chart using the toolbar icons. Click on the line chart icon on the top right of the chart to switch to the line chart view. Likewise, select the bar chart icon in the toolbar to switch to the bar chart view.
> Note that these views are only temporary and any settings you specify using the toolbar will not be saved to the dashboard.
### Undo Chart Toolbar Changes
You can easily restore the changes you applied to your chart using the chart toolbar options, including zoom, switch to line chart, and switch to bar chart. The restore option, which is the last icon in the toolbar, allows you to undo any changes you made and return to the original chart configuration.
# Dashboard Utilities
Source: https://docs.fiddler.ai/observability/dashboards/dashboard-utilities
Discover dashboard utilities on Fiddler’s platform. Learn to rename, save, share, copy links, or delete dashboards to manage your collection effortlessly.
## Dashboard Name
To rename your dashboard, simply click on the "Untitled Dashboard" title on the top-left corner of the dashboard studio. This will allow you to give your dashboard a more descriptive name that reflects its purpose and contents, making it easier to find and manage among your other dashboards.
Once you've clicked on the "Untitled Dashboard" title to rename your dashboard, simply type in the desired name and hit "Enter" on your keyboard to save the new name.
If you change your mind and want to discard the changes, simply click anywhere on the page outside of the name box. This will cancel the renaming process and leave the dashboard name as it was before.
## Save, Copy Link, and Delete
You can easily manage your dashboard by using the control panel located on the top left of the dashboard studio. This panel allows you to save your dashboard, copy a link to it, or delete it entirely. By using these controls, you can easily share your dashboard with others or remove it from your collection if it is no longer needed.
## Save
It's important to note that dashboards are not automatically saved, so you'll need to manually save your dashboard in order to lock in the current charts and filters. Once you've made the desired changes to your dashboard, simply click the "Save" button to save your progress. This will also enable you to share or delete your dashboard as needed. By saving your dashboard frequently, you can ensure that you never lose important information or data visualizations.
## Copy Link
If you want to share your dashboard with other users on Fiddler, the first step is to ensure that they have access to the project that the dashboard belongs to. Once you've confirmed that they have access, you can easily share the dashboard by copying the dashboard link and sending it to them. This makes it simple to collaborate and share insights with others who are working on the same project or who have an interest in your findings. Note that you can't share a dashboard link until you've saved the dashboard.
## Delete
To delete a dashboard, click the overflow button next to the copy link icon. NOTE: Once a dashboard has been deleted it cannot be recovered.
# Creating Dashboards
Source: https://docs.fiddler.ai/observability/dashboards/dashboards-creating
Navigate our guide for the Dashboard page. Learn how to select new or existing dashboards and access them to monitor performance, drift, integrity, and traffic.
### Creating Dashboards
To begin using our dashboard feature, navigate to the dashboard page by clicking on "Dashboards" from the top-level navigation bar. On the Dashboards page, you can choose to either select from previously created dashboards or create a new one. This simple process allows you to quickly access your dashboards and begin monitoring your models' performance, data drift, data integrity, and traffic.
When creating a new dashboard, it's important to note that each dashboard is tied to a specific project space. This means that only models and charts associated with that project can be added to the dashboard. To ensure you're working within the correct project space, select the desired project space before entering the dashboard editor page, then click "Continue." This will ensure that you can add relevant charts and models to your dashboard.
#### Auto-Generated Dashboards
Fiddler will automatically generate model monitoring dashboards for all models registered to the platform. Depending on the task type, these dashboards will include charts spanning Performance, Traffic, Drift, and Data Integrity metrics.
Auto-generated model monitoring dashboard
Auto-generated dashboard are automatically set as the default dashboards for each model, and can be accessed via the Insights button on the Homepage or Model Schema pages, or alternatively all dashboards are always accessible in the Dashboards list page. Default dashboards and their charts can easily be modified to display the desired set of [monitoring](/observability/platform/monitoring-charts-platform) and [embedding](/observability/llm/embedding-visualization-with-umap) visualizations to meet any use case.
### Add Monitoring Charts
Once you’ve created a dashboard, you can add previously saved monitoring charts that display these metrics over time, making it easy to track changes and identify patterns.
To create a new monitoring chart for your dashboard, simply select "New Monitoring Chart" from the "Add" dropdown menu. For more information on creating and customizing monitoring charts, check out our [Monitoring Charts UI Guide](/observability/platform/monitoring-charts-platform).
If you'd like to add an existing chart to your dashboard, select "Saved Charts" to display a full list of monitoring charts that are available in your project space. This makes it easy to quickly access and add the charts you need to your dashboard for monitoring and reporting purposes.
To further customize your dashboard, you can select the saved monitoring charts of interest by clicking on their respective cards. For instance, you might choose to add charts for Accuracy, Drift, Traffic, and Range Violation to your dashboard for a more comprehensive model overview. By adding these charts to your dashboard, you can quickly access important metrics and visualize your model's performance over time, enabling you to identify trends and patterns that might require further investigation.
### Dashboard Filters
There are three main filters that can be applied to all the charts within dashboards, these include date range, time zone, and bin size.
#### Date Range
When the `Default` time range is selected, the data range, time zone, and bin size that each monitoring chart was originally saved with will be applied. This enables you to create a dashboard where each chart shows a unique filter set to highlight what matters to each team. Updating the date range will unlock the time zone and bin size filters. You can select from a number of predefined ranges or choose `Custom` to select a start and end date-time.
#### Bin Size
Bin size controls the frequency at which data is displayed on your monitoring charts. You can select from the following bin sizes: `Hour`, `Day`, `Week`, or `Month`.
> 📘 Note: Hour bin sizes are not supported for time ranges above 90 days.
>
> For example, if we select the `6M` data range, we see that the `Hourly` bin selection is disabled. This is disabled to avoid long computation and dashboard loading times.
#### Saved Model Updates
If you recently created or updated a saved chart and are not seeing the changes either on the dashboard itself or the Saved Charts list, click the refresh button on the main dashboard studio or within the saved charts list to reflect updates.
### Model Comparison
With our dashboard feature, you can also compare multiple models side-by-side, making it easy to see which models are performing the best and which may require additional attention. To create model-to-model comparison dashboards, ensure the models you wish to compare belong to the same project. Create the desired charts for each model and then add them to a single dashboard. By creating a single dashboard that tracks the health of all of your models, you can save time and simplify your AI monitoring efforts. With these dashboards, you can easily share insights with your team, management, or stakeholders, and ensure that everyone is up-to-date on your AI performance.
Refer to our [Dashboard Utilities ](/observability/dashboards/dashboard-utilities)and [Dashboard Interactions](/observability/dashboards/dashboard-interactions) pages for more info on dashboard usage.
# Executive Dashboard
Source: https://docs.fiddler.ai/observability/dashboards/executive-dashboard
A cross-project overview of GenAI and ML health for organization leadership, with pre-aggregated metrics that outlive raw data retention.
This capability is currently in [private preview](/reference/feature-maturity-definitions#private-preview) and is not yet available to all customers. Contact your Fiddler Customer Success Manager to request access.
### Overview
The Executive Dashboard gives organization leadership a single, at-a-glance
view of GenAI adoption, quality, and cost across every application — without
stitching together per-application dashboards.
* **Top-line KPIs** — Total Apps, Active Apps, Total Traces, Total Spans,
Errors, Sessions, and Evaluations Run over a selectable 7-day, 14-day, or
30-day window.
* **Evaluator scores** — Average Quality alongside Answer Relevance, Context
Relevance, RAG Faithfulness, Response Faithfulness, Conciseness, Prompt
Safety, and PII Detection, aggregated across all applications.
* **Usage and latency** — Traffic and token-usage trends, Average Trace
Latency, and P90 Span Latency.
* **Application Health Overview** — a per-application table (traces, average
latency, tokens, average quality) with drill-down links into each
application.
* **Period-over-period change** — each metric card shows the change versus
the immediately preceding window of equal length.
### How metrics are captured
Executive Dashboard metrics are not computed from raw traces at page load.
Instead, Fiddler continuously pre-aggregates them into a dedicated **metric
store**:
* **Hourly aggregation.** Background jobs run every hour and summarize the
most recently completed hour of traces, spans, and evaluator results into
per-application metric values (counts, sums, averages, and latency
percentiles). Coarser views — daily or weekly trends — are derived from
these hourly values.
* **Metric history outlives raw data.** The metric store is retained
independently of your raw trace retention policy. When raw traces and spans
are deleted at the end of their configured retention period, the aggregated
metric history remains, so long-range trends and period-over-period
comparisons keep working.
* **Freshness.** Because aggregation runs hourly on completed hours, the most
recent data point on the dashboard can lag real time by up to an hour.
Evaluator scores that are produced asynchronously are incorporated by
subsequent aggregation passes.
* **What is aggregated.** Only numeric metric values are stored in the metric
store — counts, token totals, latency statistics, and evaluator score
aggregates per application and hour. No prompt, response, or other span
content is copied into it.
Period-over-period change compares the selected window against the window of
equal length immediately before it — a 30-day view reads 60 days of metric
history. Deltas appear once enough history has accumulated in the metric
store.
### Related
* [Dashboards overview](/observability/dashboards)
* [Creating dashboards](/observability/dashboards/dashboards-creating)
# Dashboards
Source: https://docs.fiddler.ai/observability/dashboards/index
Explore our guide to Fiddler’s dashboards for centralized monitoring. Discover key features like filters, utilities, default dashboards, and performance.
### Overview
With Fiddler, you can create comprehensive dashboards that bring together all of your monitoring data in one place. This includes monitoring charts for data drift, traffic, data integrity, and performance metrics. Adding monitoring charts to your dashboards lets you create a detailed view of your model's performance. These dashboards can inform your team, management, or stakeholders, and help make data-driven decisions that improve your AI performance. View a list of the [**available metrics for monitoring charts here**](/observability/platform/monitoring-charts-platform#supported-metric-types).
### Key Features
Dashboards offer a powerful way to analyze the overall health and performance of your models, as well as to compare multiple models.
#### Dashboard Filters
* [Flexible filters](/observability/dashboards/dashboards-creating#dashboard-filters) including date range, time zone, and bin size to customize your view
#### Chart Utilities
* [Leverage the chart toolbar](/observability/dashboards/dashboard-interactions#zoom-into-details) to zoom into data and toggle between line and bar chart types
#### Automatic & Default Dashboards
* Fiddler automatically creates a [monitoring dashboard](/observability/dashboards/dashboards-creating#auto-generated-dashboards) for all your models that can be accessed as Insights throughout the product.
#### [Dashboard Basics](/observability/dashboards/dashboard-utilities)
* Perform model-to-model comparison
* Plot drift or data integrity for multiple columns in one view
* Easily save and share your dashboard
Checkout more on the [Dashboards Guide](/observability/dashboards/dashboards-creating).
# Fairness
Source: https://docs.fiddler.ai/observability/fairness
Explore our walkthrough of ML model fairness and bias. Review the sample calculations you can customize to your data and use with Fiddler's custom metrics.
Thanks to Fiddler charts visuals and our [custom metrics](/observability/platform/custom-metrics), you can define your fairness metric of choice to detect model bias on your dataset or production data and set up alerts.
## Definitions of Fairness
Models are trained on real-world examples to mimic past outcomes on unseen data. The training data could be biased, which means the model will perpetuate the biases in the decisions it makes.
While there is not a universally agreed upon definition of fairness, we define a ‘fair’ ML model as a model that does not favor any group of people based on their characteristics.
Ensuring fairness is key before deploying a model in production. For example, in the US, the government prohibited discrimination in credit and real-estate transactions with fair lending laws like the Equal Credit Opportunity Act (ECOA) and the Fair Housing Act (FHAct).
The Equal Employment Opportunity Commission (EEOC) acknowledges 12 factors of discrimination:[\[1\]](#reference) age, disability, equal pay/compensation, genetic information, harassment, national origin, pregnancy, race/color, religion, retaliation, sex, sexual harassment. These are what we call protected attributes.
## Fairness Metrics
Among others, some popular metrics are:
* Disparate Impact
* Group Benefit
* Equal Opportunity
* Demographic Parity
The choice of the metric is use case-dependent. An important point to make is that it's impossible to optimize all the metrics at the same time. This is something to keep in mind when analyzing fairness metrics.
## Disparate Impact
Disparate impact is a form of **indirect and unintentional discrimination**, in which certain decisions disproportionately affect members of a protected group.
Mathematically, disparate impact compares the pass rate of one group to that of another.
The pass rate is the rate of positive outcomes for a given group. It's defined as follows:
pass rate = passed / (num of ppl in the group)
Disparate impact is calculated by:
`DI = (pass rate of group 1) / (pass rate of group 2)`
Groups 1 and 2 are interchangeable. Therefore, the following formula can be used to calculate disparate impact:
`DI = min{pr_1, pr_2} / max{pr_1, pr_2}.`
The disparate impact value is between 0 and 1. The Four-Fifths rule states that the disparate impact has to be greater than 80%.
For example:
`pass-rate_1 = 0.3, pass-rate_2 = 0.4, DI = 0.3/0.4 = 0.75`
`pass-rate_1 = 0.4, pass-rate_2 = 0.3, DI = 0.3/0.4 = 0.75`
> 📘 Info
>
> Disparate impact is the only legal metric available. The other metrics are not yet codified in US law.
## Demographic Parity
Demographic parity states that the proportion of each segment of a protected class should receive the positive outcome at equal rates.
Mathematically, demographic parity compares the pass rate of two groups.
The pass rate is the rate of positive outcomes for a given group. It is defined as follows:
`pass rate = passed / (num of ppl in the group)`
If the decisions are fair, the pass rates should be the same.
## Group Benefit
Group benefit aims to measure the rate at which a particular event is predicted to occur within a subgroup compared to the rate at which it actually occurs.
Mathematically, group benefit for a given group is defined as follows:
`Group Benefit = (TP+FP) / (TP + FN).`
Group benefit equality compares the group benefit between two groups. If the two groups are treated equally, the group benefit should be the same.
## Equal Opportunity
Equal opportunity means that all people will be treated equally or similarly and not disadvantaged by prejudices or bias.
Mathematically, equal opportunity compares the true positive rate (TPR) between two groups. TPR is the probability that an actual positive will test positive. The true positive rate is defined as follows:
`TPR = TP/(TP+FN)`
If the two groups are treated equally, the TPR should be the same.
## Intersectional Fairness
We believe fairness should be ensured to all subgroups of the population. We extended the classical metrics (which are defined for two classes) to multiple classes. In addition, we allow multiple protected features (e.g. race *and* gender). By measuring fairness along overlapping dimensions, we introduce the concept of intersectional fairness.
To understand why we decided to go with intersectional fairness, we can take a simple example. In the figure below, we observe that equal numbers of black and white people pass. Similarly, there is an equal number of men and women passing. However, this classification is unfair because we don’t have any black women and white men that passed, and all black men and white women passed. Here, we observe bias within subgroups when we take race and gender as protected attributes.
The EEOC provides examples of past intersectional discrimination/harassment cases.[\[2\]](#reference)
For more details about the implementation of the intersectional framework, please refer to this [research paper](https://arxiv.org/pdf/2101.01673.pdf).
## Model Behavior
In addition to the fairness metrics, we suggest tracking model outcomes and model performance for each subgroup of population.
## Reference
\[^1]: USEEOC article on [*Discrimination By Type*](https://www.eeoc.gov/know-your-rights-workplace-discrimination-illegal) \[^2]: USEEOC article on [*Intersectional Discrimination/Harassment*](https://www.eeoc.gov/initiatives/e-race/significant-eeoc-racecolor-casescovering-private-and-federal-sectors#intersectional)
# Embedding Visualizations
Source: https://docs.fiddler.ai/observability/llm/embedding-visualization-with-umap
Explore our guide on embedding visualization to enhance LLM monitoring. Discover UMAP techniques, analyze high-dimensional data, and uncover patterns with ease.
Discover how embedding visualization enhances LLM monitoring and analysis by simplifying complex data relationships and delivering actionable insights. This guide explains how to implement Fiddler’s LLM visualization techniques, such as UMAP, to uncover patterns, clusters, and anomalies in high-dimensional data.
### What is Embedding Visualization and Why It Matters for LLM Monitoring
Embedding visualization is a powerful technique for understanding and interpreting complex relationships in high-dimensional data. By reducing the dimensionality of custom features into a 2D or 3D space, patterns, clusters, and outliers become easier to identify, offering valuable insights for LLM embedding analysis and vector embedding visualization.
In Fiddler, high-dimensional data like embeddings and vectors are ingested as a [Custom Feature](/observability/platform/vector-monitoring-platform#define-custom-features).
This approach allows you to visualize embeddings and perform vector embedding visualization effectively, enabling detailed analysis and uncovering hidden patterns in your data.
### Using UMAP for LLM Embedding Visualization in High-Dimensional Data
We use the [UMAP](https://umap-learn.readthedocs.io/en/latest/) (Uniform Manifold Approximation and Projection) technique to embed visualizations. UMAP is a dimensionality reduction method particularly effective at preserving the local structure of data, making it ideal for visualizing embeddings, including those from embedded LLM models. Reducing high-dimensional embeddings to a 3D space allows for easier interpretation and analysis.
UMAP supports both text embedding visualization and image embedding as a [Custom Feature](/observability/platform/vector-monitoring-platform#define-custom-features). Our [guide](#using-umap-for-llm-embedding-visualization-in-high-dimensional-data) teaches you how to apply UMAP visualizations to your application data, unlocking deeper insights into your embedded LLM data.
### How UMAP Embedding Visualization Enhances Generative Applications
UMAP embedding visualizations are extremely helpful in understanding common themes and topics in the data corpus for generative AI applications. When evaluating prompts and responses, it is paramount to see which concept clusters emerge and which exhibit the most problems. Users can identify the clusters with the most issues by coloring them with various LLM and GenAI correctness and safety metrics.
To create an embedding visualization chart, follow the Guide [here](#using-umap-for-llm-embedding-visualization-in-high-dimensional-data).
# Enrichments
Source: https://docs.fiddler.ai/observability/llm/enrichments
Explore our guide on how Fiddler can enrich your LLM application's data to help analyze and evaluate application behavior and performance.
For a complete list of all LLM enrichments with output columns and sub-metrics, see the [LLM Observability Metrics Reference](/reference/llm-observability-metrics).
## Introduction to Enrichments
Enrichments are specialized evaluation features that augment your LLM application data with automatically generated trust and safety metrics. By defining enrichments during model onboarding, you instruct Fiddler to analyze published prompts and responses using purpose-built models that assess dimensions like faithfulness, toxicity, relevance, and safety compliance.
The enrichment framework processes your application's inputs and outputs to generate quantitative scores that integrate directly with Fiddler's monitoring dashboards, alerting systems, and root cause analysis tools. This approach enables proactive detection of model drift, content safety violations, and performance degradation without requiring manual evaluation or external API dependencies.
* Enrichments are [custom features](/observability/platform/vector-monitoring-platform#define-custom-features) designed to augment the data provided in events
* Enrichments augment existing columns with new metrics that are defined during model onboarding
* The new metrics are available for use within the analysis, charting, and alerting functionalities in Fiddler
The following example demonstrates how to configure a TextEmbedding enrichment to enable vector-based monitoring and visualization of your LLM application's text inputs. TextEmbedding enrichments convert unstructured text into high-dimensional vector representations that capture semantic meaning, enabling Fiddler to detect drift in the topics and themes of your prompts or responses over time.
In this configuration, the enrichment transforms text from the `question` column into numerical embeddings stored in `question_embedding`. These embeddings power Fiddler's 3D UMAP visualizations, allowing you to visually identify clusters of similar content, detect outliers, and spot shifts in user behavior patterns. The `TextEmbedding` feature also enables drift detection by comparing the distribution of embeddings between your baseline and production data, providing early warning when your application encounters significantly different types of content than expected.
```py theme={null}
import fiddler as fdl
# Define a TextEmbedding enrichment for vector-based monitoring
fiddler_custom_features = [
fdl.TextEmbedding(
name='question_cf', # Internal name for the custom feature
source_column='question', # Original text column to analyze
column='question_embedding', # Generated embedding vector column
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
***
### Embedding
Embeddings are numerical representations (vectors) generated by a model for input text. Each number within the vector represents a different dimension of the text input. The meaning of each number depends on how the embedding generating model was trained.
Fiddler uses publicly available embeddings to power the 3D UMAP experience. Because the same model generates all embeddings, the points will naturally cluster, enabling quick visual anomaly detection.
To create embeddings and leverage them for the UMAP visualization, you must create a new TextEmbedding enrichment on your unstructured text column. If you want to bring your own embeddings onto the Fiddler platform, you can direct Fiddler to consume the embeddings vector directly from your data.
**Example 1: Fiddler-Generated Embeddings**
This example automatically generates text embeddings on a text column called `prompt`:
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.TextEmbedding(
name='Prompt TextEmbedding', # name of generated column (for internal use, required)
source_column='prompt', # source - raw text
column='Enrichment Prompt Embedding', # name of the vector output
n_clusters=5, # Number of clusters for k-means clustering
n_tags=5, # Top n tags used as summary tags per cluster
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ----------------------------------------- | ------- | -------------------------------------------------- |
| Enrichment Prompt Embedding | vector | Embeddings corresponding to string column `prompt` |
| Prompt Text Embedding | integer | Column for internal use |
| Prompt Text Embedding - Centroid Distance | float | Centroid Distance to string column `prompt` |
**Example 2: User-Provided Embeddings**
This example demonstrates leveraging an existing vector column of text embeddings called `pre_existing_prompt_embedding`:
Vector embeddings must use data type List(float) for compatibility. Using CSV for source data often requires preprocessing after initial loading into a pandas dataframe.
```python theme={null}
import ast
import numpy as np
import pandas as pd
import fiddler as fdl
df = pd.read_csv(PATH_TO_SAMPLE_CSV)
# Convert string representation to list
df['pre_existing_prompt_embedding'] = df['pre_existing_prompt_embedding'].apply(
ast.literal_eval
)
fiddler_custom_features = [
fdl.TextEmbedding(
name='User-provided Text Embedding', # name of the generated column
source_column='prompt', # source - raw text
column='pre_existing_prompt_embedding', # name of your vector column
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt'],
metadata=['pre_existing_prompt_embedding'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------------------------ | ------- | ------------------------------------------- |
| User-provided Text Embedding | integer | Column for internal use |
| User-provided Text Embedding - Centroid Distance | float | Centroid Distance to string column `prompt` |
***
### Centroid Distance
Fiddler uses KMeans to determine cluster membership for a given enrichment. The Centroid Distance enrichment provides information about the distance between the selected point and the closest centroid. Centroid Distance is automatically added if the TextEmbedding enrichment is created for any given model.
Centroid Distance columns are automatically generated when you create a TextEmbedding enrichment. See the [Embedding](#embedding) section above for complete usage examples.
***
### Personally Identifiable Information
The PII (Personally Identifiable Information) enrichment tool is a critical tool for detecting and flagging sensitive information in textual data. Whether user-entered or system-generated, this enrichment aims to identify instances where PII may be exposed, helping prevent privacy breaches and misuse of personal data. In an era where digital privacy concerns are paramount, mishandling or unintentionally leaking PII can have serious repercussions, including privacy violations, identity theft, and significant legal and reputational damage.
PII enrichment is integrated with [Presidio](https://microsoft.github.io/presidio/analyzer/languages/) for entity detection.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Rag PII',
enrichment='pii',
columns=['question'], # one or more columns
allow_list=['fiddler'], # Optional: list of strings that are white listed
score_threshold=0.85, # Optional: float value for minimum possible confidence
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------- | ---- | --------------------------------------------------------------------------------------- |
| FDL Rag PII (question) | bool | Whether any PII was detected |
| FDL Rag PII (question) Matches | str | What matches in raw text were flagged as potential PII (ex. 'Douglas MacArthur,Korean') |
| FDL Rag PII (question) Entities | str | What entities these matches were tagged as (ex. 'PERSON') |
**Supported PII Entity Types:**
CREDIT\_CARD, CRYPTO, DATE\_TIME, EMAIL\_ADDRESS, IBAN\_CODE, IP\_ADDRESS, LOCATION, PERSON, PHONE\_NUMBER, URL, US\_SSN, US\_DRIVER\_LICENSE, US\_ITIN, US\_PASSPORT
***
### Evaluate
This enrichment provides n-gram-based metrics for comparing two passages of text, such as BLEU, ROUGE, and METEOR. Created initially to compare an AI-generated translation or summary to a human-generated one, these metrics have some use in RAG summarization tasks. They score highest when the reference and generated texts contain overlapping sequences. Additionally, these metrics are not as effective for long passages of text.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='QA Evaluate',
enrichment='evaluate',
columns=['correct_answer', 'generated_answer'],
config={
'reference_col': 'correct_answer', # required
'prediction_col': 'generated_answer', # required
'metrics': ['bleu', 'rouge', 'meteor'], # optional, this is the default
}
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------------------- | ----- | ------------------------------------------------------ |
| FDL QA Evaluate (bleu) | float | BLEU score: Measures precision of word n-grams |
| FDL QA Evaluate (rouge1) | float | ROUGE-1 score: Unigram recall |
| FDL QA Evaluate (rouge2) | float | ROUGE-2 score: Bigram recall |
| FDL QA Evaluate (rougel) | float | ROUGE-L score: Longest common subsequence |
| FDL QA Evaluate (rougelsum) | float | ROUGE-L summary score |
| FDL QA Evaluate (meteor) | float | METEOR score: Precision, recall, and semantic matching |
***
### Textstat
The Textstat enrichment generates various text statistics such as character/letter count, Flesch-Kincaid, and other metrics on the target text column.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Text Statistics',
enrichment='textstat',
columns=['question'],
config={
'statistics': [
'char_count',
'dale_chall_readability_score',
]
},
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| -------------------------------------------------------------- | ----- | ------------------------------------------------ |
| FDL Text Statistics (question) char\_count | int | Character count of string in `question` column |
| FDL Text Statistics (question) dale\_chall\_readability\_score | float | Readability score of string in `question` column |
**Supported Statistics:**
char\_count, letter\_count, miniword\_count, words\_per\_sentence, polysyllabcount, lexicon\_count, syllable\_count, sentence\_count, flesch\_reading\_ease, smog\_index, flesch\_kincaid\_grade, coleman\_liau\_index, automated\_readability\_index, dale\_chall\_readability\_score, difficult\_words, linsear\_write\_formula, gunning\_fog, long\_word\_count, monosyllabcount
***
### Sentiment
The Sentiment enrichment uses NLTK's VADER lexicon to generate a score and corresponding sentiment for all specified columns. To enable, set the enrichment parameter to `sentiment`.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Question Sentiment',
enrichment='sentiment',
columns=['question'],
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------------------- | ------ | ------------------------------------------- |
| FDL Question Sentiment (question) compound | float | Raw score of sentiment |
| FDL Question Sentiment (question) sentiment | string | One of `positive`, `negative`, or `neutral` |
***
### Profanity
The Profanity enrichment is designed to detect and flag the use of offensive or inappropriate language within textual content. This enrichment is essential for maintaining the integrity and professionalism of digital platforms, forums, social media, and any user-generated content areas.
The profanity enrichment searches the target text for words from the two sources below:
* The Obscenity List from [SurgeAI](https://github.com/surge-ai/)
* Google banned words \<[https://github.com/coffee-and-fun/google-profanity-words/blob/main/data/en.txt](https://github.com/coffee-and-fun/google-profanity-words/blob/main/data/en.txt)>
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Profanity',
enrichment='profanity',
columns=['prompt', 'response'],
config={'output_column_name': 'contains_profanity'},
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| -------------------------------------------- | ---- | ------------------------------------------------------------------------- |
| FDL Profanity (prompt) contains\_profanity | bool | Indicates if input contains profanity in the value of the prompt column |
| FDL Profanity (response) contains\_profanity | bool | Indicates if input contains profanity in the value of the response column |
***
### Regex Match
The Regex Match enrichment evaluates text responses or content for adherence to specific patterns defined by regular expressions (regex). By accepting a regex as input, this metric offers a highly customizable way to check if a string column in the dataset matches the given pattern. This functionality is essential for scenarios requiring precise formatting, specific keyword inclusion, or adherence to particular linguistic structures.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Regex - only digits',
enrichment='regex_match',
columns=['prompt', 'response'],
config={
'regex': '^\d+$',
}
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'doc_0', 'doc_1', 'doc_2', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ----------------------- | -------- | ---------------------------------------------------------------------------------------- |
| FDL Regex - only digits | category | Match or No Match, depending on the regex specified in the config matching in the string |
***
### Topic
The Topic enrichment leverages the capabilities of [Zero Shot Classifier](https://huggingface.co/tasks/zero-shot-classification) models to categorize textual inputs into a predefined list of topics, even without having been explicitly trained on those topics. This approach to text classification is known as zero-shot learning, a groundbreaking method in natural language processing (NLP) that enables models to intelligently classify text they haven't encountered during training. It's beneficial for applications that require understanding and organizing content dynamically across a broad range of subjects or themes.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Topics',
enrichment='topic_model',
columns=['response'],
config={'topics': ['politics', 'economy', 'astronomy']},
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'doc_0', 'doc_1', 'doc_2', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------------------ | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| FDL Topics (response) topic\_model\_scores | list\[float] | Probability of the given column in each of the topics specified in the Enrichment config. Each float value indicates the probability of the given input being classified in the corresponding topic, in the same order as topics. Each value will be between 0 and 1. The sum of values does not equal 1, as each classification is performed independently of other topics. |
| FDL Topics (response) max\_score\_topic | string | Topic with the maximum score from the list of topic names specified in the Enrichment config |
***
### Banned Keyword Detector
The Banned Keyword Detector enrichment is designed to scrutinize textual inputs for the presence of specified terms, with a particular focus on identifying content that includes potentially undesirable or restricted keywords. This enrichment operates based on a list of terms defined in its configuration, making it highly adaptable to various content moderation, compliance, and content filtering needs.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Banned KW',
enrichment='banned_keywords',
columns=['prompt', 'response'],
config={
'output_column_name': 'contains_banned_kw',
'banned_keywords': ['nike', 'adidas', 'puma'],
},
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'doc_0', 'doc_1', 'doc_2', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------------------------------------- | ---- | ---------------------------------------------------------------------------------------------------- |
| FDL Banned KW (prompt) contains\_banned\_kw | bool | Indicates if input contains one of the specified banned keywords in the value of the prompt column |
| FDL Banned KW (response) contains\_banned\_kw | bool | Indicates if input contains one of the specified banned keywords in the value of the response column |
***
### Language Detector
The Language Detector enrichment identifies the language of the source text. This enrichment is based on a pretrained text identification model and leverages [fasttext models](https://fasttext.cc/docs/en/language-identification.html) for language detection.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Language',
enrichment='language_detection',
columns=['prompt'],
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------------------- | ------ | --------------------------------------------- |
| FDL Language (prompt) language | string | Language prediction for input text |
| FDL Language (prompt) language\_probability | float | Confidence probability of language prediction |
***
### Answer Relevance
The Answer Relevance enrichment evaluates the pertinence of AI-generated responses to their corresponding prompts. This enrichment assesses whether a response accurately addresses the question or topic posed by the initial prompt, providing a simple yet effective binary outcome: relevant or not. Its primary function is to ensure that the output of AI systems, such as chatbots, virtual assistants, and content generation models, remains aligned with the user's informational needs and intentions.
Looking for more granular relevance scoring? The [Evals SDK](/developers/quick-starts/experiments-quick-start) and [Agentic Monitoring](/getting-started/agentic-monitoring) offer **Answer Relevance 2.0** with ordinal scoring (High/Medium/Low) as part of the [RAG Health Metrics](/concepts/rag-health-diagnostics) diagnostic triad.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
answer_relevance_config = {
'prompt': 'prompt_col',
'response': 'response_col',
}
fiddler_custom_features = [
fdl.Enrichment(
name='Answer Relevance',
enrichment='answer_relevance',
columns=['prompt_col', 'response_col'],
config=answer_relevance_config,
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt_col', 'response_col'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| -------------------- | ---- | ---------------------------------------------------------------------- |
| FDL Answer Relevance | bool | Binary metric, which is True if `response` is relevant to the `prompt` |
***
### Faithfulness
The Faithfulness (Groundedness) enrichment is a binary indicator that evaluates the accuracy and reliability of the facts presented in AI-generated text responses. It specifically assesses whether the information used in the response aligns with and is grounded in the provided context, often through referenced documents or data. This enrichment plays a critical role in ensuring that the AI's outputs are not only relevant but also factually accurate, given the context it was provided.
**Faithfulness (LLM-as-a-Judge) vs Faithfulness (Centor Model):** This LLM-based Faithfulness enrichment (`faithfulness`) uses `context` and `response` config keys. [Faithfulness (Centor Model)](#faithfulness-centor-model) (`ftl_response_faithfulness`) uses Fiddler's proprietary Centor Model for Faithfulness for lower latency. For RAG pipeline diagnostics with detailed reasoning, see [RAG Faithfulness](/concepts/rag-health-diagnostics#rag-faithfulness-vs-faithfulness-centor-model) in the Evals SDK and Agentic Monitoring.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
faithfulness_config = {
'context': ['doc_0', 'doc_1', 'doc_2'],
'response': 'response_col',
}
fiddler_custom_features = [
fdl.Enrichment(
name='Faithfulness',
enrichment='faithfulness',
columns=['doc_0', 'doc_1', 'doc_2', 'response_col'],
config=faithfulness_config,
),
]
model_spec = fdl.ModelSpec(
inputs=['doc_0', 'doc_1', 'doc_2', 'response_col'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ---------------- | ---- | ---------------------------------------------------------------------------------------------------------- |
| FDL Faithfulness | bool | Binary metric, which is True if the facts used in `response` are correctly used from the `context` columns |
***
### Coherence
The Coherence enrichment assesses the logical flow and clarity of AI-generated text responses, ensuring they are structured in a way that makes sense from start to finish. This enrichment is crucial for evaluating whether the content produced by AI maintains a consistent theme, argument, or narrative, without disjointed thoughts or abrupt shifts in topic. Coherence is key to making AI-generated content not only understandable but also engaging and informative for the reader.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
coherence_config = {
'response': 'response_col',
}
fiddler_custom_features = [
fdl.Enrichment(
name='Coherence',
enrichment='coherence',
columns=['response_col'],
config=coherence_config,
),
]
model_spec = fdl.ModelSpec(
inputs=['doc_0', 'doc_1', 'doc_2', 'response_col'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------- | ---- | ---------------------------------------------------------------------------------- |
| FDL Coherence | bool | Binary metric, which is True if `response` makes coherent arguments that flow well |
***
### Conciseness
The Conciseness enrichment evaluates the brevity and clarity of AI-generated text responses, ensuring that the information is presented in a straightforward and efficient manner. This enrichment identifies and rewards responses that effectively communicate their message without unnecessary elaboration or redundancy. In the realm of AI-generated content, where verbosity can dilute the message's impact or confuse the audience, maintaining conciseness is crucial for enhancing readability and user engagement.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
conciseness_config = {
'response': 'response_col',
}
fiddler_custom_features = [
fdl.Enrichment(
name='Conciseness',
enrichment='conciseness',
columns=['response'],
config=conciseness_config,
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'doc_0', 'doc_1', 'doc_2', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------- | ---- | ---------------------------------------------------------------------------- |
| FDL Conciseness | bool | Binary metric, which is True if `response` is concise and not overly verbose |
***
### Safety
The Safety enrichment evaluates the safety of the text along eleven different dimensions: `illegal, hateful, harassing, racist, sexist, violent, sexual, harmful, unethical, jailbreaking, roleplaying`. The Safety enrichment is powered by the [Centor Model for Safety](/observability/llm/llm-based-metrics).
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Prompt Safety',
enrichment='ftl_prompt_safety',
columns=['prompt'],
config={'classifiers': ['jailbreaking', 'illegal']} # Optional: specify dimensions
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
For each dimension specified (or all 11 dimensions if not specified):
| Column | Type | Description |
| -------------------------------------------- | ----- | --------------------------------------------------------------------------- |
| FDL Prompt Safety (prompt) `dimension` | bool | Binary metric, which is True if the input is deemed unsafe, False otherwise |
| FDL Prompt Safety (prompt) `dimension` score | float | Confidence probability of safety prediction |
**Supported Dimensions:** illegal, hateful, harassing, racist, sexist, violent, sexual, harmful, unethical, jailbreaking, roleplaying
***
### Faithfulness (Centor Model)
The Faithfulness (Centor Model) enrichment is designed to evaluate the accuracy and reliability of facts presented in AI-generated text responses. It is powered by the [Centor Model for Faithfulness](/observability/llm/llm-based-metrics).
The faithfulness threshold defaults to 0.5. A response is labeled unfaithful when its faithfulness score falls below the threshold, so raising the threshold labels more responses as unfaithful (fewer are labeled faithful), while lowering it is more permissive. The ideal threshold is data-dependent — tune it against a labeled dataset so it flags as many genuinely unfaithful responses as possible without over-flagging faithful ones.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Faithfulness',
enrichment='ftl_response_faithfulness',
columns=['context', 'response'],
config={
'context_field': 'context',
'response_field': 'response',
'threshold': '0.5' # Optional parameter, default is 0.5
}
),
]
model_spec = fdl.ModelSpec(
inputs=['context', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------- | ----- | ---------------------------------------------------------------------------------------------------------- |
| FDL Faithfulness faithful | bool | Binary metric, which is True if the facts used in `response` are correctly used from the `context` columns |
| FDL Faithfulness faithful score | float | Confidence probability of faithfulness prediction |
***
### Token Count
The Token Count enrichment counts the number of tokens in a string.
This enrichment uses the [tiktoken](https://github.com/openai/tiktoken) library for token counting.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='TokenCount',
enrichment='token_count',
columns=['question'],
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------------------- | ---- | ------------------------------ |
| FDL Token Counts (question) | int | Number of tokens in the string |
***
### SQL Validation
The SQL Validation enrichment is designed to evaluate different query dialects for syntax correctness.
Query validation is syntax based and does not check against any existing schema or databases for validity.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
# The following dialects are supported:
# 'athena', 'bigquery', 'clickhouse', 'databricks', 'doris', 'drill', 'duckdb',
# 'hive', 'materialize', 'mysql', 'oracle', 'postgres', 'presto', 'prql',
# 'redshift', 'risingwave', 'snowflake', 'spark', 'spark2', 'sqlite',
# 'starrocks', 'tableau', 'teradata', 'trino', 'tsql'
fiddler_custom_features = [
fdl.Enrichment(
name='SQLValidation',
enrichment='sql_validation',
columns=['query_string'],
config={
'dialect': 'mysql'
}
),
]
model_spec = fdl.ModelSpec(
inputs=['query_string'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| -------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------- |
| SQL Validator valid | bool | True if the query string is syntactically valid for the specified dialect, False if not |
| SQL Validator errors | str | If syntax errors are found they will be present as a JSON serialized string containing a list of dictionaries describing the errors |
***
### JSON Validation
The JSON Validation enrichment is designed to validate JSON for correctness and optionally against a user-defined schema for validation.
This enrichment uses the [python-jsonschema](https://python-jsonschema.readthedocs.io) library for JSON schema validation. The defined `validation_schema` must be a valid python-jsonschema schema.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='JSONValidation',
enrichment='json_validation',
columns=['json_string'],
config={
'strict': 'true',
'validation_schema': {
'$schema': 'https://json-schema.org/draft/2020-12/schema',
'type': 'object',
'properties': {
'prop_1': {'type': 'number'}
# ... additional properties
},
'required': ['prop_1'], # ... additional required fields
'additionalProperties': False
}
}
),
]
model_spec = fdl.ModelSpec(
inputs=['json_string'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------------- | ---- | ------------------------------------------------------------------------------------------------------------------- |
| JSON Validator valid | bool | String is valid JSON |
| JSON Validator errors | str | If the string failed to parse to JSON any parsing errors will be returned as a serialized JSON list of dictionaries |
# LLM Monitoring
Source: https://docs.fiddler.ai/observability/llm/index
Explore our guide to LLM application monitoring. Learn how Fiddler generates enrichments using trust and safety metrics for alerting, analysis, and debugging.
### Monitoring LLM Applications with Fiddler
Monitoring Large Language Model (LLM) applications with Fiddler requires publishing the LLM application's inputs and outputs, including prompts, prompt context, responses, and the source documents retrieved (for RAG-based applications). Fiddler will then generate enrichments, which are LLM trust and safety metrics, for alerting, analysis, or debugging purposes.
Fiddler is a pioneer in the AI Trust domain and, as such, offers the most extensive set of AI safety and trust metrics available today.
### Selecting Enrichments to Enhance Monitoring
Fiddler offers many enrichments that each measure different aspects of an LLM application. For detailed information about which enrichment to select for any specific use case, visit [this](/observability/llm/selecting-enrichments) page. Some enrichments use Fiddler Centor Models to generate these scores. Compare Centor Models cost to third-party LLM evaluators with the [Fiddler Evals TCO calculator](https://www.fiddler.ai/evals-tco-calculator).
### Generating Enrichments with Fiddler
LLM application owners must specify the enrichments to be generated by Fiddler during model onboarding. The enrichment pipeline then generates enrichments for the LLM application's inputs and outputs as events are published to Fiddler.
Figure 1. The Fiddler Enrichment Framework
After the LLM application's raw, unstructured inputs and outputs are published to Fiddler, the enrichment framework augments them with various AI trust and safety metrics. These metrics can monitor the application's overall health and alert users to any performance degradation.
Figure 2. A Fiddler dashboard showing LLM application performance
Using the metrics produced by the enrichment framework, users can monitor LLM application performance over time and conduct root-cause analysis when problematic trends are identified.
During model onboarding, application owners can opt in to the various, ever-expanding Fiddler enrichments by specifying [Enrichments](/observability/llm/enrichments) as custom features in the Fiddler ModelSpec object.
If your application routes LLM traffic through a third-party AI gateway — Kong, AgentGateway, or LiteLLM — see [AI Gateway Integrations](/integrations/agentic-ai/ai-gateways) to capture prompts, responses, and token usage with no application code changes.
# LLM-Based Metrics
Source: https://docs.fiddler.ai/observability/llm/llm-based-metrics
Explore our guide on LLM-specific metrics useful for evaluating AI-generated content for use cases like chatbots, writing assistants, or content creation tools.
For a complete reference of all LLM enrichments, see the [LLM Observability Metrics Reference](/reference/llm-observability-metrics).
LLM-based metrics use large language models to evaluate the quality of text generated by AI. This approach is much closer to how humans judge text, making these metrics particularly useful for evaluating AI-generated content for use cases such as chatbots, writing assistants, or content creation tools.
LLM-based metrics can adapt to different topics and types of text because LLMs have been trained on a wide range of information, making them a valuable tool for developers and researchers looking to enhance the quality of AI-generated text.
Currently, Fiddler supports three types of LLM-based metrics: LLM-as-a-Judge evaluators (including RAG Health Metrics), OpenAI-based enrichments, and Fiddler Centor Model metrics.
## RAG Health Metrics (LLM-as-a-Judge Evaluators)
RAG Health Metrics are a purpose-built diagnostic triad for evaluating RAG applications. These evaluators use LLM-as-a-Judge approaches and are available in **Agentic Monitoring** and **Experiments**:
* **Answer Relevance 2.0** — Ordinal scoring (High/Medium/Low = 1.0/0.5/0.0) measuring how well the response addresses the query. Also available in LLM Observability.
* **Context Relevance** — Ordinal scoring measuring whether retrieved documents support the query. Available in Agentic Monitoring and Experiments only.
* **RAG Faithfulness** — Binary scoring (Yes/No = 1/0) assessing whether the response is grounded in retrieved documents. Also available in LLM Observability.
See [RAG Health Diagnostics](/concepts/rag-health-diagnostics) for a conceptual guide to using these evaluators together for root cause analysis.
**RAG Faithfulness vs Faithfulness (Centor Model):** These are **separate evaluators** with different architectures. RAG Faithfulness is an LLM-as-a-Judge evaluator using external LLMs with inputs `user_query`, `rag_response`, and `retrieved_documents`. Faithfulness (Centor Model) (below) is powered by the Centor Model for Faithfulness using `context` and `response`. See the [enrichment glossary](/glossary/enrichment) for a detailed comparison.
## OpenAI-based metrics
* These metrics are generated through the OpenAI API, which may introduce latency due to network communication and processing time.
* OpenAI API access token MUST BE provided by the user, which will be configured during onboarding.
* The specific model to be used for these metrics will also be chosen during onboarding.
Currently, the below metrics are OpenAI-based:
* [Answer Relevance](/observability/llm/enrichments#answer-relevance)
* [Faithfulness](/observability/llm/enrichments#faithfulness)
* [Coherence](/observability/llm/enrichments#coherence)
* [Conciseness](/observability/llm/enrichments#conciseness)
## Fiddler Centor Model metrics
* These metrics are generated through Fiddler's in-house, purpose-built SLMs.
* These metrics can be generated in air-gapped environments and do not rely on any over-the-network connection to generate such scores.
Currently, the below metrics are Fiddler Centor Model-based:
* [Safety](/observability/llm/enrichments#safety) — Evaluates safety across 11 dimensions including jailbreaking, toxicity, and harmful content. Powered by the Centor Model for Safety.
* [Faithfulness (Centor Model)](/observability/llm/enrichments#faithfulness-centor-model) — Powered by the Centor Model for Faithfulness for hallucination detection. Not to be confused with RAG Faithfulness above.
# LLM Evaluation Prompt Specs
Source: https://docs.fiddler.ai/observability/llm/llm-evaluation-prompt-specs
Prompt specs is a framework Fiddler provides for leveraging a general-purpose LLM to quickly create custom scoring functions without the need to manually tune an evaluation prompt.
## The Challenge With LLM Evaluation
When building production LLM applications, evaluation is critical, but it's often the most time-consuming bottleneck in your development workflow. Traditional approaches to creating LLM-as-a-Judge evaluators require you to:
* **Hand-craft natural language prompts** for each new use case or data schema
* **Iteratively tune prompts** through trial-and-error until the model produces reliable outputs
* **Manually rewrite entire prompt templates** every time your input fields or output categories change
* **Struggle with inconsistent results** as small prompt variations lead to unpredictable model behavior
* **Spend weeks perfecting prompts** before you can move to production monitoring
This manual process doesn't scale when you need to evaluate multiple aspects of your LLM application or adapt to evolving requirements.
## How Prompt Specs Solve This Problem
Fiddler's Prompt Specs eliminate the prompt engineering bottleneck by letting you **declare your evaluation logic using simple JSON schemas** instead of writing prompts. Here's the key difference:
### Manual Prompt Engineering (Traditional Approach)
```
You are an expert classifier. Given a news summary, classify it into one of these topics: World, Sports, Business, Technology.
News summary: {news_summary}
Consider the main subject matter and classify accordingly. Provide your reasoning.
Output format:
Topic: [your classification]
Reasoning: [your explanation]
```
*Every schema change requires rewriting and retuning the entire prompt*
### Schema-Based Evaluation Using Prompt Specs
```json theme={null}
{
"input_fields": {
"news_summary": { "type": "string" }
},
"output_fields": {
"topic": {
"type": "string",
"choices": ["Business", "Sci/Tech", "Sports", "World"]
},
"reasoning": { "type": "string" }
}
}
```
*Schema changes are simple JSON updates—no prompt rewriting needed*
## When to Use Prompt Specs
Prompt Specs are ideal for scenarios where you need:
**Domain-Specific Classification**
* Content moderation with custom policy categories for example finance-specific metrics
* Topic classification
* Product categorization using your taxonomy
**Quality Assessment**
* Response relevance scoring for your specific use case
* Conciseness and coherence evaluation
* Completeness checking for required information elements
**Safety and Compliance**
* Custom content policy enforcement
* Regulatory compliance checking (e.g., financial disclosures, medical claims)
* Brand safety evaluation with organization-specific criteria
Prompt Specs work particularly well when you need multiple output fields, such as confidence ranges and reasoning, alongside the classification and when classification accuracy is more important than the flexibility of fully custom prompts.
## Benefits for Technical Teams
**Faster Development Cycles**
* Reduce evaluation setup from weeks to hours
* Eliminate iterative prompt tuning cycles
* Enable rapid schema evolution without prompt rewrites
**Improved Reliability**
* Consistent structured output guaranteed by schema validation
* Higher classification accuracy compared to guided choice decoding
* Built-in reasoning capture for debugging and explainability
**Better Maintainability**
* Version control friendly JSON schemas
* Clear separation between evaluation logic and implementation
* Seamless integration with existing Fiddler monitoring workflows
## Benefits for Risk and Compliance Teams
**Enhanced Auditability**
* Clear, declarative evaluation criteria in version-controlled schemas
* Consistent evaluation logic across all model versions
* Traceable reasoning for every evaluation decision
**Improved Governance**
* Schema-based approach prevents evaluation drift over time
* Standardized evaluation framework across teams and use cases
* Integration with Fiddler's monitoring and alerting capabilities for continuous oversight
**Regulatory Compliance**
* Documented evaluation criteria that can be reviewed by compliance teams
* Consistent application of evaluation logic for audit purposes
* Historical tracking of schema changes and their impacts on evaluation outcomes
## How Prompt Specs Work
Prompt Specs follow a simple three-step workflow that takes you from evaluation idea to production monitoring:
### 1. Define Your Evaluation Schema
Create a JSON schema specifying your input fields and desired output format. Supported field types include:
* `string` (with optional `choices` array for categorical outputs)
* `integer`
* `boolean`
* `number`
### 2. Validate and Test Your Schema
Use Fiddler's validation and prediction APIs to ensure your schema works correctly with sample data.
### 3. Deploy for Production Monitoring
Deploy your validated Prompt Spec as a Fiddler [enrichment](/glossary/enrichment) for ongoing monitoring.
**Ready to try it yourself?** Follow our step-by-step [Quickstart Guide](/evaluate-and-test/prompt-specs-quick-start) to build your first evaluation in minutes.
## Performance and Cost Considerations
### Prompt Specs Benefits
* **Higher accuracy**: Classification accuracy is very high using meaningful field names and labels
* **Consistent structure**: Yields reliably structured LLM output without guided decoding
* **Performance optimization**: System prompts remain consistent across invocations, enabling KV caching benefits
* **Customization**: Easy to add descriptions to fields and tasks for better accuracy
**Trade-offs:**
* Structured output is not 100% guaranteed
* Less flexibility than fully custom prompts for highly nuanced evaluations
* Best suited for classification and structured scoring tasks rather than open-ended evaluation
## Integration With Fiddler's Monitoring Platform
Prompt Specs integrate seamlessly with Fiddler's existing monitoring capabilities:
**Enrichment Pipeline Integration**
* Deploy Prompt Specs as [enrichments](/glossary/enrichment) alongside other Fiddler [trust and safety metrics](/glossary/trust-score)
* Combine schema-based evaluations with [built-in enrichments](/observability/llm/enrichments) like toxicity detection and PII identification
* Use evaluation results in [alerting rules](/observability/platform/alerts-platform) and [dashboard visualizations](/observability/dashboards)
**Monitoring and Alerting**
* Set up automated alerts based on Prompt Spec outputs (e.g., alert when helpfulness scores drop below threshold)
* Track evaluation trends over time alongside other model performance metrics
* Use reasoning outputs for [root cause analysis](/observability/analytics) when issues are detected
## Validation Rules and Schema Requirements
When creating Prompt Specs, ensure your schema follows these rules:
* **Required fields**: Input fields and output fields must each have at least one item
* **Field naming**: Field names must be valid Python variable names
* **Uniqueness**: Field names must be unique between input and output sections
* **Type specification**: `type` is required for all fields and must be one of: `string`, `integer`, `number`, `boolean`
* **Optional elements**: Top-level `task_instruction` and field-level `description` are optional
* **Choices arrays**: Must have at least 2 items and are only allowed when `type` is `string`
## When Not to Use Prompt Specs
Prompt Specs may not be the best choice when:
* You need highly nuanced evaluation requiring extensive domain context that can't be captured in field descriptions
* Your evaluation criteria change frequently and unpredictably in ways that require prompt-level modifications
* You require real-time evaluation with sub-100ms latency requirements
* Your use case is already well-served by existing Fiddler enrichments like Faithfulness (Centor Model)
* You need the full flexibility of custom prompt templates
## Comparison With Other Evaluation Approaches
| Approach | Setup Time | Accuracy | Consistency | Flexibility | Maintenance |
| ----------------------------- | ---------- | --------- | ----------- | ----------- | ----------- |
| **Prompt Specs** | Hours | High | High | Medium | Low |
| **Manual Prompt Engineering** | Weeks | Very High | Medium | Very High | High |
| **Pre-built Enrichments** | Minutes | High | High | Low | None |
Prompt Specs occupy a sweet spot between the quick deployment of pre-built enrichments and the full customization of manual prompt engineering, making them ideal for customers who need domain-specific evaluation without extensive prompt tuning effort.
## Getting Started
### Prompt Specs Quick Start
The fastest way to get started is with the Prompt Spec Quick Start Guide:
**🚀** [**Follow the Quickstart Guide**](/evaluate-and-test/prompt-specs-quick-start) - Build and deploy your first evaluation in minutes
# Selecting Enrichments
Source: https://docs.fiddler.ai/observability/llm/selecting-enrichments
Learn about Fiddler’s enrichments and monitor key aspects of LLM applications. Discover the different factors to analyze for your specific use case.
For a complete reference of all LLM enrichments with output columns and sub-metrics, see the [LLM Observability Metrics Reference](/reference/llm-observability-metrics).
Fiddler offers enrichments out of the box to monitor different aspects of LLM applications. Use the below table to select the right enrichment for your specific use case.
This table provides high level information on the metric, the enrichment to use to measure the metric, if the metric uses LLMs, and if so, what LLM it uses.
If you have a use case not covered by the below enrichments out of the box, please contact your administrator.
| Metric | Metric Category | Description | Enrichment | LLM Used? | LLM Type |
| --------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | --------- | ------------ |
| [Faithfulness](/observability/llm/enrichments#faithfulness) | Hallucination | This enrichment identifies the accuracy and reliability of facts presented in AI-generated texts | `faithfulness` | Yes | OpenAI |
| [Faithfulness (Centor Model)](/observability/llm/enrichments#faithfulness-centor-model) | Hallucination | This enrichment identifies the accuracy and reliability of facts presented in AI-generated texts. It is powered by the Centor Model for Faithfulness | `ftl_response_faithfulness` | Yes | Centor Model |
| [Answer Relevance](/observability/llm/enrichments#answer-relevance) | Hallucination | This enrichment measures the pertinence of AI-generated responses to their inputs | `answer_relevance` | Yes | OpenAI |
| [Conciseness](/observability/llm/enrichments#conciseness) | Hallucination | This enrichment evaluates the brevity and clarity of AI-generated responses | `conciseness` | Yes | OpenAI |
| [Coherence](/observability/llm/enrichments#coherence) | Hallucination | This enrichment assesses the logical flow and clarity of AI-generated responses | `coherence` | Yes | OpenAI |
| [Safety](/observability/llm/enrichments#safety) | Safety | This enrichment generates 11 different safety metrics to measure texts upon. These metrics are: `illegal, hateful, harassing, racist, sexist, violent, sexual, harmful, unethical, jailbreaking, roleplaying` | `ftl_prompt_safety` | Yes | Centor Model |
| [PII](/observability/llm/enrichments#personally-identifiable-information) | Safety | This enrichment flags the presence of sensitive information within textual data | `pii` | No | |
| [Regex Match](/observability/llm/enrichments#regex-match) | Safety | This enrichment compares the text with a regular expression string | `regex_match` | No | |
| [Topic](/observability/llm/enrichments#topic) | Safety | This enrichment classifies the text into several preset dimensions using a zero-shot classifier | `topic_model` | No | |
| [Banned Keywords](/observability/llm/enrichments#banned-keyword-detector) | Safety | This enrichment detects the presence of banned keywords configured by the user | `banned_keywords` | No | |
| [Profanity](/observability/llm/enrichments#profanity) | Safety | This enrichment flags the use of offensive or inappropriate language | `profanity` | No | |
| [Language Detection](/observability/llm/enrichments#language-detector) | Safety | This enrichment identifies the language of the source text | `language_detection` | No | |
| [Evaluate](/observability/llm/enrichments#evaluate) | Text Statistics | This enrichment provides classic text evaluation methods such as BLEU, ROUGE, and Meteor | `evaluate` | No | |
| [Sentiment](/observability/llm/enrichments#sentiment) | Text Statistics | This enrichment provides sentiment analysis of the target text | `sentiment` | No | |
| [TextStat](/observability/llm/enrichments#textstat) | Text Statistics | This enrichment provides various text statistics such as character/letter count, Flesch-Kincaid, and others | `textstat` | No | |
| [Token Count](/observability/llm/enrichments#token-count) | Text Statistics | The Token Count enrichment is designed to count the number of tokens in a string. | `token_count` | No | |
| [SQLValidation](/observability/llm/enrichments#sql-validation) | Text Validation | Evaluates different query dialects for syntax correctness. | `sql_validation` | No | |
| [JSONValidation](/observability/llm/enrichments#json-validation) | Text Validation | Validates JSON for correctness and optionally against a user-defined schema. | `json_validation` | No | |
# Model UI
Source: https://docs.fiddler.ai/observability/model-ui/index
Learn about Fiddler's no-code Model Editor for streamlined ML model onboarding, featuring draft mode for iterative development and team collaboration.
### Adding and Editing Models in the UI
#### Overview
UI-based Model Onboarding lets you add models to Fiddler without writing Python code. You can define schemas, validate data, and register models directly through the Fiddler interface—making model onboarding accessible to more team members and streamlining the process.
Key benefits:
* No coding required - Onboard models through an intuitive web interface
* Faster deployment - Use automatic schema inference and guided validation to reduce setup time
* Early error detection - Identify issues like missing categories, incorrect ranges, or invalid data types immediately
* Draft mode flexibility - Save work-in-progress schemas, consult with your team, and validate sample data before publishing
### Model Onboarding Modes
Fiddler offers two approaches to model onboarding through the UI: Standard Onboarding and enhanced Draft Mode.
#### Standard Onboarding
With Standard Onboarding, you proceed directly through the complete process by:
* Uploading your dataset
* Reviewing the inferred schema
* Making necessary adjustments
* Publishing the model immediately
#### Draft Mode
Draft Mode provides additional flexibility when you need to refine your model configuration before publication:
* Iterate without commitment - Save a draft version to refine column names, data types, and inferred ranges
* Collaborate with your team - Share and review the draft before finalizing
* Validate with sample data - Upload sample data to visualize charts and validate before production rollout
* Refine as needed - Return to the onboarding process to make additional adjustments at any time
#### Technical Considerations
* File size limits: CSV uploads are limited to 2 GB
* Schema validation: All fundamental schema mismatches must be corrected before publication
* Draft limitations: While you can visualize sample data in drafts, some features like saving charts have limitations
For detailed instructions and best practices, see the [Model Editor](/observability/model-ui/model-editor) documentation.
# Model Editor
Source: https://docs.fiddler.ai/observability/model-ui/model-editor
Step-by-step instructions for onboarding ML models using Fiddler's UI-based editor, from dataset upload to schema validation and publication.
## Model Onboarding Process
1. Access your project
* Sign in to Fiddler and navigate to the Projects tab
* Select an existing project or create a new one
2. Add a model
* Select Add Model in the top-right corner
* Choose a model task type: Not Set, Regression, Binary Classification, or Multi-Class Classification
3. Upload and analyze your dataset
* Upload your CSV file
* Review the automatically inferred schema
* Verify flagged columns that need additional attention
4. Choose your onboarding approach
* For immediate publication: Enter a model name and select Create Model
* For draft mode: Select Save as Draft
5. Work with your draft (Draft Mode)
* Access your draft from the Draft Models tab
* Edit schema details as needed
* Publish sample data to validate your configuration
* When ready, publish the finalized model
For specific details and best practices see the [Model Schema Editing Guide](/observability/model-ui/model-schema-editing).
# Model Schema Editing
Source: https://docs.fiddler.ai/observability/model-ui/model-schema-editing
Learn how to modify numeric ranges, edit categorical features, and add metadata columns to keep your model schema aligned with evolving production data.
## Overview
This guide explains how to edit your model's schema in Fiddler to better align with production data. Schema editing helps you maintain accurate monitoring as your data evolves.
Key capabilities
* Adjust numeric feature ranges when real-world data deviates from your original sample data
* Edit categorical feature values to add or remove categories as new patterns emerge
* Add metadata columns to include additional contextual information for improved insights
* Customize histogram bin boundaries for numerical columns to better represent data distributions
### Adjusting numeric feature ranges
**Access the Schema tab**
1. Navigate to the Model Page of your desired model
2. Select the Schema tab
**Edit numeric column range**
1. Find the numeric column you want to adjust
2. Select the edit icon (✏️) next to the column name
3. In the dialog box, modify the minimum and/or maximum values
4. Select Update to save your changes
**Impact of changes**
* **Data drift metrics**: Changes apply to all data, including historical data
* A job will run to recalculate aggregates and update metrics
* **Data integrity metrics**: Changes only apply to new data going forward
#### Editing histogram bins
You can customize the histogram bin boundaries for numerical columns to better represent your data distribution (e.g., quantile-based bins instead of uniform).
**Edit bins for a numeric column**
1. Find the numeric column you want to adjust
2. Select the edit icon (✏️) next to the column name
3. In the dialog box, enter comma-separated bin boundary values in the **Bins** field
* Example: `350, 450, 550, 650, 750, 850`
* Values must be strictly increasing, span the column's \[min, max] range, and have at most 16 boundary values (15 bins)
4. To revert to auto-generated uniform bins, clear the Bins field
5. Select Update to save your changes
**Impact of changes**
* **Data drift metrics**: Changes apply to all data, including historical data
* A job will run to recalculate aggregates and update histogram metrics
* **Feature distribution charts**: Histogram visualizations will use the new bin boundaries
#### Editing categorical variables
**Access the Schema tab**
1. Navigate to the Model Page of your desired model
2. Select the **Schema** tab
**Edit categorical column**
1. Locate the categorical column you want to modify
2. Select the edit icon (✏️) next to the column name
3. Add or remove categories as needed
4. Select Update to save your changes
**Impact of changes**
* For both data drift and data integrity metrics:
* Changes only apply to new data going forward
* Historical data remains unchanged
#### Adding metadata columns
**Access the Schema tab**
1. Navigate to the Model Page of your desired model
2. Select the Schema tab
**Add a Metadata Column**
1. Select Add Metadata
2. Provide the required information:
* Column Name: Specify the name of the new metadata column
* Data Type: Choose a data type (integer, float, string, or boolean)
* Range: For numeric types, define minimum and maximum values
3. Select Add to save
**Impact of Changes**
* New metadata columns are effective immediately for new data
### Best Practices
* Analyze production data to set realistic range values and identify useful metadata columns
* Monitor metrics after adjustments to ensure changes effectively address your needs
* Use chart annotations for transparency to maintain a clear history of schema changes
Schema updates that change numeric column properties (min, max, bins) trigger re-computation of historical aggregates. Models with more than 10 million events in any environment are not eligible for these updates. Contact support for assistance with large-scale schema changes.
### Frequently Asked Questions
**Can I change column names or data types?**
No, changing column names or data types is not supported.
**What if I make a mistake?**
You can edit the values again and save the updated schema.
**How long do changes take to apply?**
Application time depends on dataset size and complexity. For example, processing 10 million rows over six months takes approximately 12 minutes.
**Can I delete a metadata column?**
No, metadata columns cannot be deleted once added.
**Can I customize histogram bins?**
Yes, you can set custom bin boundaries for numerical columns via the Bins field in the schema editor, or programmatically via the Python client using `model.update()`. Leave the field empty to use auto-generated uniform bins.
**What happens if I add a category that doesn't exist in the data?**
The category will be listed but won't impact existing calculations.
# Overview
Source: https://docs.fiddler.ai/observability/monitoring
Monitor production models in real-time with comprehensive observability
The future of AI is agentic—autonomous systems that reason, plan, and coordinate across multiple agents to solve complex problems. Fiddler Observability is built for this future, providing comprehensive monitoring across traditional ML models, LLM applications, and emerging multi-agent systems.
## The Challenge: Exponential Complexity
As AI evolves from static models to autonomous agents, observability complexity grows exponentially:
* **Multi-agent systems require [26x more monitoring resources](https://www.capgemini.com/insights/expert-perspectives/ai-lab-the-efficient-use-of-tokens-for-multi-agent-systems/)** than single-agent applications
* **Non-deterministic behavior** breaks traditional APM frameworks designed for predictable code
* **Cascading failures** across agent hierarchies create unprecedented debugging challenges
* **[90% of enterprises](https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-agent-survey.html)** cite security, trust, and compliance as top concerns for agentic AI
Fiddler provides the unified observability platform that scales from simple models to complex agentic workflows—all powered by the same family of Fiddler Centor Models.
## Agentic Observability
Fiddler's agentic observability provides hierarchical visibility into multi-agent systems, tracking the complete lifecycle of autonomous reasoning and coordination.
### The Five Observable Stages
Every agent operates through five distinct stages that require specialized monitoring:
```mermaid theme={null}
graph LR
Thought[1. Thought Ingest, Retrieve, Interpret] --> Action[2. Action Plan and Select Tools]
Action --> Execute[3. Execution Perform Tasks]
Execute --> Reflect[4. Reflection Evaluate and Adapt]
Reflect --> Align[5. Alignment Enforce Trust & Safety]
Align -.->|Next Iteration| Thought
style Thought fill:#e1f5ff
style Action fill:#fff4e6
style Execute fill:#e6ffe6
style Reflect fill:#f0e6ff
style Align fill:#ffe6e6
```
**Stage-by-Stage Observability:**
1. **Thought**: Monitor how agents ingest data, retrieve context, and interpret information
2. **Action**: Track planning processes, tool selection, and decision-making logic
3. **Execution**: Observe task performance, API calls, and external integrations
4. **Reflection**: Capture self-evaluation, learning signals, and adaptation decisions
5. **Alignment**: Verify trust, safety, and policy enforcement at every step
### Hierarchical Monitoring Architecture
Agentic systems operate across multiple levels of abstraction. Fiddler provides observability at each layer:
```mermaid theme={null}
graph TD
App[Application Level Overall system performance & health]
App --> Session1[Session Level User interaction & conversation flow]
App --> Session2[Session Level Parallel user sessions]
Session1 --> Agent1[Agent Level Individual agent behavior & decisions]
Session1 --> Agent2[Agent Level Multi-agent coordination]
Agent1 --> Span1[Span Level Tool calls, LLM requests, actions]
Agent1 --> Span2[Span Level Granular operation tracing]
Agent2 --> Span3[Span Level Inter-agent communication]
style App fill:#0891b2,color:#fff
style Session1 fill:#06b6d4
style Session2 fill:#06b6d4
style Agent1 fill:#22d3ee
style Agent2 fill:#22d3ee
style Span1 fill:#a5f3fc
style Span2 fill:#a5f3fc
style Span3 fill:#a5f3fc
```
**Hierarchical Root Cause Analysis:**
* Trace issues from user-facing symptoms down to individual tool calls
* Understand cross-agent dependencies and coordination failures
* Analyze patterns across sessions to identify systemic issues
* Full context preservation for debugging non-deterministic behavior
### Framework & Integration Support
**Supported Frameworks:**
* **LangGraph** - Full SDK integration with native tracing
* **Strands Agents** - Strands agent application monitoring
* **OpenTelemetry** - Standard instrumentation for custom agents
* **Custom Agents** - Fiddler Client SDK for any framework
## Unified Observability Platform
All Fiddler observability capabilities—from traditional ML to agentic systems—are powered by a unified architecture built on Fiddler Centor Models:
```mermaid theme={null}
graph TB
subgraph Trust[Fiddler Centor Models]
Safety[Centor Model for Safety 11 Dimensions]
PII[Centor Model for PII/PHI 35+ Entities]
Faith[Centor Model for Faithfulness Hallucination Detection]
Custom[Custom Metrics Domain-Specific]
end
Trust --> ML[ML Observability Drift, Performance, Integrity]
Trust --> LLM[LLM Observability Quality, Safety, RAG]
Trust --> Agent[Agentic Observability Lifecycle, Coordination, Trust]
ML --> Dash[Unified Dashboards & Analytics]
LLM --> Dash
Agent --> Dash
style Trust fill:#f0f0f0
style Safety fill:#e1f5ff
style PII fill:#e1f5ff
style Faith fill:#e1f5ff
style Custom fill:#e1f5ff
style ML fill:#fff4e6
style LLM fill:#e6ffe6
style Agent fill:#f0e6ff
style Dash fill:#ffe6e6
```
**Centor Models Advantages:**
* **10-100x faster** than general-purpose LLMs for evaluation tasks
* **Purpose-built models** optimized for safety, quality, and accuracy assessment
* **Consistent, deterministic** evaluation at scale
* **Air-gapped deployment** options for data sovereignty
* **GDPR, HIPAA, CCPA** compliant monitoring
## Core Capabilities
### LLM Monitoring
Comprehensive observability for generative AI applications with trust and safety at the core.
**Key Features:**
* **14+ Enrichment Metrics**: Auto-generated trust, safety, and quality scores
* **RAG Monitoring**: Retrieval quality, source relevance, groundedness
* **Embedding Analysis**: UMAP visualization, drift detection, clustering
* **Prompt & Response Tracking**: Full conversation history and context
**Trust & Safety Metrics:**
* Safety (toxicity, jailbreaking, harmful content)
* Privacy (PII/PHI detection across 35+ entity types)
* Quality (faithfulness, coherence, conciseness, relevance)
* Sentiment and tone analysis
* [llm](/observability/llm)
### ML Model Observability
Battle-tested monitoring for traditional machine learning models in production.
**Key Features:**
* **Drift Detection**: JSD and PSI metrics for distribution shifts
* **Performance Tracking**: Accuracy, precision, recall, F1 across all deployments
* **Data Integrity**: Missing values, type mismatches, range violations
* **Traffic Monitoring**: Volume patterns and anomaly detection
* **Vector Monitoring**: Specialized tools for embedding-based applications
**Advanced Capabilities:**
* Model segmentation and cohort analysis
* Class imbalance handling
* Statistical analysis (mean, std, distributions)
* Model version comparison
* Custom formula-based metrics
* [platform](/observability/platform)
### Analytics & Root Cause Analysis
Deep-dive investigation tools for understanding performance issues and data quality problems.
**Four-Part Analysis Experience:**
1. **Events**: Browse sample of 1,000 recent events for pattern recognition
2. **Data Drift**: Feature-by-feature drift breakdown with prediction impact
3. **Data Integrity**: Violation summaries (range, type, missing value issues)
4. **Analyze**: Interactive charts for performance and feature analytics
**Chart Types:**
* Performance Analytics (confusion matrices, prediction scatterplots)
* Feature Analytics (distributions, correlations, feature matrices)
* Metric Cards (single KPI visualization)
* [analytics](/observability/analytics)
### Dashboards & Visualization
Customizable dashboards for monitoring your entire AI portfolio.
**Features:**
* **Auto-Generated Insights**: Every model gets an out-of-the-box dashboard
* **Custom Dashboards**: Build your own views with flexible layouts
* **Model Comparison**: Side-by-side performance tracking
* **Multi-Column Plots**: Drift and integrity across all features
* **Interactive Controls**: Date ranges, timezones, bin sizes, zoom
* **Collaboration**: Save and share dashboards across teams
* [dashboards](/observability/dashboards)
### Alerting & Response
Proactive monitoring with intelligent alerting across all AI systems.
**Alert Types:**
* **Drift Alerts**: Detect distribution shifts in production data
* **Data Integrity Alerts**: Flag missing values, type mismatches, range violations
* **Performance Alerts**: Monitor accuracy degradation over time
* **Custom Metric Alerts**: Formula-based alerts for business KPIs
* **Traffic Alerts**: Volume and pattern anomaly detection
**Alert Features:**
* Warning and critical threshold configuration
* Multiple notification channels (email, Slack, PagerDuty, webhooks)
* Triggered revisions with real-time updates
* Template-based alert creation
* Alert history and audit logs
## Getting Started
### Choose Your Path
**For LLM Applications:**
* [LLM Monitoring Quick Start](/getting-started/llm-monitoring) - Set up enrichments and quality tracking
* [LLM-Based Metrics Guide](/observability/llm/llm-based-metrics) - Configure trust and safety metrics
**For Traditional ML Models:**
* [ML Observability Quick Start](/getting-started/ml-observability) - Deploy drift detection and performance monitoring
* [Monitoring Platform Guide](/observability/platform) - Configure alerts and data integrity checks
**For Agentic Systems:**
* [Agentic Monitoring Quick Start](/getting-started/agentic-monitoring) - Set up hierarchical tracing with LangGraph
* [Agentic Observability Concepts](/glossary/agentic-observability) - Understand the agent lifecycle and monitoring approach
### Additional Resources
**Platform Guides:**
* [Analytics Deep Dive](/observability/analytics) - Root cause analysis and investigation
* [Custom Dashboards](/observability/dashboards) - Build monitoring views for your team
**Integration Documentation:**
* [Python Client SDK Reference](/sdk-api/python-client/connection) - Programmatic access to all features
# Alerts
Source: https://docs.fiddler.ai/observability/platform/alerts-platform
Discover how to enhance monitoring with Alerts. Learn about alert types and how to set up and view them using the alerts tab in the navigation bar.
### Overview
Fiddler enables users to set up alert rules to track a model's health and performance over time. Fiddler alerts also enable users to dig into triggered alerts and perform root-cause analysis to identify what is causing a model to degrade. Users can set up alerts using the [Fiddler Python client SDK](/sdk-api/python-client/connection) and the Fiddler UI as demonstrated below.
### Supported Metric Types
You can get alerts for the following metrics:
* [**Traffic**](/observability/platform/traffic-platform)
* The volume of traffic received by the model over time indicates the overall system's health.
* [**Statistics**](/observability/platform/statistics)
* Metrics for monitoring basic column aggregations.
* [**Data Drift**](/observability/platform/data-drift-platform)
* Model performance can be poor if models trained on a specific dataset encounter different data in production.
* [**Data Integrity**](/observability/platform/data-integrity-platform)
* Three types of violations can occur at model inference: missing feature values, type mismatches (e.g. sending a float input for a categorical feature type) or range mismatches (e.g. sending an unknown US State for a State categorical feature).
* [**Performance**](/observability/platform/performance-tracking-platform)
* The model performance tells us how well a model performs on its task. A poorly performing model can have significant business implications.
* [**Custom Metrics**](/observability/platform/custom-metrics)
* Create your own custom metric formulas and create Alerts on those metrics giving you ultimate flexibility in alerting behavior.
### Alert Configurations
#### Comparison Types
There are two options for alert threshold comparison:
* **Absolute** — Compare the metric to an absolute value
* Example: if traffic for a given hour is less than a threshold of 1,000, trigger alert.
* **Relative** — Compare the metric to a previous period
* Example: if traffic is down 10% or more than it was at the same time one week ago, trigger alert.
#### Alert Rule Priority & Severity
* **Priority**: Whether you're setting up an alert rule to keep tabs on a model in a test environment or for production scenarios, Fiddler has you covered. Easily set the Alert Rule Priority to indicate the importance of any given Alert Rule. Users can select from Low, Medium, and High priorities.
* **Severity**: Up to two threshold values can be specified for additional flexibility. A **Critical** severity threshold value is always required when setting up an Alert Rule, and a **Warning** threshold value is optional.
### Why do we need alerts?
* It’s not possible to manually track all metrics 24/7.
* Sensible alerts are your first line of defense, and they are meant to warn about issues in production.
#### Setting up Alert Rules
To create a new alert in the Fiddler UI, click Add Alert on the Alerts tab.
1. Fill in the Alert Rule form with basic details like alert name, project, and model.
2. Choose an Alert Type (Traffic, Data Drift, Data Integrity, Performance, Statistic, or Custom Metric) and set up specific metrics, bin size, and columns.
3. Define comparison methods, thresholds, and notification preferences. Click Add Alert Rule to finish.
In order to create and configure alerts using the Fiddler Python client SDK see [Alert Configuration with Fiddler Client](/developers/python-client-guides/alerts-with-fiddler-client).
#### Alert Notification options
You can select the following notification types for your alert.
#### Delete an Alert Rule
Delete an existing alert by clicking on the overflow button (⋮) on the right-hand side of any Alert Rule record and clicking `Delete`. To make any other changes to an Alert Rule, you will need to delete the alert and create a new one with the desired specifications.
#### Triggered Alert Revisions
Say goodbye to stale alerts! Triggered Alert Revisions mark a leap forward in alert intelligence, giving you the confidence to act decisively and optimize your operations.
Alerts now adapt to changing data. If new information emerges that alters an alert's severity or value, the alert automatically updates you with highlights in the user interface and revised notifications. This approach empowers you to:
* Make informed decisions based on real-time data: No more relying on outdated or inaccurate alerts.
* Focus on critical issues: Updated alerts prioritize the most relevant information.
Inspect Alert experience
### Sample Alert Email
Here's a sample of an email that's sent if an alert is triggered:
### Integrations
The Integrations tab is a read-only view of all the integrations your Admin has enabled for use. As of today, users can configure their Alert Rules to notify them via email or Pager Duty services.
Admins can add new integrations by clicking on the setting cog icon in the main navigation bar and selecting the integration tab of interest.
#### Pause alert notification
This feature allows users to temporarily pause and resume notifications for specific alerts without affecting their evaluation and triggering mechanisms. It enhances user experience by providing efficient notification management.\\
**How to Use**
**Using the Fiddler User Interface (UI)**
* Locate the Alert Tool:
Navigate to the alert rule table and identify the desired alert.
* Toggle Notifications:
* Click the notification bell icon.
* The icon updates to indicate the new state (paused or resumed).
* Confirm Action:
* A loading indicator and a toast notification confirm the action.
**Using the Fiddler Client SDK**
For programmatic control, use the Fiddler client SDK's alert-rules method with the enable\_notification argument.
* Details:
Refer to the [Fiddler Python client SDK Reference](/sdk-api/langgraph/fiddler-client) for a complete explanation of SDK features.
**Note**
* No Impact on Evaluation:
Pausing notifications does not affect the evaluation of alert conditions. The alert tool will continue to assess conditions and trigger alerts as usual.
# Class Imbalanced Data
Source: https://docs.fiddler.ai/observability/platform/class-imbalanced-data
Explore how Fiddler uses weighting to help improve drift detection when class distribution is highly imbalanced.
### Discover How Class Imbalanced Data Impacts Feature Drift
Drift is a measure of how different the production distribution is from the baseline distribution on which the model was trained. In practice, the distributions are approximated using histograms and then compared using divergence metrics like Jensen–Shannon divergence or Population Stability Index. Generally, when constructing the histograms, every event contributes equally to the bin counts.
However, for scenarios with large class imbalance the minority class’ contribution to the histograms would be minimal. Hence, any change in production distribution with respect to the minority class would not lead to a significant change in the production histograms. Consequently, even if there is a significant change in distribution with respect to the minority class, the drift value would not change significantly.
To solve this issue, Fiddler monitoring provides a way for events to be weighted based on the class distribution. For such models, when computing the histograms, events belonging to the minority class would be up-weighted whereas those belonging to the majority class would be down-weighted.
### Solving Issues with Class Imbalanced Data
Fiddler has implemented two solutions for class imbalance use cases.
#### Workflow 1: User provided global class weights
* The user computes the class distribution on baseline data and then provides the class weights via the Model-Info object.
* Class weights can either be manually entered by the user or computed from their dataset
* To tease out drift in a class-imbalanced fraud use case, check the [class-imbalanced-notebook](/developers/tutorials/ml-monitoring/class-imbalance-monitoring-example)
#### Workflow 2: User-provided event-level weights
User provides event-level weights as a metadata column in baseline data and provides them while publishing events:
* Users will add a `_weight` column of type metadata in the model's ModelSpec.
* The baseline dataset requires this `_weight` column. Note that all rows must contain valid float values. We expect the user to enforce this assumption.
* Note that using weighting parameters requires a model output in the baseline dataset.
# Custom Metrics
Source: https://docs.fiddler.ai/observability/platform/custom-metrics
Dive into our guide to enhancing ML and LLM insights with custom metrics. Learn to define, add, access, modify, and delete custom metrics in charts and alerts.
### Overview
Custom metrics offer the capability to define metrics that align precisely with your machine learning requirements. Whether it's tracking business KPIs, crafting specialized performance assessments, or computing weighted averages, custom metrics empower you to tailor measurements to your specific needs. Seamlessly integrate these custom metrics throughout Fiddler, leveraging them in dashboards, alerting, and performance tracking.
Create user-defined metrics by employing a simple query language we call [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language). FQL enables you to leverage your model's features, metadata, predictions, and outcomes for new data fields using a rich array of aggregations, operators, and metric functions, thereby expanding the depth of your analytical insights.
### How to Define a Custom Metric
Build custom metrics effortlessly with Fiddler's intuitive Excel-formula-like syntax. Once a custom metric is defined, Fiddler distinguishes itself by seamlessly managing time granularity and ranges within the charting, dashboarding, and analytics experience. This empowers you to effortlessly adjust time range and granularity without the need to modify your query, ensuring a smooth and efficient analytical experience.
### Adding a Custom Metric
From the model schema page, you can access the model's custom metrics by clicking the **Custom Metrics** tab at the top of the page. Then click **Add Custom Metric** to add a new Custom Metric. Finally, enter the name, description, and FQL definition for your custom metric and click **Save**.
### Accessing Custom Metrics in Charts and Alerts
After your custom metric is saved, you can use it in your chart and alert definitions.
#### Charts
Set `Metric Type` to `Custom Metric` and select your desired custom metric.
#### Alerts
When creating a new alert rule, set `Metric Type` to `Custom Metric`, and under the `Metric` field select your desired custom metric or author a new metric to use.
### Modifying Custom Metrics
Since alerts can be set on Custom Metrics, making modifications to a metric may introduce inconsistencies in alerts.
> 🚧 Therefore, custom metrics cannot be modified once they are created.
If you'd like to try out a new metric, you can create a new one with a different name and definition.
### Deleting Custom Metrics
To delete a custom metric using the Python client, see [CustomMetric.delete()](/sdk-api/python-client/connection). Alternatively, from the custom metrics tab, you can delete a metric by clicking the trash icon next to the metric record.
### Examples
> 📘 Custom metrics must return either:
>
> * an aggregate (produced by aggregate functions or built-in metric functions)
> * a combination of aggregates
#### Simple Metric
Given this example use case:
> If an event is a false negative, assign a value of -40. If the event is a false positive, assign a value of -400. If the event is a true positive or true negative, then assign a value of 250.
Create a new Custom Metric with the following FQL formula:
```text theme={null}
average(if(fn(), -40, if(fp(), -400, 250)))
```
Fiddler offers many convenience functions such as `fp()` and `fn()`.
Alternatively, we could also identify false positives and false negatives the old fashioned way.
```text theme={null}
average(if(Prediction < 0.5 and Target == 1, -40, if(Prediction >= 0.5 and Target == 0, -400, 250)))
```
Here, we assume `Prediction` is the name of the output column for a binary classifier and `Target` is the name of our label column.
#### Tweedie Loss
In our next example, we provide an example implementation of the Tweedie Loss Function. Here, `Target` is the name of the target column and `Prediction` is the name of the prediction/output column.
```text theme={null}
average((Target * Prediction ^ (1 - 0.5)) / (1 - 0.5) + Prediction ^ (2 - 0.5) / (2 - 0.5))
```
#### Quantile/Percentile Metrics
Quantile metrics are essential for understanding the distribution of your data and tracking percentile-based performance indicators. Unlike averages, which can be skewed by outliers, quantiles provide robust insights into the actual behavior of your model across different percentiles.
**Use Case: Monitoring ML Model Latency**
Track the median (50th percentile) inference time to understand typical model performance:
```text theme={null}
quantile(inference_time_ms, level=0.5)
```
**Use Case: SLA Compliance Tracking**
Monitor 95th percentile latency to ensure most requests meet performance SLAs:
```text theme={null}
quantile(response_time_ms, level=0.95)
```
**Use Case: Outlier Detection**
Track the 99th percentile of prediction scores to identify extreme predictions:
```text theme={null}
quantile(prediction_score, level=0.99)
```
**Use Case: Statistical Distribution Analysis**
Compute quartiles to understand the distribution of a feature or prediction:
```text theme={null}
quantile(feature_value, level=0.25) # First quartile (Q1)
quantile(feature_value, level=0.5) # Median (Q2)
quantile(feature_value, level=0.75) # Third quartile (Q3)
```
> 📘 **Why use quantiles instead of averages?**
>
> Quantiles provide a more complete picture of your data's distribution. While averages can be heavily influenced by outliers, percentiles show you the actual values at specific points in your distribution. For example, a 95th percentile latency of 500ms means 95% of your requests complete in 500ms or less, regardless of how slow the remaining 5% might be.
#### Gini Coefficient
The Gini coefficient, derived from the Lorenz curve, measures how well a model's predicted scores rank actual values. It is available for all ML model task types, including regression. Higher values indicate better ranking ability (the normalized Gini coefficient yields 1 for a perfect model and 0 for a random model).
`gini()` is available for ML custom metrics only. It is not supported for GenAI or agentic applications.
Both `actual` and `predicted` are required keyword arguments of type `Number`. Column names must match your model schema exactly (case-sensitive).
**Use case: Basic Gini coefficient**
```text theme={null}
gini(actual=Target, predicted=Score)
```
**Use case: Normalized Gini coefficient**
The `gini()` function returns the raw Gini coefficient. To obtain the normalized version — scaled so a perfect model yields 1 — divide by the ideal Gini where actuals are perfectly ranked by themselves:
```text theme={null}
gini(actual=Target, predicted=Score) / gini(actual=Target, predicted=Target)
```
**Use case: Binary classification with categorical targets**
For binary classification models where the target column is categorical (e.g. `'yes'`/`'no'`), use an `if` expression to convert to numeric 0/1 before passing to `gini()`:
```text theme={null}
gini(actual=if(churn == 'yes', 1, 0), predicted=predicted_churn) / gini(actual=if(churn == 'yes', 1, 0), predicted=if(churn == 'yes', 1, 0))
```
**Use case: Deriving normalized Gini from AUC (binary classification only)**
For binary classification models, the normalized Gini coefficient is mathematically equivalent to `2 * AUC - 1`. You can use the built-in `auroc()` function to compute this directly without needing a separate `actual` column:
```text theme={null}
2 * auroc() - 1
```
#### Min and Max
Use `min()` and `max()` to compute the minimum or maximum value of a column or expression across all rows in a time window.
**Use case: Feature range (spread)**
Track how wide the distribution of a feature is over time. A growing range may indicate data drift.
```text theme={null}
max(Prediction) - min(Prediction)
```
**Use case: Min/max ratio**
Monitor how concentrated predictions are. A ratio close to 1 indicates a narrow spread; a ratio near 0 suggests extreme outliers are present.
```text theme={null}
min(Prediction) / max(Prediction)
```
`min(x)` and `max(x)` aggregate a single expression **across rows**. To compare multiple already-computed aggregates, use `least(...)` and `greatest(...)` instead.
#### Null-Safe Expressions
Use the `null` keyword as a return value in conditional expressions to explicitly propagate missing data rather than substituting a placeholder.
**Use case: Null-safe transformation**
Apply a transformation to a column, but return `null` for rows where the input is missing instead of coercing a default value.
```text theme={null}
average(if(is_null(Prediction), null, Prediction * 100))
```
This computes the average only over rows where `Prediction` is non-null, which is equivalent to `average(Prediction) * 100` but makes the null-handling intent explicit.
To test whether a value is null, always use `is_null(x)` or `is_not_null(x)`. The `null` keyword is intended for use as a **return value** only in the `if(condition, value, null)` expression.
### Modifying Custom Metrics
Since alerts can be set on Custom Metrics, modifying a metric may introduce inconsistencies in those alerts.
> 🚧 Therefore, custom metrics cannot be modified once they are created.
If you'd like to try out a new metric, you can create a new one with a different name and definition.
# Data Drift
Source: https://docs.fiddler.ai/observability/platform/data-drift-platform
Learn about data drift and how Fiddler can monitor your ML model data for drift to provide early detection of issues that could impact model performance.
For a complete list of all built-in ML metrics including drift, see the [ML Metrics Reference](/reference/ml-metrics-reference).
### Monitor Model Performance Changes with Fiddler's Insights
#### Monitoring Data Drift in ML Models
Model performance can be poor if models trained on a specific dataset encounter different data in production. This is called data drift and it is a metric which is available on model inputs, outputs, and custom features. On the **Insights** dashboard for your model, Fiddler gives you a diverse set of visuals to explore different metrics.
Leverage the data drift chart to identify what data is drifting, when it’s drifting, and how it’s drifting. This is the first step in identifying possible model performance issues.
#### Drift Metrics Details
Fiddler supports the following:
* ***Drift Metrics***
* **Jensen–Shannon distance (JSD)**
* A distance metric calculated between the distribution of a field in the [baseline](/glossary/baseline) and that same distribution for the time period of interest. For numerical columns, distributions are approximated using histograms with [configurable bin boundaries](/developers/python-client-guides/model-onboarding/customizing-your-model-schema#modifying-the-histogram-bins).
* For more information on JSD, click [here](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.jensenshannon.html).
* **Population Stability Index (PSI)**
* A drift metric based on the multinomial classification of a variable into bins or categories. For numerical columns, the bin boundaries can be customized via the column's `bins` property — see [Customizing Your Model Schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema#modifying-the-histogram-bins). By default, 10 uniform bins are auto-generated from the column's min and max values. The differences in each bin between the baseline and the time period of interest are then utilized to calculate it as follows:
> 🚧 Note
>
> There is a possibility that PSI can shoot to infinity. To avoid this, PSI calculation in Fiddler is done such that each bin count is incremented with a base\_count=1. Thus, there might be a slight difference in the PSI values obtained from manual calculations.
* ***Average Values*** – The mean of a field (feature or prediction) over time. This can be thought of as an intuitive drift score.
* ***Drift Analytics*** – You can drill down into the features responsible for the prediction drift using the table at the bottom.
* ***Feature Impact***: The contribution of a feature to the model’s predictions, averaged over the [baseline](/glossary/baseline). The contribution is calculated using random ablation feature impact.
* ***Feature Drift***: Drift of the feature, calculated using the drift metric of choice.
* ***Prediction Drift Impact***: A heuristic calculated using the product of the feature impact and the feature drift. The higher the score, the more this feature is likely to have contributed to the prediction drift.
### Determining Drift Root Cause
In the Root Cause Analysis table of your drift charts, you can select a feature to see the feature distribution for both the time period under consideration and the baseline dataset.
### Why Track Data Drift?
* Data drift is a great proxy metric for **performance decline**, especially if there is delay in getting labels for production events. (e.g. In a credit lending use case, an actual default may happen after months or years.)
* Monitoring data drift also helps you stay informed about **distributional shifts in the data for features of interest**, which could have business implications even if there is no decline in model performance.
#### Taking Action on Observed Data Drift
* High drift can occur as a result of *data integrity issues* (bugs in the data pipeline), or as a result of *an actual change in the distribution of data* due to external factors (e.g. a dip in income due to COVID). The former is more in our control to solve directly. The latter may not be solvable directly, but can serve as an indicator that further investigation (and possible retraining) may be needed.
* You can drill down deeper into the data by examining it in the Analyze tab.
# Data Integrity
Source: https://docs.fiddler.ai/observability/platform/data-integrity-platform
Dive into our guide on ensuring data integrity in ML models and LLMs. Learn to monitor violations with Fiddler’s auto-generated charts and alerts.
For a complete list of all built-in ML metrics including data integrity, see the [ML Metrics Reference](/reference/ml-metrics-reference).
### Overview
ML models are increasingly driven by complex feature pipelines and automated workflows that involve dynamic data. Data is transformed from source to model input which can result in data inconsistencies and errors.
There are three types of violations that can occur at model inference: **missing values**, **type mismatches** (e.g. sending a float input for a categorical feature type) or **range violations** (e.g. sending an unknown US State for a State categorical feature).
You can monitor all these violations with auto-generated Data Integrity charts and alerts, or create your own custom alerts and charts. The time series shown below tracks the violations of data integrity constraints set up for this model.
### What Is Being Tracked?
The time series chart above tracks the violations of data integrity constraints set up for this model. Note that both raw count and percentage are available for data integrity metrics.
* **Any Violation Any Column** — The count of any type of data integrity violation over all features for a given period of time.
* **% Any Violation Any Column** — The percentage of any type of data integrity violation over all features for a given period of time.
* **NULL Count Any Column** — The count of missing value violations over all features for a given period of time.
* **Range Violation Count Any Column** — The count of range violations over all features for a given period of time.
* **Type Violation Count Any Column** — The count of data type violations over all features for a given period of time.
### Why is it being tracked?
Data integrity issues can cause incorrect data to flow into the model, leading to poor model performance and negatively impacting the business or end-user experience.
### How does it work?
Setting up constraints for individual features when they number in the tens or hundreds can be tedious. To avoid this, the schema of a model is used as a reference to detect when features in the incoming production logs deviate from expected patterns established during model training. For example, feature values may be out of range (numerical inputs) or contain unknown values (categorical inputs). The minimums, maximums, and distinct categorical values for a model's features are collected during initial model onboarding and stored in the Fiddler model's `ModelSchema`.
Fiddler will automatically generate constraints based on the data distribution of the sample data used to generate the model schema during model onboarding.
* **Type mismatch**: A data integrity violation will be triggered when the type of a feature value differs from what was specified for that feature in the model's schema.
* **Range mismatch**:
* For categorical features, a data integrity violation will be triggered when it sees any value other than the ones specified in the model's schema.
* For continuous variables, the violation will be triggered if the values are outside the range specified in the model's schema.
* For [vector datatype](/observability/platform/vector-monitoring-platform), a range mismatch will be triggered when a dimension mismatch occurs compared to the expected dimension from the model's schema.
# Embedding Visualization
Source: https://docs.fiddler.ai/observability/platform/embedding-visualization-with-umap
Dive into our guide on embedding visualization with UMAP in Fiddler. Learn to create charts, select parameters, and interact with visualizations.
Embedding visualization is a powerful technique for understanding and interpreting complex relationships in high-dimensional data. Reducing the dimensionality of custom features into a 2D or 3D space makes identifying patterns, clusters, and outliers easier.
In Fiddler, high-dimensional data like embeddings and vectors are ingested as a [custom feature](/observability/platform/vector-monitoring-platform#define-custom-features).
Our goal in this document is to visualize these custom features.
### UMAP Technique for Embedding Visualization
We use the [UMAP](https://umap-learn.readthedocs.io/en/latest/) (Uniform Manifold Approximation and Projection) technique for embedding visualizations. UMAP is a dimension reduction technique that is particularly good at preserving the local structure of the data, making it ideal for visualizing embeddings. We reduce the high-dimensional embeddings to a 3D space.
UMAP is supported for both Text and Image embeddings using a custom feature
### Creating an Embedding Visualization Chart
To create an embedding visualization chart, follow these steps:
1. Navigate to the **Charts** tab in your Fiddler AI instance
2. Click on the **Add Chart** button on the top right
3. In the modal, select the project that has a model with Custom features
4. Select **Embedding Visualization**.
#### Chart Parameters
When creating an embedding visualization chart, you will need to specify the following parameters:
* Model and model version
* Embedding column
* Display columns
* Baseline
* Segment
* Date range
* Sample size
* Advanced fields
* Number of neighbors
* Minimum distance
* Distance metric
Please see below for details on these parameters.
#### Model
Select the model containing at least one embedding column. You may further refine to a model version if required.
#### Embedding Column
Choose the embedding column from your dataset that you wish to visualize.
#### Display Columns
Select the columns for which you want to display additional information when hovering over points in the visualization. When plot points are selected, these additional display columns will also be available in the data cards.
#### Baseline
Select a baseline for comparison. This is optional and will be helpful when comparing datasets, such as a pre-production dataset with a production dataset or two time periods in production.
#### Segment
Select an existing segment (or define a new segment) to filter the chart to a particular data cohort. This is optional, but it will be helpful when focusing on a specific cohort.
#### Sample Size
Decide the number of samples you want to include for performance and clarity in the visualization. Currently, sample sizes between 100 and 10,000 can be selected. In future releases, we will enable support for larger sample sizes.
#### Number of Neighbors
This parameter controls how UMAP balances local versus global structure in the data. It determines the number of neighboring points used in the manifold approximation. Low values of this parameter, such as 5, will lead UMAP to focus too much on the local structure, losing sight of the big picture. Conversely, bigger values will lead to a focus on the broader data. It is important to experiment on your dataset and use case to identify the value that provides the best results. Values from 2 to 100 are supported.
#### Minimum Distance
Controls how closely points can be placed to each other in the visualization. A smaller value (such as 0.1) allows points to cluster more tightly, revealing finer details and local structures in your data. A larger value forces points to spread out more evenly across the visualization space.
### Interactions on Embedding Visualization
### Choose Different Periods
When generating the embedding visualization, you can choose different periods of production data to analyze. To do this:
* Access the Date Range selector.
* Choose the start and end dates for the period you are interested in.
* The visualization will update to reflect the embeddings from the selected date range.
### Color By
The 'Color By' feature enriches the visualization by categorizing your data points using different colors based on attributes.
* Find the 'Color By' dropdown in your control panel.
* Choose a categorical feature to color-code the data points. For example, select "data source" to color the data points according to whether they are baseline or production data.
Using the 'Color By' feature can help uncover patterns in your data. For instance, in the above image, data points with varying 'target' column values demonstrate clustering, where similar values tend to group.
You can also select points to delve deeper for further inspection. This ability to interactively color and select data points may be very useful for root cause analysis.
### Zoom
Zooming in on the UMAP chart provides a closer look at clusters and individual data points.
* Use the mouse scroll wheel to zoom in or out.
* Click and drag the mouse to move the zoomed-in area around the chart.
* Zooming helps to focus on areas of interest or to distinguish between closely packed points.
### Selection of Data Points
You can select individual or groups of data points to analyze further.
* Click on a data point to select it. Or use the Selector on the top right to select multiple points
### Data Cards
* Selected points will be highlighted on the chart, and details of the display columns of these cards are displayed in data cards, as shown below
* Use this feature to identify and analyze specific data points
In the following example, we use the categorical attribute "feedback", which contains three possible values: like, dislike, or None, as the legend indicates. After applying the 'color by' feature, the user selects specific data points to examine in greater detail. The selected data points are then presented as data cards below.
### Hover on a Data Point
Hovering over a data point reveals additional information about it, providing immediate insight without the need for selection.
* Move the cursor over a data point on the chart
* A tooltip will appear, displaying the data associated with that point, such as values of different display columns
* Use this feature to quickly look up data without altering your current selection on the chart
## Saving the Chart
Once you're satisfied with your visualization, you can save the chart. This chart can then be added to a dashboard. This allows you to revisit the UMAP visualization at any time easily, either directly from the Chart or from the dashboard.
# Fiddler Query Language
Source: https://docs.fiddler.ai/observability/platform/fiddler-query-language
Explore our guide on using Fiddler Query Language to build custom metrics to drive additional business value in dashboards and extra capability in alerting.
## Overview
[Custom Metrics](/observability/platform/custom-metrics) and [Segments](/observability/platform/segments) are defined using the **Fiddler Query Language (FQL)**, a flexible set of constants, operators, and functions which can accommodate a large variety of metrics.
## Definitions
| Term | Definition |
| ------------------ | -------------------------------------------------------------------------------------- |
| Row-level function | A function which executes row-wise for a set of data. Returns a value for each row. |
| Aggregate function | A function which executes across rows. Returns a single value for a given set of rows. |
## FQL Rules
* Column names can be referenced by name either with double quotes ("my\_column") or with no quotes (my\_column).
* Single quotes (') are used to represent string values.
## Data Types
FQL distinguishes between three data types:
| Data type | Supported values | Examples | Supported Model Schema Data Types |
| --------- | --------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------- |
| Number | Any numeric value (integers and floats are both included) |
10 2.34
|
DataType.INTEGER DataType.FLOAT
|
| Boolean | Only `true` and `false` |
true false
| `DataType.BOOLEAN` |
| String | Any value wrapped in single quotes (`'`) |
'This is a string.' '200.0'
|
DataType.CATEGORY DataType.STRING
|
## Constants
| Symbol | Description |
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `true` | Boolean constant for true expressions |
| `false` | Boolean constant for false expressions |
| `null` | Represents an absent or missing value. Use as a return value in conditional expressions — for example: `if(is_null(x), null, x * 2)`. To test whether a value is null, use `is_null()` or `is_not_null()`. |
## Operators
| Symbol | Description | Syntax | Returns | Examples |
| ------ | ------------------------ | --------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `^` | Exponentiation | `Number ^ Number` | `Number` |
|
| `and` | Logical AND | `Boolean and Boolean` | `Boolean` |
true and false column1 and column2
|
| `or` | Logical OR | `Boolean or Boolean` | `Boolean` |
true or false column1 or column2
|
The comparison operators are not limited to numbers. Each requires its two operands to share the **same** data type — numbers, strings, or booleans — and rejects mixed-type comparisons. Equality (`==`, `!=`) and the ordering operators apply the same rule, so ordering is not limited to numeric operands (string comparisons are lexicographic, and `true` is greater than `false`). The `null` literal is not a valid comparison operand — use `is_null()` or `is_not_null()` instead.
## Constant functions
| Symbol | Description | Syntax | Returns | Examples |
| ------ | ----------------------------------------------------- | ------ | -------- | --------------------------- |
| `e()` | Base of the natural logarithm | `e()` | `Number` | `e() == 2.718281828459045` |
| `pi()` | The ratio of a circle's circumference to its diameter | `pi()` | `Number` | `pi() == 3.141592653589793` |
## Row-level functions
Row-level functions can be applied either to a single value or to a column/row expression (in which case they are mapped element-wise to each value in the column/row expression).
| Symbol | Description | Syntax | Returns | Examples |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | --------- | -------------------------------------------------------------------------------------------------- |
| `if(condition, value1, value2)` |
Evaluates condition and returns value1 if true, otherwise returns value2. value1 and value2 must have the same type.
|
| `in(x, ...values)` | Returns `true` if `x` equals any of the listed values. Values must be literals of one type, matching the type of `x`. A null value for `x` returns null, which does not satisfy a filter. | `in(Any, Any, ...)` | `Boolean` | `in(country, 'US', 'CA')` |
| `not_in(x, ...values)` | Returns `true` if `x` equals none of the listed values. Same value rules as `in`. A null value for `x` returns null, which does not satisfy a filter. | `not_in(Any, Any, ...)` | `Boolean` | `not_in(country, 'US', 'CA')` |
| `abs(x)` | Returns the absolute value of number `x`. | `abs(Number)` | `Number` | `abs(-3) == 3` |
| `exp(x)` | Returns `e^x`, where `e` is the base of the natural logarithm. | `exp(Number)` | `Number` | `exp(1) == 2.718281828459045` |
| `log(x)` | Returns the natural logarithm (base `e`) of number `x`. | `log(Number)` | `Number` | `log(e) == 1` |
| `log2(x)` | Returns the binary logarithm (base `2`) of number `x`. | `log2(Number)` | `Number` | `log2(16) == 4` |
| `log10(x)` | Returns the common logarithm (base `10`) of number `x`. | `log10(Number)` | `Number` | `log10(1000) == 3` |
| `sqrt(x)` | Returns the positive square root of number `x`. | `sqrt(Number)` | `Number` | `sqrt(144) == 12` |
**Using `in(...)` and `not_in(...)`**
* **Every value must be a literal** — a quoted string, a number, or a boolean. Expressions are not allowed in the value list, so `in(country, region)` is rejected even when `region` is a valid column.
* **All values must share one data type**, and that type must match the tested expression. `in(country, 'US', 5)` is rejected because the values are mixed.
* **At least one value is required.** `in(country)` is a parse error.
* **A null value for `x` returns null.** Both `in(x, ...)` and `not_in(x, ...)` return null when `x` is null, and a null result does not satisfy a filter — so a null row is excluded by both. To also match missing values, combine the expression with the `or` operator — `not_in(country, 'US') or is_null(country)`.
## Aggregate functions
Every Custom Metric must be wrapped in an aggregate function or be a combination of aggregate functions.
| Symbol | Description | Syntax | Returns | Examples |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sum(x)` | Returns the sum of a numeric column or row expression `x`. | `sum(Number)` | `Number` | `sum(column1 + column2)` |
| `average(x)` | Returns the arithmetic mean/average value of a numeric column or row expression `x`. | `average(Number)` | `Number` | `average(2 * column1)` |
| `count(x)` | Returns the number of non-null rows of a column or row expression `x`. | `count(Any)` | `Number` | `count(column1)` |
| `min(x)` | Returns the minimum value of a numeric column or row expression `x`. | `min(Number)` | `Number` | `min(column1)` |
| `max(x)` | Returns the maximum value of a numeric column or row expression `x`. | `max(Number)` | `Number` | `max(column1)` |
| `quantile(x, level)` | Returns the quantile (percentile) of a numeric column or row expression `x` at the specified `level`. The `level` must be a constant between 0.0 and 1.0 (defaults to 0.5 if not specified). | `quantile(Number, level=Number)` | `Number` |
|
| `greatest(...aggs)` | Returns the maximum value across one or more numerical aggregates. | `greatest(Number, ...)` | `Number` | `greatest(sum(col1), sum(col2), sum(col3))` |
| `least(...aggs)` | Returns the minimum value across one or more numerical aggregates. | `least(Number, ...)` | `Number` | `least(sum(col1), sum(col2), sum(col3))` |
**`min(x)` / `max(x)` vs `least(...)` / `greatest(...)`**: `min(x)` and `max(x)` aggregate a single row-level expression **across rows** (e.g., `min(column1)` returns the smallest value of `column1` across all rows in the time window). `least(...)` and `greatest(...)` compare **multiple aggregate results** and return the smallest or largest among them (e.g., `least(sum(col1), sum(col2))` compares two already-computed sums).
Built-in metric functions are available for **ML models only** (classification, regression, and ranking tasks). They are not supported in custom metrics for agentic or GenAI applications. For agentic applications, use the `attribute()` function with aggregate functions instead — see [Custom Metrics for Agentic Applications](/observability/agentic/custom-metrics).
## Built-in metric functions
| Symbol | Description | Syntax | Returns | Examples |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `jsd(column, baseline)` | The Jensen-Shannon distance of column `column` with respect to baseline `baseline`. | `jsd(Any, String)` | `Number` | `jsd(column1, 'my_baseline')` |
| `psi(column, baseline)` | The population stability index of column `column` with respect to baseline `baseline`. | `psi(Any, String)` | `Number` | `psi(column1, 'my_baseline')` |
| `null_violation_count(column)` | Number of rows with null values in column `column`. | `null_violation_count(Any)` | `Number` | `null_violation_count(column1)` |
| `range_violation_count(column)` | Number of rows with out-of-range values in column `column`. | `range_violation_count(Any)` | `Number` | `range_violation_count(column1)` |
| `type_violation_count(column)` | Number of rows with invalid data types in column `column`. | `type_violation_count(Any)` | `Number` | `type_violation_count(column1)` |
| `any_violation_count(column)` | Number of rows with at least one Data Integrity violation in `column`. | `any_violation_count(Any)` | `Number` | `any_violation_count(column1)` |
| `null_violation_percentage(column)` | Percentage of rows with null values in column `column`, out of total rows. | `null_violation_percentage(Any)` | `Number` | `null_violation_percentage(column1)` |
| `range_violation_percentage(column)` | Percentage of rows with out-of-range values in column `column`, out of total rows. Not supported for string or timestamp columns. | `range_violation_percentage(Any)` | `Number` | `range_violation_percentage(column1)` |
| `type_violation_percentage(column)` | Percentage of rows with invalid data types in column `column`, out of total rows. Not supported for string or timestamp columns. | `type_violation_percentage(Any)` | `Number` | `type_violation_percentage(column1)` |
| `any_violation_percentage(column)` | Percentage of rows with at least one Data Integrity violation in `column`, out of total rows. | `any_violation_percentage(Any)` | `Number` | `any_violation_percentage(column1)` |
| `traffic()` | Total row count. Includes null rows. | `traffic()` | `Number` | `traffic()` |
| `tp(class)` | True positive boolean state. Available for binary classification and multiclass classification models. For multiclass, `class` is used to specify the positive class. | `tp(class=Optional[String])` | `Boolean` |
tp() tp(class='class1')
|
| `tn(class)` | True negative boolean state. Available for binary classification and multiclass classification models. For multiclass, `class` is used to specify the positive class. | `tn(class=Optional[String])` | `Boolean` |
tn() tn(class='class1')
|
| `fp(class)` | False positive boolean state. Available for binary classification and multiclass classification models. For multiclass, `class` is used to specify the positive class. | `fp(class=Optional[String])` | `Boolean` |
fp() fp(class='class1')
|
| `fn(class)` | False negative boolean state. Available for binary classification and multiclass classification models. For multiclass, `class` is used to specify the positive class. | `fn(class=Optional[String])` | `Boolean` |
fn() fn(class='class1')
|
| `precision(target, threshold)` |
Precision between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
Recall between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
F1 score between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
False positive rate between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
Area under the ROC curve between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
| `auroc(target=Optional[Any])` | `Number` |
auroc() auroc(target=column1)
|
| `data_count(target)` |
Total count of scored, labeled predictions. Available for binary classification model tasks. Unlike traffic(), this counts only rows that have both a prediction and a label. If target is specified, it will be used in place of the default target column.
| `data_count(target=Optional[Any])` | `Number` |
data\_count() data\_count(target=column1)
|
| `geometric_mean(target, threshold)` |
Geometric mean score between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
Expected calibration error between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
Log loss (binary cross entropy) between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
| `log_loss(target=Optional[Any])` | `Number` |
log\_loss() log\_loss(target=column1)
|
| `calibrated_threshold(target)` |
Optimal threshold value for a high TPR and a low FPR. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.
Accuracy score between target and outputs. Available for multiclass classification model tasks. If target is specified, it will be used in place of the default target column.
Log loss score between target and outputs. Available for multiclass classification model tasks. If target is specified, it will be used in place of the default target column.
| `log_loss(target=Optional[Any])` | `Number` |
log\_loss() log\_loss(target=column1)
|
| `r2(target)` |
R-squared score between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.
| `r2(target=Optional[Any])` | `Number` |
r2() r2(target=column1)
|
| `mse(target)` |
Mean squared error between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.
| `mse(target=Optional[Any])` | `Number` |
mse() mse(target=column1)
|
| `mae(target)` |
Mean absolute error between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.
| `mae(target=Optional[Any])` | `Number` |
mae() mae(target=column1)
|
| `mape(target)` |
Mean absolute percentage error between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.
| `mape(target=Optional[Any])` | `Number` |
mape() mape(target=column1)
|
| `wmape(target)` |
Weighted mean absolute percentage error between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.
| `wmape(target=Optional[Any])` | `Number` |
wmape() wmape(target=column1)
|
| `map(target)` |
Mean average precision score. Available for ranking model tasks. If target is specified, it will be used in place of the default target column.
| `map(target=Optional[Any])` | `Number` |
map() map(target=column1)
|
| `ndcg_mean(target)` |
Mean normalized discounted cumulative gain score. Available for ranking model tasks. If target is specified, it will be used in place of the default target column.
| `ndcg_mean(target=Optional[Any])` | `Number` |
ndcg\_mean() ndcg\_mean(target=column1)
|
| `query_count(target)` |
Count of ranking queries. Available for ranking model tasks. If target is specified, it will be used in place of the default target column.
|
| `gini(actual, predicted)` | Gini coefficient derived from the Lorenz curve. Measures how well a model's predicted scores rank actual values. Available for ML custom metrics (all model task types, including regression). Both `actual` and `predicted` are required keyword arguments of type `Number`. To compute the **normalized Gini coefficient**, divide by the ideal Gini: `gini(actual=Target, predicted=Score) / gini(actual=Target, predicted=Target)`. For binary classification, the normalized Gini can also be derived from AUC: `2 * auroc() - 1`. | `gini(actual=Number, predicted=Number)` | `Number` | `gini(actual=Target, predicted=Score)` |
# Monitoring Platform
Source: https://docs.fiddler.ai/observability/platform/index
Dive into our guide to optimizing ML models and LLM applications with Fiddler’s monitoring tools. Learn key metrics to track data drift, performance, and more.
Fiddler Monitoring helps you detect performance issues in your production ML models and LLM applications. It provides five built-in metric types, plus support for custom metrics that you can monitor and track.
* [**Data Drift**](/observability/platform/data-drift-platform)
* [**Performance**](/observability/platform/performance-tracking-platform)
* [**Data Integrity**](/observability/platform/data-integrity-platform)
* [**Traffic**](/observability/platform/traffic-platform)
* [**Statistic**](/observability/platform/statistics)
* [**Custom Metrics**](/observability/platform/custom-metrics)
These metrics can be visualized through Fiddler's Charts and Dashboards, alerted on using Alerts, and extended with Custom Metrics, allowing for quick root cause analysis alerts.
# Model Versions
Source: https://docs.fiddler.ai/observability/platform/model-versions
Discover model versions in Fiddler. Learn structured approaches to managing related models, their use cases, capabilities, and how to create a model version.
### Overview
Model versions in Fiddler provide a structured approach to managing related models, offering enhanced efficiency in tasks such as model retraining and champion vs. challenger analyses. Rather than creating entirely new model instances for updates, users can opt for creating versions of existing models, maintaining their foundational structure while accommodating necessary changes. These changes can span schema modifications, including column additions or removals, adjustments to data types and value ranges, updates to model specifications, and refinement of task parameters or Explainable AI (XAI) settings.
### Use Cases
Model versions are particularly useful in scenarios where:
* **Model Retraining**: Models require updating with new data or improved algorithms without disrupting existing deployments.
* **Champion vs. Challenger Analyses**: Comparing the performance of different model versions to identify the most effective one for deployment.
* **Historical Tracking**: Maintaining a clear record of model iterations and changes over time for auditing or analysis purposes.
### Capabilities
With model versions, users can:
* **Maintain Model Lineage**: Easily trace the evolution of models by keeping track of different versions and their respective changes.
* **Efficiently Manage Updates**: Streamline the process of updating models by creating new versions with incremental changes, avoiding the need to re-upload entire model instances.
* **Flexibly Modify Schemas**: Modify model schemas, including column structures, data types, and other specifications, to adapt models to evolving requirements.
* **Adjust Parameters**: Refine task parameters, XAI settings, and other configurations to improve explainability, or tailor the task to better suit the model's purpose.
* **Ensure Consistency**: Ensure consistency in model deployments by managing related models within the same versioning system, facilitating comparisons and deployments.
### Example of Creating a Model Version
Utilizing from\_name() and duplicate() methods, we can efficiently create a new model version with modifications based on an existing model. First, we retrieve the existing model by specifying its name, project ID, and version. Subsequently, we duplicate this model while updating its version, transitioning, for instance, from `v3` to `v4`. Within the new version (`v4`), we tailor the value ranges of the 'Age' column to meet our requirements. Finally, the create() method is invoked to publish the newly minted model version `v4`.
```python theme={null}
# Define project ID and model name
PROJECT_ID = 'your_project_id'
MODEL_NAME = 'your_model_name'
# Retrieve the existing model version by name and project ID
existing_model = fdl.Model.from_name(name=MODEL_NAME, project_id=PROJECT_ID, version='v3')
# Duplicate the existing model to create a new version
new_model = existing_model.duplicate(version='v4')
# Modify the schema of the new version
new_model.schema['Age'].min = 18
new_model.schema['Age'].max = 60
# Create the new version of the model
new_model.create()
```
# Monitoring Charts
Source: https://docs.fiddler.ai/observability/platform/monitoring-charts-platform
Explore our guide to the monitoring charts UI. Learn how to create charts, explore functions, customize tabs, and track LLM metrics effectively.
### Getting Started with Fiddler’s Charts UI
The Chart UI on Fiddler AI’s platform is designed to help you effectively monitor and analyze model performance metrics. Navigating to the Charts tab in the navigation bar allows you to easily access the tools needed to visualize key data and track your LLM and ML models’ performance. Choose to open a previously saved chart for ongoing analysis or create a new one tailored to specific metrics, such as response faithfulness, jailbreak attempts, data drift, and more.
#### Supported Metric Types
Monitoring charts enable you to plot any combination of the following metric types for a given model:
* [**Traffic**](/observability/platform/traffic-platform)
* Model traffic volume over time provides insights into system usage patterns and overall operational health.
* [**Statistics**](/observability/platform/statistics)
* Aggregated metrics that monitor trends and changes in your model's data columns
* [**Data Drift**](/observability/platform/data-drift-platform)
* Data drift occurs when production data differs significantly from training data, potentially degrading model performance and accuracy.
* [**Data Integrity**](/observability/platform/data-integrity-platform)
* Monitors three critical data integrity issues: missing values, incorrect data types, and out-of-range values that violate model input requirements.
* [**Performance**](/observability/platform/performance-tracking-platform)
* Model performance measures how accurately your model accomplishes its intended task, directly impacting business outcomes and decision quality.
* [**Custom Metrics**](/observability/platform/custom-metrics)
* Build tailored monitoring metrics that align with your specific business rules and success criteria using custom formulas.
### What Are the Key Features and Functions of the Monitoring Chart UI?
#### Multiple Charting Options
You can [plot up to 20 columns](#customizing-columns-for-monitoring-charts) and 6 metric queries for a model enabling you to easily perform model-to-model comparisons and plot multiple metrics in a single chart view.
#### Embedded Root Cause Analytics
Root cause analysis information covers data drift and data integrity, and performance analytics charts for binary classification, multiclass classification, and regression models.
#### Download Your Raw Data for Further Analysis
You can [easily download the raw chart data](#breakdown-summary) to CSV or parquet files. This feature allows you to analyze your data further.
### How to Create a New Monitoring Chart
To create a new monitoring chart, click on the Add Chart button on the Charts page. Search for and select the [project](/glossary#projects) to create the chart, and press Add Chart.
#### Save and Share Your Monitoring Chart
Save your chart by clicking the Save button in the chart studio's top right corner. You can then share your chart by copying its link and sending it to other users who have access to the project.
#### Undo and Redo Actions
Easily control the following actions with the undo and redo buttons:
* Metric query selection
* Time range selections
* Bin size selections
To learn how to undo actions taken using the chart toolbar, see the Toolbar information in the next section.
### Exploring Chart Metric Queries & Filters
#### Creating and Managing Metric Queries
Start building your monitoring chart by creating a metric query. First, select a model from your project to analyze. Then, specify which metrics and columns you want to visualize. Note that you can only select models that exist within your current project.
Select the type of metric you want to monitor: Performance, Data Drift, Data Integrity, Statistic, Traffic, or Custom Metric. Then choose specific metrics within that category. For instance, if you're monitoring a binary classification model, you might select Performance metrics and track accuracy over time.
#### Customizing Columns for Monitoring Charts
If you choose to chart data drift or data integrity, you can choose to plot up to 20 different columns from the following column categories: inputs, outputs, targets, metadata, and custom features.
#### Adding Multiple Metrics or Models
Add up to 6 metric queries that allow you to chart different metrics from one or more models in a single chart view.
#### Using Chart Filters for Better Analysis
Analyze specific slices of data using three key filtering tools: chart filters, the chart toolbar, and the zoom slider. These tools work together to help you investigate important patterns and trends in your data.
#### Tailored Filters
You can customize your chart view using the date range and bin size filters. The date range can be one of the pre-defined time ranges or a custom range. The bin size selected controls the frequency for which the data is displayed. So selecting Day will show daily data over the date range selected.
Customize your chart's time-based display using the date range and bin size filters. Choose from preset date ranges or define a custom period. The bin size determines how your data is grouped - for example, selecting 'Day' will show daily aggregated data across your chosen date range.
#### Toolbar Functions
The charts toolbar is made up of 5 functions:
* Drag to zoom
* Reset zoom
* Toggle to a line chart
* Toggle to a bar chart
* Undo all toolbar actions
> 📘 Note: If the zoom reset or toolbar undo is selected, this will also undo any actions taken with the zoom slider.
**Line & Bar Chart Toggles**
Toggle between line and bar chart views using the toolbar icons in the top right corner. While you can freely switch between these visualization types, please note that toolbar settings are temporary and won't be saved with your chart.
**Zoom Slider**
Use the horizontal zoom bar at the bottom of the chart to focus on specific time periods. Select your desired time range by dragging the zoom bar handles, then slide the selected range left or right to analyze different periods. For example, to examine week-by-week data over six months, zoom to your preferred view and drag the range to explore different weeks.
#### Breakdown Summary
You can easily visualize your charts' raw data as a table within the fiddler chart studio, or download the content as a CSV or parquet file for further analysis. If you choose to chart multiple columns, as shown below, you can search for and sort by Model name, Metric name, Column name, or values for a specific date.
### How to Customize Your Monitoring Chart
#### Adjusting Scale and Range
The Customize tab lets you adjust your chart's y-axis settings to improve data visualization. You can set minimum and maximum values to better utilize chart space, and apply logarithmic scaling to clearly display data with large variations in values.
#### Assigning Y-axis for Metric Queries
Select the y-axis for your metric queries with enhanced flexibility to customize the scale and range for each axis.
### Mastering our Chart UI to Effectively Track LLM and ML Performance
Understanding how to use the chart UI is key to measuring and maintaining the performance of LLM and ML models. Fiddler’s model monitoring tools allow you to track key metrics, identify data drift, and gain valuable insights into model behavior. Customize your Fiddler charts to proactively address issues and ensure models remain accurate, reliable, and high-performing.
# Performance Tracking
Source: https://docs.fiddler.ai/observability/platform/performance-tracking-platform
Learn to track performance with Fiddler. Discover why performance metrics matter and the steps to take when your model isn’t performing as expected.
For a complete list of all built-in ML metrics, see the [ML Metrics Reference](/reference/ml-metrics-reference).
## Overview
The model performance tells us how well a model performs on its task. A poorly performing model can have significant business implications.
## What is being tracked?
***Performance metrics***
| Model Task Type | Metric | Description |
| --------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Binary Classification | Accuracy | (TP + TN) / (TP + TN + FP + FN) |
| Binary Classification | True Positive Rate/Recall | TP / (TP + FN) |
| Binary Classification | False Positive Rate | FP / (FP + TN) |
| Binary Classification | Precision | TP / (TP + FP) |
| Binary Classification | F1 Score | 2 \* ( Precision \* Recall ) / ( Precision + Recall ) |
| Binary Classification | AUROC | Area Under the Receiver Operating Characteristic (ROC) curve, which plots the true positive rate against the false positive rate |
| Binary Classification | Binary Cross Entropy | Measures the difference between the predicted probability distribution and the true distribution |
| Binary Classification | Geometric Mean | Square Root of ( Precision \* Recall ) |
| Binary Classification | Calibrated Threshold | A threshold that balances precision and recall at a particular operating point |
| Binary Classification | Data Count | The number of events where target and output are both not NULL. ***This will be used as the denominator when calculating accuracy***. |
| Binary Classification | Expected Calibration Error | Measures the difference between predicted probabilities and empirical probabilities |
| Multi Classification | Accuracy | (Number of correctly classified samples) / ( Data Count ). Data Count refers to the number of events where the target and output are both not NULL |
| Multi Classification | Log Loss | Measures the difference between the predicted probability distribution and the true distribution, in a logarithmic scale |
| Regression | Coefficient of determination (R-squared) | Measures the proportion of variance in the dependent variable that is explained by the independent variables |
| Regression | Mean Squared Error (MSE) | Average of the squared differences between the predicted and true values |
| Regression | Mean Absolute Error (MAE) | Average of the absolute differences between the predicted and true values |
| Regression | Mean Absolute Percentage Error (MAPE) | Average of the absolute percentage differences between the predicted and true values |
| Regression | Weighted Mean Absolute Percentage Error (WMAPE) | The weighted average of the absolute percentage differences between the predicted and true values |
| Ranking | Mean Average Precision (MAP)—for binary relevance ranking only | Measures the average precision of the relevant items in the top-k results |
| Ranking | Normalized Discounted Cumulative Gain (NDCG) | Measures the quality of the ranking of the retrieved items, by discounting the relevance scores of items at lower ranks |
## Why is it being tracked?
* Model performance tells us how well a model is doing on its task. A poorly performing model can have significant business implications.
* The volume of decisions made on the basis of the predictions give visibility into the business impact of the model.
## What steps should I take based on this information?
* For changes in model performance—again, the best way to cross-verify the results is by checking the [Data Drift Tab](/observability/platform/data-drift-platform). Once you confirm that the performance issue is not due to the data, you need to assess if the change in performance is due to temporary factors, or due to longer-lasting issues.
* You can check if there are any lightweight changes you can make to help recover performance—for example, you could try modifying the decision threshold.
* Retraining the model with the latest data and redeploying it is usually the solution that yields the best results, although it may be time-consuming and expensive.
# Segments
Source: https://docs.fiddler.ai/observability/platform/segments
Learn to use model segments for monitoring diverse dimensions. Define, add, and modify segments to gain valuable insights into specific cohorts and dimensions.
### Overview
A segment, sometimes referred to as a cohort or slice, represents a distinct subset of model values crucial for performance analysis and troubleshooting. Model segments can be defined using various model dimensions, such as specific time periods or sets of features. Analyzing segments proves invaluable for understanding or troubleshooting specific cohorts of interest, particularly in tasks like bias detection, where overarching datasets might obscure statistical intricacies.
### How to Define a Segment
Fiddler makes it easy to define custom segments using either the Fiddler UI or the Fiddler Python client. Instructions for both approaches are covered in more detail below. In either case, Fiddler Segments are constructed using the [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language).
You can use any of the constants, operators, and functions mentioned in the page linked above in a Segment definition.
However, every Segment definition must return **a boolean row-level expression**. In other words, each inference will either satisfy the segment expression and thus belong to the segment or it will not.
### Examples
Let us illustrate further by providing a few examples. A segment can be defined by:
* A condition on some column (e.g. `age > 50`)
* A condition on some combination of columns (e.g. `(age / max_age) < 1.0`)
For details on all supported functions, see the [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language) page.
### Adding a Segment Using the Python Client
```python theme={null}
model_id = 'your_model_identifier'
segment_name = 'your_segment_name'
model = fdl.Model.get(id_=model_id)
# Use Fiddler Query Language (FQL) to define your custom segments
segment = fdl.Segment(
name=segment_name,
model_id=model.id,
definition="Age < 60",
description='Users with Age under 60',
).create()
```
### Applied Segments
When using segments in the UI for Analytics or Monitoring Charts, applied segments offer a flexible way to define segments on the fly for exploratory analysis. These segments are not saved to the model by default, but will persist locally if the chart they are applied to is saved.
At any time, an applied segment can be saved to the model. However, once a segment is saved to the model, it cannot be altered.
### Modifying Saved Segments
Since alerts can be set on Segments, making modifications to a Segment may introduce inconsistencies in alerts.
> 🚧 Therefore, **Saved segments cannot be modified once they are created**.
If you'd like to experiment with a new segment, you can create one with a different definition or use applied segments within charts.
#### Deleting Segments
Delete a segment using the [Python client SDK](/sdk-api/python-client/connection). Alternatively, from the segments tab, you can delete a segment by clicking the trash icon next to the segment record.
# Statistics
Source: https://docs.fiddler.ai/observability/platform/statistics
Discover Fiddler’s statistical metrics guide to monitor column aggregations. Learn what’s tracked, how to monitor metrics, and how to set up alerts.
### Overview
Fiddler supports some simple statistic metrics which can be used to monitor basic aggregations over columns. These can be particularly useful when you have a custom metadata field which you would like to monitor over time in addition to Fiddler's other out-of-the-box metrics.
### What is being tracked?
Specifically, we support:
* **Average**: Takes the arithmetic mean of a numeric column
* **Sum**: Calculates the sum of a numeric column
* **Frequency**: Shows the count of occurrences for each value in a categorical or boolean column
### Monitoring Statistic Metrics
#### Charting Statistic Metrics
These metrics can be accessed in Charts and Alerts by selecting the Statistic Metric Type.
#### Alerting on Statistic Metrics
Alert rules can be established based on statistics too. Like an alert rule, these can be setup using the Fiddler UI, the Fiddler python client or using Fiddler's RESTful API.
# Template-Based Alerts
Source: https://docs.fiddler.ai/observability/platform/template-based-alerts
Learn how to create and deploy template-based alerts in Fiddler using Google Sheets and YAML configurations for efficient model monitoring.
The Template-Based Alerts feature streamlines alert rule configuration and generation, significantly reducing manual effort. By using a Google Sheet to input key data and generate a recipe YAML file, this feature enables the efficient and accurate creation of alert rules. This approach ensures more relevant alerts, enhancing monitoring capabilities and enabling proactive issue resolution.
This guide outlines the process of generating and deploying template-based alerts to monitor your model's performance and data.
### Overview
Template-based alerts involve two primary steps:
1. **Generate a YAML configuration:** This involves using a provided Google Sheet to define your alert rules.
2. **Consume the YAML configuration:** This utilizes a Python notebook from the Alert Recipes repository to create the alerts in your Fiddler deployment.
### Step 1: Generating the Alert Recipe YAML Configuration
The first step is to create a YAML configuration file containing your desired alert rules. This is done using a pre-configured Google Sheet.
#### 1.1 Copying the Alert Recipes Sheet
To begin, you need to copy the Alert Recipes Google Sheet. This ensures you have a personal, editable version without affecting the base sheet.
* **Action**: Copy the entire content of the Alert Recipes sheet, including the recipe generator script it contains.
#### 1.2 Understanding the Sheet Structure
Once copied, note that the Google Sheet is organized into several worksheets:
* `AlertRecipes`: This is the main sheet where you define individual alert rules.
* `RecipeParameters` and `NotificationPacks`: These sheets need to be edited according to your preferences.
* `MetricIdMap` and `ThresholdTypeMap`: These sheets are locked and should not be edited unless you have a deep understanding of their function.
#### 1.3 Configuring Recipe Parameters
The `RecipeParameters` sheet is crucial for defining the context of your alerts.
* **Columns:**
* **Project name**: Enter the name of your project.
* **Model name**: Enter the name of your model.
* **High-priority features**: List the key features for this project and model, separated by commas.
* **Medium-priority features**: Provide a comma-separated list of features you consider medium priority for this project and model.
* **Low-priority features**: Provide a comma-separated list of features you consider low priority for this project and model.
**Important Note**: The script only considers the first row of this sheet. Any additional rows will be ignored.
#### 1.4 Configuring notification packs
Notification packs allow you to bundle email, PagerDuty, or webhook notifications to be applied to your alert rules.
* **Definition**: A notification pack is a collection of one or more notification types (email, PagerDuty, webhook) that can be attached to an alert rule.
* **Customization**:
* You can create custom notification packs in new rows.
* Each new pack can contain any combination of email, PagerDuty service, and webhook UUID.
* **Important Note**: The webhook user ID must be the actual UUID of the webhook, not its service name.
* **Availability**: Any notification packs you define or modify in this sheet will be available for selection in the main `AlertRecipes` sheet.
#### 1.5 Defining Alert Rules in AlertRecipes
The `AlertRecipes` sheet is where you configure individual alert rules.
* **Columns:**
* **Monitor type**: Specifies the type of metric on which the alert rule will be created.
* **Alert rule name**: If provided, this name will be used for the alert rule. Otherwise, a name will be inferred from the monitor type, appended with a unique identifier.
* **Time window**: Defines the time bin over which evaluations will happen and alerts will be raised.
* **Enabled (Y or N)**: Determines whether this alert rule will be included in the generated YAML configuration.
* **Features**: Select from 'High-Priority', 'Low-Priority', 'Medium-Priority', or 'All'. The specific features are chosen from the RecipeParameters sheet based on your selection.
* **Threshold algo**: Currently, only 'standard deviation' is supported.
* **Critical multiplier**: This multiplier is used to calculate the critical threshold dynamically. For example, a value of `2` means a critical alert will be raised if the metric value is 2 standard deviations away from the mean of the reference data.
* **Warning multiplier**: This multiplier is used to calculate the warning threshold dynamically. For example, a value of `1` means a warning alert will be raised if the metric value is 1 standard deviation away from the mean of the reference data. These multipliers are customizable (e.g., 0.5, 1.5).
* **Notification pack**: Select a pre-defined notification pack from the `NotificationPacks` sheet to attach to this alert rule.
#### 1.6 Generating the Alert Recipe YAML
Once you have configured the `RecipeParameters`, `NotificationPacks`, and `AlertRecipe` sheets, you can generate the YAML file.
* **Action**: Click the "Generate Alert Recipe" button in the `AlertRecipe` Sheet.
* **Permissions**: You will be prompted to grant specific permissions for the script to run. These permissions are necessary for the script to access your Google Drive (to save the YAML file) and to run within the Google Sheet environment.
* **Output**: The YAML file will be generated. Copy its content.
### Step 2: Consuming the YAML Configuration
After generating the YAML file, the next step is to use it to create the alert rules in your Fiddler deployment. This is done using the Alert Recipes repository and a Python notebook.
#### 2.1 Setting Up the Alert Recipes Repository
You'll need to work within the Alert Recipes repository.
* **File Creation**: Inside the root of the repository, create two files:
* `config.py`
* `alert_recipes.yaml`
* **Configuration**:
* `config.py`: Populate this file with the correct URL and authentication token for your Fiddler deployment. You can copy the structure from existing example files(`config.py.example`).
* `alert_recipes.yaml`: Paste the content of the YAML file you copied from the Google Sheet into this file.
#### 2.2 Running the Python Notebook
With the configuration files set up, you can now run the Python notebook to create the alert rules.
* **Action**: Open the `create_alert_rules.ipynb` notebook within the Alert Recipes repository.
* **Execution**: Run all cells within the notebook.
* **Verification**: The logs in the notebook will show which new alert rules have been created. You can then verify the creation of these rules within your Fiddler deployment.
# Traffic
Source: https://docs.fiddler.ai/observability/platform/traffic-platform
Learn how Fiddler tracks your ML and GenAI models' traffic patterns and when to take action when traffic patterns deviate from normal.
Traffic as a service metric gives you basic insights into the operational health of your model's service in production.
## What is Being Tracked?
* ***Traffic*** — The volume of traffic received by the model over time.
## Why is it Being Tracked?
* Traffic is a basic high-level metric that informs us of the model's output activity.
## What Steps Should I Take When I See an Outlier?
* A dip or spike in traffic needs to be investigated. For example, a dip could be due to a production model server going down; a spike could be an adversarial attack.
# Vector Monitoring
Source: https://docs.fiddler.ai/observability/platform/vector-monitoring-platform
Dive into our vector monitoring guide to learn about model inputs represented as vectors and how to use Fiddler's custom features to monitor and detect drift.
## Detecting Drift in Multi-Dimensional ML and GenAI Model Data
Many modern machine learning systems use input features that cannot be represented as a single number, such as text or image data. These complex features are typically represented by high-dimensional vectors obtained through vectorization methods like text embeddings generated by NLP models. Fiddler users often need to monitor groups of univariate features together and detect data drift in multidimensional feature spaces.
To address these needs, Fiddler provides vector monitoring capabilities that enable you to define [custom features](#define-custom-features) and use advanced methods for monitoring data drift in multidimensional spaces.
You can define custom features by grouping columns together in baseline and inference data. For NLP or image data, you can define custom features using columns that contain embedding vectors.
### Define Custom Features
Users can use the Fiddler client to define one or more custom features. Custom features can be specified by:
Use the Fiddler client to define one or more custom features. You can specify custom features in three ways:
1. Group dataset columns that need to be monitored together as a vector (custom\_feature\_1, custom\_feature\_2)
2. Use existing embedding vectors with the source column (custom\_feature\_3, custom\_feature\_4)
3. Define an enrichment that instructs Fiddler to generate embedding vectors automatically on ingestion (custom\_feature\_5)
After you define and pass a list of custom features to Fiddler, Fiddler runs a clustering-based data drift detection algorithm for each custom feature. The system calculates a corresponding drift value between the baseline and published events for the selected time period.
```python theme={null}
from fiddler import CustomFeature, TextEmbedding, ImageEmbedding
# Group columns into vectors
custom_feature_1 = CustomFeature.from_columns(
['f1', 'f2', 'f3'], custom_name='vector1'
)
custom_feature_2 = CustomFeature.from_columns(
['f1', 'f2', 'f3'], n_clusters=5, custom_name='vector2'
)
# Use existing embeddings
custom_feature_3 = TextEmbedding(
name='Document Text Embedding', column='text_embedding_col', source_column='text'
)
custom_feature_4 = ImageEmbedding(
name='Image Embedding', column='image_embedding_col', source_column='image_url'
)
# Define automated text embedding enrichment
custom_feature_5 = TextEmbedding(
name='Document Text Embedding',
source_column='doc_col',
column='Enrichment Unstructured Embedding',
n_tags=10,
)
```
### Passing Custom Features List to ModelSpec
After you define custom features for vector monitoring, add them to the `ModelSpec` and onboard the Model to Fiddler.
```python theme={null}
from fiddler import ModelSpec, Model, Project
model_spec = ModelSpec(
inputs=[
'creditscore',
'geography',
'gender',
'age',
'tenure',
'balance',
'numofproducts',
'hascrcard',
'isactivemember',
'estimatedsalary',
'doc_col',
],
outputs=['predicted_churn'],
targets=['churn'],
# Note: Embedding columns you pass in must be included with the metadata columns.
metadata=['customer_id', 'timestamp', 'text_embedding_col', 'image_embedding_col'],
custom_features=[
custom_feature_1,
custom_feature_2,
custom_feature_3,
custom_feature_4,
custom_feature_5,
],
)
model = Model.from_data(
name='your_model_name',
project_id=Project.from_name('your_project_name').id,
source=sample_df,
spec=model_spec,
task=model_task,
task_params=task_params,
event_id_col=id_column,
event_ts_col=timestamp_column,
)
model.create()
```
## Understanding Drift Detection Algorithm
Fiddler's vector monitoring uses a clustering-based approach to detect drift in multidimensional spaces:
1. **Baseline clustering:** The system analyzes your baseline data to identify natural clusters using k-means clustering
2. **Production comparison:** New production data is compared against these established clusters
3. **Drift calculation:** The system calculates drift scores based on changes in cluster distributions and centroid distances
### Performance considerations
* **Computational cost:** Vector monitoring requires more computational resources than univariate monitoring
* **Memory usage:** High-dimensional vectors and clustering algorithms increase memory requirements
* **Processing time:** Drift calculations may take longer for large datasets or high-dimensional vectors
### Best practices
* **Choose appropriate cluster numbers:** Start with 3-8 clusters and adjust based on your data's natural groupings
* **Monitor cluster stability:** Regularly review cluster formations to ensure they remain meaningful
* **Set reasonable thresholds:** Establish drift thresholds that balance sensitivity with false positive rates
## Risk Considerations for AI/ML Applications
When implementing vector monitoring, consider these potential risks:
* **Bias amplification:** Clustering algorithms may amplify existing biases in your training data
* **Concept drift detection:** Traditional clustering may miss subtle concept drift that affects model performance
* **Interpretability challenges:** High-dimensional clusters can be difficult to interpret and explain to stakeholders
> **Note:** For a complete example of NLP monitoring, see our [NLP monitoring quick start guide](/developers/tutorials/ml-monitoring/simple-nlp-monitoring-quick-start), which demonstrates embedding generation for unstructured inputs.
## Related topics
* [Data drift platform](/observability/platform/data-drift-platform)
* [Enrichments](/observability/llm/enrichments)
* [Embedding visualization with UMAP](/observability/platform/embedding-visualization-with-umap)
# AgentGateway Guardrails
Source: https://docs.fiddler.ai/protection/agentgateway-guardrails
Use Fiddler as a guardrail provider for AgentGateway — redacting or blocking PII and secrets in real time before requests reach your LLM.
## Overview
[AgentGateway](https://agentgateway.dev/) can call Fiddler's guardrail webhook adapter
before and after every LLM call, redacting or blocking personally identifiable information (PII)
and secrets at the proxy layer.
Guardrails run *inline* on the request path — they can rewrite or block the call —
while [AgentGateway tracing](/integrations/agentic-ai/agentgateway-integration)
exports spans out-of-band for observability.
They are independent; for a single config that runs both, see the
[combined example](/integrations/agentic-ai/agentgateway-integration#combined-tracing-and-guardrails).
| You want to… | Use | Page |
| ----------------------------------------------- | --------------------------------- | ----------------------------------------------------------------------------- |
| Observe prompts, responses, tokens, and latency | `frontendPolicies.tracing` | [AgentGateway Integration](/integrations/agentic-ai/agentgateway-integration) |
| Redact or block PII and secrets | `llm.models[].guardrails` webhook | This page |
Guardrails require **AgentGateway v1.4.0 or later** (the first stable release whose webhook
`headers` CEL config can route calls to Fiddler's canonical, versioned endpoints), and are
**available in Fiddler 26.17 and later** — the release that first contains Fiddler's AgentGateway
guardrail adapter. On earlier Fiddler releases the endpoints below return HTTP 404.
***
## How It Works
```mermaid theme={null}
graph LR
Client["Client (coding agent, app, curl)"]
Client -->|"POST /v1/chat/completions"| AG["AgentGateway"]
AG -->|"POST /v3/guardrails/agentgateway/request"| Fiddler["Fiddler Guardrails API"]
Fiddler -->|"action: reject"| AG
Fiddler -->|"action: pass or mask → proceed"| AG
AG -->|"Sanitized request"| LLM["LLM Provider (Vertex AI, OpenAI, Anthropic, …)"]
LLM -->|"Response"| AG
AG -->|"POST /v3/guardrails/agentgateway/response"| Fiddler
Fiddler -->|"action: reject"| AG
Fiddler -->|"action: pass or mask → return"| AG
AG -->|"Response (or sanitized response)"| Client
style Client fill:#e1f5ff
style AG fill:#fff4e6
style Fiddler fill:#ffe6e6
style LLM fill:#e6ffe6
```
Each gateway speaks its own wire format, so Fiddler exposes a dedicated adapter per gateway
(`/v3/guardrails/agentgateway/*`, `/v3/guardrails/kong`, and `/v3/guardrails/litellm/*` — see each
page's own Endpoints/API Reference section). AgentGateway wraps each request or response in its own
envelope and expects one of three actions back:
| Action | Meaning | AgentGateway behavior |
| -------- | -------------------------- | --------------------------------------------------- |
| `pass` | No issues detected | Forwards the request/response unchanged |
| `mask` | Sensitive content redacted | Forwards the request/response with redacted content |
| `reject` | Request must not proceed | Returns an HTTP error to the client |
Unlike the [Kong integration](/protection/kong-guardrails), which blocks only, AgentGateway can
**redact in place** (`mask`) so the sanitized call still reaches the model — with the streaming
caveat in [Failure Mode, Timeouts, and Streaming](#failure-mode-timeouts-and-streaming).
***
## Prerequisites
* **Fiddler guardrails enabled on your deployment**, on **Fiddler 26.17 or later** (the release that
contains the AgentGateway guardrail adapter). If guardrails are not enabled, the adapter returns
HTTP 403 *"Guardrails is not enabled for this cluster"* or HTTP 404 *"not available on freemium
deployments"* — see [Troubleshooting](#troubleshooting).
* **Your Fiddler instance URL** and a **Fiddler API key** (organizational settings).
* **AgentGateway v1.4.0+** (see the callout above for the earlier-version constraint).
* **Network egress** from AgentGateway to your Fiddler instance over HTTPS.
* **About ten minutes.**
***
## Quick Start
Add a named backend for the Fiddler guardrail webhook, then reference it from each model's
`guardrails` block. This uses AgentGateway's `llm.models` configuration shape:
```yaml theme={null}
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
backends:
- name: fiddler-guardrail
host: :443
policies:
# Required for any HTTPS backend host — without it AgentGateway speaks
# plain HTTP to the host:port above and the connection hangs.
backendTLS: {}
backendAuth:
# AgentGateway injects the "Bearer" scheme itself — set the raw token.
# Restrict this file's permissions and inject it at deploy time from a
# secret manager; see the note below on environment expansion.
key: ""
llm:
models:
- name: gpt-5-mini
provider: openai
params:
model: gpt-5-mini
guardrails:
request:
- webhook:
target:
backend: /fiddler-guardrail
headers:
# AgentGateway defaults webhook calls to the backend's root-level
# /request and /response paths. Setting :path via a CEL string
# literal routes to Fiddler's canonical, versioned endpoints.
":path": '"/v3/guardrails/agentgateway/request"'
response:
- webhook:
target:
backend: /fiddler-guardrail
headers:
":path": '"/v3/guardrails/agentgateway/response"'
```
On standalone (non-Kubernetes) AgentGateway deployments, backend names with no namespace are stored
with a leading slash — reference them as `/fiddler-guardrail`, not `fiddler-guardrail`
(see [agentgateway/agentgateway#2220](https://github.com/agentgateway/agentgateway/pull/2220)).
A named `backend:` reference (rather than an inline `host:` on the webhook target) is required to
attach `backendAuth`/`backendTLS` policies to the guardrail call.
AgentGateway supports `$VAR` expansion for credentials in its config; if it applies to
`backendAuth.key` on your version you can set `key: "$FIDDLER_API_KEY"` instead of a literal token.
Verify it against your deployment — Fiddler has not confirmed `$VAR` expansion for this specific
field.
```bash theme={null}
agentgateway -f agentgateway_config.yaml
```
With no `gateways` block, the `llm.models` config implies a default gateway serving LLM traffic on
port 4000.
Send a request containing a synthetic (correctly formatted but not real) secret:
```bash theme={null}
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"My API key is sk-ant-api03-abcdefghijklmnopqrstu"}]}'
```
With the default request guard, the secret is **redacted in place** — the model receives
`[REDACTED
***
## What This Configuration Enables
The quick-start config sends no `x-fiddler-guardrails` header, so the adapter runs **both** of
its checks on the request path:
| Check | What it detects | Default action | Threshold |
| ----------- | ----------------------------------- | --------------- | ------------- |
| **PII** | Personally identifiable information | Redact (`mask`) | 0.8 |
| **Secrets** | Credentials and API keys | Redact (`mask`) | Any detection |
One thing is important:
* **There is a 50,000-character scan cap.** Under the default fail-open behavior, text over the cap is
skipped and passes to the model **unscanned**; under fail-closed it is rejected — see
[Failure Mode, Timeouts, and Streaming](#failure-mode-timeouts-and-streaming).
***
## What Gets Scanned
| Check | Request guard | Response guard | Default action |
| ----------- | ------------- | -------------- | -------------- |
| **PII** | ✓ | ✓ | Redact |
| **Secrets** | ✓ | ✓ | Redact |
Only free-text message `content` is scanned. AgentGateway's webhook protocol sends a simplified
message shape — `{role, content}` only — and strips `tool_calls`, `name`, and other OpenAI
chat-completions fields before calling the webhook, so there is nothing beyond message text for
Fiddler to scan or redact on this integration.
***
## Check Behavior
Checks are configured server-side with the same defaults used across all Fiddler guardrail
integrations — see [Guardrails](/protection/guardrails) for the PII model and the
[secrets detection tutorial](/developers/tutorials/guardrails/guardrails-secrets) for secrets.
Because AgentGateway's wire body carries no per-request config field, those defaults are overridden
**per route** via static HTTP headers in the webhook's `headers` CEL config — the same mechanism used
for the `:path` override in [Step 1](#quick-start).
### Header-Based Configuration
| Header | Applies to | Value | Effect |
| ------------------------- | ---------- | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `x-fiddler-guardrails` | All | Comma-separated check names — `pii`, `secrets` | Restricts which checks run. Case-insensitive and whitespace-tolerant. Absent, blank, or naming no recognized check at all runs every check with defaults. |
| `x-fiddler-pii-mode` | `pii` | `redact` (default) or `block` | Overrides the PII action. |
| `x-fiddler-pii-threshold` | `pii` | Float, e.g. `0.8` | Overrides the PII detection threshold. An unparseable value is ignored (falls back to the default), not rejected. |
| `x-fiddler-pii-entities` | `pii` | Comma-separated entity names | Switches PII detection **out of preset mode** to check only the named entity types — see the [PII Detection tutorial](/developers/tutorials/guardrails/guardrails-pii#supported-entity-types). |
| `x-fiddler-secrets-mode` | `secrets` | `redact` (default) or `block` | Overrides the secrets action. |
| `x-fiddler-failure-mode` | All | `open` or `closed` | What happens when a *check itself* fails to complete (detector timeout or error) — default `open`. Separate from AgentGateway's own `failureMode`, which governs an unreachable webhook — see [Failure Mode](#failure-mode-timeouts-and-streaming). |
| `x-fiddler-timeout` | All | Seconds, e.g. `12` | Fiddler's internal wall-clock budget for running checks — default 12s, capped at 60s. |
```yaml theme={null}
guardrails:
request:
- webhook:
target:
backend: /fiddler-guardrail
headers:
":path": '"/v3/guardrails/agentgateway/request"'
"x-fiddler-guardrails": '"pii,secrets"'
"x-fiddler-pii-mode": '"block"'
```
Each header value must be a CEL string literal — note the nested quotes (`'"pii,secrets"'`).
A bare, unquoted value (for example `"x-fiddler-pii-mode": "block"`) is parsed as an unresolvable CEL
field reference; AgentGateway **drops that header instead of raising an error**, so the override is
silently ignored.
**Naming a subset silently disables the rest.** `x-fiddler-guardrails` restricts checks to those
named, so `x-fiddler-guardrails: pii` runs only PII and **silently disables secrets**. A header
naming a check that is not `pii` or `secrets` (for example a typo) is dropped with a
server-side warning; every check runs with defaults only when **none** of the requested names is
recognized. A partial typo like `pii,screts` still runs `pii` and silently skips `secrets`. Per-check
override headers take effect only when their check is present in `x-fiddler-guardrails`; but
`x-fiddler-failure-mode` and `x-fiddler-timeout` are parsed before check selection, so they do take
effect even when `x-fiddler-guardrails` is absent.
***
## Action Mapping
AgentGateway's webhook protocol uses
[serde untagged deserialization](https://serde.rs/enum-representations.html#untagged) on the response
body — there is no explicit `"type"` discriminator. The action shapes are distinguished by their JSON
structure:
```json theme={null}
// pass (allow)
{"action": {}}
// reject (block) — body is a string
{"action": {"body": "Request blocked by guardrail.", "status_code": 403, "reason": "..."}}
// mask (redact), request guard — body is an object with messages
{"action": {"body": {"messages": [{"role": "user", "content": "my email is [REDACTED EMAIL_ADDRESS]"}]}, "reason": "PII/secrets redacted"}}
// mask (redact), response guard — body is an object with choices
{"action": {"body": {"choices": [{"message": {"role": "assistant", "content": "your key is [REDACTED SECRET]"}}]}, "reason": "PII/secrets redacted"}}
```
`mask` and `reject` are disambiguated by the type of `body`: an object
(`{"messages": [...]}` or `{"choices": [...]}`) means `mask`; a string means `reject`.
Redacted content is a labelled in-place substitution — `[REDACTED