--set kafka.security_protocol=SSL
--set kafka.ssl_cafile=cafile
--set kafka.ssl_certfile=certfile
--set kafka.ssl_keyfile=keyfile
--set-string kafka.ssl_check_hostname=False
```
This creates a deployment that reads event data from the Kafka topic and publishes it to the configured Fiddler model. The deployment can be scaled as needed; however, note that if the Kafka topic is not partitioned, scaling will not result in any gains.
#### Limitations
1. The connector assumes that there is a single dedicated topic containing production events for a given model. Multiple deployments can be created, one for each model, and scaled independently.
2. The connector assumes that events are published as JSON-serialized dictionaries of key-value pairs. Support for other formats can be added on request. As an example, a Kafka message should look like the following:
```json theme={null}
{
“feature_1”: 20.7,
“feature_2”: 45000,
“feature_3”: true,
“output_column”: 0.79,
“target_column”: 1,
“ts”: 1637344470000,
}
```
# SageMaker Pipelines
Source: https://docs.fiddler.ai/integrations/data-platforms/sagemaker-integration
Learn how integrating SageMaker with Fiddler simplifies model monitoring. Explore our guide on using AWS Lambda with the Fiddler Python client.
## Amazon SageMaker Integration
### Introduction
Integrate Amazon SageMaker with Fiddler to monitor your deployed models effectively. This guide shows you how to create an AWS Lambda function that uses the Fiddler Python client to process SageMaker inference logs from Amazon S3 and send them to your Fiddler instance. This integration provides real-time monitoring capabilities and valuable insights into your model's performance and behavior.
Fiddler AI Observability Platform is now available within Amazon SageMaker AI in SageMaker Unified Studio. This native integration lets SageMaker customers monitor ML models privately and securely without leaving the SageMaker environment.
Learn more about the Amazon SageMaker AI with Fiddler native integration [here](https://www.fiddler.ai/blog/fiddler-delivers-native-enterprise-grade-ai-observability-to-amazon-sagemaker-ai-customers).
### Prerequisites
Before you begin, ensure you have:
1. An active SageMaker model with:
* Data capture enabled
* Inference logs saved to S3 in JSONL format
2. Access to a Fiddler environment
3. Your SageMaker model is onboarded to Fiddler (See the [ML Monitoring Quick Start Guide](/developers/quick-starts/simple-ml-monitoring))
4. Latest Fiddler Python client version
### Implementation Steps
#### 1. Configure SageMaker Data Capture
Ensure your SageMaker endpoint has data capture properly configured:
1. Open the SageMaker console
2. Navigate to your model endpoint
3. Verify data capture is enabled and configured to save to your S3 bucket
4. Confirm captured data is in JSONL format
#### 2. Create an AWS Lambda Function
1. Open the AWS Lambda console
2. Click "Create function"
3. Configure the basic settings:
* Name your function (for example, "fiddler-sagemaker-integration")
* Select Python 3.10 or later as the runtime
* Choose execution permissions that allow S3 access
#### 3. Set Up Environment Variables
Configure these environment variables in your Lambda function:
| Variable | Description | Example |
| -------------------- | ------------------------------------------ | ------------------------------------ |
| `FIDDLER_URL` | Your Fiddler environment URL | `https://your_company.fiddler.ai` |
| `FIDDLER_TOKEN` | Your Fiddler authorization token | *(secure token value)* |
| `FIDDLER_MODEL_UUID` | Your model's unique identifier in Fiddler | 8a86cc43-71c1-49e7-a01b-d98ae91975bb |
| `MODEL_COLUMNS` | Comma-separated list of input column names | `feature1,feature2,feature3` |
| `MODEL_OUTPUT` | Name of the model output column | `prediction` |
| `MODEL_TIMESTAMP` | Name of the timestamp column (optional) | `event_time` |
If you provisioned Fiddler via the [SageMaker AI marketplace](https://docs.aws.amazon.com/sagemaker/latest/dg/partner-apps.html), add these additional variables:
* `AWS_PARTNER_APP_AUTH`: Set to `True`
* `AWS_PARTNER_APP_ARN`: The ARN of your SageMaker AI Fiddler instance
* `AWS_PARTNER_APP_URL`: The URL of your SageMaker AI Fiddler instance
![AWS Lambda function console on the environment variables tab showing the example variables used in this guide.]()
#### 4. Configure S3 Trigger
Set up your Lambda to run automatically when new data arrives:
1. In the Lambda console, select your function
2. Choose the "Add trigger" option
3. Select "S3" as the trigger type
4. Configure these settings:
* Bucket: Select your SageMaker inference logs bucket
* Event type: "All object create events"
* Prefix: (Optional) Specify a path prefix if needed
* Suffix: `.jsonl` (to only process JSON Lines files)
![AWS Lambda function console on the triggers tab showing the details of the trigger used in this guide.]()
#### 5. Add Lambda Function Code
![AWS Lambda function console on the main view showing the code tab and some of the code used in this guide.]()
Copy this code into your Lambda function editor:
```python theme={null}
import os
import json
import uuid
import boto3
import logging
from typing import Dict, List, Any
import fiddler as fdl
# Configure logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Load environment variables, customize to model and use case
url = os.getenv('FIDDLER_URL')
token = os.getenv('FIDDLER_TOKEN')
model_uuid = os.getenv('FIDDLER_MODEL_UUID')
model_columns = os.getenv('MODEL_COLUMNS')
model_output_column = os.getenv('MODEL_OUTPUT')
timestamp_column = os.getenv('MODEL_TIMESTAMP')
# Initialize AWS clients
s3_client = boto3.client('s3')
# Initialize Fiddler connection and Fiddler Model to receive events
fdl.init(url=url, token=token)
fiddler_model = fdl.Model.get(id_=model_uuid)
def get_all_columns():
# The types of columns needed when publishing depend on use case. Typically,
# you would expect to pass at least your model inputs and output(s) and often
# metadata such as IDs, dates, data segments, etc.
return model_columns.split(',') + [timestamp_column] + [model_output_column]
def process_jsonl_content(event_data: str) -> Dict[str, Any]:
input_data = event_data['captureData']['endpointInput']['data']
input_values = input_data.split(',') # Split the CSV string into a list
# Extract the model prediction from 'captureData/endpointOutput/data'
model_prediction = event_data['captureData']['endpointOutput']['data']
# Optionally, you can set your own timestamp value on the inference occurrence time,
# or let Fiddler default it to the time of publish.
timestamp_value = event_data['eventMetadata']['inferenceTime']
# Combine inputs and any metadata values with the output into a single row
all_values = input_values + [timestamp_value] + [model_prediction]
# Create dictionary using zip to pair column names with their values
return dict(zip(get_all_columns(), all_values))
def parse_sagemaker_log(log_file_path: str) -> List[Dict[str, Any]]:
try:
# Collect all events in a List, 1 per JSON-line in the file
event_rows = []
with open(log_file_path, 'r') as file:
for line in file:
event = json.loads(line.strip())
row = process_jsonl_content(event)
event_rows.append(row)
return {
'status': 'success',
'record_count': len(event_rows),
'data': event_rows
}
except json.JSONDecodeError as e:
logger.error(f'Error parsing JSONL content: {str(e)}')
raise
def publish_to_fiddler(inferences: List[Dict[str, Any]], model: fdl.Model):
# There are multiple options for publishing data to Fiddler, check
# the online documentation for batch, streaming, and REST API options.
# publish_stream() sends events synchronously; for large datasets
# consider publish_batch() with a file path or DataFrame instead.
event_ids = model.publish_stream(
events=inferences,
environment=fdl.EnvType.PRODUCTION
)
return event_ids
def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
# Process each record in the event, streaming to Fiddler in batches
for record in event['Records']:
# Extract bucket and key information
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
logger.info(f'Processing new file: {key} from bucket: {bucket}')
# Persist log file to a temporary location
tmp_key = key.replace('/', '')
download_path = f'/tmp/{uuid.uuid4()}{tmp_key}'
s3_client.download_file(bucket, key, download_path)
# Retrieve the inference event(s) from the log file
results = parse_sagemaker_log(download_path)
# Check if the log file was processed successfully
if results['status'] != 'success':
logger.error(f'Error processing log file: {key}')
return {
'statusCode': 500,
'body': {'message': 'Error processing log file', 'results': results},
}
# Push the inference events to Fiddler
event_ids = publish_to_fiddler(results["data"], fiddler_model)
logger.info(f'Published events to Fiddler with ID(s): {event_ids}')
return {
'statusCode': 200,
'body': {'message': 'Successfully processed events', 'results': results},
}
```
# Snowflake
Source: https://docs.fiddler.ai/integrations/data-platforms/snowflake-integration
Learn how to extract baseline or production data from Snowflake for model onboarding and publishing production data to Fiddler for ML and LLM monitoring.
In this article, we will be looking at loading data from Snowflake tables and using the data for the following tasks:
1. Onboarding a model to Fiddler
2. Uploading baseline data to Fiddler
3. Publishing production data to Fiddler
## Import data from Snowflake
In order to import data from Snowflake to a Jupyter notebook, we will use the snowflake library which can be installed using the following command in your Python environment.
```bash theme={null}
pip install snowflake-connector-python
```
The following information is required in order to establish a connection to Snowflake:
* Snowflake Warehouse
* Snowflake Role
* Snowflake Account
* Snowflake User
* Snowflake Password
These values can be obtained from your Snowflake account under the ‘Admin’ option in the Menu as shown below or by running the queries below:
* Warehouse - select CURRENT\_WAREHOUSE()
* Role - select CURRENT\_ROLE()
* Account - select CURRENT\_ACCOUNT()
'User' and 'Password' are the same that you use when logging in to your Snowflake account.
![Example Snowflake dashboard showing the Warehouse tab.]()
Once you have this information, you can set up a Snowflake connector using the following code:
```python theme={null}
# establish Snowflake connection
connection = connector.connect(
user=snowflake_username,
password=snowflake_password,
account=snowflake_account,
role=snowflake_role,
warehouse=snowflake_warehouse
)
```
You can then write a custom SQL query and import the data to a pandas dataframe.
```python theme={null}
# sample SQL query
sql_query = 'select * from FIDDLER.FIDDLER_SCHEMA.CHURN_BASELINE LIMIT 100'
# create cursor object
cursor = connection.cursor()
# execute SQL query inside Snowflake
cursor.execute(sql_query)
baseline_df = cursor.fetch_pandas_all()
```
## Publish Production Events
Now that we have data imported from Snowflake to a dataframe, we can refer to the following pages to:
1. [Onboard a model](/developers/python-client-guides/model-onboarding/create-a-project-and-model) using the baseline dataset for the model schema inference sample.
2. [Upload a Baseline dataset](/developers/python-client-guides/publishing-production-data/creating-a-baseline-dataset), which is optional but recommended for monitoring comparisons.
3. [Publish production events](/developers/client-library-reference/publishing-production-data) for continuous monitoring.
# Integrations
Source: https://docs.fiddler.ai/integrations/index
Connect Fiddler to your AI stack with native SDKs, cloud platform integrations, data pipeline connectors, and framework support. From agentic workflows to traditional ML models, Fiddler integrates with the tools you already use.
## Quick Start
Choose your integration path based on what you're building:
* [**Monitor LangGraph Agents →**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) Auto-instrument agentic applications with the LangGraph SDK
* [**Deploy on AWS →**](/integrations/cloud-platforms-and-deployment/cloud-platforms) Run Fiddler as a Partner AI App in SageMaker
* [**Connect Your Data →**](/integrations/data-platforms-and-pipelines/data-platforms) Integrate with Snowflake, BigQuery, S3, and more
## Integration Categories
### 🎯 Agentic AI & LLM Frameworks
Native instrumentation for agentic workflows and LLM applications—Fiddler's competitive differentiator.
**Native SDKs:**
* [Fiddler LangGraph SDK](/integrations/agentic-ai/langgraph-sdk) - Auto-instrument LangGraph agents with OpenTelemetry
* [Fiddler Strands SDK](/integrations/agentic-ai/strands-sdk) - Monitor Strands Agents
* [Fiddler Evals SDK](/integrations/agentic-ai/evals-sdk) - LLM experiments framework with pre-built evaluators
**Platform Access:**
* Python Client SDK - Core SDK client for ML/LLM monitoring
* [REST API](/sdk-api/rest-api/alert-rules) - Complete HTTP API for all platform features
[**View all agentic AI integrations →**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai)
### ☁️ Cloud Platforms & Deployment
Deploy Fiddler on your preferred cloud infrastructure.
**Featured:**
* [AWS SageMaker Partner AI App](/integrations/cloud-platforms-and-deployment/aws-sagemaker) - Fully managed deployment in SageMaker
**Data Storage:**
* [Amazon S3](/integrations/data-platforms/integration-with-s3) - S3 bucket integration for data ingestion
* [Google BigQuery](/integrations/data-platforms/bigquery-integration) - BigQuery connector for ML pipelines
[**View all cloud integrations →**](/integrations/cloud-platforms-and-deployment/cloud-platforms)
### 🗄️ Data Platforms & Pipelines
Connect Fiddler to your data infrastructure for seamless model monitoring.
**Data Warehouses:**
* [Snowflake](/integrations/data-platforms/snowflake-integration) - Snowflake connector for production data
* [BigQuery](/integrations/data-platforms/bigquery-integration) - Google BigQuery integration
**Data Streaming:**
* [Apache Kafka](/integrations/data-platforms/kafka-integration) - Real-time streaming data integration
**Orchestration:**
* [Apache Airflow](/integrations/data-platforms/airflow-integration) - Workflow automation for model deployments
[**View all data integrations →**](/integrations/data-platforms-and-pipelines/data-platforms)
### 🤖 ML Platforms & Tools
Integrate with your existing ML infrastructure and experiment tracking tools.
**MLOps Platforms:**
* [Databricks](/integrations/ml-platforms/databricks-integration) - Databricks ML integration via MLflow and Spark
* [MLflow](/integrations/ml-platforms/ml-flow-integration) - Model registry and experiment tracking
[**View all ML platform integrations →**](/integrations/ml-platforms-and-tools/ml-platforms)
### 📡 Monitoring & Alerting
Route alerts and metrics to your observability stack.
**Observability Platforms:**
* [Datadog](/integrations/monitoring-alerting/datadog-integration) - Export metrics to Datadog for centralized monitoring
**Incident Management:**
* [PagerDuty](/integrations/monitoring-alerting/pagerduty) - Alert routing and on-call management
**Notifications:**
* Webhooks - Generic webhook support for custom integrations
* Email Alerts - Built-in email notification system
[**View all monitoring integrations →**](/integrations/monitoring-and-alerting/monitoring-alerting)
### 🔐 Authentication & Access
Enterprise SSO and authentication providers for secure access control.
**Enterprise SSO:**
* [Okta Integration](/reference/access-control/okta-integration) - Enterprise single sign-on
* [Google Workspace](/reference/access-control/google-integration) - Google SSO integration
**API Authentication:**
* API Keys - Token-based API authentication
* OAuth - OAuth 2.0 flow support
[**View all authentication options →**](/reference/access-control)
## Integration Types
* **🏆 Native SDK** - Built and maintained by Fiddler; included with platform
* **🤝 Partner Integration** - Built in collaboration with technology partner
* **🔌 Connector** - Pre-built component for data source integration
* **📚 Framework Guide** - Code examples and patterns for specific frameworks
## What's Next?
* **Getting Started:** Follow our [quick start guides](/developers) to set up your first integration
* **SDK Documentation:** Dive into detailed [SDK API reference](/sdk-api) documentation
***
**Ready to integrate?** Choose a category above to explore specific integrations and get started in minutes.
# ML Platforms Overview
Source: https://docs.fiddler.ai/integrations/ml-platforms-and-tools/ml-platforms
Integrate Fiddler with MLOps platforms, experiment tracking tools, and ML frameworks
Integrate Fiddler into your MLOps workflow to monitor models across the entire machine learning lifecycle. From experiment tracking to production deployment, Fiddler works with the ML platforms you already use.
## Why ML Platform Integrations Matter
Modern ML teams use sophisticated platforms for experimentation, training, and deployment. Fiddler's integrations ensure you can:
* **Unified Model Governance** - Track models from experiment to production in one platform
* **Automated Monitoring Setup** - Auto-configure monitoring when models are registered
* **Seamless Workflow Integration** - Add observability without changing existing processes
* **Bi-Directional Sync** - Share metrics between Fiddler and your ML platform
* **Experiment Comparison** - Compare production performance against training experiments
## MLOps Platform Integrations
### Databricks
Integrate Fiddler with Databricks for unified ML development and monitoring.
**Why Databricks + Fiddler:**
* **Lakehouse Architecture** - Monitor models trained on Delta Lake data
* **MLflow Integration** - Automatic sync of registered models to Fiddler
* **Notebook Integration** - Use Fiddler SDK directly in Databricks notebooks
* **Production Monitoring** - Monitor models served via Databricks Model Serving
**Key Features:**
* **Automatic Model Registration** - Models registered in Databricks MLflow automatically appear in Fiddler
* **Feature Store Integration** - Monitor drift using Databricks Feature Store definitions
* **Collaborative Debugging** - Share Fiddler insights in Databricks notebooks
* **Unified Data Access** - Use Delta Lake as data source for baselines and production data
[**Get Started with Databricks →**](/integrations/ml-platforms/databricks-integration)
**Quick Start:**
```python theme={null}
from databricks import mlflow as dbx_mlflow
from fiddler import FiddlerClient
# Register model in Databricks MLflow
model_uri = "models:/credit_risk_model/Production"
model_version = dbx_mlflow.register_model(model_uri, "credit_risk_model")
# Automatically sync to Fiddler
client = FiddlerClient(api_key="fid_...")
client.sync_from_databricks(
model_name="credit_risk_model",
version=model_version,
enable_monitoring=True
)
```
### MLflow
Connect Fiddler to MLflow for experiment tracking and model registry integration.
**Why MLflow + Fiddler:**
* **Open-Source Standard** - Works with any MLflow deployment (Databricks, AWS, GCP, self-hosted)
* **Model Registry Sync** - Automatically monitor models when they transition to "Production"
* **Experiment Tracking** - Compare production metrics with training experiment metrics
* **Model Versioning** - Track performance across model versions
**Key Features:**
* **Automatic Model Onboarding** - Models in MLflow registry auto-configure in Fiddler
* **Metric Synchronization** - Export Fiddler metrics back to MLflow for unified view
* **Artifact Integration** - Link model artifacts between MLflow and Fiddler
* **Stage-Based Monitoring** - Different monitoring configs for Staging vs Production
[**Get Started with MLflow →**](/integrations/ml-platforms/ml-flow-integration)
**Quick Start:**
```python theme={null}
import mlflow
from fiddler import FiddlerClient
# Set MLflow tracking URI
mlflow.set_tracking_uri("https://mlflow.example.com")
# Configure Fiddler to sync with MLflow
client = FiddlerClient(api_key="fid_...")
client.add_mlflow_integration(
tracking_uri="https://mlflow.example.com",
auto_sync_on_stage_transition=True,
stages=["Production", "Staging"]
)
# Models transitioning to "Production" will automatically be monitored in Fiddler
```
## Experiment Tracking & Model Registry
### Unified Model Lifecycle
Track models from experimentation through production:
```
Experiment → Training → Registration → Staging → Production → Monitoring
↓ ↓ ↓ ↓ ↓ ↓
MLflow MLflow MLflow MLflow MLflow Fiddler
Runs Runs Registry Registry Registry + MLflow
```
**Integration Benefits:**
* **Single Source of Truth** - MLflow registry as canonical model inventory
* **Automated Workflows** - Monitoring setup triggered by model registration
* **Version Comparison** - Compare production metrics across model versions
* **Rollback Readiness** - Quick rollback with historical performance data
### Experiment-to-Production Comparison
Compare production model performance against training experiments:
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Get production metrics
prod_metrics = client.get_metrics(
project="fraud-detection",
model="fraud_model_v3",
start_time="2024-11-01",
end_time="2024-11-10"
)
# Compare with training experiment (from MLflow)
experiment_metrics = client.get_experiment_metrics(
mlflow_experiment_id="exp_12345",
mlflow_run_id="run_67890"
)
# Generate comparison report
report = client.compare_metrics(
production=prod_metrics,
experiment=experiment_metrics,
metrics=["accuracy", "precision", "recall", "auc"]
)
```
## ML Framework Support
While Fiddler is framework-agnostic, we provide enhanced support for popular ML frameworks:
### Supported ML Frameworks
**Classical ML:**
* **Scikit-Learn** - Full support for all estimators
* **XGBoost** - Native explainability for tree models
* **LightGBM** - Fast SHAP explanations
* **CatBoost** - Categorical feature support
**Deep Learning:**
* **TensorFlow/Keras** - Model analysis and monitoring
* **PyTorch** - Dynamic graph model support
* **JAX** - High-performance model monitoring
* **ONNX** - Framework-agnostic model format
**AutoML:**
* **H2O.ai** - AutoML model monitoring
* **AutoGluon** - Tabular model support
* **TPOT** - Pipeline optimization monitoring
### Framework-Specific Features
**Tree-Based Models (XGBoost, LightGBM, CatBoost):**
* Fast SHAP explanations using native implementations
* Feature importance tracking over time
* Tree structure analysis for debugging
**Deep Learning (TensorFlow, PyTorch):**
* Layer-wise activation monitoring
* Embedding drift detection
* Custom metric support for complex architectures
**Example - XGBoost Monitoring:**
```python theme={null}
import xgboost as xgb
from fiddler import FiddlerClient
# Train XGBoost model
model = xgb.XGBClassifier()
model.fit(X_train, y_train)
# Upload to Fiddler with automatic feature importance
client = FiddlerClient(api_key="fid_...")
client.upload_model(
project="credit-risk",
model_name="xgb_risk_model",
model=model,
task="binary_classification",
enable_shap=True # Native XGBoost SHAP support
)
```
## Integration Architecture Patterns
### Pattern 1: MLflow-Centric Workflow
Use MLflow as the central hub for all ML operations:
```
Data Preparation
↓
Experimentation (MLflow Tracking)
↓
Model Registry (MLflow)
↓ (webhook on stage transition)
Fiddler Auto-Onboarding
↓
Production Monitoring (Fiddler + MLflow metrics export)
```
**Configuration:**
```python theme={null}
# One-time setup: Configure MLflow webhook
client = FiddlerClient(api_key="fid_...")
client.configure_mlflow_webhook(
mlflow_tracking_uri="https://mlflow.example.com",
webhook_secret="webhook_secret_key",
on_stage_transition={
"Production": "enable_full_monitoring",
"Staging": "enable_basic_monitoring",
"Archived": "disable_monitoring"
}
)
```
### Pattern 2: Databricks Unity Catalog Integration
Leverage Databricks Unity Catalog for governance and Fiddler for monitoring:
```
Unity Catalog (Model Registry)
↓
Databricks Model Serving
↓
Production Traffic
↓ (streaming predictions)
Fiddler Monitoring
↓
Alerts → Databricks Workflow Jobs (retraining)
```
**Configuration:**
```python theme={null}
# Connect Fiddler to Unity Catalog
client = FiddlerClient(api_key="fid_...")
client.add_unity_catalog_integration(
workspace_url="https://dbc-xxxxx.cloud.databricks.com",
catalog="ml_models",
schema="production",
access_token=dbutils.secrets.get("fiddler", "databricks_token")
)
```
### Pattern 3: Multi-Platform Model Tracking
Monitor models across multiple ML platforms:
```
Training Platform Mix:
├── Databricks (Lakehouse models)
├── SageMaker (AWS-native models)
├── Vertex AI (GCP models)
└── On-Premises (Legacy models)
↓ (all models sync to)
Fiddler (Unified Monitoring)
```
## Getting Started
### Prerequisites
* **Fiddler Account** - Cloud or on-premises deployment
* **ML Platform Access** - Databricks workspace or MLflow server
* **Credentials** - Fiddler access token + ML platform credentials
* **Network Connectivity** - Firewall rules for integration
### General Setup Steps
**1. Configure ML Platform Connection**
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(
api_key="fid_...",
url="https://app.fiddler.ai"
)
# Add ML platform integration
client.add_integration(
type="databricks", # or "mlflow"
config={
"workspace_url": "https://dbc-xxxxx.cloud.databricks.com",
"access_token": "dapi...",
"auto_sync": True
}
)
```
**2. Sync Existing Models (Optional)**
```python theme={null}
# One-time sync of existing models
models = client.sync_models_from_platform(
platform="databricks",
filter={"stage": "Production"}
)
print(f"Synced {len(models)} models to Fiddler")
```
**3. Enable Auto-Monitoring**
```python theme={null}
# Future model registrations will automatically be monitored
client.configure_auto_monitoring(
platform="databricks",
enabled=True,
default_config={
"enable_drift_detection": True,
"enable_performance_tracking": True,
"enable_explainability": True
}
)
```
## Advanced Integration Features
### Feature Store Integration
Monitor models using features from Databricks Feature Store:
```python theme={null}
from databricks.feature_store import FeatureStoreClient
fs = FeatureStoreClient()
# Create feature spec
feature_spec = fs.create_feature_spec(
table_name="ml_features.user_features",
primary_keys=["user_id"]
)
# Monitor model with feature store schema
client.upload_model(
project="recommendations",
model="user_model",
feature_spec=feature_spec, # Auto-generates schema from Feature Store
enable_drift_detection=True
)
```
### Automated Retraining Triggers
Trigger retraining workflows when drift is detected:
```python theme={null}
# Configure alert to trigger Databricks job
client.create_alert(
name="High Drift - Retrain Model",
trigger_type="drift",
threshold=0.15,
model="credit_risk_model",
actions=[{
"type": "databricks_job",
"job_id": "12345",
"parameters": {
"model_name": "credit_risk_model",
"reason": "drift_detected"
}
}]
)
```
### Model Lineage Tracking
Track complete model lineage from data to deployment:
```python theme={null}
# Capture full model lineage
lineage = {
"data_source": "s3://bucket/training-data-v2.parquet",
"feature_transformations": "feature_pipeline_v1",
"training_framework": "xgboost==1.7.0",
"mlflow_run_id": "run_67890",
"parent_model": "credit_risk_model_v1"
}
client.update_model_metadata(
project="credit-risk",
model="credit_risk_model_v2",
lineage=lineage
)
```
## Integration Selector
Choose the right ML platform integration for your workflow:
| Your ML Platform | Recommended Integration | Why |
| -------------------- | -------------------------- | ------------------------------------------- |
| Databricks Lakehouse | **Databricks integration** | Native MLflow, Unity Catalog, Feature Store |
| Self-hosted MLflow | **MLflow integration** | Open-source, cloud-agnostic |
| AWS SageMaker | **SageMaker Pipelines** | AWS-native, Partner AI App compatible |
| Azure ML | **MLflow integration** | Azure ML uses MLflow under the hood |
| Vertex AI (GCP) | **MLflow integration** | Vertex AI supports MLflow |
| Multiple platforms | **MLflow integration** | Universal compatibility |
## Bi-Directional Metric Sync
Share metrics between Fiddler and your ML platform:
### Export Fiddler Metrics to MLflow
```python theme={null}
# Log Fiddler metrics to MLflow experiments
client.export_metrics_to_mlflow(
fiddler_project="fraud-detection",
fiddler_model="fraud_model_v3",
mlflow_experiment_name="production_monitoring",
metrics=["drift_score", "accuracy", "f1_score"],
time_range="last_7_days"
)
```
### Import MLflow Metrics to Fiddler
```python theme={null}
# Import custom metrics from MLflow
client.import_metrics_from_mlflow(
mlflow_run_id="run_67890",
fiddler_project="fraud-detection",
fiddler_model="fraud_model_v3",
metrics=["custom_business_metric", "validation_loss"]
)
```
## Security & Access Control
### Authentication Methods
**Databricks:**
* Personal Access Tokens (development)
* Service Principal OAuth (production)
* Azure AD Integration (enterprise)
**MLflow:**
* HTTP Basic Authentication
* Token-Based Authentication
* Custom Auth Plugins
### Permission Requirements
**Databricks Permissions:**
* `CAN_MANAGE` on registered models
* `CAN_READ` on Feature Store tables
* `CAN_USE` on clusters (for SHAP computation)
**MLflow Permissions:**
* Read access to Model Registry
* Read access to Experiment Tracking
* Write access for metric export (optional)
## Monitoring MLOps Pipeline Health
### Track Integration Health
```python theme={null}
# Check integration status
status = client.get_integration_status("databricks")
print(f"Status: {status.connected}")
print(f"Last sync: {status.last_sync_time}")
print(f"Models synced: {status.models_count}")
```
### Alerts for Sync Failures
```python theme={null}
# Alert on integration failures
client.create_alert(
name="MLflow Sync Failure",
trigger_type="integration_error",
integration="mlflow",
notification_channels=["email", "slack"]
)
```
## Troubleshooting
### Common Issues
**Models Not Syncing:**
* Verify MLflow/Databricks credentials are valid
* Check network connectivity from Fiddler to ML platform
* Ensure models are in the correct stage (e.g., "Production")
* Validate webhook endpoint is reachable (for event-driven sync)
**Schema Mismatches:**
* Ensure feature names match between training and production
* Verify data types are consistent
* Check for missing features in production data
**Performance Issues:**
* For large models, use SHAP sampling instead of full computation
* Enable lazy loading for model artifacts
* Use incremental sync for model registry (don't sync all historical versions)
## Related Integrations
* [**Data Platforms**](/integrations/data-platforms-and-pipelines/data-platforms) - Connect to Snowflake, BigQuery for training data
* [**Cloud Platforms**](/integrations/cloud-platforms-and-deployment/cloud-platforms) - Deploy Fiddler on AWS, Azure, GCP
* [**Agentic AI**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) - Monitor LangGraph and LLM applications
* [**Monitoring & Alerting**](/integrations/monitoring-and-alerting/monitoring-alerting) - Alert on model issues
***
# Databricks
Source: https://docs.fiddler.ai/integrations/ml-platforms/databricks-integration
Discover how Fiddler helps monitor, explain, and analyze models in Databricks Workspace. Integrate with MLFlow and Spark to manage, validate, and monitor models.
Fiddler allows your team to monitor, explain and analyze your models developed and deployed in [Databricks Workspace](https://docs.databricks.com/introduction/index.html) by integrating with [MLflow](https://docs.databricks.com/mlflow/index.html) for model asset management and utilizing Databricks Spark environment for data management.
To validate and monitor models built on Databricks using Fiddler, you can follow these steps:
1. [Create a Fiddler model](/developers/python-client-guides/model-onboarding/create-a-project-and-model) using sample data or model information from MLflow
2. [Publish production data](/developers/client-library-reference/publishing-production-data) streaming live or in batches
#### Prerequisites
This guide assumes you have:
* A Databricks account and valid credentials
* A Fiddler environment with an account and valid credentials
* Know how to [connect and use ](/developers/python-client-guides/installation-and-setup)the [Fiddler Python Client SDK](/sdk-api/python-client/connection)
#### Begin with a Databricks Notebook
Launch a [Databricks notebook](https://docs.databricks.com/notebooks/index.html) from your workspace and run the following code:
```bash theme={null}
pip install -q fiddler-client
```
```python theme={null}
import fiddler as fdl
```
Now that you have the Fiddler library installed, you can connect to your Fiddler environment. You will need your authentication token from the [Credentials](/reference/administration/settings#credentials) tab in Application Settings.
```python theme={null}
URL = ""
AUTH_TOKEN = ""
fdl.init(url=URL, token=AUTH_TOKEN)
```
Finally, you can set up a new [project](/sdk-api/python-client/connection) using:
```python theme={null}
# The project.id is required when creating models
project = fdl.Project(name='YOUR_PROJECT_NAME')
project.create()
```
#### Creating the Fiddler Model
**Quickest Option: Let Fiddler Automate Model Creation**
The quickest way to onboard a Fiddler model is to get a sample of data from which Fiddler can infer model schema and metadata. Ideally you will have baseline, testing, or training data that is representative of your model schema. Fiddler can infer your model schema from this sample dataset. You can download baseline or training data from a [delta table](https://docs.databricks.com/getting-started/dataframes-python.html) and share it with Fiddler as a baseline dataset:
```python theme={null}
sample_dataset = spark.read.table("YOUR_DATASET").select("*").toPandas()
```
Now that you have sample data, you can easily create a Fiddler model, as demonstrated in our [Simple Monitoring Quick Start Guide](/developers/quick-starts/simple-ml-monitoring). A rough outline of the steps follows:
```py theme={null}
# Define a ModelSpec that tells Fiddler what role each column
# in your model schema serves.
model_spec = fdl.ModelSpec(
inputs=['feature_input_column', ...],
outputs=['output_column'],
targets=['label_column'],
metadata=['id_column', 'data_segment_column', ...],
)
# Identify the task your ML model performs as Fiddler will use this
# to generate the performance metrics appropriate to the task.
# ModelTask.NOT_SET is also an option if performance metrics are not needed.
model_task = fdl.ModelTask.BINARY_CLASSIFICATION
task_params = fdl.ModelTaskParams(target_class_order=['no', 'yes'])
# Use Model.from_data() to define your model's ModelSchema automatically by
# passing the sample_dataset in the source parameter for schema inference.
model = fdl.Model.from_data(
name='name_for_display_in_Fiddler',
project_id=project.id,
source=sample_dataset,
spec=model_spec,
task=model_task,
task_params=task_params,
event_id_col='your_unique_event_id_column',
event_ts_col='event_timestamp_column'
)
# Create the model in Fiddler
model.create()
```
**Option: Using the MLflow Model Registry**
Another option is to manually construct your model's schema from the details contained in the MLflow registry. Using the MLflow API, you can query the model registry and get the model signature, which describes the inputs and outputs as a dictionary. You can use this dictionary to build the Model, ModelSchema, and ModelSpec objects that define the tabular schema of your model.
```python theme={null}
import mlflow
from mlflow.tracking import MlflowClient
# Initiate MLFlow Client
client = MlflowClient()
# Get the model URI
model_version_info = client.get_model_version(model_name, model_version)
model_uri = client.get_model_version_download_uri(model_name, model_version_info)
#Get the Model Signature
mlflow_model_info = mlflow.models.get_model_info(model_uri)
model_inputs_schema = mlflow_model_info.signature.inputs.to_dict()
model_inputs = [ sub['name'] for sub in model_inputs_schema ]
```
Refer to this [example notebook](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/examples/create_model_from_constructor.ipynb) in GitHub, which demonstrates manually defining your Fiddler model's schema.
#### Publishing Events
Now you can publish all the events from your models. You can do this in two ways:
**Batch Models**
If your models run batch processes with your models or your aggregate model outputs over a time frame, then you can use the table change feed from Databricks to select only the new events and send them to Fiddler:
```python theme={null}
import fiddler as fdl
from pyspark.sql import SparkSession
# Get the active Spark session
spark = SparkSession.builder.getOrCreate()
changes_df = (
spark.read.format("delta")
.option("readChangeFeed", "true")
.option("startingVersion", last_version)
.option("endingVersion", new_version)
.table("inferences")
.toPandas()
)
# Assumes an initialized Python client session and instantiated Model
job = model.publish_batch(
source=changes_df,
environment=fdl.EnvType.PRODUCTION,
)
print(f'Initiated Production dataset upload with Job ID = {job.id}')
```
**Live Models**
For models with live predictions or real-time applications, you can add the following code snippet to your prediction pipeline and send every event to Fiddler in real-time:
```python theme={null}
# Turn your model's output in a pandas dataframe
example_event = model_output.toJSON().map(lambda x: json.loads(x)).collect()
# Assumes an initialized Python client session and instantiated Model
event_id = model.publish_stream(
events=example_event,
environment=fdl.EnvType.PRODUCTION,
)
print(f'Published {event_id_list}')
```
# MLflow
Source: https://docs.fiddler.ai/integrations/ml-platforms/ml-flow-integration
Explore how Fiddler helps your team onboard, monitor, explain, and analyze models with MLFlow. Learn to ingest model metadata and artifacts for observability.
Fiddler allows your team to onboard, monitor, explain, and analyze your models developed with [MLflow](https://mlflow.org/).
This guide shows you how to ingest the model metadata and artifacts stored in your MLflow model registry and use them to set up model observability in the Fiddler Platform:
1. Exporting Model Metadata from MLflow to Fiddler
2. Uploading Model Artifacts to Fiddler for XAI
### Onboarding a Model
Refer to this [section](/integrations/ml-platforms/databricks-integration#creating-the-fiddler-model) of the Databricks integration guide for onboarding your model to Fiddler using model information from MLflow.
### Uploading Model Artifacts
Using the [**MLflow API**](https://mlflow.org/docs/latest/python_api/mlflow.html) you can query the model registry and get the **model signature** which describes the inputs and outputs as a dictionary.
#### Uploading Model Files
Sharing your model artifacts helps Fiddler explain your models. By leveraging the MLflow API you can download these model files:
```python theme={null}
import os
import mlflow
from mlflow.store.artifact.models_artifact_repo import ModelsArtifactRepository
model_name = "example-model-name"
model_stage = "Staging" # Should be either 'Staging' or 'Production'
mlflow.set_tracking_uri("databricks")
os.makedirs("model", exist_ok=True)
local_path = ModelsArtifactRepository(
f'models:/{model_name}/{model_stage}'
).download_artifacts("", dst_path="model")
print(f'{model_stage} Model {model_name} is downloaded at {local_path}')
```
Once you have the model file, you can create a package.py file in this model directory that describes how to access this model.
Finally, you can upload all the model artifacts to Fiddler:
```python theme={null}
# Assumes an initialized Python client session and instantiated Model
job = model.add_artifact(
model_dir=MODEL_ARTIFACTS_DIR,
)
job.wait()
```
Alternatively, you can skip uploading your model and use Fiddler to generate a surrogate model to get low-fidelity explanations for your model.
Please refer to the Explainability guide for detailed information on model artifacts, packages, and surrogate models.
# Datadog
Source: https://docs.fiddler.ai/integrations/monitoring-alerting/datadog-integration
Learn about Fiddler’s Datadog integration to bring AI Observability metrics into your dashboards. Follow steps to centralize ML model and application monitoring.
Fiddler offers an integration with Datadog which allows Fiddler and Datadog customers to bring their AI Observability metrics from Fiddler into their centralized Datadog dashboards. Additionally, a Fiddler license can now be procured through the [Datadog Marketplace](https://www.datadoghq.com/blog/tag/datadog-marketplace/). This [integration](https://www.datadoghq.com/blog/monitor-machine-learning-models-fiddler/) enables you to centralize your monitoring of ML models and the applications that utilize them within one unified platform.
### Integrating Fiddler with Datadog
Instructions for integrating Fiddler with Datadog can be found on the "Integrations" section of your Datadog console. Simply search for "Fiddler" and follow the installation instructions provided on the "Configure" tab. Please reach out to \<[support@fiddler.ai](mailto:support@fiddler.ai)> with any issues or questions.
![Fiddler displayed as installed in the Datadog marketplace Integrations tab]()
![The Fiddler Datadog integration app Overview tab]()
![The Fiddler Datadog integration app Configuration tab]()
# PagerDuty
Source: https://docs.fiddler.ai/integrations/monitoring-alerting/pagerduty
Discover how customer churn prediction works with Fiddler’s AI observability platform. Follow our example to detect and diagnose issues step by step.
Fiddler offers powerful alerting tools for monitoring models. By integrating with
PagerDuty services, you gain the ability to trigger PagerDuty events within your monitoring
workflow.
> 📘 If your organization has already integrated with PagerDuty, then you may skip to the [Setup: In Fiddler](#setup-in-fiddler) section to learn more about setting up PagerDuty within Fiddler.
### Setup: In PagerDuty
1. Within your PagerDuty Team, navigate to **Services** → **Service Directory**.
![]()
2. Within the Service Directory:
* If you are creating a new service for integration, select **+New Service** and follow the prompts to create your service.
* Click the **name of the service** you want to integrate with.
![]()
3. Navigate to **Integrations** within your service, and select **Add a new integration to this service**.
![]()
4. Enter an **Integration Name**, and under **Integration Type** select the option **Use our API directly**. Then, select the **Add Integration** button to save your new integration. You will be redirected to the Integrations page for your service.
![]()
5. Copy the **Integration Key** for your new integration.
![]()
### Setup: In Fiddler
1. Within **Fiddler**, navigate to the **Settings** page, and then to the **PagerDuty Integration** menu. If your organization **already has a PagerDuty service integrated with Fiddler**, you will be able to find it in the list of services.
![]()
2. If you are looking to integrate with a new service, select the **`+`** box on the top right. Then, enter the name of your service, as well as the Integration Key copied from the end of the [Setup: In PagerDuty](#setup-in-pagerduty) section above. After creation, confirm that your new entry is now in the list of available services.
![]()
> 🚧 Creating, editing, and deleting these services is an **ADMINISTRATOR**-only privilege. Please contact an **ADMINISTRATOR** within your organization to set up any new PagerDuty services
### PagerDuty Alerts in Fiddler
1. Within the **Projects** page, select the model you wish to use with PagerDuty.
![]()
2. Select **Monitor** → **Alerts** → **Add Alert**.
![]()
3. Enter the condition you would like to alert on, and under **PagerDuty Services**, select all services you would like the alert to trigger for. Additionally, select the **Severity** of this alert, and hit **Save**.
![]()
4. After creation, the alert will now trigger for the specified PagerDuty services.
> 📘 Info
>
> Check out the [alerts documentation](/developers/python-client-guides/alerts-with-fiddler-client) for more information on setting up alerts.
### FAQ
**Can Fiddler integrate with multiple PagerDuty services?**
* Yes. So long as the service is present within **Settings** → **PagerDuty Services**, anyone within your organization can select that service to be a recipient for an alert.
# Monitoring & Alerting Overview
Source: https://docs.fiddler.ai/integrations/monitoring-and-alerting/monitoring-alerting
Connect Fiddler alerts to incident management, observability, and communication tools
Route Fiddler AI observability alerts to your existing incident management and communication tools. Integrate with observability platforms, on-call systems, and team collaboration tools to ensure AI issues are detected, triaged, and resolved quickly.
## Why Alert Integration Matters
AI models fail in unique ways—drift, data quality issues, performance degradation, safety violations. Fiddler's alert integrations ensure your team responds immediately:
* **Unified Incident Management** - AI alerts flow into the same systems as infrastructure alerts
* **Faster Response Times** - On-call engineers notified via existing escalation policies
* **Context-Rich Alerts** - Model context, affected predictions, and root cause analysis included
* **Reduced Alert Fatigue** - Intelligent grouping and deduplication across tools
* **Automated Remediation** - Trigger workflows to rollback models or scale resources
## Integration Categories
### 📊 Observability Platforms
Send Fiddler metrics and alerts to enterprise observability platforms for unified monitoring.
**Supported Platforms:**
* [**Datadog**](/integrations/monitoring-alerting/datadog-integration) - Application performance monitoring and infrastructure observability ✓ **GA**
**Common Use Cases:**
* Correlate AI model issues with infrastructure metrics
* Build unified dashboards combining Fiddler + infrastructure data
* Use Datadog's anomaly detection on Fiddler metrics
* Alert on compound conditions (model drift + high latency)
### 🚨 Incident Management
Connect alerts to on-call systems for immediate engineer notification.
**Supported Platforms:**
* [**PagerDuty**](/integrations/monitoring-alerting/pagerduty) - Incident management and on-call scheduling ✓ **GA**
**Common Use Cases:**
* Page on-call ML engineers for critical model failures
* Escalate unresolved AI incidents automatically
* Track MTTR (Mean Time To Resolution) for model issues
* Integrate with incident runbooks and response workflows
### 💬 Team Collaboration
Send alerts to team communication tools for visibility and collaboration.
**Supported Platforms:**
* **Slack** - Team messaging and collaboration ✓ **GA** *(Coming Soon)*
* **Microsoft Teams** - Enterprise communication platform ✓ **GA** *(Coming Soon)*
**Common Use Cases:**
* Notify ML team channel when drift is detected
* Alert data science team on data quality issues
* Share model performance reports automatically
* Collaborative incident triage in team channels
## Observability Platform Integrations
### Datadog
Integrate Fiddler with Datadog for unified application and AI monitoring.
**Why Datadog + Fiddler:**
* **Unified Dashboards** - Combine infrastructure, application, and AI model metrics
* **Correlated Alerts** - Alert on compound conditions (e.g., "high model drift + high API latency")
* **Service Map Integration** - See model health in Datadog service dependency graphs
* **Anomaly Detection** - Leverage Datadog's ML-based alerting on Fiddler metrics
**Key Features:**
* **Metric Export** - Send Fiddler drift, performance, and data quality metrics to Datadog
* **Event Streaming** - Stream model events (predictions, drift detections) as Datadog events
* **Alert Forwarding** - Route Fiddler alerts to Datadog for unified incident management
* **Tag Propagation** - Maintain consistent tagging across platforms (model, environment, team)
**Status:** ✓ **GA** - Production-ready
[**Get Started with Datadog →**](/integrations/monitoring-alerting/datadog-integration)
**Quick Start:**
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Configure Datadog integration
client.add_datadog_integration(
api_key="datadog_api_key",
app_key="datadog_app_key",
site="datadoghq.com", # or datadoghq.eu
# Metric export configuration
export_metrics=True,
metric_prefix="fiddler.model.",
tags=["env:production", "team:ml-platform"],
# Event export configuration
export_events=True,
event_priority="normal" # or "low"
)
```
**Example Datadog Dashboard:**
```yaml theme={null}
# Datadog dashboard combining infrastructure + AI metrics
widgets:
- title: "Model Latency vs API Latency"
type: timeseries
queries:
- metric: fiddler.model.latency.p95
scope: model:fraud_detector
- metric: trace.flask.request.duration.p95
scope: service:fraud-api
- title: "Model Drift Detection"
type: query_value
queries:
- metric: fiddler.model.drift.score
aggregation: max
conditional_formats:
- comparator: ">"
value: 0.1
palette: "red"
```
## Incident Management Integrations
### PagerDuty
Route critical AI alerts to on-call engineers via PagerDuty.
**Why PagerDuty + Fiddler:**
* **On-Call Escalation** - Page the right ML engineer based on escalation policies
* **Incident Deduplication** - Prevent alert storms from related model issues
* **Incident Timeline** - Track when AI issues were detected, acknowledged, resolved
* **Postmortem Integration** - Include model context in incident reports
**Key Features:**
* **Severity Mapping** - Map Fiddler alert criticality to PagerDuty severity levels
* **Service Integration** - Associate alerts with PagerDuty services (e.g., "Fraud Detection Service")
* **Custom Payloads** - Include model metadata, drift scores, affected predictions
* **Bidirectional Updates** - Acknowledge/resolve incidents in PagerDuty or Fiddler
**Status:** ✓ **GA** - Production-ready
[**Get Started with PagerDuty →**](/integrations/monitoring-alerting/pagerduty)
**Quick Start:**
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Configure PagerDuty integration
client.add_pagerduty_integration(
integration_key="pagerduty_integration_key",
service_name="ML Models - Production",
# Alert routing rules
severity_mapping={
"critical": "critical", # Fiddler → PagerDuty severity
"high": "error",
"medium": "warning",
"low": "info"
}
)
# Create alert with PagerDuty notification
client.create_alert(
name="Critical Model Drift",
project="fraud-detection",
model="fraud_detector_v3",
metric="drift_score",
threshold=0.15,
severity="critical",
notification_channels=["pagerduty"]
)
```
**Example PagerDuty Incident:**
```json theme={null}
{
"incident_key": "fiddler_drift_fraud_detector_v3",
"type": "trigger",
"description": "High drift detected on fraud_detector_v3",
"details": {
"model": "fraud_detector_v3",
"project": "fraud-detection",
"drift_score": 0.23,
"affected_features": ["transaction_amount", "merchant_category"],
"time_window": "2024-11-10 14:00 - 14:30 UTC",
"fiddler_url": "https://app.fiddler.ai/projects/fraud-detection/models/fraud_detector_v3/drift"
},
"client": "Fiddler AI Observability",
"client_url": "https://app.fiddler.ai"
}
```
## Team Collaboration Integrations
### Slack *(Coming Soon)*
**Planned Features:**
* **Channel Notifications** - Post alerts to team Slack channels
* **Interactive Messages** - Acknowledge, snooze, or resolve alerts from Slack
* **Scheduled Reports** - Daily/weekly model performance summaries
* **Threaded Discussions** - Collaborate on incident resolution in threads
**Example Configuration:**
```python theme={null}
# Future Slack integration API
client.add_slack_integration(
webhook_url="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXX",
channel="#ml-alerts",
username="Fiddler",
icon_emoji=":robot_face:"
)
```
### Microsoft Teams *(Coming Soon)*
**Planned Features:**
* **Adaptive Cards** - Rich, interactive alert notifications
* **Team Channels** - Route alerts to relevant team channels
* **Bot Commands** - Query model status from Teams chat
* **Integration with Workflows** - Trigger Teams workflows on alerts
## Alert Routing Patterns
### Pattern 1: Severity-Based Routing
Route alerts to different channels based on severity:
```python theme={null}
# Critical alerts → PagerDuty (page on-call)
client.create_alert(
name="Model Offline",
severity="critical",
notification_channels=["pagerduty"]
)
# High alerts → Slack (notify team)
client.create_alert(
name="High Drift Detected",
severity="high",
notification_channels=["slack"]
)
# Medium alerts → Datadog (track as events)
client.create_alert(
name="Minor Data Quality Issue",
severity="medium",
notification_channels=["datadog"]
)
```
### Pattern 2: Team-Based Routing
Different teams get different alerts:
```python theme={null}
# ML Engineering team
client.create_alert(
name="Model Performance Degradation",
project="fraud-detection",
notification_channels=["slack"],
slack_channel="#ml-engineering",
tags=["team:ml-engineering"]
)
# Data Engineering team
client.create_alert(
name="Data Pipeline Failure",
project="fraud-detection",
notification_channels=["slack"],
slack_channel="#data-engineering",
tags=["team:data-engineering"]
)
# On-Call team (escalation)
client.create_alert(
name="Critical Model Failure",
project="fraud-detection",
notification_channels=["pagerduty"],
pagerduty_service="ml-models-production",
tags=["team:on-call"]
)
```
### Pattern 3: Composite Alerting
Alert on compound conditions across multiple platforms:
```python theme={null}
# Example: Alert if BOTH model drift is high AND API latency is high
# (Indicates model update needed + production impact)
# Configure in Datadog (using Fiddler + Datadog metrics)
datadog_composite_alert = {
"name": "Model Drift + High Latency",
"query": """
avg(last_5m):fiddler.model.drift.score{model:fraud_detector} > 0.1
AND
avg(last_5m):trace.flask.request.duration.p95{service:fraud-api} > 500
""",
"message": "@pagerduty-ml-oncall Model drift detected with production latency impact",
"tags": ["model:fraud_detector", "alert:composite"]
}
```
## Metric Export Patterns
### Export Fiddler Metrics to Datadog
```python theme={null}
from fiddler import FiddlerClient
client = FiddlerClient(api_key="fid_...")
# Configure metric export
client.configure_metric_export(
destination="datadog",
metrics=[
"drift_score",
"data_quality_score",
"prediction_latency_p95",
"prediction_count",
"error_rate"
],
export_interval=60, # seconds
tags=["source:fiddler", "env:production"]
)
```
**Exported Metrics:**
* `fiddler.model.drift.score` - Overall drift score (0-1)
* `fiddler.model.drift.feature.<feature_name>` - Per-feature drift
* `fiddler.model.performance.<metric>` - Model performance metrics
* `fiddler.model.data_quality.score` - Data quality score
* `fiddler.model.predictions.count` - Prediction volume
* `fiddler.model.predictions.latency` - Prediction latency percentiles
### Query Fiddler Metrics in Datadog
```python theme={null}
# Datadog Metrics Query Language (MQL)
queries = {
"high_drift_models": """
avg:fiddler.model.drift.score{*} by {model} > 0.1
""",
"low_performance_models": """
avg:fiddler.model.performance.accuracy{*} by {model} < 0.85
""",
"high_volume_models": """
sum:fiddler.model.predictions.count{*} by {model}.as_count()
"""
}
```
## Alert Lifecycle Management
### Alert States
Fiddler alerts transition through these states:
```
Triggered → Acknowledged → Investigating → Resolved → Closed
↓ ↑
Escalated --------------------------------┘
```
**Synchronization with External Tools:**
* **PagerDuty**: Bidirectional state sync (acknowledge, resolve)
* **Datadog**: Event-based updates
* **Slack**: Interactive message updates
### Alert Deduplication
Prevent alert storms with intelligent deduplication:
```python theme={null}
# Configure deduplication rules
client.configure_alert_deduplication(
project="fraud-detection",
model="fraud_detector_v3",
# Group related alerts
grouping_window=300, # seconds
grouping_keys=["model", "metric"],
# Suppress similar alerts
suppression_window=3600, # seconds
max_alerts_per_window=3
)
```
## Custom Webhook Integrations
For platforms not natively supported, use generic webhooks:
```python theme={null}
# Send alerts to any webhook endpoint
client.add_webhook_integration(
name="custom-alerting-system",
url="https://your-system.com/webhook/fiddler",
method="POST",
headers={
"Authorization": "Bearer your-token",
"Content-Type": "application/json"
},
payload_template={
"alert_name": "{{alert.name}}",
"model": "{{model.name}}",
"severity": "{{alert.severity}}",
"timestamp": "{{alert.timestamp}}",
"details": "{{alert.details}}"
}
)
# Use webhook in alerts
client.create_alert(
name="Custom Alert",
notification_channels=["webhook:custom-alerting-system"]
)
```
**Webhook Payload Example:**
```json theme={null}
{
"alert_name": "High Drift Detected",
"model": "fraud_detector_v3",
"project": "fraud-detection",
"severity": "high",
"timestamp": "2024-11-10T14:32:01Z",
"metric": "drift_score",
"current_value": 0.23,
"threshold": 0.15,
"fiddler_url": "https://app.fiddler.ai/projects/fraud-detection/models/fraud_detector_v3",
"affected_features": ["transaction_amount", "merchant_category"],
"recommended_actions": [
"Investigate feature distribution changes",
"Consider model retraining",
"Review data pipeline for issues"
]
}
```
## Monitoring Integration Health
### Track Integration Status
```python theme={null}
# Check integration health
integrations = client.list_integrations()
for integration in integrations:
status = client.get_integration_health(integration.name)
print(f"{integration.name}: {status.status}")
print(f" Last successful send: {status.last_success}")
print(f" Failed alerts (24h): {status.failed_count}")
```
### Alerts on Integration Failures
```python theme={null}
# Alert if alert delivery fails
client.create_meta_alert(
name="Alert Delivery Failure",
trigger="integration_failure",
integration="pagerduty",
threshold=3, # failures
time_window=3600, # seconds
notification_channels=["email"] # use different channel!
)
```
## Best Practices
### Alert Fatigue Prevention
**1. Use Appropriate Severity Levels:**
```python theme={null}
# Reserve "critical" for page-worthy issues
severity_guidelines = {
"critical": "Model offline, safety violations, production outages",
"high": "Significant drift, performance degradation",
"medium": "Minor data quality issues, gradual drift",
"low": "Informational, trend observations"
}
```
**2. Implement Alert Throttling:**
```python theme={null}
client.create_alert(
name="Drift Detection",
threshold=0.1,
evaluation_window=300, # 5 minutes
cooldown_period=3600, # Don't re-alert for 1 hour
max_alerts_per_day=5 # Cap daily alerts
)
```
**3. Use Alert Grouping:**
```python theme={null}
# Group related alerts into single notification
client.configure_alert_grouping(
group_name="Feature Drift Alerts",
alerts=["drift_feature_1", "drift_feature_2", "drift_feature_3"],
send_as_digest=True,
digest_interval=1800 # 30 minutes
)
```
### Incident Response Runbooks
Include runbook links in alert payloads:
```python theme={null}
client.create_alert(
name="High Model Drift",
metadata={
"runbook_url": "https://wiki.company.com/ml/runbooks/drift-response",
"on_call_team": "ml-platform",
"escalation_policy": "ml-models-production",
"sla": "30 minutes to acknowledge, 2 hours to investigate"
}
)
```
## Security & Compliance
### Secure Credential Management
**Never hardcode credentials:**
```python theme={null}
import os
# Use environment variables
client.add_pagerduty_integration(
integration_key=os.environ['PAGERDUTY_INTEGRATION_KEY']
)
# Or use secret management systems
from secret_manager import get_secret
client.add_datadog_integration(
api_key=get_secret('datadog-api-key'),
app_key=get_secret('datadog-app-key')
)
```
### Alert Data Privacy
**PII Redaction in Alerts:**
```python theme={null}
client.configure_alert_privacy(
redact_pii=True,
pii_fields=["email", "ssn", "phone_number"],
redaction_string="[REDACTED]"
)
```
### Audit Logging
**Track alert delivery:**
```python theme={null}
# Query alert delivery audit log
audit_log = client.get_alert_audit_log(
start_time="2024-11-01",
end_time="2024-11-10",
include_payload=True
)
for entry in audit_log:
print(f"Alert: {entry.alert_name}")
print(f"Delivered to: {entry.channel}")
print(f"Status: {entry.status}")
print(f"Timestamp: {entry.timestamp}")
```
## Troubleshooting
### Common Issues
**Alerts Not Delivered:**
* Verify integration credentials are valid and not expired
* Check network connectivity from Fiddler to external platform
* Ensure webhook endpoints are reachable (not blocked by firewall)
* Validate alert thresholds are actually being triggered
**Duplicate Alerts:**
* Enable alert deduplication with appropriate time windows
* Check if multiple notification channels are configured
* Verify integration isn't configured twice
**Missing Alert Context:**
* Ensure `include_context=True` in alert configuration
* Check payload template includes necessary fields
* Verify external platform supports rich payloads (some SMS gateways don't)
## Integration Selector
Choose the right integration for your use case:
| Your Need | Recommended Integration | Why |
| -------------------------- | ------------------------- | ------------------------------------------ |
| On-call engineer paging | **PagerDuty** | Escalation policies, incident management |
| Infrastructure correlation | **Datadog** | Unified metrics, correlated dashboards |
| Team notifications | **Slack** *(Coming Soon)* | Channel-based, collaborative triage |
| Custom internal tools | **Generic Webhooks** | Flexible, integrate with any HTTP endpoint |
| Multi-tool strategy | **Datadog + PagerDuty** | Metrics + incidents in one workflow |
## Related Integrations
* [**Cloud Platforms**](/integrations/cloud-platforms-and-deployment/cloud-platforms) - Deploy Fiddler on AWS, Azure, GCP
* [**Data Platforms**](/integrations/data-platforms-and-pipelines/data-platforms) - Ingest data from Snowflake, Kafka
* [**ML Platforms**](/integrations/ml-platforms-and-tools/ml-platforms) - Integrate with Databricks, MLflow
* [**Agentic AI**](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) - Monitor LangGraph and Strands Agents
***
# Custom Metrics for Agentic Applications
Source: https://docs.fiddler.ai/observability/agentic/custom-metrics
Define custom metrics for your agentic and GenAI applications using FQL and span attributes to track business KPIs, quality scores, and operational signals beyond built-in metrics.
### Overview
Custom metrics let you define measurements that align precisely with your agentic application's requirements. Whether tracking business KPIs, aggregating quality scores from enrichments, or computing cost and latency signals, custom metrics let you tailor observability to your specific needs. Once defined, they are available in charts and dashboards.
Custom metrics for agentic applications are defined using [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language) and reference **span attributes** captured from your application's OpenTelemetry traces. This differs from ML custom metrics, which reference model schema columns.
Custom metrics can be **organization-level** (visible across all projects) or **project-scoped** (visible only within a specific project). See [Metric Visibility](#metric-visibility-organization-vs-project) below.
### Metric Visibility: Organization vs Project
When creating a custom metric, you choose whether it is **organization-level** (global) or **project-scoped**. This determines who can see it and who can delete it, and cannot be changed after creation.
#### Organization-level metrics
Created without selecting a project. Visible across all projects in the organization. The metric name is reserved org-wide — no other metric in the organization can share the same name.
| Action | Roles |
| ------ | --------------------- |
| Create | Org Admin, Org Member |
| View | Org Admin, Org Member |
| Delete | Org Admin |
#### Project-scoped metrics
Created with a specific project selected. Visible only to users who have access to that project. The same metric name can be reused in a different project as long as the projects don't overlap.
| Action | Roles |
| ------ | --------------------------------------------------------------- |
| Create | Project Admin, Project Writer (on that project) |
| View | Project Admin, Project Writer, Project Viewer (on that project) |
| Delete | Project Admin (on that project) |
Metric visibility is **immutable** — an organization-level metric cannot be changed to project-scoped or vice versa after creation.
### The `attribute()` Function
The `attribute()` function is the GenAI-specific FQL primitive for referencing span data. It replaces the column references used in ML custom metrics.
#### Syntax
```
attribute('name', type='user', scope='span')
attribute('name', type='user', scope='span', value='category_value')
attribute('name', type='system', scope='span')
```
| Parameter | Required | Description |
| --------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | Yes | The attribute name as it appears in your trace data (e.g., `gen_ai.usage.input_tokens`) |
| `scope` | Yes | The attribute scope. Currently only `'span'` is supported. |
| `type` | Yes | The attribute source: `'user'` for attributes your application sets, or `'system'` for attributes emitted by OpenTelemetry instrumentation (e.g., `gen_ai.usage.input_tokens`). |
| `value` | No | Filters to spans where the attribute equals this string; returns the value if matched, `null` otherwise. Used for categorical attributes (e.g., `attribute('status', value='error')`). When set, the attribute is always treated as a string. |
#### Type inference
Fiddler infers the attribute type from context — you do not need to declare it explicitly:
* When used inside a numeric aggregate like `sum()` or `average()`, the attribute is treated as a number.
* When used with string functions like `length()` or `match()`, the attribute is treated as a string.
* When the `value` keyword is provided, the attribute is always treated as a string.
### Adding a Custom Metric
1. Navigate to the **Custom Metrics** section in the Fiddler UI.
2. Click **Add Custom Metric**.
3. Enter a **Metric name**, an optional **Description**, and the **Metric definition**.
4. Optionally select a **project** to scope the metric to. If no project is selected, the metric is created as an **organization-level** metric visible across all projects.
5. Click **Create Metric**.
![Custom metric creation form in the Fiddler UI]()
### Using Custom Metrics in Charts
After saving a custom metric, you can use it in chart definitions:
1. Open or create a chart in the Fiddler UI.
2. Set **Metric Type** to **Custom Metric**.
3. Select your custom metric from the list.
### Deleting Custom Metrics
To delete a custom metric, click the trash icon next to the metric in the **Custom Metrics** tab. Deletion runs as a background job that automatically:
* Removes the metric from any charts that reference it
* Deletes charts that have no remaining metrics after cleanup
* Updates dashboard layouts to remove deleted charts
* Deletes dashboards that become empty as a result
### Examples
Custom metrics must return either an aggregate (produced by aggregate functions) or a combination of aggregates. See the [FQL reference](/observability/platform/fiddler-query-language) for the full list of supported operators and functions.
#### Average input token usage
Track the mean number of input tokens consumed per span to monitor LLM cost drivers over time.
```
average(attribute('gen_ai.usage.input_tokens', type='system', scope='span'))
```
→ Returns a `Number` (e.g., `312.4`)
#### Premium user ratio
Measure the fraction of spans attributed to premium-tier users by filtering on a categorical attribute.
```
count(attribute('tier', type='user', scope='span', value='premium')) / count(attribute('tier', type='user', scope='span'))
```
→ Returns a `Number` between 0 and 1 (e.g., `0.34`)
#### P95 response latency
Use the `quantile()` function to track the 95th-percentile response time. This is more robust than averages for catching tail latency issues.
```
quantile(attribute('response_time_ms', type='user', scope='span'), level=0.95)
```
→ Returns a `Number` in the same unit as the attribute (e.g., `1420.0` ms)
#### Conditional cost (weighted by outcome)
Apply different weights to successful and failed spans to surface the true cost impact of errors. The `if(condition, true_value, false_value)` function evaluates the condition per span and returns one of two values.
```
average(if(attribute('status', type='user', scope='span') == 'error', attribute('cost', type='user', scope='span') * 2, attribute('cost', type='user', scope='span')))
```
→ Returns a `Number` (e.g., `0.0042`)
#### Latency range
Track the spread of response times across spans in a time window. A widening range can signal instability or the emergence of slow outlier requests.
```
max(attribute('response_time_ms', type='user', scope='span')) - min(attribute('response_time_ms', type='user', scope='span'))
```
→ Returns a `Number` in the same unit as the attribute (e.g., `3850.0` ms)
#### Minimum token usage
Find the smallest input token count across all spans in a window. Useful for detecting unusually short requests that may indicate truncated inputs or misconfigured clients.
```
min(attribute('gen_ai.usage.input_tokens', type='system', scope='span'))
```
→ Returns a `Number` (e.g., `12`)
#### Null-safe cost with markup
Apply a price adjustment to spans that have a cost attribute, while preserving `null` for spans where cost data is absent — avoiding accidental zero-inflation of the average.
```
average(if(is_null(attribute('cost', type='user', scope='span')), null, attribute('cost', type='user', scope='span') * 1.15))
```
→ Returns a `Number` representing the average marked-up cost, excluding null-cost spans (e.g., `0.0048`)
Use `is_null()` to test whether an attribute is absent. The `null` keyword is for use as a **return value** in expressions (e.g., `if(condition, value, null)`) to propagate missing data explicitly.
### Related Resources
* [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language) — full reference for operators, aggregate functions, and expression syntax
* [Custom Metrics for ML Models](/observability/platform/custom-metrics) — custom metrics using model schema columns instead of span attributes
* [Agentic Observability](/observability/agentic/index) — overview of dashboards, metrics, and integrations for agentic applications
* [Custom Metrics glossary entry](/glossary/custom-metrics)
# Agentic Observability
Source: https://docs.fiddler.ai/observability/agentic/index
Monitor AI agents and multi-step workflows with specialized dashboards, metrics, and trace visualization
Agentic observability provides specialized observability for AI agents and multi-step workflows through dedicated dashboards, metrics, and trace visualization. Unlike traditional monitoring, which tracks single-shot inferences, agentic observability captures the complete lifecycle of autonomous-agent behavior—from initial reasoning through tool execution and final response.
## Dashboards & Visualization
Agentic observability uses a dedicated UI with Projects → Applications (instead of the legacy Projects → Models structure) designed specifically for observing agent workflows.
### Agentic Dashboards
Access pre-built dashboards optimized for agentic workflows:
* **Agent Performance Overview** - Monitor success rates, latency, and throughput across all agents
* **Workflow Execution Traces** - Visualize complete multi-step reasoning chains from start to finish
* **Tool Usage Analytics** - Track which external tools and APIs your agents are calling
* **Error & Exception Tracking** - Identify where agent workflows fail and why
### Trace Visualization
Every agent interaction is captured as a hierarchical trace showing:
* **Agent Steps** - Each decision point in the agent's reasoning process
* **LLM Calls** - All language model interactions with inputs, outputs, and metadata
* **Tool Invocations** - External function calls, API requests, and data retrievals
* **Timing Information** - Duration of each step to identify performance bottlenecks
* **Parent-Child Relationships** - How multi-agent systems coordinate and delegate tasks
**Viewing Traces:**
1. Navigate to your Application in the Fiddler UI
2. Select a conversation or workflow from the list
3. Click on any trace to expand the full execution tree
4. Drill down into individual spans to see inputs, outputs, and metadata
### Custom Dashboards
Create custom dashboards to monitor specific agent behaviors:
* Combine multiple charts to track KPIs relevant to your use case
* Filter by agent type, user segments, or time periods
* Share dashboards with team members
* Set up alerts based on dashboard metrics
## Metrics & Analytics
Agentic observability provides specialized metrics that go beyond traditional model monitoring:
### Agent-Specific Metrics
* **Agent Success Rate** - Percentage of workflows that complete successfully
* **Tool Call Distribution** - Which tools agents use most frequently
* **Reasoning Chain Length** - Average number of steps per workflow
* **Agent Handoffs** - How often agents delegate to other agents
* **Retry & Recovery Rate** - How often agents recover from errors
### Performance Metrics
* **End-to-End Latency** - Total time from user request to final response
* **Per-Step Latency** - Duration of individual reasoning steps, LLM calls, and tool invocations
* **Token Usage** - Track LLM consumption across all agent interactions
* **API Call Volume** - Monitor external tool and API usage
### Quality Metrics
* **Response Accuracy** - Validate agent outputs against expected results (requires ground truth)
* **Hallucination Detection** - Identify when agents generate unsupported claims
* **Safety & Guardrails** - Track safety violations and guardrail activations
* **User Satisfaction** - Capture feedback signals from end users
### Analyzing Metrics
All metrics are available in:
* **Real-time Dashboards** - Monitor live agent performance
* **Historical Trends** - Analyze patterns over days, weeks, or months
* **Comparative Analysis** - Compare different agent versions or configurations
* **Custom Queries** - Use Fiddler Query Language (FQL) for advanced analysis
## Integration Options
Agentic observability ingests OpenTelemetry spans and traces from your agent applications. Choose the integration method that fits your framework:
### LangGraph Applications
**Best for:** Applications built with LangGraph or LangChain
The Fiddler LangGraph SDK provides automatic instrumentation with zero code changes required.
```bash theme={null}
pip install fiddler-langgraph
```
→ [**LangGraph SDK Quick Start**](/developers/quick-starts/langgraph-sdk-quick-start) - Get started in 10 minutes
→ [**LangGraph SDK Documentation**](/sdk-api/langgraph/fiddler-client) - Complete integration guide
### Strands Agent Framework
**Best for:** Applications built with Strands Agents
The Fiddler Strands SDK integrates directly with the Strands framework for seamless monitoring.
```bash theme={null}
pip install fiddler-strands
```
→ [**Strands Agent SDK Quick Start**](/developers/quick-starts/strands-agent-quick-start) - Get started in 15 minutes
→ [**Strands SDK Documentation**](/sdk-api/strands/strands-agent-instrumentor) - Complete integration guide
### Custom Instrumentation (OpenTelemetry)
**Best for:** Custom agent frameworks or non-Python applications
Use OpenTelemetry directly for maximum flexibility and control.
```bash theme={null}
pip install opentelemetry-api opentelemetry-sdk
```
→ [**OpenTelemetry Quick Start**](/developers/quick-starts/opentelemetry-quick-start) - Get started in 20 minutes
→ [**OpenTelemetry Documentation**](/integrations/agentic-ai/opentelemetry-integration) - Complete integration guide
### Integration Comparison
| Feature | LangGraph SDK | Strands SDK | OpenTelemetry |
| ----------------------------- | -------------------- | -------------------- | -------------------------- |
| **Automatic Instrumentation** | ✅ Zero code changes | ✅ Native integration | ❌ Manual setup |
| **Framework Support** | LangGraph, LangChain | Strands Agents | Any framework |
| **Language Support** | Python | Python | Python, Java, Go, JS, etc. |
| **Setup Time** | \~10 minutes | \~10 minutes | \~15 minutes |
| **Customization** | Medium | Medium | High |
## Next Steps
* **Getting Started:** Learn the concepts behind agentic observability in our [Agentic Observability Getting Started Guide](/getting-started/agentic-monitoring)
* **Choose Integration:** Pick your framework and follow the quick start guide above
* **Explore Features:** Set up dashboards, configure alerts, and analyze agent performance
* **Advanced Topics:** Learn about [custom metrics](/observability/platform/custom-metrics), [alerts](/observability/platform/alerts-platform), and [segmentation](/observability/platform/segments)
# Explorer
Source: https://docs.fiddler.ai/observability/agentic/trace-explorer
Explore, filter, and search every span ingested into your GenAI application with the Explorer DataGrid.
Explorer is in **Public Preview**. See [Feature Maturity Definitions](/reference/feature-maturity-definitions#public-preview) for what this means.
## Overview
The Explorer is a spans-first exploration surface for GenAI applications. It gives you a paginated, filterable, searchable view of every span ingested into a single application — useful for debugging individual requests, investigating evaluator failures, drilling into latency outliers, and validating instrumentation.
Open the **Explorer** tab on any GenAI Application Details page.
## Concepts
| Term | Meaning |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Span** | A single unit of work in your application — an LLM call, a tool invocation, an agent step, or a chain step. The Explorer grid is one row per span. |
| **Trace** | A collection of spans that share a `trace_id` and represent one end-to-end request through your application. |
| **Session** | A higher-level grouping (typically a multi-turn conversation) identified by a `session_id`. A session may contain multiple traces. |
| **Time range** | Mandatory. The Explorer always filters by a time window. The UI defaults to the last 7 calendar days (ending at end-of-day UTC). The time range is enforced server-side to keep queries fast against the underlying ClickHouse partitioning. |
***
## Using the UI
### Where it lives
Open any GenAI Application Details page, then select the **Explorer** tab.
### The DataGrid
Each row in the grid is a span. The grid is server-side paginated and sorted, with rich column controls.
#### Columns visible by default
| Column | Notes |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Type | Span type — typically one of `llm`, `tool`, `agent`, `chain` |
| Name | Span operation name |
| Input | LLM or tool input content. Query via the search bar with the **Input** scope selector (see [Searching in the UI](#searching-in-the-ui)). |
| Output | LLM or tool output content. Query via the search bar with the **Output** scope selector (see [Searching in the UI](#searching-in-the-ui)). |
| Timestamp | When the span started |
| Duration | Span duration (displayed in human-friendly units) |
| Status | OpenTelemetry status code (`Ok`, `Error`, `Unset`) |
| **Actions** | Pinned to the right of the grid; always visible. Contains the **View Trace** icon. |
#### Columns hidden by default
These columns are available via the column picker but are not shown out of the box:
* Session ID
* Trace ID
* Span ID
* Agent
* Agent ID
#### Dynamic columns
Explorer adds columns automatically based on your application's schema:
* **Evaluator output columns** — one column per output of every evaluator rule configured for the application.
* **User-defined span attribute columns** — one column per attribute you've instrumented (for example, `user_id`, `model_version`, `environment`).
* **Token count columns** — `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.total_tokens`.
See [Span and Resource Attributes](/integrations/agentic-ai/attributes) for the full attribute catalog and naming conventions.
### Filtering in the UI
Open the filter panel from any filterable column header, or from the toolbar. Filters are applied server-side. By default, filter rules are combined with **AND**; the MUI filter panel also lets you switch to **OR** within a single column.
* **ID columns** (`Session ID`, `Trace ID`, `Span ID`) — `equals` only in the UI (enable the column via the column picker first; these are hidden by default).
* **String columns** (Name, Agent, span attributes) — `is`, `is not`, `is any of` (autocomplete) or `equals` / `does not equal` (free text), depending on whether the field has a known value set.
* **Numeric columns** (Duration, token counts, numeric attributes) — `=`, `!=`, `>`, `>=`, `<`, `<=`.
* **Enum columns** (Type, Status) — single- or multi-select.
* **Boolean evaluator outputs** — `is` or `is not` with values `true` or `false`.
### Searching in the UI
Use the search bar at the top of the grid to run a case-insensitive substring match against prompt and response content captured on your spans.
| Setting | Detail |
| ---------- | ----------------------------------------------------------------------------------------------- |
| **Length** | 3 to 64 characters |
| **Match** | Case-insensitive substring |
| **Index** | Backed by a dedicated content table with n-gram and token bloom-filter indexes for fast lookups |
The **scope selector** next to the search bar restricts which content fields are searched:
| Scope | Searches |
| ----------------- | ---------------------------------------------- |
| **All** (default) | LLM input, LLM output, tool input, tool output |
| **Input** | LLM input + tool input |
| **Output** | LLM output + tool output |
Search runs in addition to any structured filters — both must match for a span to appear in results.
### Sorting in the UI
The grid is sorted by **Timestamp** (most recent first) by default. Click the **Timestamp** or **Duration** column header to change the sort. Other columns are not sortable in the UI in this release.
### Opening the trace drawer
To inspect the full trace context for a span, click the **View Trace** (eye) icon in the rightmost **Actions** column. This opens a side drawer that loads every span in the row's session and renders them as a tree.
* The drawer is **session-scoped**, not trace-scoped — if a session contains multiple traces, all of them are visible in the drawer.
* The View Trace icon is **disabled** for spans without a `session_id`, with the tooltip *"No trace found"*.
***
## Related Resources
* [Custom Metrics for Agentic Applications](/observability/agentic/custom-metrics) — define custom metrics over the same span data using FQL
* [Span and Resource Attributes](/integrations/agentic-ai/attributes) — full attribute catalog and naming conventions
* [Agentic Observability](/observability/agentic/index) — overview of dashboards, metrics, and integrations
* [OpenTelemetry Trace Export](/integrations/agentic-ai/otel-trace-export) — export pre-existing OpenTelemetry traces to Fiddler from your own storage or pipeline
***
## Limitations & Roadmap (Public Preview)
The following are not available in this release and are tracked for future work:
* **Session-attribute filters.** You can filter by `session_id`, but filters over session-level attributes are not yet supported.
* **Multi-pattern structural filters.** Constructs such as "trace contains tool X *then* LLM Y" or "trace where root has positive feedback" — sometimes written with `->` (temporal) or `=>` (parent / child) operators in other tools — are not yet supported.
* **Saved filter sets.** Filters live for the current session; they cannot be named, saved, or shared yet.
* **CSV / notebook export.** Bulk export of filtered results is not yet available.
We're prioritizing these based on customer feedback during public preview — please reach out to your Customer Success Manager or file a ticket if any of the above are blocking your workflow.
# Events Table in RCA
Source: https://docs.fiddler.ai/observability/analytics/data-table-in-rca
Learn how to use Fiddler's root cause analysis features to quickly hone in on the data issues adversely impacting your ML models and LLM applications.
Allow visualizing events corresponding to a certain bin in a monitoring chart. Note that this view is to provide example rows used for the computation. The maximum number of rows that can viewed is 1000.
### Analyzing a Sample of Events
* Navigate to the `Charts` tab in your Fiddler AI instance
* Click on the `Add Chart` button on the top right
* In the modal, select a project
* Select **Monitoring**
* Create a Monitoring chart and click on a time range
* This will display the RCA (Root Cause Analysis) tab
* In RCA, select the `Events Table` tab
### Support
This visualization is supported for any model and data type.
### Represented data
The displayed events are production events coming from the selected model and bin, and segment if it was selected in the monitoring chart.
![Monitoring Chart Configuration]()
![Event Table Example]()
### Available Controls
* **Column selection**: On the top right side of the table, select the columns to be displayed. By default all non-vector columns are displayed.
![Event Table Column Selection]()
* **Vector columns**: By default the vector columns are not fetched for latency reasons. Toggle on if vectors need to be fetched.
![Event Table Vectors displayed]()
* **Download**: Download the sample events to `CSV` or `PARQUET` format.
![Event Table Vectors Download]()
# Feature Analytics
Source: https://docs.fiddler.ai/observability/analytics/feature-analytics-chart
Dive into our guide on creating feature analytics charts and visualizations for important features in your ML Models and LLM applications.
### Creating a Feature Analytics Chart
To create a Feature Analytics chart, follow these steps:
* Navigate to the `Charts` tab in your Fiddler AI instance
* Click on the `Add Chart` button on the top right
* In the modal, select a project
* Select **Feature Analytics**
![Add Chart modal dialog]()
### Support
Feature analytics is supported for any model task type, but does not support time stamps or columns of type `string` or `vector`.
### Available Right-Side Controls
| Parameter | Value |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model | List of models in the project |
| Version | List of versions for the selected model |
| Environment | `Production` or `Pre-Production` |
| Dataset | Displayed only if `Pre-Production` is selected. List of pre-production env uploaded for the model version. |
| Visual | List of possible visualizations, Feature Distribution or Feature Correlation. |
| Segment | \- Selecting a saved segment\
- Defining an applied (on the fly) segment. This applied segment isn’t saved (unless specifically required by the user) and is applied for analysis purposes.\
|
| Feature | \- Feature Distribution: Select a single column of a supported data type to view the distribution of its values\
- Feature Correlation: Select two columns to visualize their correlation\
- Correlation Matrix: Select up to eight columns to visualize their pairwise correlations\
|
### Available In-Chart Controls
| Control | Value |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Time range selection | Selecting start time and end time or time label for production data. Default is last 30 days |
| Display | Select how to display the chart, Histogram or Kernel Density Estimate, depending on the data type of the feature selected. |
### Displays for Feature Distribution
#### Kernel Density Estimate
For numeric column types, visualize feature distribution as a Kernel Density Estimate chart. ![Feature distribution chart configured with Kernel Density Estimate rendering]()
#### Histogram
Categorical and boolean value may only be displayed as Histograms, but other column types may also be displayed as such. ![Feature distribution chart configured with histogram rendering]()
### Displays for Feature Correlation
#### Line Plot
![Line plot example of Feature Correlation chart]()
#### Scatter Plot
![Scatter plot example of Feature Correlation chart]()
#### Stacked Bar Plot
![Stacked bar example of Feature Correlation chart]()
#### Box Plot
![Box plot example of Feature Correlation chart]()
### Correlation Matrix Interactions
Clicking a cell in the correlation matrix opens the Feature Correlation chart, enabling a more detailed analysis of the relationship between the selected features. ![Correlation matrix interactions]()
# Analytics
Source: https://docs.fiddler.ai/observability/analytics/index
Explore our UI guide to Fiddler analytics. Learn about interfaces for various analytics charts and root cause analysis to better understand your models.
## Analytics Charts
There are three supported analytics chart types:
1. ***Performance Analytics*** — Visualize various performance metrics tailored to different task types, such as confusion matrices, prediction scatterplots, and more.
2. ***Feature Analytics*** — Analyze the behavior and relationships between features, helping to identify patterns and insights within the data, using feature distribution, feature correlation, and correlation matrix charts.
3. ***Metric Card*** — Display a single numeric representation of a metric, offering a concise overview of performance, data quality, data integrity, or custom metrics on a card.
## Root Cause Analysis
The Root Cause Analysis (RCA) experience consists of four parts, all based on the FQL segment and time range provided in the monitoring chart:
1. ***Events*** — Browse a sample of 1,000 events.
2. ***Data Drift*** — View a breakdown of drift for your features, along with prediction drift impact.
3. ***Data Integrity*** — Review a summary of data integrity violations, including counts across range, type, and missing value issues.
4. ***Analyze*** — View performance and feature analytics charts.
![Root Cause Analyze Experience]()
# Metric Card
Source: https://docs.fiddler.ai/observability/analytics/metric-card
Dive into our guide for metric card creation. Follow step-by-step instructions to create metric cards, use custom metrics, right-side controls, and save charts.
### Creating a Metric Card Chart
To create a Metric Card, follow these steps:
* Navigate to the `Charts` tab in your Fiddler AI instance
* Click on the `Add Chart` button on the top right
* In the modal, select a project
* Select **Metric Card**
### Support
Metric card is supported for any model task type and for both production and pre-production data. Metric card allows displaying of Custom Metric, Data Drift, Data Integrity, Performance, Traffic, and Statistic metrics.
### Available Right-Side Controls
| Parameter | Value |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Model | List of models in the project |
| Version | List of versions for the selected model |
| Environment | `Production` or `Pre-Production` |
| Dataset | Displayed only if `Pre-Production` is selected. List of pre-production env uploaded for the model version. |
| Metric | Selecting a metric across Custom Metrics, Data Drift, Data Integrity, Performance, Traffic, and Statistic to display as an aggregated number. Select up to 4 metrics to display on the chart. |
| Segment | \- Selecting a saved segment\
- Defining an applied (on the fly) segment. This applied segment isn’t saved (unless specifically required by the user) and is applied for analysis purposes.\
|
### Available In-Chart Controls
| Control | Model Task | Value |
| ------------------------ | --------------------- | -------------------------------------------------------------------------------------------- |
| Time range selection | All | Selecting start time and end time or time label for production data. Default is last 30 days |
| Positive class threshold | Binary classification | For performance metrics, select a threshold to be applied for computation. Default is 0.5 |
| Top-K | Ranking | For performance metrics, select a top-k value to be applied for computation. Default is 20. |
### Saving the Chart
Once you're satisfied with your metric card Chart, you can save it which allows it to be added to Dashboards and viewed individually from the Charts page.
![Root Cause Analyze Experience]()
# Performance Charts Creation
Source: https://docs.fiddler.ai/observability/analytics/performance-charts-creation
Discover our guide to creating performance charts. Learn key steps to select charts, use right-side and in-chart controls, and save your customized charts.
### Creating a Performance Chart
To create a Performance chart, follow these steps:
* Navigate to the `Charts` tab in your Fiddler AI instance
* Click on the `Add Chart` button on the top right
* In the modal, Select the project that has a model with Custom features
* Select **Performance Analytics**
![Add Chart modal dialog]()
### Available Performance charts
| Model task | Available Chart(s) |
| -------------------------- | ------------------------------------------------------------------------------------------ |
| Binary classification | \- Confusion Matrix\
- ROC\
- Precision Recall\
- Calibration Plot Charts\
|
| Multi-class Classification | - Confusion Matrix |
| Regression | \- Prediction Scatterplot\
- Error Distribution\
|
| Ranking / LLM / Not Set | *Not available* |
### Available right side controls
| Parameter | Value |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model | List of models in the project |
| Version | List of versions for the selected model |
| Environment | `Production` or `Pre-Production` |
| Dataset | Displayed only if `Pre-Production` is selected. List of pre-production env uploaded for the model version. |
| Visual | List of possible [performance visualization](/observability/analytics/performance-charts-visualization) depending on the model task. |
| Segment | \- Selecting a saved segment\
- Defining an applied (on the fly) segment. This applied segment isn’t saved (unless specifically required by the user) and is applied for analysis purposes.\
|
| Max Sample size | \Integer representing the maximum number of rows used for computing the chart, up to 500,000. If the data selected has less rows, we will use all the available rows with non null target and output(s).\
Fiddler select the \n\ first number of rows from the selected slice.\
\
\Note: Clickhouse is configured using multiple shards, which means slightly different results can be observed if data is only selected on a specific shard (usually when little observation are queried).\\
|
### Available in-chart controls
| Control | Model Task | Value |
| ------------------------ | -------------------------- | -------------------------------------------------------------------------------------------------- |
| Time range selection | All | Selecting start time and end time or time label for production data. Default to last 30 days |
| Positive class threshold | Binary classification | Selecting threshold applied for computation / visualization. Default to 0.5 |
| Displayed labels | Multi-class Classification | Selecting the labels to display on the confusion matrix (up to 12). Default to the 12 first labels |
### Saving the Chart
Once you're satisfied with your visualization, you can save the chart. This chart can then be added to a Dashboard. This allows you to revisit the Performance visualization at any time easily either directly going to the Chart or to the dashboard.
# Performance Charts Visualization
Source: https://docs.fiddler.ai/observability/analytics/performance-charts-visualization
Dive into our guide on performance charts and visualizations used to monitor the behavior and performance of your ML models.
### Performance Charts Visualizations for ML and LLM Models
List of possible performance visualization depending on the model task. To see how to create a Performance chart, visit [this page](/observability/analytics/performance-charts-creation).
### Binary Classification
#### Confusion Matrix
A 2x2 table that shows how many predicted and actual values exist for positive and negative classes. Also referred to as an error matrix. The percentage is computed per row.
![]()
#### Receiver Operating Characteristic (ROC) Curve
A graph showing the performance of a classification model at different classification thresholds. Plots the true positive rate (TPR), also known as recall, against the false positive rate (FPR).
![]()
#### Precision-Recall Curve
A graph that plots the precision against the recall for different classification thresholds.
![]()
#### Calibration Plot
A graph that tell us how well the model is calibrated. The plot is obtained by dividing the predictions into 10 quantile buckets (0-10th percentile, 10-20th percentile, etc.). The average predicted probability is plotted against the true observed probability for that set of points.
![]()
### Multi-class Classification
#### Confusion Matrix
A table that shows how many predicted and actual values exist for different classes. Also referred to as an error matrix. The percentage is computed per row (using all classes). In the full view, up to 15 classes can be displayed. In the chart creation mode, up to 12 classes can be displayed. The displayed labels can be controlled in the chart.
![]()
### Regression
#### Prediction Scatterplot
Plots the predicted values against the actual values. The more closely the plot hugs the `y=x line`, the better the fit of the model.
![]()
#### Error Distribution
A histogram showing the distribution of errors (differences between model predictions and actuals). The closer to 0 the errors are, the better the fit of the model.
![]()
# Dashboard Interactions
Source: https://docs.fiddler.ai/observability/dashboards/dashboard-interactions
Explore our guide to dashboard interactions. Learn to remove, edit, zoom into charts, switch between bar and line views, and undo toolbar changes.
### Remove a Chart
If you want to remove a chart from your dashboard, simply click on the "X" located at the top right of the chart. This will remove the chart from the dashboard, but it will still be available in the saved charts list for future use. If you change your mind and decide to add the chart back to the dashboard, you can simply find it in the saved charts list and add it back to the dashboard at any time.
![Example Auto Insurance Dashboard with drift and accuracy charts.]()
### Edit a Saved Chart
To edit a saved chart, simply click on the chart title within your dashboard. This will open the chart studio in a new tab, where you can make any necessary changes. Once you have made your changes, be sure to select the `Default` time range and then use the Dashboard refresh button to see your updated chart.
![Displays chart title selected for editing mode.]()
### Zoom Into Details
To zoom into a chart within your dashboard, you have two different utilities at your disposal. The first one is located on the top right of the chart component, in the toolbar. After clicking on the **zoom icon**, you can drag your cursor over the data points you wish to zoom into.
![Example of a region of an accuracy chart selected for zoom.]()
You can also use the **horizontal zoom bar** located at the base of the chart to zoom in. Once you've identified the time range you want to focus on, you can use the zoom bar to drag the range across time. For instance, if you want to analyze your data weekly over the past six months, you can use the toolbar or horizontal zoom bar to zoom in on the desired time range and then click and drag the selected range using the base horizontal zoom bar.
![Examples of using the horizontal time range control underneath charts to narrow the date range.]()
### Bar & Line Charts
You can switch between visualizing your chart as a line or a bar chart using the toolbar icons. Click on the line chart icon on the top right of the chart to switch to the line chart view. Likewise, select the bar chart icon in the toolbar to switch to the bar chart view.
> Note that these views are only temporary and any settings you specify using the toolbar will not be saved to the dashboard.
![Highlight of the line/bar chart type toggle button.]()
### Undo Chart Toolbar Changes
You can easily restore the changes you applied to your chart using the chart toolbar options, including zoom, switch to line chart, and switch to bar chart. The restore option, which is the last icon in the toolbar, allows you to undo any changes you made and return to the original chart configuration.
![Highlight of the undo button used to restore the chart to its original configuration.]()
# Dashboard Utilities
Source: https://docs.fiddler.ai/observability/dashboards/dashboard-utilities
Discover dashboard utilities on Fiddler’s platform. Learn to rename, save, share, copy links, or delete dashboards to manage your collection effortlessly.
## Dashboard Name
To rename your dashboard, simply click on the "Untitled Dashboard" title on the top-left corner of the dashboard studio. This will allow you to give your dashboard a more descriptive name that reflects its purpose and contents, making it easier to find and manage among your other dashboards.
![]()
Once you've clicked on the "Untitled Dashboard" title to rename your dashboard, simply type in the desired name and hit "Enter" on your keyboard to save the new name.
![]()
![]()
If you change your mind and want to discard the changes, simply click anywhere on the page outside of the name box. This will cancel the renaming process and leave the dashboard name as it was before.
## Save, Copy Link, and Delete
You can easily manage your dashboard by using the control panel located on the top left of the dashboard studio. This panel allows you to save your dashboard, copy a link to it, or delete it entirely. By using these controls, you can easily share your dashboard with others or remove it from your collection if it is no longer needed.
![]()
## Save
It's important to note that dashboards are not automatically saved, so you'll need to manually save your dashboard in order to lock in the current charts and filters. Once you've made the desired changes to your dashboard, simply click the "Save" button to save your progress. This will also enable you to share or delete your dashboard as needed. By saving your dashboard frequently, you can ensure that you never lose important information or data visualizations.
![]()
## Copy Link
If you want to share your dashboard with other users on Fiddler, the first step is to ensure that they have access to the project that the dashboard belongs to. Once you've confirmed that they have access, you can easily share the dashboard by copying the dashboard link and sending it to them. This makes it simple to collaborate and share insights with others who are working on the same project or who have an interest in your findings. Note that you can't share a dashboard link until you've saved the dashboard.
![]()
## Delete
To delete a dashboard, click the overflow button next to the copy link icon. NOTE: Once a dashboard has been deleted it cannot be recovered.
![]()
# Creating Dashboards
Source: https://docs.fiddler.ai/observability/dashboards/dashboards-creating
Navigate our guide for the Dashboard page. Learn how to select new or existing dashboards and access them to monitor performance, drift, integrity, and traffic.
### Creating Dashboards
To begin using our dashboard feature, navigate to the dashboard page by clicking on "Dashboards" from the top-level navigation bar. On the Dashboards page, you can choose to either select from previously created dashboards or create a new one. This simple process allows you to quickly access your dashboards and begin monitoring your models' performance, data drift, data integrity, and traffic.
![]()
When creating a new dashboard, it's important to note that each dashboard is tied to a specific project space. This means that only models and charts associated with that project can be added to the dashboard. To ensure you're working within the correct project space, select the desired project space before entering the dashboard editor page, then click "Continue." This will ensure that you can add relevant charts and models to your dashboard.
![]()
#### Auto-Generated Dashboards
Fiddler will automatically generate model monitoring dashboards for all models registered to the platform. Depending on the task type, these dashboards will include charts spanning Performance, Traffic, Drift, and Data Integrity metrics.
![Auto-generated model monitoring dashboard]()
Auto-generated model monitoring dashboard
Auto-generated dashboard are automatically set as the default dashboards for each model, and can be accessed via the Insights button on the Homepage or Model Schema pages, or alternatively all dashboards are always accessible in the Dashboards list page. Default dashboards and their charts can easily be modified to display the desired set of [monitoring](/observability/platform/monitoring-charts-platform) and [embedding](/observability/llm/embedding-visualization-with-umap) visualizations to meet any use case.
### Add Monitoring Charts
Once you’ve created a dashboard, you can add previously saved monitoring charts that display these metrics over time, making it easy to track changes and identify patterns.
![]()
To create a new monitoring chart for your dashboard, simply select "New Monitoring Chart" from the "Add" dropdown menu. For more information on creating and customizing monitoring charts, check out our [Monitoring Charts UI Guide](/observability/platform/monitoring-charts-platform).
If you'd like to add an existing chart to your dashboard, select "Saved Charts" to display a full list of monitoring charts that are available in your project space. This makes it easy to quickly access and add the charts you need to your dashboard for monitoring and reporting purposes.
![]()
To further customize your dashboard, you can select the saved monitoring charts of interest by clicking on their respective cards. For instance, you might choose to add charts for Accuracy, Drift, Traffic, and Range Violation to your dashboard for a more comprehensive model overview. By adding these charts to your dashboard, you can quickly access important metrics and visualize your model's performance over time, enabling you to identify trends and patterns that might require further investigation.
### Dashboard Filters
There are three main filters that can be applied to all the charts within dashboards, these include date range, time zone, and bin size.
![]()
#### Date Range
When the `Default` time range is selected, the data range, time zone, and bin size that each monitoring chart was originally saved with will be applied. This enables you to create a dashboard where each chart shows a unique filter set to highlight what matters to each team. Updating the date range will unlock the time zone and bin size filters. You can select from a number of predefined ranges or choose `Custom` to select a start and end date-time.
![]()
#### Bin Size
Bin size controls the frequency at which data is displayed on your monitoring charts. You can select from the following bin sizes: `Hour`, `Day`, `Week`, or `Month`.
> 📘 Note: Hour bin sizes are not supported for time ranges above 90 days.
>
> For example, if we select the `6M` data range, we see that the `Hourly` bin selection is disabled. This is disabled to avoid long computation and dashboard loading times.
![]()
#### Saved Model Updates
If you recently created or updated a saved chart and are not seeing the changes either on the dashboard itself or the Saved Charts list, click the refresh button on the main dashboard studio or within the saved charts list to reflect updates.
![]()
### Model Comparison
With our dashboard feature, you can also compare multiple models side-by-side, making it easy to see which models are performing the best and which may require additional attention. To create model-to-model comparison dashboards, ensure the models you wish to compare belong to the same project. Create the desired charts for each model and then add them to a single dashboard. By creating a single dashboard that tracks the health of all of your models, you can save time and simplify your AI monitoring efforts. With these dashboards, you can easily share insights with your team, management, or stakeholders, and ensure that everyone is up-to-date on your AI performance.
![]()
Refer to our [Dashboard Utilities ](/observability/dashboards/dashboard-utilities)and [Dashboard Interactions](/observability/dashboards/dashboard-interactions) pages for more info on dashboard usage.
# Dashboards
Source: https://docs.fiddler.ai/observability/dashboards/index
Explore our guide to Fiddler’s dashboards for centralized monitoring. Discover key features like filters, utilities, default dashboards, and performance.
### Overview
With Fiddler, you can create comprehensive dashboards that bring together all of your monitoring data in one place. This includes monitoring charts for data drift, traffic, data integrity, and performance metrics. Adding monitoring charts to your dashboards lets you create a detailed view of your model's performance. These dashboards can inform your team, management, or stakeholders, and help make data-driven decisions that improve your AI performance. View a list of the [**available metrics for monitoring charts here**](/observability/platform/monitoring-charts-platform#supported-metric-types).
![Model monitoring dashboard including performance, drift, traffic, custom metrics, and segment analysis.]()
### Key Features
Dashboards offer a powerful way to analyze the overall health and performance of your models, as well as to compare multiple models.
#### Dashboard Filters
* [Flexible filters](/observability/dashboards/dashboards-creating#dashboard-filters) including date range, time zone, and bin size to customize your view
#### Chart Utilities
* [Leverage the chart toolbar](/observability/dashboards/dashboard-interactions#zoom-into-details) to zoom into data and toggle between line and bar chart types
#### Automatic & Default Dashboards
* Fiddler automatically creates a [monitoring dashboard](/observability/dashboards/dashboards-creating#auto-generated-dashboards) for all your models that can be accessed as Insights throughout the product.
#### [Dashboard Basics](/observability/dashboards/dashboard-utilities)
* Perform model-to-model comparison
* Plot drift or data integrity for multiple columns in one view
* Easily save and share your dashboard
Checkout more on the [Dashboards Guide](/observability/dashboards/dashboards-creating).
# Fairness
Source: https://docs.fiddler.ai/observability/fairness
Explore our walkthrough of ML model fairness and bias. Review the sample calculations you can customize to your data and use with Fiddler's custom metrics.
Thanks to Fiddler charts visuals and our [custom metrics](/observability/platform/custom-metrics), you can define your fairness metric of choice to detect model bias on your dataset or production data and set up alerts.
## Definitions of Fairness
Models are trained on real-world examples to mimic past outcomes on unseen data. The training data could be biased, which means the model will perpetuate the biases in the decisions it makes.
While there is not a universally agreed upon definition of fairness, we define a ‘fair’ ML model as a model that does not favor any group of people based on their characteristics.
Ensuring fairness is key before deploying a model in production. For example, in the US, the government prohibited discrimination in credit and real-estate transactions with fair lending laws like the Equal Credit Opportunity Act (ECOA) and the Fair Housing Act (FHAct).
The Equal Employment Opportunity Commission (EEOC) acknowledges 12 factors of discrimination:[\[1\]](#reference) age, disability, equal pay/compensation, genetic information, harassment, national origin, pregnancy, race/color, religion, retaliation, sex, sexual harassment. These are what we call protected attributes.
## Fairness Metrics
Among others, some popular metrics are:
* Disparate Impact
* Group Benefit
* Equal Opportunity
* Demographic Parity
The choice of the metric is use case-dependent. An important point to make is that it's impossible to optimize all the metrics at the same time. This is something to keep in mind when analyzing fairness metrics.
## Disparate Impact
Disparate impact is a form of **indirect and unintentional discrimination**, in which certain decisions disproportionately affect members of a protected group.
Mathematically, disparate impact compares the pass rate of one group to that of another.
The pass rate is the rate of positive outcomes for a given group. It's defined as follows:
pass rate = passed / (num of ppl in the group)
Disparate impact is calculated by:
`DI = (pass rate of group 1) / (pass rate of group 2)`
Groups 1 and 2 are interchangeable. Therefore, the following formula can be used to calculate disparate impact:
`DI = min{pr_1, pr_2} / max{pr_1, pr_2}.`
The disparate impact value is between 0 and 1. The Four-Fifths rule states that the disparate impact has to be greater than 80%.
For example:
`pass-rate_1 = 0.3, pass-rate_2 = 0.4, DI = 0.3/0.4 = 0.75`
`pass-rate_1 = 0.4, pass-rate_2 = 0.3, DI = 0.3/0.4 = 0.75`
> 📘 Info
>
> Disparate impact is the only legal metric available. The other metrics are not yet codified in US law.
## Demographic Parity
Demographic parity states that the proportion of each segment of a protected class should receive the positive outcome at equal rates.
Mathematically, demographic parity compares the pass rate of two groups.
The pass rate is the rate of positive outcomes for a given group. It is defined as follows:
`pass rate = passed / (num of ppl in the group)`
If the decisions are fair, the pass rates should be the same.
## Group Benefit
Group benefit aims to measure the rate at which a particular event is predicted to occur within a subgroup compared to the rate at which it actually occurs.
Mathematically, group benefit for a given group is defined as follows:
`Group Benefit = (TP+FP) / (TP + FN).`
Group benefit equality compares the group benefit between two groups. If the two groups are treated equally, the group benefit should be the same.
## Equal Opportunity
Equal opportunity means that all people will be treated equally or similarly and not disadvantaged by prejudices or bias.
Mathematically, equal opportunity compares the true positive rate (TPR) between two groups. TPR is the probability that an actual positive will test positive. The true positive rate is defined as follows:
`TPR = TP/(TP+FN)`
If the two groups are treated equally, the TPR should be the same.
## Intersectional Fairness
We believe fairness should be ensured to all subgroups of the population. We extended the classical metrics (which are defined for two classes) to multiple classes. In addition, we allow multiple protected features (e.g. race *and* gender). By measuring fairness along overlapping dimensions, we introduce the concept of intersectional fairness.
To understand why we decided to go with intersectional fairness, we can take a simple example. In the figure below, we observe that equal numbers of black and white people pass. Similarly, there is an equal number of men and women passing. However, this classification is unfair because we don’t have any black women and white men that passed, and all black men and white women passed. Here, we observe bias within subgroups when we take race and gender as protected attributes.
![]()
The EEOC provides examples of past intersectional discrimination/harassment cases.[\[2\]](#reference)
For more details about the implementation of the intersectional framework, please refer to this [research paper](https://arxiv.org/pdf/2101.01673.pdf).
## Model Behavior
In addition to the fairness metrics, we suggest tracking model outcomes and model performance for each subgroup of population.
## Reference
\[^1]: USEEOC article on [*Discrimination By Type*](https://www.eeoc.gov/know-your-rights-workplace-discrimination-illegal) \[^2]: USEEOC article on [*Intersectional Discrimination/Harassment*](https://www.eeoc.gov/initiatives/e-race/significant-eeoc-racecolor-casescovering-private-and-federal-sectors#intersectional)
# Embedding Visualizations
Source: https://docs.fiddler.ai/observability/llm/embedding-visualization-with-umap
Explore our guide on embedding visualization to enhance LLM monitoring. Discover UMAP techniques, analyze high-dimensional data, and uncover patterns with ease.
Discover how embedding visualization enhances LLM monitoring and analysis by simplifying complex data relationships and delivering actionable insights. This guide explains how to implement Fiddler’s LLM visualization techniques, such as UMAP, to uncover patterns, clusters, and anomalies in high-dimensional data.
![Charts: UMAP embedding visualization]()
### What is Embedding Visualization and Why It Matters for LLM Monitoring
Embedding visualization is a powerful technique for understanding and interpreting complex relationships in high-dimensional data. By reducing the dimensionality of custom features into a 2D or 3D space, patterns, clusters, and outliers become easier to identify, offering valuable insights for LLM embedding analysis and vector embedding visualization.
In Fiddler, high-dimensional data like embeddings and vectors are ingested as a [Custom Feature](/observability/platform/vector-monitoring-platform#define-custom-features).
This approach allows you to visualize embeddings and perform vector embedding visualization effectively, enabling detailed analysis and uncovering hidden patterns in your data.
### Using UMAP for LLM Embedding Visualization in High-Dimensional Data
We use the [UMAP](https://umap-learn.readthedocs.io/en/latest/) (Uniform Manifold Approximation and Projection) technique to embed visualizations. UMAP is a dimensionality reduction method particularly effective at preserving the local structure of data, making it ideal for visualizing embeddings, including those from embedded LLM models. Reducing high-dimensional embeddings to a 3D space allows for easier interpretation and analysis.
UMAP supports both text embedding visualization and image embedding as a [Custom Feature](/observability/platform/vector-monitoring-platform#define-custom-features). Our [guide](#using-umap-for-llm-embedding-visualization-in-high-dimensional-data) teaches you how to apply UMAP visualizations to your application data, unlocking deeper insights into your embedded LLM data.
### How UMAP Embedding Visualization Enhances Generative Applications
UMAP embedding visualizations are extremely helpful in understanding common themes and topics in the data corpus for generative AI applications. When evaluating prompts and responses, it is paramount to see which concept clusters emerge and which exhibit the most problems. Users can identify the clusters with the most issues by coloring them with various LLM and GenAI correctness and safety metrics.
![Identifying a cluster of prompts with Jailbreak attempts via UMAP]()
To create an embedding visualization chart, follow the Guide [here](#using-umap-for-llm-embedding-visualization-in-high-dimensional-data).
# Enrichments
Source: https://docs.fiddler.ai/observability/llm/enrichments
Explore our guide on how Fiddler can enrich your LLM application's data to help analyze and evaluate application behavior and performance.
For a complete list of all LLM enrichments with output columns and sub-metrics, see the [LLM Observability Metrics Reference](/reference/llm-observability-metrics).
## Introduction to Enrichments
Enrichments are specialized evaluation features that augment your LLM application data with automatically generated trust and safety metrics. By defining enrichments during model onboarding, you instruct Fiddler to analyze published prompts and responses using purpose-built models that assess dimensions like faithfulness, toxicity, relevance, and safety compliance.
The enrichment framework processes your application's inputs and outputs to generate quantitative scores that integrate directly with Fiddler's monitoring dashboards, alerting systems, and root cause analysis tools. This approach enables proactive detection of model drift, content safety violations, and performance degradation without requiring manual evaluation or external API dependencies.
* Enrichments are [custom features](/observability/platform/vector-monitoring-platform#define-custom-features) designed to augment the data provided in events
* Enrichments augment existing columns with new metrics that are defined during model onboarding
* The new metrics are available for use within the analysis, charting, and alerting functionalities in Fiddler
The following example demonstrates how to configure a TextEmbedding enrichment to enable vector-based monitoring and visualization of your LLM application's text inputs. TextEmbedding enrichments convert unstructured text into high-dimensional vector representations that capture semantic meaning, enabling Fiddler to detect drift in the topics and themes of your prompts or responses over time.
In this configuration, the enrichment transforms text from the `question` column into numerical embeddings stored in `question_embedding`. These embeddings power Fiddler's 3D UMAP visualizations, allowing you to visually identify clusters of similar content, detect outliers, and spot shifts in user behavior patterns. The `TextEmbedding` feature also enables drift detection by comparing the distribution of embeddings between your baseline and production data, providing early warning when your application encounters significantly different types of content than expected.
```py theme={null}
import fiddler as fdl
# Define a TextEmbedding enrichment for vector-based monitoring
fiddler_custom_features = [
fdl.TextEmbedding(
name='question_cf', # Internal name for the custom feature
source_column='question', # Original text column to analyze
column='question_embedding', # Generated embedding vector column
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
***
### Embedding
Embeddings are numerical representations (vectors) generated by a model for input text. Each number within the vector represents a different dimension of the text input. The meaning of each number depends on how the embedding generating model was trained.
Fiddler uses publicly available embeddings to power the 3D UMAP experience. Because the same model generates all embeddings, the points will naturally cluster, enabling quick visual anomaly detection.
To create embeddings and leverage them for the UMAP visualization, you must create a new TextEmbedding enrichment on your unstructured text column. If you want to bring your own embeddings onto the Fiddler platform, you can direct Fiddler to consume the embeddings vector directly from your data.
**Example 1: Fiddler-Generated Embeddings**
This example automatically generates text embeddings on a text column called `prompt`:
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.TextEmbedding(
name='Prompt TextEmbedding', # name of generated column (for internal use, required)
source_column='prompt', # source - raw text
column='Enrichment Prompt Embedding', # name of the vector output
n_clusters=5, # Number of clusters for k-means clustering
n_tags=5, # Top n tags used as summary tags per cluster
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ----------------------------------------- | ------- | -------------------------------------------------- |
| Enrichment Prompt Embedding | vector | Embeddings corresponding to string column `prompt` |
| Prompt Text Embedding | integer | Column for internal use |
| Prompt Text Embedding - Centroid Distance | float | Centroid Distance to string column `prompt` |
**Example 2: User-Provided Embeddings**
This example demonstrates leveraging an existing vector column of text embeddings called `pre_existing_prompt_embedding`:
Vector embeddings must use data type List(float) for compatibility. Using CSV for source data often requires preprocessing after initial loading into a pandas dataframe.
```python theme={null}
import ast
import numpy as np
import pandas as pd
import fiddler as fdl
df = pd.read_csv(PATH_TO_SAMPLE_CSV)
# Convert string representation to list
df['pre_existing_prompt_embedding'] = df['pre_existing_prompt_embedding'].apply(
ast.literal_eval
)
fiddler_custom_features = [
fdl.TextEmbedding(
name='User-provided Text Embedding', # name of the generated column
source_column='prompt', # source - raw text
column='pre_existing_prompt_embedding', # name of your vector column
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt'],
metadata=['pre_existing_prompt_embedding'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------------------------ | ------- | ------------------------------------------- |
| User-provided Text Embedding | integer | Column for internal use |
| User-provided Text Embedding - Centroid Distance | float | Centroid Distance to string column `prompt` |
***
### Centroid Distance
Fiddler uses KMeans to determine cluster membership for a given enrichment. The Centroid Distance enrichment provides information about the distance between the selected point and the closest centroid. Centroid Distance is automatically added if the TextEmbedding enrichment is created for any given model.
Centroid Distance columns are automatically generated when you create a TextEmbedding enrichment. See the [Embedding](#embedding) section above for complete usage examples.
***
### Personally Identifiable Information
The PII (Personally Identifiable Information) enrichment tool is a critical tool for detecting and flagging sensitive information in textual data. Whether user-entered or system-generated, this enrichment aims to identify instances where PII may be exposed, helping prevent privacy breaches and misuse of personal data. In an era where digital privacy concerns are paramount, mishandling or unintentionally leaking PII can have serious repercussions, including privacy violations, identity theft, and significant legal and reputational damage.
PII enrichment is integrated with [Presidio](https://microsoft.github.io/presidio/analyzer/languages/) for entity detection.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Rag PII',
enrichment='pii',
columns=['question'], # one or more columns
allow_list=['fiddler'], # Optional: list of strings that are white listed
score_threshold=0.85, # Optional: float value for minimum possible confidence
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------- | ---- | --------------------------------------------------------------------------------------- |
| FDL Rag PII (question) | bool | Whether any PII was detected |
| FDL Rag PII (question) Matches | str | What matches in raw text were flagged as potential PII (ex. 'Douglas MacArthur,Korean') |
| FDL Rag PII (question) Entities | str | What entities these matches were tagged as (ex. 'PERSON') |
**Supported PII Entity Types:**
CREDIT\_CARD, CRYPTO, DATE\_TIME, EMAIL\_ADDRESS, IBAN\_CODE, IP\_ADDRESS, LOCATION, PERSON, PHONE\_NUMBER, URL, US\_SSN, US\_DRIVER\_LICENSE, US\_ITIN, US\_PASSPORT
***
### Evaluate
This enrichment provides n-gram-based metrics for comparing two passages of text, such as BLEU, ROUGE, and METEOR. Created initially to compare an AI-generated translation or summary to a human-generated one, these metrics have some use in RAG summarization tasks. They score highest when the reference and generated texts contain overlapping sequences. Additionally, these metrics are not as effective for long passages of text.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='QA Evaluate',
enrichment='evaluate',
columns=['correct_answer', 'generated_answer'],
config={
'reference_col': 'correct_answer', # required
'prediction_col': 'generated_answer', # required
'metrics': ['bleu', 'rouge', 'meteor'], # optional, this is the default
}
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------------------- | ----- | ------------------------------------------------------ |
| FDL QA Evaluate (bleu) | float | BLEU score: Measures precision of word n-grams |
| FDL QA Evaluate (rouge1) | float | ROUGE-1 score: Unigram recall |
| FDL QA Evaluate (rouge2) | float | ROUGE-2 score: Bigram recall |
| FDL QA Evaluate (rougel) | float | ROUGE-L score: Longest common subsequence |
| FDL QA Evaluate (rougelsum) | float | ROUGE-L summary score |
| FDL QA Evaluate (meteor) | float | METEOR score: Precision, recall, and semantic matching |
***
### Textstat
The Textstat enrichment generates various text statistics such as character/letter count, Flesch-Kincaid, and other metrics on the target text column.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Text Statistics',
enrichment='textstat',
columns=['question'],
config={
'statistics': [
'char_count',
'dale_chall_readability_score',
]
},
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| -------------------------------------------------------------- | ----- | ------------------------------------------------ |
| FDL Text Statistics (question) char\_count | int | Character count of string in `question` column |
| FDL Text Statistics (question) dale\_chall\_readability\_score | float | Readability score of string in `question` column |
**Supported Statistics:**
char\_count, letter\_count, miniword\_count, words\_per\_sentence, polysyllabcount, lexicon\_count, syllable\_count, sentence\_count, flesch\_reading\_ease, smog\_index, flesch\_kincaid\_grade, coleman\_liau\_index, automated\_readability\_index, dale\_chall\_readability\_score, difficult\_words, linsear\_write\_formula, gunning\_fog, long\_word\_count, monosyllabcount
***
### Sentiment
The Sentiment enrichment uses NLTK's VADER lexicon to generate a score and corresponding sentiment for all specified columns. To enable, set the enrichment parameter to `sentiment`.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Question Sentiment',
enrichment='sentiment',
columns=['question'],
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------------------- | ------ | ------------------------------------------- |
| FDL Question Sentiment (question) compound | float | Raw score of sentiment |
| FDL Question Sentiment (question) sentiment | string | One of `positive`, `negative`, or `neutral` |
***
### Profanity
The Profanity enrichment is designed to detect and flag the use of offensive or inappropriate language within textual content. This enrichment is essential for maintaining the integrity and professionalism of digital platforms, forums, social media, and any user-generated content areas.
The profanity enrichment searches the target text for words from the two sources below:
* The Obscenity List from [SurgeAI](https://github.com/surge-ai/)
* Google banned words \<[https://github.com/coffee-and-fun/google-profanity-words/blob/main/data/en.txt](https://github.com/coffee-and-fun/google-profanity-words/blob/main/data/en.txt)>
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Profanity',
enrichment='profanity',
columns=['prompt', 'response'],
config={'output_column_name': 'contains_profanity'},
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| -------------------------------------------- | ---- | ------------------------------------------------------------------------- |
| FDL Profanity (prompt) contains\_profanity | bool | Indicates if input contains profanity in the value of the prompt column |
| FDL Profanity (response) contains\_profanity | bool | Indicates if input contains profanity in the value of the response column |
***
### Regex Match
The Regex Match enrichment evaluates text responses or content for adherence to specific patterns defined by regular expressions (regex). By accepting a regex as input, this metric offers a highly customizable way to check if a string column in the dataset matches the given pattern. This functionality is essential for scenarios requiring precise formatting, specific keyword inclusion, or adherence to particular linguistic structures.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Regex - only digits',
enrichment='regex_match',
columns=['prompt', 'response'],
config={
'regex': '^\d+$',
}
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'doc_0', 'doc_1', 'doc_2', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ----------------------- | -------- | ---------------------------------------------------------------------------------------- |
| FDL Regex - only digits | category | Match or No Match, depending on the regex specified in the config matching in the string |
***
### Topic
The Topic enrichment leverages the capabilities of [Zero Shot Classifier](https://huggingface.co/tasks/zero-shot-classification) models to categorize textual inputs into a predefined list of topics, even without having been explicitly trained on those topics. This approach to text classification is known as zero-shot learning, a groundbreaking method in natural language processing (NLP) that enables models to intelligently classify text they haven't encountered during training. It's beneficial for applications that require understanding and organizing content dynamically across a broad range of subjects or themes.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Topics',
enrichment='topic_model',
columns=['response'],
config={'topics': ['politics', 'economy', 'astronomy']},
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'doc_0', 'doc_1', 'doc_2', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------------------ | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| FDL Topics (response) topic\_model\_scores | list\[float] | Probability of the given column in each of the topics specified in the Enrichment config. Each float value indicates the probability of the given input being classified in the corresponding topic, in the same order as topics. Each value will be between 0 and 1. The sum of values does not equal 1, as each classification is performed independently of other topics. |
| FDL Topics (response) max\_score\_topic | string | Topic with the maximum score from the list of topic names specified in the Enrichment config |
***
### Banned Keyword Detector
The Banned Keyword Detector enrichment is designed to scrutinize textual inputs for the presence of specified terms, with a particular focus on identifying content that includes potentially undesirable or restricted keywords. This enrichment operates based on a list of terms defined in its configuration, making it highly adaptable to various content moderation, compliance, and content filtering needs.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Banned KW',
enrichment='banned_keywords',
columns=['prompt', 'response'],
config={
'output_column_name': 'contains_banned_kw',
'banned_keywords': ['nike', 'adidas', 'puma'],
},
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'doc_0', 'doc_1', 'doc_2', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------------------------------------- | ---- | ---------------------------------------------------------------------------------------------------- |
| FDL Banned KW (prompt) contains\_banned\_kw | bool | Indicates if input contains one of the specified banned keywords in the value of the prompt column |
| FDL Banned KW (response) contains\_banned\_kw | bool | Indicates if input contains one of the specified banned keywords in the value of the response column |
***
### Language Detector
The Language Detector enrichment identifies the language of the source text. This enrichment is based on a pretrained text identification model and leverages [fasttext models](https://fasttext.cc/docs/en/language-identification.html) for language detection.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Language',
enrichment='language_detection',
columns=['prompt'],
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------------------- | ------ | --------------------------------------------- |
| FDL Language (prompt) language | string | Language prediction for input text |
| FDL Language (prompt) language\_probability | float | Confidence probability of language prediction |
***
### Answer Relevance
The Answer Relevance enrichment evaluates the pertinence of AI-generated responses to their corresponding prompts. This enrichment assesses whether a response accurately addresses the question or topic posed by the initial prompt, providing a simple yet effective binary outcome: relevant or not. Its primary function is to ensure that the output of AI systems, such as chatbots, virtual assistants, and content generation models, remains aligned with the user's informational needs and intentions.
Looking for more granular relevance scoring? The [Evals SDK](/developers/quick-starts/experiments-quick-start) and [Agentic Monitoring](/getting-started/agentic-monitoring) offer **Answer Relevance 2.0** with ordinal scoring (High/Medium/Low) as part of the [RAG Health Metrics](/concepts/rag-health-diagnostics) diagnostic triad.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
answer_relevance_config = {
'prompt': 'prompt_col',
'response': 'response_col',
}
fiddler_custom_features = [
fdl.Enrichment(
name='Answer Relevance',
enrichment='answer_relevance',
columns=['prompt_col', 'response_col'],
config=answer_relevance_config,
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt_col', 'response_col'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| -------------------- | ---- | ---------------------------------------------------------------------- |
| FDL Answer Relevance | bool | Binary metric, which is True if `response` is relevant to the `prompt` |
***
### Faithfulness
The Faithfulness (Groundedness) enrichment is a binary indicator that evaluates the accuracy and reliability of the facts presented in AI-generated text responses. It specifically assesses whether the information used in the response aligns with and is grounded in the provided context, often through referenced documents or data. This enrichment plays a critical role in ensuring that the AI's outputs are not only relevant but also factually accurate, given the context it was provided.
**Faithfulness (LLM-as-a-Judge) vs Faithfulness (Centor Model):** This LLM-based Faithfulness enrichment (`faithfulness`) uses `context` and `response` config keys. [Faithfulness (Centor Model)](#faithfulness-centor-model) (`ftl_response_faithfulness`) uses Fiddler's proprietary Centor Model for Faithfulness for lower latency. For RAG pipeline diagnostics with detailed reasoning, see [RAG Faithfulness](/concepts/rag-health-diagnostics#rag-faithfulness-vs-faithfulness-centor-model) in the Evals SDK and Agentic Monitoring.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
faithfulness_config = {
'context': ['doc_0', 'doc_1', 'doc_2'],
'response': 'response_col',
}
fiddler_custom_features = [
fdl.Enrichment(
name='Faithfulness',
enrichment='faithfulness',
columns=['doc_0', 'doc_1', 'doc_2', 'response_col'],
config=faithfulness_config,
),
]
model_spec = fdl.ModelSpec(
inputs=['doc_0', 'doc_1', 'doc_2', 'response_col'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ---------------- | ---- | ---------------------------------------------------------------------------------------------------------- |
| FDL Faithfulness | bool | Binary metric, which is True if the facts used in `response` are correctly used from the `context` columns |
***
### Coherence
The Coherence enrichment assesses the logical flow and clarity of AI-generated text responses, ensuring they are structured in a way that makes sense from start to finish. This enrichment is crucial for evaluating whether the content produced by AI maintains a consistent theme, argument, or narrative, without disjointed thoughts or abrupt shifts in topic. Coherence is key to making AI-generated content not only understandable but also engaging and informative for the reader.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
coherence_config = {
'response': 'response_col',
}
fiddler_custom_features = [
fdl.Enrichment(
name='Coherence',
enrichment='coherence',
columns=['response_col'],
config=coherence_config,
),
]
model_spec = fdl.ModelSpec(
inputs=['doc_0', 'doc_1', 'doc_2', 'response_col'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------- | ---- | ---------------------------------------------------------------------------------- |
| FDL Coherence | bool | Binary metric, which is True if `response` makes coherent arguments that flow well |
***
### Conciseness
The Conciseness enrichment evaluates the brevity and clarity of AI-generated text responses, ensuring that the information is presented in a straightforward and efficient manner. This enrichment identifies and rewards responses that effectively communicate their message without unnecessary elaboration or redundancy. In the realm of AI-generated content, where verbosity can dilute the message's impact or confuse the audience, maintaining conciseness is crucial for enhancing readability and user engagement.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
conciseness_config = {
'response': 'response_col',
}
fiddler_custom_features = [
fdl.Enrichment(
name='Conciseness',
enrichment='conciseness',
columns=['response'],
config=conciseness_config,
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt', 'doc_0', 'doc_1', 'doc_2', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------- | ---- | ---------------------------------------------------------------------------- |
| FDL Conciseness | bool | Binary metric, which is True if `response` is concise and not overly verbose |
***
### Safety
The Safety enrichment evaluates the safety of the text along eleven different dimensions: `illegal, hateful, harassing, racist, sexist, violent, sexual, harmful, unethical, jailbreaking, roleplaying`. The Safety enrichment is powered by the [Centor Model for Safety](/observability/llm/llm-based-metrics).
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Prompt Safety',
enrichment='ftl_prompt_safety',
columns=['prompt'],
config={'classifiers': ['jailbreaking', 'illegal']} # Optional: specify dimensions
),
]
model_spec = fdl.ModelSpec(
inputs=['prompt'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
For each dimension specified (or all 11 dimensions if not specified):
| Column | Type | Description |
| -------------------------------------------- | ----- | --------------------------------------------------------------------------- |
| FDL Prompt Safety (prompt) `dimension` | bool | Binary metric, which is True if the input is deemed unsafe, False otherwise |
| FDL Prompt Safety (prompt) `dimension` score | float | Confidence probability of safety prediction |
**Supported Dimensions:** illegal, hateful, harassing, racist, sexist, violent, sexual, harmful, unethical, jailbreaking, roleplaying
***
### Faithfulness (Centor Model)
The Faithfulness (Centor Model) enrichment is designed to evaluate the accuracy and reliability of facts presented in AI-generated text responses. It is powered by the [Centor Model for Faithfulness](/observability/llm/llm-based-metrics).
The faithfulness threshold defaults to 0.5 but can be adjusted in the configuration to control the sensitivity of the faithfulness scoring. Lower thresholds result in stricter faithfulness detection, while higher thresholds are more permissive.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='Faithfulness',
enrichment='ftl_response_faithfulness',
columns=['context', 'response'],
config={
'context_field': 'context',
'response_field': 'response',
'threshold': '0.5' # Optional parameter, default is 0.5
}
),
]
model_spec = fdl.ModelSpec(
inputs=['context', 'response'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| ------------------------------- | ----- | ---------------------------------------------------------------------------------------------------------- |
| FDL Faithfulness faithful | bool | Binary metric, which is True if the facts used in `response` are correctly used from the `context` columns |
| FDL Faithfulness faithful score | float | Confidence probability of faithfulness prediction |
***
### Token Count
The Token Count enrichment counts the number of tokens in a string.
This enrichment uses the [tiktoken](https://github.com/openai/tiktoken) library for token counting.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='TokenCount',
enrichment='token_count',
columns=['question'],
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------------------- | ---- | ------------------------------ |
| FDL Token Counts (question) | int | Number of tokens in the string |
***
### SQL Validation
The SQL Validation enrichment is designed to evaluate different query dialects for syntax correctness.
Query validation is syntax based and does not check against any existing schema or databases for validity.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
# The following dialects are supported:
# 'athena', 'bigquery', 'clickhouse', 'databricks', 'doris', 'drill', 'duckdb',
# 'hive', 'materialize', 'mysql', 'oracle', 'postgres', 'presto', 'prql',
# 'redshift', 'risingwave', 'snowflake', 'spark', 'spark2', 'sqlite',
# 'starrocks', 'tableau', 'teradata', 'trino', 'tsql'
fiddler_custom_features = [
fdl.Enrichment(
name='SQLValidation',
enrichment='sql_validation',
columns=['query_string'],
config={
'dialect': 'mysql'
}
),
]
model_spec = fdl.ModelSpec(
inputs=['query_string'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| -------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------- |
| SQL Validator valid | bool | True if the query string is syntactically valid for the specified dialect, False if not |
| SQL Validator errors | str | If syntax errors are found they will be present as a JSON serialized string containing a list of dictionaries describing the errors |
***
### JSON Validation
The JSON Validation enrichment is designed to validate JSON for correctness and optionally against a user-defined schema for validation.
This enrichment uses the [python-jsonschema](https://python-jsonschema.readthedocs.io) library for JSON schema validation. The defined `validation_schema` must be a valid python-jsonschema schema.
**Python Configuration:**
```python theme={null}
import fiddler as fdl
fiddler_custom_features = [
fdl.Enrichment(
name='JSONValidation',
enrichment='json_validation',
columns=['json_string'],
config={
'strict': 'true',
'validation_schema': {
'$schema': 'https://json-schema.org/draft/2020-12/schema',
'type': 'object',
'properties': {
'prop_1': {'type': 'number'}
# ... additional properties
},
'required': ['prop_1'], # ... additional required fields
'additionalProperties': False
}
}
),
]
model_spec = fdl.ModelSpec(
inputs=['json_string'],
custom_features=fiddler_custom_features,
)
```
**Generated Columns:**
| Column | Type | Description |
| --------------------- | ---- | ------------------------------------------------------------------------------------------------------------------- |
| JSON Validator valid | bool | String is valid JSON |
| JSON Validator errors | str | If the string failed to parse to JSON any parsing errors will be returned as a serialized JSON list of dictionaries |
# LLM Monitoring
Source: https://docs.fiddler.ai/observability/llm/index
Explore our guide to LLM application monitoring. Learn how Fiddler generates enrichments using trust and safety metrics for alerting, analysis, and debugging.
### Monitoring LLM Applications with Fiddler
Monitoring Large Language Model (LLM) applications with Fiddler requires publishing the LLM application's inputs and outputs, including prompts, prompt context, responses, and the source documents retrieved (for RAG-based applications). Fiddler will then generate enrichments, which are LLM trust and safety metrics, for alerting, analysis, or debugging purposes.
Fiddler is a pioneer in the AI Trust domain and, as such, offers the most extensive set of AI safety and trust metrics available today.
### Selecting Enrichments to Enhance Monitoring
Fiddler offers many enrichments that each measure different aspects of an LLM application. For detailed information about which enrichment to select for any specific use case, visit [this](/observability/llm/selecting-enrichments) page. Some enrichments use Fiddler Centor Models to generate these scores. Compare Centor Models cost to third-party LLM evaluators with the [Fiddler Evals TCO calculator](https://www.fiddler.ai/evals-tco-calculator).
### Generating Enrichments with Fiddler
LLM application owners must specify the enrichments to be generated by Fiddler during model onboarding. The enrichment pipeline then generates enrichments for the LLM application's inputs and outputs as events are published to Fiddler.
![Fiddler Enrichment Framework diagram displaying sample inputs and outputs flowing into the Fiddler enrichment pipeline.]()
Figure 1. The Fiddler Enrichment Framework
After the LLM application's raw, unstructured inputs and outputs are published to Fiddler, the enrichment framework augments them with various AI trust and safety metrics. These metrics can monitor the application's overall health and alert users to any performance degradation.
![Fiddler dashboard showing LLM application performance using enrichment metrics.]()
Figure 2. A Fiddler dashboard showing LLM application performance
Using the metrics produced by the enrichment framework, users can monitor LLM application performance over time and conduct root-cause analysis when problematic trends are identified.
During model onboarding, application owners can opt in to the various, ever-expanding Fiddler enrichments by specifying [Enrichments](/observability/llm/enrichments) as custom features in the Fiddler ModelSpec object.
# LLM-Based Metrics
Source: https://docs.fiddler.ai/observability/llm/llm-based-metrics
Explore our guide on LLM-specific metrics useful for evaluating AI-generated content for use cases like chatbots, writing assistants, or content creation tools.
For a complete reference of all LLM enrichments, see the [LLM Observability Metrics Reference](/reference/llm-observability-metrics).
LLM-based metrics use large language models to evaluate the quality of text generated by AI. This approach is much closer to how humans judge text, making these metrics particularly useful for evaluating AI-generated content for use cases such as chatbots, writing assistants, or content creation tools.
LLM-based metrics can adapt to different topics and types of text because LLMs have been trained on a wide range of information, making them a valuable tool for developers and researchers looking to enhance the quality of AI-generated text.
Currently, Fiddler supports three types of LLM-based metrics: LLM-as-a-Judge evaluators (including RAG Health Metrics), OpenAI-based enrichments, and Fiddler Centor Model metrics.
## RAG Health Metrics (LLM-as-a-Judge Evaluators)
RAG Health Metrics are a purpose-built diagnostic triad for evaluating RAG applications. These evaluators use LLM-as-a-Judge approaches and are available in **Agentic Monitoring** and **Experiments**:
* **Answer Relevance 2.0** — Ordinal scoring (High/Medium/Low = 1.0/0.5/0.0) measuring how well the response addresses the query. Also available in LLM Observability.
* **Context Relevance** — Ordinal scoring measuring whether retrieved documents support the query. Available in Agentic Monitoring and Experiments only.
* **RAG Faithfulness** — Binary scoring (Yes/No = 1/0) assessing whether the response is grounded in retrieved documents. Also available in LLM Observability.
See [RAG Health Diagnostics](/concepts/rag-health-diagnostics) for a conceptual guide to using these evaluators together for root cause analysis.
**RAG Faithfulness vs Faithfulness (Centor Model):** These are **separate evaluators** with different architectures. RAG Faithfulness is an LLM-as-a-Judge evaluator using external LLMs with inputs `user_query`, `rag_response`, and `retrieved_documents`. Faithfulness (Centor Model) (below) is powered by the Centor Model for Faithfulness using `context` and `response`. See the [enrichment glossary](/glossary/enrichment) for a detailed comparison.
## OpenAI-based metrics
* These metrics are generated through the OpenAI API, which may introduce latency due to network communication and processing time.
* OpenAI API access token MUST BE provided by the user, which will be configured during onboarding.
* The specific model to be used for these metrics will also be chosen during onboarding.
Currently, the below metrics are OpenAI-based:
* [Answer Relevance](/observability/llm/enrichments#answer-relevance)
* [Faithfulness](/observability/llm/enrichments#faithfulness)
* [Coherence](/observability/llm/enrichments#coherence)
* [Conciseness](/observability/llm/enrichments#conciseness)
## Fiddler Centor Model metrics
* These metrics are generated through Fiddler's in-house, purpose-built SLMs.
* These metrics can be generated in air-gapped environments and do not rely on any over-the-network connection to generate such scores.
Currently, the below metrics are Fiddler Centor Model-based:
* [Safety](/observability/llm/enrichments#safety) — Evaluates safety across 11 dimensions including jailbreaking, toxicity, and harmful content. Powered by the Centor Model for Safety.
* [Faithfulness (Centor Model)](/observability/llm/enrichments#faithfulness-centor-model) — Powered by the Centor Model for Faithfulness for hallucination detection. Not to be confused with RAG Faithfulness above.
# LLM Evaluation Prompt Specs
Source: https://docs.fiddler.ai/observability/llm/llm-evaluation-prompt-specs
Prompt specs is a framework Fiddler provides for leveraging a general-purpose LLM to quickly create custom scoring functions without the need to manually tune an evaluation prompt.
## The Challenge With LLM Evaluation
When building production LLM applications, evaluation is critical, but it's often the most time-consuming bottleneck in your development workflow. Traditional approaches to creating LLM-as-a-Judge evaluators require you to:
* **Hand-craft natural language prompts** for each new use case or data schema
* **Iteratively tune prompts** through trial-and-error until the model produces reliable outputs
* **Manually rewrite entire prompt templates** every time your input fields or output categories change
* **Struggle with inconsistent results** as small prompt variations lead to unpredictable model behavior
* **Spend weeks perfecting prompts** before you can move to production monitoring
This manual process doesn't scale when you need to evaluate multiple aspects of your LLM application or adapt to evolving requirements.
## How Prompt Specs Solve This Problem
Fiddler's Prompt Specs eliminate the prompt engineering bottleneck by letting you **declare your evaluation logic using simple JSON schemas** instead of writing prompts. Here's the key difference:
### Manual Prompt Engineering (Traditional Approach)
```
You are an expert classifier. Given a news summary, classify it into one of these topics: World, Sports, Business, Technology.
News summary: {news_summary}
Consider the main subject matter and classify accordingly. Provide your reasoning.
Output format:
Topic: [your classification]
Reasoning: [your explanation]
```
*Every schema change requires rewriting and retuning the entire prompt*
### Schema-Based Evaluation Using Prompt Specs
```json theme={null}
{
"input_fields": {
"news_summary": { "type": "string" }
},
"output_fields": {
"topic": {
"type": "string",
"choices": ["Business", "Sci/Tech", "Sports", "World"]
},
"reasoning": { "type": "string" }
}
}
```
*Schema changes are simple JSON updates—no prompt rewriting needed*
## When to Use Prompt Specs
Prompt Specs are ideal for scenarios where you need:
**Domain-Specific Classification**
* Content moderation with custom policy categories for example finance-specific metrics
* Topic classification
* Product categorization using your taxonomy
**Quality Assessment**
* Response relevance scoring for your specific use case
* Conciseness and coherence evaluation
* Completeness checking for required information elements
**Safety and Compliance**
* Custom content policy enforcement
* Regulatory compliance checking (e.g., financial disclosures, medical claims)
* Brand safety evaluation with organization-specific criteria
Prompt Specs work particularly well when you need multiple output fields, such as confidence ranges and reasoning, alongside the classification and when classification accuracy is more important than the flexibility of fully custom prompts.
## Benefits for Technical Teams
**Faster Development Cycles**
* Reduce evaluation setup from weeks to hours
* Eliminate iterative prompt tuning cycles
* Enable rapid schema evolution without prompt rewrites
**Improved Reliability**
* Consistent structured output guaranteed by schema validation
* Higher classification accuracy compared to guided choice decoding
* Built-in reasoning capture for debugging and explainability
**Better Maintainability**
* Version control friendly JSON schemas
* Clear separation between evaluation logic and implementation
* Seamless integration with existing Fiddler monitoring workflows
## Benefits for Risk and Compliance Teams
**Enhanced Auditability**
* Clear, declarative evaluation criteria in version-controlled schemas
* Consistent evaluation logic across all model versions
* Traceable reasoning for every evaluation decision
**Improved Governance**
* Schema-based approach prevents evaluation drift over time
* Standardized evaluation framework across teams and use cases
* Integration with Fiddler's monitoring and alerting capabilities for continuous oversight
**Regulatory Compliance**
* Documented evaluation criteria that can be reviewed by compliance teams
* Consistent application of evaluation logic for audit purposes
* Historical tracking of schema changes and their impacts on evaluation outcomes
## How Prompt Specs Work
Prompt Specs follow a simple three-step workflow that takes you from evaluation idea to production monitoring:
### 1. Define Your Evaluation Schema
Create a JSON schema specifying your input fields and desired output format. Supported field types include:
* `string` (with optional `choices` array for categorical outputs)
* `integer`
* `boolean`
* `number`
### 2. Validate and Test Your Schema
Use Fiddler's validation and prediction APIs to ensure your schema works correctly with sample data.
### 3. Deploy for Production Monitoring
Deploy your validated Prompt Spec as a Fiddler [enrichment](/glossary/enrichment) for ongoing monitoring.
**Ready to try it yourself?** Follow our step-by-step [Quickstart Guide](/evaluate-and-test/prompt-specs-quick-start) to build your first evaluation in minutes.
## Performance and Cost Considerations
### Prompt Specs Benefits
* **Higher accuracy**: Classification accuracy is very high using meaningful field names and labels
* **Consistent structure**: Yields reliably structured LLM output without guided decoding
* **Performance optimization**: System prompts remain consistent across invocations, enabling KV caching benefits
* **Customization**: Easy to add descriptions to fields and tasks for better accuracy
**Trade-offs:**
* Structured output is not 100% guaranteed
* Less flexibility than fully custom prompts for highly nuanced evaluations
* Best suited for classification and structured scoring tasks rather than open-ended evaluation
## Integration With Fiddler's Monitoring Platform
Prompt Specs integrate seamlessly with Fiddler's existing monitoring capabilities:
**Enrichment Pipeline Integration**
* Deploy Prompt Specs as [enrichments](/glossary/enrichment) alongside other Fiddler [trust and safety metrics](/glossary/trust-score)
* Combine schema-based evaluations with [built-in enrichments](/observability/llm/enrichments) like toxicity detection and PII identification
* Use evaluation results in [alerting rules](/observability/platform/alerts-platform) and [dashboard visualizations](/observability/dashboards)
**Monitoring and Alerting**
* Set up automated alerts based on Prompt Spec outputs (e.g., alert when helpfulness scores drop below threshold)
* Track evaluation trends over time alongside other model performance metrics
* Use reasoning outputs for [root cause analysis](/observability/analytics) when issues are detected
## Validation Rules and Schema Requirements
When creating Prompt Specs, ensure your schema follows these rules:
* **Required fields**: Input fields and output fields must each have at least one item
* **Field naming**: Field names must be valid Python variable names
* **Uniqueness**: Field names must be unique between input and output sections
* **Type specification**: `type` is required for all fields and must be one of: `string`, `integer`, `number`, `boolean`
* **Optional elements**: Top-level `task_instruction` and field-level `description` are optional
* **Choices arrays**: Must have at least 2 items and are only allowed when `type` is `string`
## When Not to Use Prompt Specs
Prompt Specs may not be the best choice when:
* You need highly nuanced evaluation requiring extensive domain context that can't be captured in field descriptions
* Your evaluation criteria change frequently and unpredictably in ways that require prompt-level modifications
* You require real-time evaluation with sub-100ms latency requirements
* Your use case is already well-served by existing Fiddler enrichments like Faithfulness (Centor Model)
* You need the full flexibility of custom prompt templates
## Comparison With Other Evaluation Approaches
| Approach | Setup Time | Accuracy | Consistency | Flexibility | Maintenance |
| ----------------------------- | ---------- | --------- | ----------- | ----------- | ----------- |
| **Prompt Specs** | Hours | High | High | Medium | Low |
| **Manual Prompt Engineering** | Weeks | Very High | Medium | Very High | High |
| **Pre-built Enrichments** | Minutes | High | High | Low | None |
Prompt Specs occupy a sweet spot between the quick deployment of pre-built enrichments and the full customization of manual prompt engineering, making them ideal for customers who need domain-specific evaluation without extensive prompt tuning effort.
## Getting Started
### Prompt Specs Quick Start
The fastest way to get started is with the Prompt Spec Quick Start Guide:
**🚀** [**Follow the Quickstart Guide**](/evaluate-and-test/prompt-specs-quick-start) - Build and deploy your first evaluation in minutes
# Selecting Enrichments
Source: https://docs.fiddler.ai/observability/llm/selecting-enrichments
Learn about Fiddler’s enrichments and monitor key aspects of LLM applications. Discover the different factors to analyze for your specific use case.
For a complete reference of all LLM enrichments with output columns and sub-metrics, see the [LLM Observability Metrics Reference](/reference/llm-observability-metrics).
Fiddler offers enrichments out of the box to monitor different aspects of LLM applications. Use the below table to select the right enrichment for your specific use case.
This table provides high level information on the metric, the enrichment to use to measure the metric, if the metric uses LLMs, and if so, what LLM it uses.
If you have a use case not covered by the below enrichments out of the box, please contact your administrator.
| Metric | Metric Category | Description | Enrichment | LLM Used? | LLM Type |
| --------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | --------- | ------------ |
| [Faithfulness](/observability/llm/enrichments#faithfulness) | Hallucination | This enrichment identifies the accuracy and reliability of facts presented in AI-generated texts | `faithfulness` | Yes | OpenAI |
| [Faithfulness (Centor Model)](/observability/llm/enrichments#faithfulness-centor-model) | Hallucination | This enrichment identifies the accuracy and reliability of facts presented in AI-generated texts. It is powered by the Centor Model for Faithfulness | `ftl_response_faithfulness` | Yes | Centor Model |
| [Answer Relevance](/observability/llm/enrichments#answer-relevance) | Hallucination | This enrichment measures the pertinence of AI-generated responses to their inputs | `answer_relevance` | Yes | OpenAI |
| [Conciseness](/observability/llm/enrichments#conciseness) | Hallucination | This enrichment evaluates the brevity and clarity of AI-generated responses | `conciseness` | Yes | OpenAI |
| [Coherence](/observability/llm/enrichments#coherence) | Hallucination | This enrichment assesses the logical flow and clarity of AI-generated responses | `coherence` | Yes | OpenAI |
| [Safety](/observability/llm/enrichments#safety) | Safety | This enrichment generates 11 different safety metrics to measure texts upon. These metrics are: `illegal, hateful, harassing, racist, sexist, violent, sexual, harmful, unethical, jailbreaking, roleplaying` | `ftl_prompt_safety` | Yes | Centor Model |
| [PII](/observability/llm/enrichments#personally-identifiable-information) | Safety | This enrichment flags the presence of sensitive information within textual data | `pii` | No | |
| [Regex Match](/observability/llm/enrichments#regex-match) | Safety | This enrichment compares the text with a regular expression string | `regex_match` | No | |
| [Topic](/observability/llm/enrichments#topic) | Safety | This enrichment classifies the text into several preset dimensions using a zero-shot classifier | `topic_model` | No | |
| [Banned Keywords](/observability/llm/enrichments#banned-keyword-detector) | Safety | This enrichment detects the presence of banned keywords configured by the user | `banned_keywords` | No | |
| [Profanity](/observability/llm/enrichments#profanity) | Safety | This enrichment flags the use of offensive or inappropriate language | `profanity` | No | |
| [Language Detection](/observability/llm/enrichments#language-detector) | Safety | This enrichment identifies the language of the source text | `language_detection` | No | |
| [Evaluate](/observability/llm/enrichments#evaluate) | Text Statistics | This enrichment provides classic text evaluation methods such as BLEU, ROUGE, and Meteor | `evaluate` | No | |
| [Sentiment](/observability/llm/enrichments#sentiment) | Text Statistics | This enrichment provides sentiment analysis of the target text | `sentiment` | No | |
| [TextStat](/observability/llm/enrichments#textstat) | Text Statistics | This enrichment provides various text statistics such as character/letter count, Flesch-Kincaid, and others | `textstat` | No | |
| [Token Count](/observability/llm/enrichments#token-count) | Text Statistics | The Token Count enrichment is designed to count the number of tokens in a string. | `token_count` | No | |
| [SQLValidation](/observability/llm/enrichments#sql-validation) | Text Validation | Evaluates different query dialects for syntax correctness. | `sql_validation` | No | |
| [JSONValidation](/observability/llm/enrichments#json-validation) | Text Validation | Validates JSON for correctness and optionally against a user-defined schema. | `json_validation` | No | |
# Model UI
Source: https://docs.fiddler.ai/observability/model-ui/index
Learn about Fiddler's no-code Model Editor for streamlined ML model onboarding, featuring draft mode for iterative development and team collaboration.
### Adding and Editing Models in the UI
#### Overview
UI-based Model Onboarding lets you add models to Fiddler without writing Python code. You can define schemas, validate data, and register models directly through the Fiddler interface—making model onboarding accessible to more team members and streamlining the process.
Key benefits:
* No coding required - Onboard models through an intuitive web interface
* Faster deployment - Use automatic schema inference and guided validation to reduce setup time
* Early error detection - Identify issues like missing categories, incorrect ranges, or invalid data types immediately
* Draft mode flexibility - Save work-in-progress schemas, consult with your team, and validate sample data before publishing
### Model Onboarding Modes
Fiddler offers two approaches to model onboarding through the UI: Standard Onboarding and enhanced Draft Mode.
#### Standard Onboarding
With Standard Onboarding, you proceed directly through the complete process by:
* Uploading your dataset
* Reviewing the inferred schema
* Making necessary adjustments
* Publishing the model immediately
#### Draft Mode
Draft Mode provides additional flexibility when you need to refine your model configuration before publication:
* Iterate without commitment - Save a draft version to refine column names, data types, and inferred ranges
* Collaborate with your team - Share and review the draft before finalizing
* Validate with sample data - Upload sample data to visualize charts and validate before production rollout
* Refine as needed - Return to the onboarding process to make additional adjustments at any time
#### Technical Considerations
* File size limits: CSV uploads are limited to 2 GB
* Schema validation: All fundamental schema mismatches must be corrected before publication
* Draft limitations: While you can visualize sample data in drafts, some features like saving charts have limitations
For detailed instructions and best practices, see the [Model Editor](/observability/model-ui/model-editor) documentation.
# Model Editor
Source: https://docs.fiddler.ai/observability/model-ui/model-editor
Step-by-step instructions for onboarding ML models using Fiddler's UI-based editor, from dataset upload to schema validation and publication.
## Model Onboarding Process
1. Access your project
* Sign in to Fiddler and navigate to the Projects tab
* Select an existing project or create a new one
2. Add a model
* Select Add Model in the top-right corner
* Choose a model task type: Not Set, Regression, Binary Classification, or Multi-Class Classification
3. Upload and analyze your dataset
* Upload your CSV file
* Review the automatically inferred schema
* Verify flagged columns that need additional attention
4. Choose your onboarding approach
* For immediate publication: Enter a model name and select Create Model
* For draft mode: Select Save as Draft
5. Work with your draft (Draft Mode)
* Access your draft from the Draft Models tab
* Edit schema details as needed
* Publish sample data to validate your configuration
* When ready, publish the finalized model
For specific details and best practices see the [Model Schema Editing Guide](/observability/model-ui/model-schema-editing).
# Model Schema Editing
Source: https://docs.fiddler.ai/observability/model-ui/model-schema-editing
Learn how to modify numeric ranges, edit categorical features, and add metadata columns to keep your model schema aligned with evolving production data.
## Overview
This guide explains how to edit your model's schema in Fiddler to better align with production data. Schema editing helps you maintain accurate monitoring as your data evolves.
Key capabilities
* Adjust numeric feature ranges when real-world data deviates from your original sample data
* Edit categorical feature values to add or remove categories as new patterns emerge
* Add metadata columns to include additional contextual information for improved insights
* Customize histogram bin boundaries for numerical columns to better represent data distributions
### Adjusting numeric feature ranges
**Access the Schema tab**
1. Navigate to the Model Page of your desired model
2. Select the Schema tab
**Edit numeric column range**
1. Find the numeric column you want to adjust
2. Select the edit icon (✏️) next to the column name
3. In the dialog box, modify the minimum and/or maximum values
4. Select Update to save your changes
**Impact of changes**
* **Data drift metrics**: Changes apply to all data, including historical data
* A job will run to recalculate aggregates and update metrics
* **Data integrity metrics**: Changes only apply to new data going forward
#### Editing histogram bins
You can customize the histogram bin boundaries for numerical columns to better represent your data distribution (e.g., quantile-based bins instead of uniform).
**Edit bins for a numeric column**
1. Find the numeric column you want to adjust
2. Select the edit icon (✏️) next to the column name
3. In the dialog box, enter comma-separated bin boundary values in the **Bins** field
* Example: `350, 450, 550, 650, 750, 850`
* Values must be strictly increasing, span the column's \[min, max] range, and have at most 16 boundary values (15 bins)
4. To revert to auto-generated uniform bins, clear the Bins field
5. Select Update to save your changes
**Impact of changes**
* **Data drift metrics**: Changes apply to all data, including historical data
* A job will run to recalculate aggregates and update histogram metrics
* **Feature distribution charts**: Histogram visualizations will use the new bin boundaries
#### Editing categorical variables
**Access the Schema tab**
1. Navigate to the Model Page of your desired model
2. Select the **Schema** tab
**Edit categorical column**
1. Locate the categorical column you want to modify
2. Select the edit icon (✏️) next to the column name
3. Add or remove categories as needed
4. Select Update to save your changes
**Impact of changes**
* For both data drift and data integrity metrics:
* Changes only apply to new data going forward
* Historical data remains unchanged
#### Adding metadata columns
**Access the Schema tab**
1. Navigate to the Model Page of your desired model
2. Select the Schema tab
**Add a Metadata Column**
1. Select Add Metadata
2. Provide the required information:
* Column Name: Specify the name of the new metadata column
* Data Type: Choose a data type (integer, float, string, or boolean)
* Range: For numeric types, define minimum and maximum values
3. Select Add to save
**Impact of Changes**
* New metadata columns are effective immediately for new data
### Best Practices
* Analyze production data to set realistic range values and identify useful metadata columns
* Monitor metrics after adjustments to ensure changes effectively address your needs
* Use chart annotations for transparency to maintain a clear history of schema changes
Schema updates that change numeric column properties (min, max, bins) trigger re-computation of historical aggregates. Models with more than 10 million events in any environment are not eligible for these updates. Contact support for assistance with large-scale schema changes.
### Frequently Asked Questions
**Can I change column names or data types?**
No, changing column names or data types is not supported.
**What if I make a mistake?**
You can edit the values again and save the updated schema.
**How long do changes take to apply?**
Application time depends on dataset size and complexity. For example, processing 10 million rows over six months takes approximately 12 minutes.
**Can I delete a metadata column?**
No, metadata columns cannot be deleted once added.
**Can I customize histogram bins?**
Yes, you can set custom bin boundaries for numerical columns via the Bins field in the schema editor, or programmatically via the Python client using `model.update()`. Leave the field empty to use auto-generated uniform bins.
**What happens if I add a category that doesn't exist in the data?**
The category will be listed but won't impact existing calculations.
# Overview
Source: https://docs.fiddler.ai/observability/monitoring
Monitor production models in real-time with comprehensive observability
The future of AI is agentic—autonomous systems that reason, plan, and coordinate across multiple agents to solve complex problems. Fiddler Observability is built for this future, providing comprehensive monitoring across traditional ML models, LLM applications, and emerging multi-agent systems.
## The Challenge: Exponential Complexity
As AI evolves from static models to autonomous agents, observability complexity grows exponentially:
* **Multi-agent systems require [26x more monitoring resources](https://www.capgemini.com/insights/expert-perspectives/ai-lab-the-efficient-use-of-tokens-for-multi-agent-systems/)** than single-agent applications
* **Non-deterministic behavior** breaks traditional APM frameworks designed for predictable code
* **Cascading failures** across agent hierarchies create unprecedented debugging challenges
* **[90% of enterprises](https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-agent-survey.html)** cite security, trust, and compliance as top concerns for agentic AI
Fiddler provides the unified observability platform that scales from simple models to complex agentic workflows—all powered by the same family of Fiddler Centor Models.
## Agentic Observability
Fiddler's agentic observability provides hierarchical visibility into multi-agent systems, tracking the complete lifecycle of autonomous reasoning and coordination.
### The Five Observable Stages
Every agent operates through five distinct stages that require specialized monitoring:
```mermaid theme={null}
graph LR
Thought[1. Thought
Ingest, Retrieve, Interpret] --> Action[2. Action
Plan and Select Tools]
Action --> Execute[3. Execution
Perform Tasks]
Execute --> Reflect[4. Reflection
Evaluate and Adapt]
Reflect --> Align[5. Alignment
Enforce Trust & Safety]
Align -.->|Next Iteration| Thought
style Thought fill:#e1f5ff
style Action fill:#fff4e6
style Execute fill:#e6ffe6
style Reflect fill:#f0e6ff
style Align fill:#ffe6e6
```
**Stage-by-Stage Observability:**
1. **Thought**: Monitor how agents ingest data, retrieve context, and interpret information
2. **Action**: Track planning processes, tool selection, and decision-making logic
3. **Execution**: Observe task performance, API calls, and external integrations
4. **Reflection**: Capture self-evaluation, learning signals, and adaptation decisions
5. **Alignment**: Verify trust, safety, and policy enforcement at every step
### Hierarchical Monitoring Architecture
Agentic systems operate across multiple levels of abstraction. Fiddler provides observability at each layer:
```mermaid theme={null}
graph TD
App[Application Level
Overall system performance & health]
App --> Session1[Session Level
User interaction & conversation flow]
App --> Session2[Session Level
Parallel user sessions]
Session1 --> Agent1[Agent Level
Individual agent behavior & decisions]
Session1 --> Agent2[Agent Level
Multi-agent coordination]
Agent1 --> Span1[Span Level
Tool calls, LLM requests, actions]
Agent1 --> Span2[Span Level
Granular operation tracing]
Agent2 --> Span3[Span Level
Inter-agent communication]
style App fill:#0891b2,color:#fff
style Session1 fill:#06b6d4
style Session2 fill:#06b6d4
style Agent1 fill:#22d3ee
style Agent2 fill:#22d3ee
style Span1 fill:#a5f3fc
style Span2 fill:#a5f3fc
style Span3 fill:#a5f3fc
```
**Hierarchical Root Cause Analysis:**
* Trace issues from user-facing symptoms down to individual tool calls
* Understand cross-agent dependencies and coordination failures
* Analyze patterns across sessions to identify systemic issues
* Full context preservation for debugging non-deterministic behavior
### Framework & Integration Support
**Supported Frameworks:**
* **LangGraph** - Full SDK integration with native tracing
* **Strands Agents** - Strands agent application monitoring
* **OpenTelemetry** - Standard instrumentation for custom agents
* **Custom Agents** - Fiddler Client SDK for any framework
## Unified Observability Platform
All Fiddler observability capabilities—from traditional ML to agentic systems—are powered by a unified architecture built on Fiddler Centor Models:
```mermaid theme={null}
graph TB
subgraph Trust[Fiddler Centor Models]
Safety[Centor Model for Safety
11 Dimensions]
PII[Centor Model for PII/PHI
35+ Entities]
Faith[Centor Model for Faithfulness
Hallucination Detection]
Custom[Custom Metrics
Domain-Specific]
end
Trust --> ML[ML Observability
Drift, Performance, Integrity]
Trust --> LLM[LLM Observability
Quality, Safety, RAG]
Trust --> Agent[Agentic Observability
Lifecycle, Coordination, Trust]
ML --> Dash[Unified Dashboards & Analytics]
LLM --> Dash
Agent --> Dash
style Trust fill:#f0f0f0
style Safety fill:#e1f5ff
style PII fill:#e1f5ff
style Faith fill:#e1f5ff
style Custom fill:#e1f5ff
style ML fill:#fff4e6
style LLM fill:#e6ffe6
style Agent fill:#f0e6ff
style Dash fill:#ffe6e6
```
**Centor Models Advantages:**
* **10-100x faster** than general-purpose LLMs for evaluation tasks
* **Purpose-built models** optimized for safety, quality, and accuracy assessment
* **Consistent, deterministic** evaluation at scale
* **Air-gapped deployment** options for data sovereignty
* **GDPR, HIPAA, CCPA** compliant monitoring
## Core Capabilities
### LLM Monitoring
Comprehensive observability for generative AI applications with trust and safety at the core.
**Key Features:**
* **14+ Enrichment Metrics**: Auto-generated trust, safety, and quality scores
* **RAG Monitoring**: Retrieval quality, source relevance, groundedness
* **Embedding Analysis**: UMAP visualization, drift detection, clustering
* **Prompt & Response Tracking**: Full conversation history and context
**Trust & Safety Metrics:**
* Safety (toxicity, jailbreaking, harmful content)
* Privacy (PII/PHI detection across 35+ entity types)
* Quality (faithfulness, coherence, conciseness, relevance)
* Sentiment and tone analysis
* [llm](/observability/llm)
### ML Model Observability
Battle-tested monitoring for traditional machine learning models in production.
**Key Features:**
* **Drift Detection**: JSD and PSI metrics for distribution shifts
* **Performance Tracking**: Accuracy, precision, recall, F1 across all deployments
* **Data Integrity**: Missing values, type mismatches, range violations
* **Traffic Monitoring**: Volume patterns and anomaly detection
* **Vector Monitoring**: Specialized tools for embedding-based applications
**Advanced Capabilities:**
* Model segmentation and cohort analysis
* Class imbalance handling
* Statistical analysis (mean, std, distributions)
* Model version comparison
* Custom formula-based metrics
* [platform](/observability/platform)
### Analytics & Root Cause Analysis
Deep-dive investigation tools for understanding performance issues and data quality problems.
**Four-Part Analysis Experience:**
1. **Events**: Browse sample of 1,000 recent events for pattern recognition
2. **Data Drift**: Feature-by-feature drift breakdown with prediction impact
3. **Data Integrity**: Violation summaries (range, type, missing value issues)
4. **Analyze**: Interactive charts for performance and feature analytics
**Chart Types:**
* Performance Analytics (confusion matrices, prediction scatterplots)
* Feature Analytics (distributions, correlations, feature matrices)
* Metric Cards (single KPI visualization)
* [analytics](/observability/analytics)
### Dashboards & Visualization
Customizable dashboards for monitoring your entire AI portfolio.
**Features:**
* **Auto-Generated Insights**: Every model gets an out-of-the-box dashboard
* **Custom Dashboards**: Build your own views with flexible layouts
* **Model Comparison**: Side-by-side performance tracking
* **Multi-Column Plots**: Drift and integrity across all features
* **Interactive Controls**: Date ranges, timezones, bin sizes, zoom
* **Collaboration**: Save and share dashboards across teams
* [dashboards](/observability/dashboards)
### Alerting & Response
Proactive monitoring with intelligent alerting across all AI systems.
**Alert Types:**
* **Drift Alerts**: Detect distribution shifts in production data
* **Data Integrity Alerts**: Flag missing values, type mismatches, range violations
* **Performance Alerts**: Monitor accuracy degradation over time
* **Custom Metric Alerts**: Formula-based alerts for business KPIs
* **Traffic Alerts**: Volume and pattern anomaly detection
**Alert Features:**
* Warning and critical threshold configuration
* Multiple notification channels (email, Slack, PagerDuty, webhooks)
* Triggered revisions with real-time updates
* Template-based alert creation
* Alert history and audit logs
## Getting Started
### Choose Your Path
**For LLM Applications:**
* [LLM Monitoring Quick Start](/getting-started/llm-monitoring) - Set up enrichments and quality tracking
* [LLM-Based Metrics Guide](/observability/llm/llm-based-metrics) - Configure trust and safety metrics
**For Traditional ML Models:**
* [ML Observability Quick Start](/getting-started/ml-observability) - Deploy drift detection and performance monitoring
* [Monitoring Platform Guide](/observability/platform) - Configure alerts and data integrity checks
**For Agentic Systems:**
* [Agentic Monitoring Quick Start](/getting-started/agentic-monitoring) - Set up hierarchical tracing with LangGraph
* [Agentic Observability Concepts](/glossary/agentic-observability) - Understand the agent lifecycle and monitoring approach
### Additional Resources
**Platform Guides:**
* [Analytics Deep Dive](/observability/analytics) - Root cause analysis and investigation
* [Custom Dashboards](/observability/dashboards) - Build monitoring views for your team
**Integration Documentation:**
* [Python Client SDK Reference](/sdk-api/python-client/connection) - Programmatic access to all features
# Alerts
Source: https://docs.fiddler.ai/observability/platform/alerts-platform
Discover how to enhance monitoring with Alerts. Learn about alert types and how to set up and view them using the alerts tab in the navigation bar.
### Overview
Fiddler enables users to set up alert rules to track a model's health and performance over time. Fiddler alerts also enable users to dig into triggered alerts and perform root-cause analysis to identify what is causing a model to degrade. Users can set up alerts using the [Fiddler Python client SDK](/sdk-api/python-client/connection) and the Fiddler UI as demonstrated below.
### Supported Metric Types
You can get alerts for the following metrics:
* [**Traffic**](/observability/platform/traffic-platform)
* The volume of traffic received by the model over time indicates the overall system's health.
* [**Statistics**](/observability/platform/statistics)
* Metrics for monitoring basic column aggregations.
* [**Data Drift**](/observability/platform/data-drift-platform)
* Model performance can be poor if models trained on a specific dataset encounter different data in production.
* [**Data Integrity**](/observability/platform/data-integrity-platform)
* Three types of violations can occur at model inference: missing feature values, type mismatches (e.g. sending a float input for a categorical feature type) or range mismatches (e.g. sending an unknown US State for a State categorical feature).
* [**Performance**](/observability/platform/performance-tracking-platform)
* The model performance tells us how well a model performs on its task. A poorly performing model can have significant business implications.
* [**Custom Metrics**](/observability/platform/custom-metrics)
* Create your own custom metric formulas and create Alerts on those metrics giving you ultimate flexibility in alerting behavior.
### Alert Configurations
#### Comparison Types
There are two options for alert threshold comparison:
* **Absolute** — Compare the metric to an absolute value
* Example: if traffic for a given hour is less than a threshold of 1,000, trigger alert.
* **Relative** — Compare the metric to a previous period
* Example: if traffic is down 10% or more than it was at the same time one week ago, trigger alert.
#### Alert Rule Priority & Severity
* **Priority**: Whether you're setting up an alert rule to keep tabs on a model in a test environment or for production scenarios, Fiddler has you covered. Easily set the Alert Rule Priority to indicate the importance of any given Alert Rule. Users can select from Low, Medium, and High priorities.
* **Severity**: Up to two threshold values can be specified for additional flexibility. A **Critical** severity threshold value is always required when setting up an Alert Rule, and a **Warning** threshold value is optional.
### Why do we need alerts?
* It’s not possible to manually track all metrics 24/7.
* Sensible alerts are your first line of defense, and they are meant to warn about issues in production.
#### Setting up Alert Rules
To create a new alert in the Fiddler UI, click Add Alert on the Alerts tab.
1. Fill in the Alert Rule form with basic details like alert name, project, and model.
2. Choose an Alert Type (Traffic, Data Drift, Data Integrity, Performance, Statistic, or Custom Metric) and set up specific metrics, bin size, and columns.
3. Define comparison methods, thresholds, and notification preferences. Click Add Alert Rule to finish.
![]()
In order to create and configure alerts using the Fiddler Python client SDK see [Alert Configuration with Fiddler Client](/developers/python-client-guides/alerts-with-fiddler-client).
#### Alert Notification options
You can select the following notification types for your alert.
![]()
#### Delete an Alert Rule
Delete an existing alert by clicking on the overflow button (⋮) on the right-hand side of any Alert Rule record and clicking `Delete`. To make any other changes to an Alert Rule, you will need to delete the alert and create a new one with the desired specifications.
![]()
#### Triggered Alert Revisions
Say goodbye to stale alerts! Triggered Alert Revisions mark a leap forward in alert intelligence, giving you the confidence to act decisively and optimize your operations.
Alerts now adapt to changing data. If new information emerges that alters an alert's severity or value, the alert automatically updates you with highlights in the user interface and revised notifications. This approach empowers you to:
* Make informed decisions based on real-time data: No more relying on outdated or inaccurate alerts.
* Focus on critical issues: Updated alerts prioritize the most relevant information.
![Inspect Alert experience]()
Inspect Alert experience
![Triggered Alert revision experience]()
### Sample Alert Email
Here's a sample of an email that's sent if an alert is triggered:
![]()
### Integrations
The Integrations tab is a read-only view of all the integrations your Admin has enabled for use. As of today, users can configure their Alert Rules to notify them via email or Pager Duty services.
![]()
Admins can add new integrations by clicking on the setting cog icon in the main navigation bar and selecting the integration tab of interest.
![]()
#### Pause alert notification
This feature allows users to temporarily pause and resume notifications for specific alerts without affecting their evaluation and triggering mechanisms. It enhances user experience by providing efficient notification management.\\
**How to Use**
**Using the Fiddler User Interface (UI)**
* Locate the Alert Tool:
Navigate to the alert rule table and identify the desired alert.
* Toggle Notifications:
* Click the notification bell icon.
* The icon updates to indicate the new state (paused or resumed).
* Confirm Action:
* A loading indicator and a toast notification confirm the action.
**Using the Fiddler Client SDK**
For programmatic control, use the Fiddler client SDK's alert-rules method with the enable\_notification argument.
* Details:
Refer to the [Fiddler Python client SDK Reference](/sdk-api/langgraph/fiddler-client) for a complete explanation of SDK features.
**Note**
* No Impact on Evaluation:
Pausing notifications does not affect the evaluation of alert conditions. The alert tool will continue to assess conditions and trigger alerts as usual.
# Class Imbalanced Data
Source: https://docs.fiddler.ai/observability/platform/class-imbalanced-data
Explore how Fiddler uses weighting to help improve drift detection when class distribution is highly imbalanced.
### Discover How Class Imbalanced Data Impacts Feature Drift
Drift is a measure of how different the production distribution is from the baseline distribution on which the model was trained. In practice, the distributions are approximated using histograms and then compared using divergence metrics like Jensen–Shannon divergence or Population Stability Index. Generally, when constructing the histograms, every event contributes equally to the bin counts.
However, for scenarios with large class imbalance the minority class’ contribution to the histograms would be minimal. Hence, any change in production distribution with respect to the minority class would not lead to a significant change in the production histograms. Consequently, even if there is a significant change in distribution with respect to the minority class, the drift value would not change significantly.
To solve this issue, Fiddler monitoring provides a way for events to be weighted based on the class distribution. For such models, when computing the histograms, events belonging to the minority class would be up-weighted whereas those belonging to the majority class would be down-weighted.
### Solving Issues with Class Imbalanced Data
Fiddler has implemented two solutions for class imbalance use cases.
#### Workflow 1: User provided global class weights
* The user computes the class distribution on baseline data and then provides the class weights via the Model-Info object.
* Class weights can either be manually entered by the user or computed from their dataset
* To tease out drift in a class-imbalanced fraud use case, check the [class-imbalanced-notebook](/developers/tutorials/ml-monitoring/class-imbalance-monitoring-example)
#### Workflow 2: User-provided event-level weights
User provides event-level weights as a metadata column in baseline data and provides them while publishing events:
* Users will add a `_weight` column of type metadata in the model's ModelSpec.
* The baseline dataset requires this `_weight` column. Note that all rows must contain valid float values. We expect the user to enforce this assumption.
* Note that using weighting parameters requires a model output in the baseline dataset.
# Custom Metrics
Source: https://docs.fiddler.ai/observability/platform/custom-metrics
Dive into our guide to enhancing ML and LLM insights with custom metrics. Learn to define, add, access, modify, and delete custom metrics in charts and alerts.
### Overview
Custom metrics offer the capability to define metrics that align precisely with your machine learning requirements. Whether it's tracking business KPIs, crafting specialized performance assessments, or computing weighted averages, custom metrics empower you to tailor measurements to your specific needs. Seamlessly integrate these custom metrics throughout Fiddler, leveraging them in dashboards, alerting, and performance tracking.
Create user-defined metrics by employing a simple query language we call [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language). FQL enables you to leverage your model's features, metadata, predictions, and outcomes for new data fields using a rich array of aggregations, operators, and metric functions, thereby expanding the depth of your analytical insights.
### How to Define a Custom Metric
Build custom metrics effortlessly with Fiddler's intuitive Excel-formula-like syntax. Once a custom metric is defined, Fiddler distinguishes itself by seamlessly managing time granularity and ranges within the charting, dashboarding, and analytics experience. This empowers you to effortlessly adjust time range and granularity without the need to modify your query, ensuring a smooth and efficient analytical experience.
### Adding a Custom Metric
From the model schema page, you can access the model's custom metrics by clicking the **Custom Metrics** tab at the top of the page. Then click **Add Custom Metric** to add a new Custom Metric. Finally, enter the name, description, and FQL definition for your custom metric and click **Save**.
![]()
### Accessing Custom Metrics in Charts and Alerts
After your custom metric is saved, you can use it in your chart and alert definitions.
#### Charts
Set `Metric Type` to `Custom Metric` and select your desired custom metric.
![]()
#### Alerts
When creating a new alert rule, set `Metric Type` to `Custom Metric`, and under the `Metric` field select your desired custom metric or author a new metric to use.
![]()
### Modifying Custom Metrics
Since alerts can be set on Custom Metrics, making modifications to a metric may introduce inconsistencies in alerts.
> 🚧 Therefore, custom metrics cannot be modified once they are created.
If you'd like to try out a new metric, you can create a new one with a different name and definition.
### Deleting Custom Metrics
To delete a custom metric using the Python client, see [CustomMetric.delete()](/sdk-api/python-client/connection). Alternatively, from the custom metrics tab, you can delete a metric by clicking the trash icon next to the metric record.
![]()
### Examples
> 📘 Custom metrics must return either:
>
> * an aggregate (produced by aggregate functions or built-in metric functions)
> * a combination of aggregates
#### Simple Metric
Given this example use case:
> If an event is a false negative, assign a value of -40. If the event is a false positive, assign a value of -400. If the event is a true positive or true negative, then assign a value of 250.
Create a new Custom Metric with the following FQL formula:
```text theme={null}
average(if(fn(), -40, if(fp(), -400, 250)))
```
Fiddler offers many convenience functions such as `fp()` and `fn()`.
Alternatively, we could also identify false positives and false negatives the old fashioned way.
```text theme={null}
average(if(Prediction < 0.5 and Target == 1, -40, if(Prediction >= 0.5 and Target == 0, -400, 250)))
```
Here, we assume `Prediction` is the name of the output column for a binary classifier and `Target` is the name of our label column.
#### Tweedie Loss
In our next example, we provide an example implementation of the Tweedie Loss Function. Here, `Target` is the name of the target column and `Prediction` is the name of the prediction/output column.
```text theme={null}
average((Target * Prediction ^ (1 - 0.5)) / (1 - 0.5) + Prediction ^ (2 - 0.5) / (2 - 0.5))
```
#### Quantile/Percentile Metrics
Quantile metrics are essential for understanding the distribution of your data and tracking percentile-based performance indicators. Unlike averages, which can be skewed by outliers, quantiles provide robust insights into the actual behavior of your model across different percentiles.
**Use Case: Monitoring ML Model Latency**
Track the median (50th percentile) inference time to understand typical model performance:
```text theme={null}
quantile(inference_time_ms, level=0.5)
```
**Use Case: SLA Compliance Tracking**
Monitor 95th percentile latency to ensure most requests meet performance SLAs:
```text theme={null}
quantile(response_time_ms, level=0.95)
```
**Use Case: Outlier Detection**
Track the 99th percentile of prediction scores to identify extreme predictions:
```text theme={null}
quantile(prediction_score, level=0.99)
```
**Use Case: Statistical Distribution Analysis**
Compute quartiles to understand the distribution of a feature or prediction:
```text theme={null}
quantile(feature_value, level=0.25) # First quartile (Q1)
quantile(feature_value, level=0.5) # Median (Q2)
quantile(feature_value, level=0.75) # Third quartile (Q3)
```
> 📘 **Why use quantiles instead of averages?**
>
> Quantiles provide a more complete picture of your data's distribution. While averages can be heavily influenced by outliers, percentiles show you the actual values at specific points in your distribution. For example, a 95th percentile latency of 500ms means 95% of your requests complete in 500ms or less, regardless of how slow the remaining 5% might be.
#### Gini Coefficient
The Gini coefficient, derived from the Lorenz curve, measures how well a model's predicted scores rank actual values. It is available for all ML model task types, including regression. Higher values indicate better ranking ability (the normalized Gini coefficient yields 1 for a perfect model and 0 for a random model).
`gini()` is available for ML custom metrics only. It is not supported for GenAI or agentic applications.
Both `actual` and `predicted` are required keyword arguments of type `Number`. Column names must match your model schema exactly (case-sensitive).
**Use case: Basic Gini coefficient**
```text theme={null}
gini(actual=Target, predicted=Score)
```
**Use case: Normalized Gini coefficient**
The `gini()` function returns the raw Gini coefficient. To obtain the normalized version — scaled so a perfect model yields 1 — divide by the ideal Gini where actuals are perfectly ranked by themselves:
```text theme={null}
gini(actual=Target, predicted=Score) / gini(actual=Target, predicted=Target)
```
**Use case: Binary classification with categorical targets**
For binary classification models where the target column is categorical (e.g. `'yes'`/`'no'`), use an `if` expression to convert to numeric 0/1 before passing to `gini()`:
```text theme={null}
gini(actual=if(churn == 'yes', 1, 0), predicted=predicted_churn) / gini(actual=if(churn == 'yes', 1, 0), predicted=if(churn == 'yes', 1, 0))
```
**Use case: Deriving normalized Gini from AUC (binary classification only)**
For binary classification models, the normalized Gini coefficient is mathematically equivalent to `2 * AUC - 1`. You can use the built-in `auroc()` function to compute this directly without needing a separate `actual` column:
```text theme={null}
2 * auroc() - 1
```
#### Min and Max
Use `min()` and `max()` to compute the minimum or maximum value of a column or expression across all rows in a time window.
**Use case: Feature range (spread)**
Track how wide the distribution of a feature is over time. A growing range may indicate data drift.
```text theme={null}
max(Prediction) - min(Prediction)
```
**Use case: Min/max ratio**
Monitor how concentrated predictions are. A ratio close to 1 indicates a narrow spread; a ratio near 0 suggests extreme outliers are present.
```text theme={null}
min(Prediction) / max(Prediction)
```
`min(x)` and `max(x)` aggregate a single expression **across rows**. To compare multiple already-computed aggregates, use `least(...)` and `greatest(...)` instead.
#### Null-Safe Expressions
Use the `null` keyword as a return value in conditional expressions to explicitly propagate missing data rather than substituting a placeholder.
**Use case: Null-safe transformation**
Apply a transformation to a column, but return `null` for rows where the input is missing instead of coercing a default value.
```text theme={null}
average(if(is_null(Prediction), null, Prediction * 100))
```
This computes the average only over rows where `Prediction` is non-null, which is equivalent to `average(Prediction) * 100` but makes the null-handling intent explicit.
To test whether a value is null, always use `is_null(x)` or `is_not_null(x)`. The `null` keyword is intended for use as a **return value** only in the `if(condition, value, null)` expression.
### Modifying Custom Metrics
Since alerts can be set on Custom Metrics, modifying a metric may introduce inconsistencies in those alerts.
> 🚧 Therefore, custom metrics cannot be modified once they are created.
If you'd like to try out a new metric, you can create a new one with a different name and definition.
# Data Drift
Source: https://docs.fiddler.ai/observability/platform/data-drift-platform
Learn about data drift and how Fiddler can monitor your ML model data for drift to provide early detection of issues that could impact model performance.
For a complete list of all built-in ML metrics including drift, see the [ML Metrics Reference](/reference/ml-metrics-reference).
### Monitor Model Performance Changes with Fiddler's Insights
#### Monitoring Data Drift in ML Models
Model performance can be poor if models trained on a specific dataset encounter different data in production. This is called data drift and it is a metric which is available on model inputs, outputs, and custom features. On the **Insights** dashboard for your model, Fiddler gives you a diverse set of visuals to explore different metrics.
![Fiddler monitoring dashboard]()
Leverage the data drift chart to identify what data is drifting, when it’s drifting, and how it’s drifting. This is the first step in identifying possible model performance issues.
![Fiddler drift charts]()
#### Drift Metrics Details
Fiddler supports the following:
* ***Drift Metrics***
* **Jensen–Shannon distance (JSD)**
* A distance metric calculated between the distribution of a field in the [baseline](/glossary/baseline) and that same distribution for the time period of interest. For numerical columns, distributions are approximated using histograms with [configurable bin boundaries](/developers/python-client-guides/model-onboarding/customizing-your-model-schema#modifying-the-histogram-bins).
* For more information on JSD, click [here](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.jensenshannon.html).
* **Population Stability Index (PSI)**
* A drift metric based on the multinomial classification of a variable into bins or categories. For numerical columns, the bin boundaries can be customized via the column's `bins` property — see [Customizing Your Model Schema](/developers/python-client-guides/model-onboarding/customizing-your-model-schema#modifying-the-histogram-bins). By default, 10 uniform bins are auto-generated from the column's min and max values. The differences in each bin between the baseline and the time period of interest are then utilized to calculate it as follows:
> 🚧 Note
>
> There is a possibility that PSI can shoot to infinity. To avoid this, PSI calculation in Fiddler is done such that each bin count is incremented with a base\_count=1. Thus, there might be a slight difference in the PSI values obtained from manual calculations.
* ***Average Values*** – The mean of a field (feature or prediction) over time. This can be thought of as an intuitive drift score.
* ***Drift Analytics*** – You can drill down into the features responsible for the prediction drift using the table at the bottom.
* ***Feature Impact***: The contribution of a feature to the model’s predictions, averaged over the [baseline](/glossary/baseline). The contribution is calculated using random ablation feature impact.
* ***Feature Drift***: Drift of the feature, calculated using the drift metric of choice.
* ***Prediction Drift Impact***: A heuristic calculated using the product of the feature impact and the feature drift. The higher the score, the more this feature is likely to have contributed to the prediction drift.
### Determining Drift Root Cause
In the Root Cause Analysis table of your drift charts, you can select a feature to see the feature distribution for both the time period under consideration and the baseline dataset.
![Feature drift chart with feature distribution comparison between baseline and production data]()
### Why Track Data Drift?
* Data drift is a great proxy metric for **performance decline**, especially if there is delay in getting labels for production events. (e.g. In a credit lending use case, an actual default may happen after months or years.)
* Monitoring data drift also helps you stay informed about **distributional shifts in the data for features of interest**, which could have business implications even if there is no decline in model performance.
#### Taking Action on Observed Data Drift
* High drift can occur as a result of *data integrity issues* (bugs in the data pipeline), or as a result of *an actual change in the distribution of data* due to external factors (e.g. a dip in income due to COVID). The former is more in our control to solve directly. The latter may not be solvable directly, but can serve as an indicator that further investigation (and possible retraining) may be needed.
* You can drill down deeper into the data by examining it in the Analyze tab.
# Data Integrity
Source: https://docs.fiddler.ai/observability/platform/data-integrity-platform
Dive into our guide on ensuring data integrity in ML models and LLMs. Learn to monitor violations with Fiddler’s auto-generated charts and alerts.
For a complete list of all built-in ML metrics including data integrity, see the [ML Metrics Reference](/reference/ml-metrics-reference).
### Overview
ML models are increasingly driven by complex feature pipelines and automated workflows that involve dynamic data. Data is transformed from source to model input which can result in data inconsistencies and errors.
There are three types of violations that can occur at model inference: **missing values**, **type mismatches** (e.g. sending a float input for a categorical feature type) or **range violations** (e.g. sending an unknown US State for a State categorical feature).
You can monitor all these violations with auto-generated Data Integrity charts and alerts, or create your own custom alerts and charts. The time series shown below tracks the violations of data integrity constraints set up for this model.
### What Is Being Tracked?
![Monitoring chart with missing values, type violations, and range violations]()
The time series chart above tracks the violations of data integrity constraints set up for this model. Note that both raw count and percentage are available for data integrity metrics.
* **Any Violation Any Column** — The count of any type of data integrity violation over all features for a given period of time.
* **% Any Violation Any Column** — The percentage of any type of data integrity violation over all features for a given period of time.
* **NULL Count Any Column** — The count of missing value violations over all features for a given period of time.
* **Range Violation Count Any Column** — The count of range violations over all features for a given period of time.
* **Type Violation Count Any Column** — The count of data type violations over all features for a given period of time.
### Why is it being tracked?
Data integrity issues can cause incorrect data to flow into the model, leading to poor model performance and negatively impacting the business or end-user experience.
### How does it work?
Setting up constraints for individual features when they number in the tens or hundreds can be tedious. To avoid this, the schema of a model is used as a reference to detect when features in the incoming production logs deviate from expected patterns established during model training. For example, feature values may be out of range (numerical inputs) or contain unknown values (categorical inputs). The minimums, maximums, and distinct categorical values for a model's features are collected during initial model onboarding and stored in the Fiddler model's `ModelSchema`.
![Monitoring chart with missing values, type violations, and range violations]()
Fiddler will automatically generate constraints based on the data distribution of the sample data used to generate the model schema during model onboarding.
* **Type mismatch**: A data integrity violation will be triggered when the type of a feature value differs from what was specified for that feature in the model's schema.
* **Range mismatch**:
* For categorical features, a data integrity violation will be triggered when it sees any value other than the ones specified in the model's schema.
* For continuous variables, the violation will be triggered if the values are outside the range specified in the model's schema.
* For [vector datatype](/observability/platform/vector-monitoring-platform), a range mismatch will be triggered when a dimension mismatch occurs compared to the expected dimension from the model's schema.
# Embedding Visualization
Source: https://docs.fiddler.ai/observability/platform/embedding-visualization-with-umap
Dive into our guide on embedding visualization with UMAP in Fiddler. Learn to create charts, select parameters, and interact with visualizations.
![Example of embedding data rendered as a UMAP visualization chart.]()
Embedding visualization is a powerful technique for understanding and interpreting complex relationships in high-dimensional data. Reducing the dimensionality of custom features into a 2D or 3D space makes identifying patterns, clusters, and outliers easier.
In Fiddler, high-dimensional data like embeddings and vectors are ingested as a [custom feature](/observability/platform/vector-monitoring-platform#define-custom-features).
Our goal in this document is to visualize these custom features.
### UMAP Technique for Embedding Visualization
We use the [UMAP](https://umap-learn.readthedocs.io/en/latest/) (Uniform Manifold Approximation and Projection) technique for embedding visualizations. UMAP is a dimension reduction technique that is particularly good at preserving the local structure of the data, making it ideal for visualizing embeddings. We reduce the high-dimensional embeddings to a 3D space.
UMAP is supported for both Text and Image embeddings using a custom feature
### Creating an Embedding Visualization Chart
To create an embedding visualization chart, follow these steps:
1. Navigate to the **Charts** tab in your Fiddler AI instance
2. Click on the **Add Chart** button on the top right
3. In the modal, select the project that has a model with Custom features
4. Select **Embedding Visualization**.
![The Add Chart modal dialog form.]()
#### Chart Parameters
When creating an embedding visualization chart, you will need to specify the following parameters:
* Model and model version
* Embedding column
* Display columns
* Baseline
* Segment
* Date range
* Sample size
* Advanced fields
* Number of neighbors
* Minimum distance
* Distance metric
Please see below for details on these parameters.
#### Model
Select the model containing at least one embedding column. You may further refine to a model version if required.
#### Embedding Column
Choose the embedding column from your dataset that you wish to visualize.
#### Display Columns
Select the columns for which you want to display additional information when hovering over points in the visualization. When plot points are selected, these additional display columns will also be available in the data cards.
#### Baseline
Select a baseline for comparison. This is optional and will be helpful when comparing datasets, such as a pre-production dataset with a production dataset or two time periods in production.
#### Segment
Select an existing segment (or define a new segment) to filter the chart to a particular data cohort. This is optional, but it will be helpful when focusing on a specific cohort.
#### Sample Size
Decide the number of samples you want to include for performance and clarity in the visualization. Currently, sample sizes between 100 and 10,000 can be selected. In future releases, we will enable support for larger sample sizes.
#### Number of Neighbors
This parameter controls how UMAP balances local versus global structure in the data. It determines the number of neighboring points used in the manifold approximation. Low values of this parameter, such as 5, will lead UMAP to focus too much on the local structure, losing sight of the big picture. Conversely, bigger values will lead to a focus on the broader data. It is important to experiment on your dataset and use case to identify the value that provides the best results. Values from 2 to 100 are supported.
#### Minimum Distance
Controls how closely points can be placed to each other in the visualization. A smaller value (such as 0.1) allows points to cluster more tightly, revealing finer details and local structures in your data. A larger value forces points to spread out more evenly across the visualization space.
### Interactions on Embedding Visualization
### Choose Different Periods
When generating the embedding visualization, you can choose different periods of production data to analyze. To do this:
* Access the Date Range selector.
* Choose the start and end dates for the period you are interested in.
* The visualization will update to reflect the embeddings from the selected date range.
![The chart date range filter selector.]()
### Color By
The 'Color By' feature enriches the visualization by categorizing your data points using different colors based on attributes.
* Find the 'Color By' dropdown in your control panel.
* Choose a categorical feature to color-code the data points. For example, select "data source" to color the data points according to whether they are baseline or production data.
Using the 'Color By' feature can help uncover patterns in your data. For instance, in the above image, data points with varying 'target' column values demonstrate clustering, where similar values tend to group.
You can also select points to delve deeper for further inspection. This ability to interactively color and select data points may be very useful for root cause analysis.
### Zoom
Zooming in on the UMAP chart provides a closer look at clusters and individual data points.
* Use the mouse scroll wheel to zoom in or out.
* Click and drag the mouse to move the zoomed-in area around the chart.
* Zooming helps to focus on areas of interest or to distinguish between closely packed points.
### Selection of Data Points
You can select individual or groups of data points to analyze further.
* Click on a data point to select it. Or use the Selector on the top right to select multiple points
![Example of embedding data rendered as a UMAP visualization with data points colored by category.]()
### Data Cards
* Selected points will be highlighted on the chart, and details of the display columns of these cards are displayed in data cards, as shown below
* Use this feature to identify and analyze specific data points
In the following example, we use the categorical attribute "feedback", which contains three possible values: like, dislike, or None, as the legend indicates. After applying the 'color by' feature, the user selects specific data points to examine in greater detail. The selected data points are then presented as data cards below.
![Data cards displayed below UMAP chart upon data point selection.]()
### Hover on a Data Point
Hovering over a data point reveals additional information about it, providing immediate insight without the need for selection.
* Move the cursor over a data point on the chart
* A tooltip will appear, displaying the data associated with that point, such as values of different display columns
* Use this feature to quickly look up data without altering your current selection on the chart
## Saving the Chart
Once you're satisfied with your visualization, you can save the chart. This chart can then be added to a dashboard. This allows you to revisit the UMAP visualization at any time easily, either directly from the Chart or from the dashboard.
# Fiddler Query Language
Source: https://docs.fiddler.ai/observability/platform/fiddler-query-language
Explore our guide on using Fiddler Query Language to build custom metrics to drive additional business value in dashboards and extra capability in alerting.
## Overview
[Custom Metrics](/observability/platform/custom-metrics) and [Segments](/observability/platform/segments) are defined using the **Fiddler Query Language (FQL)**, a flexible set of constants, operators, and functions which can accommodate a large variety of metrics.
## Definitions
| Term | Definition |
| ------------------ | -------------------------------------------------------------------------------------- |
| Row-level function | A function which executes row-wise for a set of data. Returns a value for each row. |
| Aggregate function | A function which executes across rows. Returns a single value for a given set of rows. |
## FQL Rules
* Column names can be referenced by name either with double quotes ("my\_column") or with no quotes (my\_column).
* Single quotes (') are used to represent string values.
## Data Types
FQL distinguishes between three data types:
| Data type | Supported values | Examples | Supported Model Schema Data Types |
| --------- | --------------------------------------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| Number | Any numeric value (integers and floats are both included) | \\10\\
\2.34\\
| \\DataType.INTEGER\\
\DataType.FLOAT\\
|
| Boolean | Only `true` and `false` | \\true\\
\false\\
| `DataType.BOOLEAN` |
| String | Any value wrapped in single quotes (`'`) | \\'This is a string.'\\
\'200.0'\\
| \\DataType.CATEGORY\\
\DataType.STRING\\
|
## Constants
| Symbol | Description |
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `true` | Boolean constant for true expressions |
| `false` | Boolean constant for false expressions |
| `null` | Represents an absent or missing value. Use as a return value in conditional expressions — for example: `if(is_null(x), null, x * 2)`. To test whether a value is null, use `is_null()` or `is_not_null()`. |
## Operators
| Symbol | Description | Syntax | Returns | Examples |
| ------- | ------------------------ | --------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `^` | Exponentiation | `Number ^ Number` | `Number` | \\2.5 ^ 4\\
\(column1 - column2)^2\\
|
| `-` | Unary negation | `-Number` | `Number` | `-column1` |
| `*` | Multiplication | `Number * Number` | `Number` | \\2 \* 10\\
\2 \* column1\\
\column1 \* column2\\
\sum(column1) \* 10\\
|
| `/` | Division | `Number / Number` | `Number` | \\2 / 10\\
\2 / column1\\
\column1 / column2\\
\sum(column1) / 10\\
|
| `%` | Modulo | `Number % Number` | `Number` | \\2 % 10\\
\2 % column1\\
\column1 % column2\\
\sum(column1) % 10\\
|
| `+` | Addition | `Number + Number` | `Number` | \\2 + 2\\
\2 + column1\\
\column1 + column2\\
\average(column1) + 2\\
|
| `-` | Subtraction | `Number - Number` | `Number` | \\2 - 2\\
\2 - column1\\
\column1 - column2\\
\average(column1) - 2\\
|
| `<` | Less than | `Number < Number` | `Boolean` | \\10 \< 20\\
\column1 \< 10\\
\column1 \< column2\\
\average(column2) \< 5\\
|
| `<=` | Less than or equal to | `Number <= Number` | `Boolean` | \\10 \<= 20\\
\column1 \<= 10\\
\column1 \<= column2\\
\average(column2) \<= 5\\
|
| `>` | Greater than | `Number > Number` | `Boolean` | \\10 > 20\\
\column1 > 10\\
\column1 > column2\\
\average(column2) > 5\\
|
| `>=` | Greater than or equal to | `Number >= Number` | `Boolean` | \\10 >= 20\\
\column1 >= 10\\
\column1 >= column2\\
\average(column2) >= 5\\
|
| `==` | Equals | `Number == Number` | `Boolean` | \\10 == 20\\
\column1 == 10\\
\column1 == column2\\
\average(column2) == 5\\
|
| `!=` | Does not equal | `Number != Number` | `Boolean` | \\10 != 20\\
\column1 != 10\\
\column1 != column2\\
\average(column2) != 5\\
|
| `not` | Logical NOT | `not Boolean` | `Boolean` | \\not true\\
\not column1\\
|
| `and` | Logical AND | `Boolean and Boolean` | `Boolean` | \\true and false\\
\column1 and column2\\
|
| `or` | Logical OR | `Boolean or Boolean` | `Boolean` | \\true or false\\
\column1 or column2\\
|
## Constant functions
| Symbol | Description | Syntax | Returns | Examples |
| ------ | ----------------------------------------------------- | ------ | -------- | --------------------------- |
| `e()` | Base of the natural logarithm | `e()` | `Number` | `e() == 2.718281828459045` |
| `pi()` | The ratio of a circle's circumference to its diameter | `pi()` | `Number` | `pi() == 3.141592653589793` |
## Row-level functions
Row-level functions can be applied either to a single value or to a column/row expression (in which case they are mapped element-wise to each value in the column/row expression).
| Symbol | Description | Syntax | Returns | Examples |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------- | --------- | ------------------------------------------------------------------------------------------------------- |
| `if(condition, value1, value2)` | \Evaluates \condition\ and returns \value1\ if true, otherwise returns \value2\.\
\value1\ and \value2\ must have the same type.\
| `if(Boolean, Any, Any)` | `Any` | \\if(false, 'yes', 'no') == 'no'\\
\if(column1 == 1, 'yes', 'no')\\
|
| `length(x)` | Returns the length of string `x`. | `length(String)` | `Number` | `length('Hello world') == 11` |
| `to_string(x)` | Converts a value `x` to a string. | `to_string(Any)` | `String` | \\to\_string(42) == '42'\\
\to\_string(true) == 'true'\\
|
| `startswith(str, prefix)` | Returns `true` if `str` starts with `prefix`. | `startswith(String, String)` | `Boolean` | `startswith('abcde', 'abc')` |
| `substring(str, offset, length)` | Returns a substring of `str` of length `length` from offset `offset`. The first character has an offset of 1. | `substring(String, Number, Number)` | `String` | `substring('abcde', 2, 3) == 'bcd'` |
| `match(str, regex)` | Returns `true` if `str` matches the pattern `regex` in [re2 regular syntax](https://github.com/google/re2/wiki/Syntax). | `match(String, String)` | `Boolean` | `match('abcde', 'a.c.*e')` |
| `is_null(x)` | Returns `true` if `x` is null, otherwise returns `false`. | `is_null(Any)` | `Boolean` | \\is\_null('') == true\\
\is\_null("column1")\\
|
| `is_not_null(x)` | Returns `true` if `x` is not null, otherwise returns `false`. | `is_null(Any)` | `Boolean` | \\is\_not\_null('') == false\\
\\`is\_not\_null("column1")\\
|
| `abs(x)` | Returns the absolute value of number `x`. | `abs(Number)` | `Number` | `abs(-3) == 3` |
| `exp(x)` | Returns `e^x`, where `e` is the base of the natural logarithm. | `exp(Number)` | `Number` | `exp(1) == 2.718281828459045` |
| `log(x)` | Returns the natural logarithm (base `e`) of number `x`. | `log(Number)` | `Number` | `log(e) == 1` |
| `log2(x)` | Returns the binary logarithm (base `2`) of number `x`. | `log2(Number)` | `Number` | `log2(16) == 4` |
| `log10(x)` | Returns the common logarithm (base `10`) of number `x`. | `log10(Number)` | `Number` | `log10(1000) == 3` |
| `sqrt(x)` | Returns the positive square root of number `x`. | `sqrt(Number)` | `Number` | `sqrt(144) == 12` |
## Aggregate functions
Every Custom Metric must be wrapped in an aggregate function or be a combination of aggregate functions.
| Symbol | Description | Syntax | Returns | Examples |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sum(x)` | Returns the sum of a numeric column or row expression `x`. | `sum(Number)` | `Number` | `sum(column1 + column2)` |
| `average(x)` | Returns the arithmetic mean/average value of a numeric column or row expression `x`. | `average(Number)` | `Number` | `average(2 * column1)` |
| `count(x)` | Returns the number of non-null rows of a column or row expression `x`. | `count(Any)` | `Number` | `count(column1)` |
| `min(x)` | Returns the minimum value of a numeric column or row expression `x`. | `min(Number)` | `Number` | `min(column1)` |
| `max(x)` | Returns the maximum value of a numeric column or row expression `x`. | `max(Number)` | `Number` | `max(column1)` |
| `quantile(x, level)` | Returns the quantile (percentile) of a numeric column or row expression `x` at the specified `level`. The `level` must be a constant between 0.0 and 1.0 (defaults to 0.5 if not specified). | `quantile(Number, level=Number)` | `Number` | \\quantile(Output, level=0.5)\\
\quantile(latency\_ms, level=0.95)\\
\quantile(response\_time, level=0.99)\\
|
| `greatest(agg1, agg2, ...)` | Returns the maximum value between multiple numerical aggregates | `greatest(Number, Number, ...)` | `Number` | `greatest(sum(col1), sum(col2), sum(col3))` |
| `least(agg1, agg2, ...)` | Returns the minimum value between multiple numerical aggregates | `least(Number, Number, ...)` | `Number` | `least(sum(col1), sum(col2), sum(col3))` |
**`min(x)` / `max(x)` vs `least(...)` / `greatest(...)`**: `min(x)` and `max(x)` aggregate a single row-level expression **across rows** (e.g., `min(column1)` returns the smallest value of `column1` across all rows in the time window). `least(...)` and `greatest(...)` compare **multiple aggregate results** and return the smallest or largest among them (e.g., `least(sum(col1), sum(col2))` compares two already-computed sums).
Built-in metric functions are available for **ML models only** (classification, regression, and ranking tasks). They are not supported in custom metrics for agentic or GenAI applications. For agentic applications, use the `attribute()` function with aggregate functions instead — see [Custom Metrics for Agentic Applications](/observability/agentic/custom-metrics).
## Built-in metric functions
| Symbol | Description | Syntax | Returns | Examples |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `jsd(column, baseline)` | The Jensen-Shannon distance of column `column` with respect to baseline `baseline`. | `jsd(Any, String)` | `Number` | `jsd(column1, 'my_baseline')` |
| `psi(column, baseline)` | The population stability index of column `column` with respect to baseline `baseline`. | `psi(Any, String)` | `Number` | `psi(column1, 'my_baseline')` |
| `null_violation_count(column)` | Number of rows with null values in column `column`. | `null_violation_count(Any)` | `Number` | `null_violation_count(column1)` |
| `range_violation_count(column)` | Number of rows with out-of-range values in column `column`. | `range_violation_count(Any)` | `Number` | `range_violation_count(column1)` |
| `type_violation_count(column)` | Number of rows with invalid data types in column `column`. | `type_violation_count(Any)` | `Number` | `type_violation_count(column1)` |
| `any_violation_count(column)` | Number of rows with at least one Data Integrity violation in `column`. | `any_violation_count(Any)` | `Number` | `any_violation_count(column1)` |
| `traffic()` | Total row count. Includes null rows. | `traffic()` | `Number` | `traffic()` |
| `tp(class)` | True positive boolean state. Available for binary classification and multiclass classification models. For multiclass, `class` is used to specify the positive class. | `tp(class=Optional[String])` | `Boolean` | \\tp()\\
\tp(class='class1')\\
|
| `tn(class)` | True negative boolean state. Available for binary classification and multiclass classification models. For multiclass, `class` is used to specify the positive class. | `tn(class=Optional[String])` | `Boolean` | \\tn()\\
\tn(class='class1')\\
|
| `fp(class)` | False positive boolean state. Available for binary classification and multiclass classification models. For multiclass, `class` is used to specify the positive class. | `fp(class=Optional[String])` | `Boolean` | \\fp()\\
\fp(class='class1')\\
|
| `fn(class)` | False negative boolean state. Available for binary classification and multiclass classification models. For multiclass, `class` is used to specify the positive class. | `fn(class=Optional[String])` | `Boolean` | \\fn()\\
\fn(class='class1')\\
|
| `precision(target, threshold)` | \Precision between target and output. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `precision(target=Optional[Any], threshold=Optional[Number])` | `Number` | \\precision()\\
\precision(target=column1)\\
\precision(threshold=0.5)\\
\precision(target=column1, threshold=0.5)\\
|
| `recall(target, threshold)` | \Recall between target and output. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `recall(target=Optional[Any], threshold=Optional[Number])` | `Number` | \\recall()\\
\recall(target=column1)\\
\recall(threshold=0.5)\\
\recall(target=column1, threshold=0.5)\\
|
| `f1_score(target, threshold)` | \F1 score between target and output. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `f1_score(target=Optional[Any], threshold=Optional[Number])` | `Number` | \\f1\_score()\\
\f1\_score(target=column1)\\
\f1\_score(threshold=0.5)\\
\f1\_score(target=column1, threshold=0.5)\\
|
| `fpr(target, threshold)` | \False positive rate between target and output. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `fpr(target=Optional[Any], threshold=Optional[Number])` | `Number` | \\fpr()\\
\fpr(target=column1)\\
\fpr(threshold=0.5)\\
\fpr(target=column1, threshold=0.5)\\
|
| `auroc(target)` | \Area under the ROC curve between target and output. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `auroc(target=Optional[Any])` | `Number` | \\auroc()\\
\auroc(target=column1)\\
|
| `geometric_mean(target, threshold)` | \Geometric mean score between target and output. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `geometric_mean(target=Optional[Any], threshold=Optional[Number])` | `Number` | \\geometric\_mean()\\
\geometric\_mean(target=column1)\\
\geometric\_mean(threshold=0.5)\\
\geometric\_mean(target=column1, threshold=0.5)\\
|
| `expected_calibration_error(target)` | \Expected calibration error between target and output. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `expected_calibration_error(target=Optional[Any])` | `Number` | \\expected\_calibration\_error()\\
\expected\_calibration\_error(target=column1)\\
|
| `log_loss(target)` | \Log loss (binary cross entropy) between target and output. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `log_loss(target=Optional[Any])` | `Number` | \\log\_loss()\\
\log\_loss(target=column1)\\
|
| `calibrated_threshold(target)` | \Optimal threshold value for a high TPR and a low FPR. Available for binary classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `calibrated_threshold(target=Optional[Any])` | `Number` | \\calibrated\_threshold()\\
\calibrated\_threshold(target=column1)\\
|
| `accuracy(target, threshold)` | \Accuracy score between target and outputs. Available for multiclass classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `accuracy(target=Optional[Any], threshold=Optional[Number])` | `Number` | \\accuracy()\\
\accuracy(target=column1)\\
\accuracy(threshold=0.5)\\
\accuracy(target=column1, threshold=0.5)\\
|
| `log_loss(target)` | \Log loss score between target and outputs. Available for multiclass classification model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `log_loss(target=Optional[Any])` | `Number` | \\log\_loss()\\
\log\_loss(target=column1)\\
|
| `r2(target)` | \R-squared score between target and output. Available for regression model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `r2(target=Optional[Any])` | `Number` | \\r2()\\
\r2(target=column1)\\
|
| `mse(target)` | \Mean squared error between target and output. Available for regression model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `mse(target=Optional[Any])` | `Number` | \\mse()\\
\mse(target=column1)\\
|
| `mae(target)` | \Mean absolute error between target and output. Available for regression model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `mae(target=Optional[Any])` | `Number` | \\mae()\\
\mae(target=column1)\\
|
| `mape(target)` | \Mean absolute percentage error between target and output. Available for regression model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `mape(target=Optional[Any])` | `Number` | \\mape()\\
\mape(target=column1)\\
|
| `wmape(target)` | \Weighted mean absolute percentage error between target and output. Available for regression model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `wmape(target=Optional[Any])` | `Number` | \\wmape()\\
\wmape(target=column1)\\
|
| `map(target)` | \Mean average precision score. Available for ranking model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `map(target=Optional[Any])` | `Number` | \\map()\\
\map(target=column1)\\
|
| `ndcg_mean(target)` | \Mean normalized discounted cumulative gain score. Available for ranking model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `ndcg_mean(target=Optional[Any])` | `Number` | \\ndcg\_mean()\\
\ndcg\_mean(target=column1)\\
|
| `query_count(target)` | \Count of ranking queries. Available for ranking model tasks.\
If \target\ is specified, it will be used in place of the default target column.\
| `query_count(target=Optional[Any])` | `Number` | \\query\_count()\\
\query\_count(target=column1)\\
|
| `gini(actual, predicted)` | Gini coefficient derived from the Lorenz curve. Measures how well a model's predicted scores rank actual values. Available for ML custom metrics (all model task types, including regression). Both `actual` and `predicted` are required keyword arguments of type `Number`. To compute the **normalized Gini coefficient**, divide by the ideal Gini: `gini(actual=Target, predicted=Score) / gini(actual=Target, predicted=Target)`. For binary classification, the normalized Gini can also be derived from AUC: `2 * auroc() - 1`. | `gini(actual=Number, predicted=Number)` | `Number` | `gini(actual=Target, predicted=Score)` |
# Monitoring Platform
Source: https://docs.fiddler.ai/observability/platform/index
Dive into our guide to optimizing ML models and LLM applications with Fiddler’s monitoring tools. Learn key metrics to track data drift, performance, and more.
Fiddler Monitoring helps you detect performance issues in your production ML models and LLM applications. It provides five built-in metric types, plus support for custom metrics that you can monitor and track.
* [**Data Drift**](/observability/platform/data-drift-platform)
* [**Performance**](/observability/platform/performance-tracking-platform)
* [**Data Integrity**](/observability/platform/data-integrity-platform)
* [**Traffic**](/observability/platform/traffic-platform)
* [**Statistic**](/observability/platform/statistics)
* [**Custom Metrics**](/observability/platform/custom-metrics)
These metrics can be visualized through Fiddler's Charts and Dashboards, alerted on using Alerts, and extended with Custom Metrics, allowing for quick root cause analysis alerts.
# Model Versions
Source: https://docs.fiddler.ai/observability/platform/model-versions
Discover model versions in Fiddler. Learn structured approaches to managing related models, their use cases, capabilities, and how to create a model version.
### Overview
Model versions in Fiddler provide a structured approach to managing related models, offering enhanced efficiency in tasks such as model retraining and champion vs. challenger analyses. Rather than creating entirely new model instances for updates, users can opt for creating versions of existing models, maintaining their foundational structure while accommodating necessary changes. These changes can span schema modifications, including column additions or removals, adjustments to data types and value ranges, updates to model specifications, and refinement of task parameters or Explainable AI (XAI) settings.
### Use Cases
Model versions are particularly useful in scenarios where:
* **Model Retraining**: Models require updating with new data or improved algorithms without disrupting existing deployments.
* **Champion vs. Challenger Analyses**: Comparing the performance of different model versions to identify the most effective one for deployment.
* **Historical Tracking**: Maintaining a clear record of model iterations and changes over time for auditing or analysis purposes.
### Capabilities
With model versions, users can:
* **Maintain Model Lineage**: Easily trace the evolution of models by keeping track of different versions and their respective changes.
* **Efficiently Manage Updates**: Streamline the process of updating models by creating new versions with incremental changes, avoiding the need to re-upload entire model instances.
* **Flexibly Modify Schemas**: Modify model schemas, including column structures, data types, and other specifications, to adapt models to evolving requirements.
* **Adjust Parameters**: Refine task parameters, XAI settings, and other configurations to improve explainability, or tailor the task to better suit the model's purpose.
* **Ensure Consistency**: Ensure consistency in model deployments by managing related models within the same versioning system, facilitating comparisons and deployments.
### Example of Creating a Model Version
Utilizing from\_name() and duplicate() methods, we can efficiently create a new model version with modifications based on an existing model. First, we retrieve the existing model by specifying its name, project ID, and version. Subsequently, we duplicate this model while updating its version, transitioning, for instance, from `v3` to `v4`. Within the new version (`v4`), we tailor the value ranges of the 'Age' column to meet our requirements. Finally, the create() method is invoked to publish the newly minted model version `v4`.
```python theme={null}
# Define project ID and model name
PROJECT_ID = 'your_project_id'
MODEL_NAME = 'your_model_name'
# Retrieve the existing model version by name and project ID
existing_model = fdl.Model.from_name(name=MODEL_NAME, project_id=PROJECT_ID, version='v3')
# Duplicate the existing model to create a new version
new_model = existing_model.duplicate(version='v4')
# Modify the schema of the new version
new_model.schema['Age'].min = 18
new_model.schema['Age'].max = 60
# Create the new version of the model
new_model.create()
```
# Monitoring Charts
Source: https://docs.fiddler.ai/observability/platform/monitoring-charts-platform
Explore our guide to the monitoring charts UI. Learn how to create charts, explore functions, customize tabs, and track LLM metrics effectively.
### Getting Started with Fiddler’s Charts UI
The Chart UI on Fiddler AI’s platform is designed to help you effectively monitor and analyze model performance metrics. Navigating to the Charts tab in the navigation bar allows you to easily access the tools needed to visualize key data and track your LLM and ML models’ performance. Choose to open a previously saved chart for ongoing analysis or create a new one tailored to specific metrics, such as response faithfulness, jailbreak attempts, data drift, and more.
![Plotting traffic on Fiddlers monitoring chart.]()
#### Supported Metric Types
Monitoring charts enable you to plot any combination of the following metric types for a given model:
* [**Traffic**](/observability/platform/traffic-platform)
* Model traffic volume over time provides insights into system usage patterns and overall operational health.
* [**Statistics**](/observability/platform/statistics)
* Aggregated metrics that monitor trends and changes in your model's data columns
* [**Data Drift**](/observability/platform/data-drift-platform)
* Data drift occurs when production data differs significantly from training data, potentially degrading model performance and accuracy.
* [**Data Integrity**](/observability/platform/data-integrity-platform)
* Monitors three critical data integrity issues: missing values, incorrect data types, and out-of-range values that violate model input requirements.
* [**Performance**](/observability/platform/performance-tracking-platform)
* Model performance measures how accurately your model accomplishes its intended task, directly impacting business outcomes and decision quality.
* [**Custom Metrics**](/observability/platform/custom-metrics)
* Build tailored monitoring metrics that align with your specific business rules and success criteria using custom formulas.
### What Are the Key Features and Functions of the Monitoring Chart UI?
#### Multiple Charting Options
You can [plot up to 20 columns](#customizing-columns-for-monitoring-charts) and 6 metric queries for a model enabling you to easily perform model-to-model comparisons and plot multiple metrics in a single chart view.
#### Embedded Root Cause Analytics
Root cause analysis information covers data drift and data integrity, and performance analytics charts for binary classification, multiclass classification, and regression models.
#### Download Your Raw Data for Further Analysis
You can [easily download the raw chart data](#breakdown-summary) to CSV or parquet files. This feature allows you to analyze your data further.
### How to Create a New Monitoring Chart
To create a new monitoring chart, click on the Add Chart button on the Charts page. Search for and select the [project](/glossary#projects) to create the chart, and press Add Chart.
![How to add a new monitoring chart to a project.]()
#### Save and Share Your Monitoring Chart
Save your chart by clicking the Save button in the chart studio's top right corner. You can then share your chart by copying its link and sending it to other users who have access to the project.
#### Undo and Redo Actions
Easily control the following actions with the undo and redo buttons:
* Metric query selection
* Time range selections
* Bin size selections
To learn how to undo actions taken using the chart toolbar, see the Toolbar information in the next section.
### Exploring Chart Metric Queries & Filters
#### Creating and Managing Metric Queries
Start building your monitoring chart by creating a metric query. First, select a model from your project to analyze. Then, specify which metrics and columns you want to visualize. Note that you can only select models that exist within your current project.
Select the type of metric you want to monitor: Performance, Data Drift, Data Integrity, Statistic, Traffic, or Custom Metric. Then choose specific metrics within that category. For instance, if you're monitoring a binary classification model, you might select Performance metrics and track accuracy over time.
![Analyzing metric queries in your monitoring chart.]()
#### Customizing Columns for Monitoring Charts
If you choose to chart data drift or data integrity, you can choose to plot up to 20 different columns from the following column categories: inputs, outputs, targets, metadata, and custom features.
![Adding multiple columns to your chart.]()
#### Adding Multiple Metrics or Models
Add up to 6 metric queries that allow you to chart different metrics from one or more models in a single chart view.
![Adding multiple metrics and/or models to a single chart.]()
#### Using Chart Filters for Better Analysis
Analyze specific slices of data using three key filtering tools: chart filters, the chart toolbar, and the zoom slider. These tools work together to help you investigate important patterns and trends in your data.
![How to add filters and use capabilities to analyze your monitoring chart.]()
#### Tailored Filters
You can customize your chart view using the date range and bin size filters. The date range can be one of the pre-defined time ranges or a custom range. The bin size selected controls the frequency for which the data is displayed. So selecting Day will show daily data over the date range selected.
Customize your chart's time-based display using the date range and bin size filters. Choose from preset date ranges or define a custom period. The bin size determines how your data is grouped - for example, selecting 'Day' will show daily aggregated data across your chosen date range.
#### Toolbar Functions
The charts toolbar is made up of 5 functions:
* Drag to zoom
* Reset zoom
* Toggle to a line chart
* Toggle to a bar chart
* Undo all toolbar actions
> 📘 Note: If the zoom reset or toolbar undo is selected, this will also undo any actions taken with the zoom slider.
![Toolbar functions in a monitoring chart.]()
**Line & Bar Chart Toggles**
Toggle between line and bar chart views using the toolbar icons in the top right corner. While you can freely switch between these visualization types, please note that toolbar settings are temporary and won't be saved with your chart.
![Visualize metrics in bar chart mode.]()
**Zoom Slider**
Use the horizontal zoom bar at the bottom of the chart to focus on specific time periods. Select your desired time range by dragging the zoom bar handles, then slide the selected range left or right to analyze different periods. For example, to examine week-by-week data over six months, zoom to your preferred view and drag the range to explore different weeks.
![Focus your analysis in a specific time range.]()
#### Breakdown Summary
You can easily visualize your charts' raw data as a table within the fiddler chart studio, or download the content as a CSV or parquet file for further analysis. If you choose to chart multiple columns, as shown below, you can search for and sort by Model name, Metric name, Column name, or values for a specific date.
![Analyze your chart’s raw data as a table to get a breakdown summary.]()
### How to Customize Your Monitoring Chart
#### Adjusting Scale and Range
The Customize tab lets you adjust your chart's y-axis settings to improve data visualization. You can set minimum and maximum values to better utilize chart space, and apply logarithmic scaling to clearly display data with large variations in values.
![Customize the scale and range of the y-axis on your monitoring chart.]()
#### Assigning Y-axis for Metric Queries
Select the y-axis for your metric queries with enhanced flexibility to customize the scale and range for each axis.
![Customize the scale and range of each axis on your monitoring chart.]()
### Mastering our Chart UI to Effectively Track LLM and ML Performance
Understanding how to use the chart UI is key to measuring and maintaining the performance of LLM and ML models. Fiddler’s model monitoring tools allow you to track key metrics, identify data drift, and gain valuable insights into model behavior. Customize your Fiddler charts to proactively address issues and ensure models remain accurate, reliable, and high-performing.
# Performance Tracking
Source: https://docs.fiddler.ai/observability/platform/performance-tracking-platform
Learn to track performance with Fiddler. Discover why performance metrics matter and the steps to take when your model isn’t performing as expected.
For a complete list of all built-in ML metrics, see the [ML Metrics Reference](/reference/ml-metrics-reference).
## Overview
The model performance tells us how well a model performs on its task. A poorly performing model can have significant business implications.
![]()
## What is being tracked?
***Performance metrics***
| Model Task Type | Metric | Description |
| --------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Binary Classification | Accuracy | (TP + TN) / (TP + TN + FP + FN) |
| Binary Classification | True Positive Rate/Recall | TP / (TP + FN) |
| Binary Classification | False Positive Rate | FP / (FP + TN) |
| Binary Classification | Precision | TP / (TP + FP) |
| Binary Classification | F1 Score | 2 \* ( Precision \* Recall ) / ( Precision + Recall ) |
| Binary Classification | AUROC | Area Under the Receiver Operating Characteristic (ROC) curve, which plots the true positive rate against the false positive rate |
| Binary Classification | Binary Cross Entropy | Measures the difference between the predicted probability distribution and the true distribution |
| Binary Classification | Geometric Mean | Square Root of ( Precision \* Recall ) |
| Binary Classification | Calibrated Threshold | A threshold that balances precision and recall at a particular operating point |
| Binary Classification | Data Count | The number of events where target and output are both not NULL. ***This will be used as the denominator when calculating accuracy***. |
| Binary Classification | Expected Calibration Error | Measures the difference between predicted probabilities and empirical probabilities |
| Multi Classification | Accuracy | (Number of correctly classified samples) / ( Data Count ). Data Count refers to the number of events where the target and output are both not NULL |
| Multi Classification | Log Loss | Measures the difference between the predicted probability distribution and the true distribution, in a logarithmic scale |
| Regression | Coefficient of determination (R-squared) | Measures the proportion of variance in the dependent variable that is explained by the independent variables |
| Regression | Mean Squared Error (MSE) | Average of the squared differences between the predicted and true values |
| Regression | Mean Absolute Error (MAE) | Average of the absolute differences between the predicted and true values |
| Regression | Mean Absolute Percentage Error (MAPE) | Average of the absolute percentage differences between the predicted and true values |
| Regression | Weighted Mean Absolute Percentage Error (WMAPE) | The weighted average of the absolute percentage differences between the predicted and true values |
| Ranking | Mean Average Precision (MAP)—for binary relevance ranking only | Measures the average precision of the relevant items in the top-k results |
| Ranking | Normalized Discounted Cumulative Gain (NDCG) | Measures the quality of the ranking of the retrieved items, by discounting the relevance scores of items at lower ranks |
## Why is it being tracked?
* Model performance tells us how well a model is doing on its task. A poorly performing model can have significant business implications.
* The volume of decisions made on the basis of the predictions give visibility into the business impact of the model.
## What steps should I take based on this information?
* For changes in model performance—again, the best way to cross-verify the results is by checking the [Data Drift Tab](/observability/platform/data-drift-platform). Once you confirm that the performance issue is not due to the data, you need to assess if the change in performance is due to temporary factors, or due to longer-lasting issues.
* You can check if there are any lightweight changes you can make to help recover performance—for example, you could try modifying the decision threshold.
* Retraining the model with the latest data and redeploying it is usually the solution that yields the best results, although it may be time-consuming and expensive.
# Segments
Source: https://docs.fiddler.ai/observability/platform/segments
Learn to use model segments for monitoring diverse dimensions. Define, add, and modify segments to gain valuable insights into specific cohorts and dimensions.
### Overview
A segment, sometimes referred to as a cohort or slice, represents a distinct subset of model values crucial for performance analysis and troubleshooting. Model segments can be defined using various model dimensions, such as specific time periods or sets of features. Analyzing segments proves invaluable for understanding or troubleshooting specific cohorts of interest, particularly in tasks like bias detection, where overarching datasets might obscure statistical intricacies.
### How to Define a Segment
Fiddler makes it easy to define custom segments using either the Fiddler UI or the Fiddler Python client. Instructions for both approaches are covered in more detail below. In either case, Fiddler Segments are constructed using the [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language).
You can use any of the constants, operators, and functions mentioned in the page linked above in a Segment definition.
However, every Segment definition must return **a boolean row-level expression**. In other words, each inference will either satisfy the segment expression and thus belong to the segment or it will not.
### Examples
Let us illustrate further by providing a few examples. A segment can be defined by:
* A condition on some column (e.g. `age > 50`)
* A condition on some combination of columns (e.g. `(age / max_age) < 1.0`)
For details on all supported functions, see the [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language) page.
### Adding a Segment Using the Python Client
```python theme={null}
model_id = 'your_model_identifier'
segment_name = 'your_segment_name'
model = fdl.Model.get(id_=model_id)
# Use Fiddler Query Language (FQL) to define your custom segments
segment = fdl.Segment(
name=segment_name,
model_id=model.id,
definition="Age < 60",
description='Users with Age under 60',
).create()
```
### Applied Segments
When using segments in the UI for Analytics or Monitoring Charts, applied segments offer a flexible way to define segments on the fly for exploratory analysis. These segments are not saved to the model by default, but will persist locally if the chart they are applied to is saved.
At any time, an applied segment can be saved to the model. However, once a segment is saved to the model, it cannot be altered.
### Modifying Saved Segments
Since alerts can be set on Segments, making modifications to a Segment may introduce inconsistencies in alerts.
> 🚧 Therefore, **Saved segments cannot be modified once they are created**.
If you'd like to experiment with a new segment, you can create one with a different definition or use applied segments within charts.
#### Deleting Segments
Delete a segment using the [Python client SDK](/sdk-api/python-client/connection). Alternatively, from the segments tab, you can delete a segment by clicking the trash icon next to the segment record.
![]()
# Statistics
Source: https://docs.fiddler.ai/observability/platform/statistics
Discover Fiddler’s statistical metrics guide to monitor column aggregations. Learn what’s tracked, how to monitor metrics, and how to set up alerts.
### Overview
Fiddler supports some simple statistic metrics which can be used to monitor basic aggregations over columns. These can be particularly useful when you have a custom metadata field which you would like to monitor over time in addition to Fiddler's other out-of-the-box metrics.
### What is being tracked?
Specifically, we support:
* **Average**: Takes the arithmetic mean of a numeric column
* **Sum**: Calculates the sum of a numeric column
* **Frequency**: Shows the count of occurrences for each value in a categorical or boolean column
### Monitoring Statistic Metrics
#### Charting Statistic Metrics
These metrics can be accessed in Charts and Alerts by selecting the Statistic Metric Type.
![]()
#### Alerting on Statistic Metrics
Alert rules can be established based on statistics too. Like an alert rule, these can be setup using the Fiddler UI, the Fiddler python client or using Fiddler's RESTful API.
![]()
# Template-Based Alerts
Source: https://docs.fiddler.ai/observability/platform/template-based-alerts
Learn how to create and deploy template-based alerts in Fiddler using Google Sheets and YAML configurations for efficient model monitoring.
The Template-Based Alerts feature streamlines alert rule configuration and generation, significantly reducing manual effort. By using a Google Sheet to input key data and generate a recipe YAML file, this feature enables the efficient and accurate creation of alert rules. This approach ensures more relevant alerts, enhancing monitoring capabilities and enabling proactive issue resolution.
This guide outlines the process of generating and deploying template-based alerts to monitor your model's performance and data.
### Overview
Template-based alerts involve two primary steps:
1. **Generate a YAML configuration:** This involves using a provided Google Sheet to define your alert rules.
2. **Consume the YAML configuration:** This utilizes a Python notebook from the Alert Recipes repository to create the alerts in your Fiddler deployment.
### Step 1: Generating the Alert Recipe YAML Configuration
The first step is to create a YAML configuration file containing your desired alert rules. This is done using a pre-configured Google Sheet.
#### 1.1 Copying the Alert Recipes Sheet
To begin, you need to copy the Alert Recipes Google Sheet. This ensures you have a personal, editable version without affecting the base sheet.
* **Action**: Copy the entire content of the Alert Recipes sheet, including the recipe generator script it contains.
#### 1.2 Understanding the Sheet Structure
Once copied, note that the Google Sheet is organized into several worksheets:
* `AlertRecipes`: This is the main sheet where you define individual alert rules.
* `RecipeParameters` and `NotificationPacks`: These sheets need to be edited according to your preferences.
* `MetricIdMap` and `ThresholdTypeMap`: These sheets are locked and should not be edited unless you have a deep understanding of their function.
#### 1.3 Configuring Recipe Parameters
The `RecipeParameters` sheet is crucial for defining the context of your alerts.
![RecipeParameters Sheet]()
* **Columns:**
* **Project name**: Enter the name of your project.
* **Model name**: Enter the name of your model.
* **High-priority features**: List the key features for this project and model, separated by commas.
* **Medium-priority features**: Provide a comma-separated list of features you consider medium priority for this project and model.
* **Low-priority features**: Provide a comma-separated list of features you consider low priority for this project and model.
**Important Note**: The script only considers the first row of this sheet. Any additional rows will be ignored.
#### 1.4 Configuring notification packs
Notification packs allow you to bundle email, PagerDuty, or webhook notifications to be applied to your alert rules.
![NotificationPacks Sheet]()
* **Definition**: A notification pack is a collection of one or more notification types (email, PagerDuty, webhook) that can be attached to an alert rule.
* **Customization**:
* You can create custom notification packs in new rows.
* Each new pack can contain any combination of email, PagerDuty service, and webhook UUID.
* **Important Note**: The webhook user ID must be the actual UUID of the webhook, not its service name.
* **Availability**: Any notification packs you define or modify in this sheet will be available for selection in the main `AlertRecipes` sheet.
#### 1.5 Defining Alert Rules in AlertRecipes
The `AlertRecipes` sheet is where you configure individual alert rules.
![AlertRecipes Sheet]()
* **Columns:**
* **Monitor type**: Specifies the type of metric on which the alert rule will be created.
* **Alert rule name**: If provided, this name will be used for the alert rule. Otherwise, a name will be inferred from the monitor type, appended with a unique identifier.
* **Time window**: Defines the time bin over which evaluations will happen and alerts will be raised.
* **Enabled (Y or N)**: Determines whether this alert rule will be included in the generated YAML configuration.
* **Features**: Select from 'High-Priority', 'Low-Priority', 'Medium-Priority', or 'All'. The specific features are chosen from the RecipeParameters sheet based on your selection.
* **Threshold algo**: Currently, only 'standard deviation' is supported.
* **Critical multiplier**: This multiplier is used to calculate the critical threshold dynamically. For example, a value of `2` means a critical alert will be raised if the metric value is 2 standard deviations away from the mean of the reference data.
* **Warning multiplier**: This multiplier is used to calculate the warning threshold dynamically. For example, a value of `1` means a warning alert will be raised if the metric value is 1 standard deviation away from the mean of the reference data. These multipliers are customizable (e.g., 0.5, 1.5).
* **Notification pack**: Select a pre-defined notification pack from the `NotificationPacks` sheet to attach to this alert rule.
#### 1.6 Generating the Alert Recipe YAML
Once you have configured the `RecipeParameters`, `NotificationPacks`, and `AlertRecipe` sheets, you can generate the YAML file.
* **Action**: Click the "Generate Alert Recipe" button in the `AlertRecipe` Sheet.
* **Permissions**: You will be prompted to grant specific permissions for the script to run. These permissions are necessary for the script to access your Google Drive (to save the YAML file) and to run within the Google Sheet environment.
* **Output**: The YAML file will be generated. Copy its content.
### Step 2: Consuming the YAML Configuration
After generating the YAML file, the next step is to use it to create the alert rules in your Fiddler deployment. This is done using the Alert Recipes repository and a Python notebook.
#### 2.1 Setting Up the Alert Recipes Repository
You'll need to work within the Alert Recipes repository.
* **File Creation**: Inside the root of the repository, create two files:
* `config.py`
* `alert_recipes.yaml`
* **Configuration**:
* `config.py`: Populate this file with the correct URL and authentication token for your Fiddler deployment. You can copy the structure from existing example files(`config.py.example`).
* `alert_recipes.yaml`: Paste the content of the YAML file you copied from the Google Sheet into this file.
#### 2.2 Running the Python Notebook
With the configuration files set up, you can now run the Python notebook to create the alert rules.
* **Action**: Open the `create_alert_rules.ipynb` notebook within the Alert Recipes repository.
* **Execution**: Run all cells within the notebook.
* **Verification**: The logs in the notebook will show which new alert rules have been created. You can then verify the creation of these rules within your Fiddler deployment.
# Traffic
Source: https://docs.fiddler.ai/observability/platform/traffic-platform
Learn how Fiddler tracks your ML and GenAI models' traffic patterns and when to take action when traffic patterns deviate from normal.
Traffic as a service metric gives you basic insights into the operational health of your model's service in production.
![Example of a Fiddler Chart displaying a model's traffic.]()
## What is Being Tracked?
* ***Traffic*** — The volume of traffic received by the model over time.
## Why is it Being Tracked?
* Traffic is a basic high-level metric that informs us of the model's output activity.
## What Steps Should I Take When I See an Outlier?
* A dip or spike in traffic needs to be investigated. For example, a dip could be due to a production model server going down; a spike could be an adversarial attack.
# Vector Monitoring
Source: https://docs.fiddler.ai/observability/platform/vector-monitoring-platform
Dive into our vector monitoring guide to learn about model inputs represented as vectors and how to use Fiddler's custom features to monitor and detect drift.
## Detecting Drift in Multi-Dimensional ML and GenAI Model Data
Many modern machine learning systems use input features that cannot be represented as a single number, such as text or image data. These complex features are typically represented by high-dimensional vectors obtained through vectorization methods like text embeddings generated by NLP models. Fiddler users often need to monitor groups of univariate features together and detect data drift in multidimensional feature spaces.
To address these needs, Fiddler provides vector monitoring capabilities that enable you to define [custom features](#define-custom-features) and use advanced methods for monitoring data drift in multidimensional spaces.
You can define custom features by grouping columns together in baseline and inference data. For NLP or image data, you can define custom features using columns that contain embedding vectors.
### Define Custom Features
Users can use the Fiddler client to define one or more custom features. Custom features can be specified by:
Use the Fiddler client to define one or more custom features. You can specify custom features in three ways:
1. Group dataset columns that need to be monitored together as a vector (custom\_feature\_1, custom\_feature\_2)
2. Use existing embedding vectors with the source column (custom\_feature\_3, custom\_feature\_4)
3. Define an enrichment that instructs Fiddler to generate embedding vectors automatically on ingestion (custom\_feature\_5)
After you define and pass a list of custom features to Fiddler, Fiddler runs a clustering-based data drift detection algorithm for each custom feature. The system calculates a corresponding drift value between the baseline and published events for the selected time period.
```python theme={null}
from fiddler import CustomFeature, TextEmbedding, ImageEmbedding
# Group columns into vectors
custom_feature_1 = CustomFeature.from_columns(
['f1', 'f2', 'f3'], custom_name='vector1'
)
custom_feature_2 = CustomFeature.from_columns(
['f1', 'f2', 'f3'], n_clusters=5, custom_name='vector2'
)
# Use existing embeddings
custom_feature_3 = TextEmbedding(
name='Document Text Embedding', column='text_embedding_col', source_column='text'
)
custom_feature_4 = ImageEmbedding(
name='Image Embedding', column='image_embedding_col', source_column='image_url'
)
# Define automated text embedding enrichment
custom_feature_5 = TextEmbedding(
name='Document Text Embedding',
source_column='doc_col',
column='Enrichment Unstructured Embedding',
n_tags=10,
)
```
### Passing Custom Features List to ModelSpec
After you define custom features for vector monitoring, add them to the `ModelSpec` and onboard the Model to Fiddler.
```python theme={null}
from fiddler import ModelSpec, Model, Project
model_spec = ModelSpec(
inputs=[
'creditscore',
'geography',
'gender',
'age',
'tenure',
'balance',
'numofproducts',
'hascrcard',
'isactivemember',
'estimatedsalary',
'doc_col',
],
outputs=['predicted_churn'],
targets=['churn'],
# Note: Embedding columns you pass in must be included with the metadata columns.
metadata=['customer_id', 'timestamp', 'text_embedding_col', 'image_embedding_col'],
custom_features=[
custom_feature_1,
custom_feature_2,
custom_feature_3,
custom_feature_4,
custom_feature_5,
],
)
model = Model.from_data(
name='your_model_name',
project_id=Project.from_name('your_project_name').id,
source=sample_df,
spec=model_spec,
task=model_task,
task_params=task_params,
event_id_col=id_column,
event_ts_col=timestamp_column,
)
model.create()
```
## Understanding Drift Detection Algorithm
Fiddler's vector monitoring uses a clustering-based approach to detect drift in multidimensional spaces:
1. **Baseline clustering:** The system analyzes your baseline data to identify natural clusters using k-means clustering
2. **Production comparison:** New production data is compared against these established clusters
3. **Drift calculation:** The system calculates drift scores based on changes in cluster distributions and centroid distances
### Performance considerations
* **Computational cost:** Vector monitoring requires more computational resources than univariate monitoring
* **Memory usage:** High-dimensional vectors and clustering algorithms increase memory requirements
* **Processing time:** Drift calculations may take longer for large datasets or high-dimensional vectors
### Best practices
* **Choose appropriate cluster numbers:** Start with 3-8 clusters and adjust based on your data's natural groupings
* **Monitor cluster stability:** Regularly review cluster formations to ensure they remain meaningful
* **Set reasonable thresholds:** Establish drift thresholds that balance sensitivity with false positive rates
## Risk Considerations for AI/ML Applications
When implementing vector monitoring, consider these potential risks:
* **Bias amplification:** Clustering algorithms may amplify existing biases in your training data
* **Concept drift detection:** Traditional clustering may miss subtle concept drift that affects model performance
* **Interpretability challenges:** High-dimensional clusters can be difficult to interpret and explain to stakeholders
> **Note:** For a complete example of NLP monitoring, see our [NLP monitoring quick start guide](/developers/tutorials/ml-monitoring/simple-nlp-monitoring-quick-start), which demonstrates embedding generation for unstructured inputs.
## Related topics
* [Data drift platform](/observability/platform/data-drift-platform)
* [Enrichments](/observability/llm/enrichments)
* [Embedding visualization with UMAP](/observability/platform/embedding-visualization-with-umap)
# Guardrails
Source: https://docs.fiddler.ai/protection/guardrails
Fiddler Guardrails is a powerful solution designed to serve as the first-line of defense to protect enterprises from costly GenAI and LLM risks in real-time environments.
## Overview
Fiddler Guardrails provide real-time protection for GenAI applications—including LLM-powered systems and agentic AI workflows—by detecting and preventing harmful content, PII leaks, and hallucinations before they reach your users. Built on Fiddler Centor Models—Fiddler's proprietary small language models (SLMs)—Guardrails deliver enterprise-grade security with low-latency, high-throughput performance optimized for production environments.
**Use Fiddler Guardrails to:**
* Detect and block harmful or inappropriate content across 11 safety dimensions
* Prevent personally identifiable information (PII) leaks in user inputs and model outputs
* Identify hallucinations in retrieval-augmented generation (RAG) applications
* Protect against prompt injection and jailbreaking attempts
## What Fiddler Guardrails Can Moderate
Fiddler Guardrails are powered by Fiddler Centor Models, and you can apply them to moderate or block three categories of risk:
* **Safety** - Detect harmful, toxic, or jailbreaking content
* **Hallucination (faithfulness)** - Identify hallucinations in RAG applications
* **PII/PHI** - Detect and redact sensitive information
Guardrails are designed for **real-time content blocking** with more sensitive thresholds than enrichments used for monitoring and analytics. See the [Enrichments guide](/observability/llm/enrichments) for batch processing and monitoring use cases.
## Getting Started with Fiddler Guardrails
### Prerequisites
* **Fiddler Environment** - Access to a Fiddler environment with Guardrails enabled
* **API Key** - Generate your API key from Settings → [Credentials](/reference/administration/settings#credentials)
* **HTTP Client** - Python 3.8+ with `requests` library, cURL, or any HTTP client
Guardrails can be invoked directly via REST API from any programming language. The examples below demonstrate usage with cURL and Python.
***
## Safety
For safety moderation, Fiddler Guardrails use the Centor Model for Safety, which evaluates the safety of text along eleven different dimensions: `illegal, hateful, harassing, racist, sexist, violent, sexual, harmful, unethical, jailbreaking, roleplaying`.
This model requires a single string input for evaluation and outputs 11 distinct scores (floats between 0 and 1). **Starting in release 26.13, scores are calibrated so that `0.5` is the default decision threshold across all 11 dimensions — a score of 0.5 or above indicates unsafe content.** Lower thresholds increase sensitivity but may over-block; tune the threshold for your data and risk tolerance.
**Threshold Guidance:** Starting in release 26.13, the Centor Model for Safety is calibrated so a single decision threshold of **0.5** applies across all 11 dimensions — no per-dimension tuning required. Lower the threshold to increase sensitivity (at the cost of more false positives), or raise it to reduce false positives. For monitoring use cases with enrichments, see [Safety Enrichment](/observability/llm/enrichments#safety) for monitoring thresholds.
**Migrating from earlier releases:** If you previously used a threshold of `0.1` (calibrated for the older, uncalibrated model), adopt `0.5` for the 26.13 calibrated model — keeping `0.1` will over-block benign content. Choosing a stricter (lower) threshold remains a deliberate option for high-sensitivity use cases.
### Safety Example Code
```bash theme={null}
curl --location 'https://{fiddler_endpoint}/v3/guardrails/ftl-safety'
--header 'Content-Type: application/json'
--header 'Authorization: Bearer {token}'
--data '{
"data": {
"input": "I am a dangerous person who will be wreaking havoc upon the world!!!"
}
}'
```
```python theme={null}
import requests
import json
token = "YOUR_FIDDLER_TOKEN_HERE"
url = "FIDDLER_ENDPOINT_HERE"
payload = json.dumps(
{
"data": {
"input": "I am a dangerous person who will be wreaking havoc upon the world!!!"
}
}
)
headers = {"Content-Type": "application/json", "Authorization": f"Bearer {token}"}
response = requests.request(
"POST", f"{url}/v3/guardrails/ftl-safety", headers=headers, data=payload
)
print(response.text)
```
**Sample Response**
```json theme={null}
{
"fdl_harmful": 0.119,
"fdl_violent": 0.073,
"fdl_unethical": 0.043,
"fdl_illegal": 0.016,
"fdl_sexual": 0.005,
"fdl_racist": 0.003,
"fdl_jailbreaking": 0.002,
"fdl_harassing": 0.001,
"fdl_hateful": 0.001,
"fdl_sexist": 0.001,
"fdl_roleplaying": 0.051
}
```
**Interpreting Safety Scores:**
Each dimension returns a score between 0 and 1:
* **Closer to 0** - Safe content
* **Closer to 1** - Unsafe content
* **0.5 or above** - Meets or exceeds the calibrated default decision threshold (tunable for your use case)
***
## Hallucination (faithfulness)
For hallucination moderation, Fiddler Guardrails use the Centor Model for Faithfulness, which evaluates the accuracy and reliability of facts presented in AI-generated text responses by comparing them to provided context documents. This model uses `response` and `context` inputs.
**Not to be confused with RAG Faithfulness.** For real-time blocking, Fiddler Guardrails use the Centor Model for Faithfulness (`ftl_response_faithfulness`). RAG Faithfulness is a separate LLM-as-a-Judge evaluator available in Agentic Monitoring and Experiments for diagnostic evaluation. See [RAG Health Diagnostics](/concepts/rag-health-diagnostics) for details.
This model requires a response string and contextual documents as input. The model outputs a single faithfulness score (float between 0 and 1). **Set a threshold of \< 0.5 for detection (any value less than 0.5 indicates unfaithful content).**
**Threshold Guidance:** A score closer to **0** means unfaithful (the LLM hallucinated relative to the provided context), while a score closer to **1** means faithful (the LLM output did not hallucinate and is well-grounded in the provided context). For real-time guardrails, a threshold of **0.5** strikes a balance between sensitivity and accuracy.
### Faithfulness Example Code
```bash theme={null}
curl --location 'https://{fiddler_endpoint}/v3/guardrails/ftl-response-faithfulness'
--header 'Content-Type: application/json'
--header 'Authorization: Bearer {token}'
--data '{
"data": {
"response": "The Yorkshire Terrier and the Cavalier King Charles Spaniel are both small breeds of companion dogs.",
"context": "The Yorkshire Terrier is a small dog breed of terrier type, developed during the 19th century in Yorkshire, England, to catch rats in clothing mills.The Cavalier King Charles Spaniel is a small spaniel classed as a toy dog by The Kennel Club and the American Kennel Club"
}
}'
```
```python theme={null}
import requests
import json
token = "YOUR_FIDDLER_TOKEN_HERE"
url = "FIDDLER_ENDPOINT_HERE"
payload = json.dumps(
{
"data": {
"response": "The Yorkshire Terrier and the Cavalier King Charles Spaniel are both small breeds of companion dogs.",
"context": "The Yorkshire Terrier is a small dog breed of terrier type, developed during the 19th century in Yorkshire, England, to catch rats in clothing mills.The Cavalier King Charles Spaniel is a small spaniel classed as a toy dog by The Kennel Club and the American Kennel Club",
}
}
)
headers = {"Content-Type": "application/json", "Authorization": f"Bearer {token}"}
response = requests.request(
"POST",
f"{url}/v3/guardrails/ftl-response-faithfulness",
headers=headers,
data=payload,
)
print(response.text)
```
**Sample Response**
```json theme={null}
{
"fdl_faithful_score": 0.194
}
```
**Interpreting Faithfulness Scores:**
* **0.0 - 0.49** - Unfaithful (likely hallucination - block or flag for review)
* **0.5 - 1.0** - Faithful (response is well-supported by the provided context)
The example above shows a score of **0.194**, which is **below the 0.5 threshold**, indicating the response may contain hallucinated information not supported by the context.
***
## PII/PHI
For PII/PHI moderation, Fiddler Guardrails use the Centor Model for PII/PHI, which detects, flags, and redacts PII leakage in both user inputs and model responses.
PII/PHI moderation supports a comprehensive set of label types, including: `person, address, email, email address, credit card number, credit card expiration date, cvv, cvc, bank account number, iban, social security number, date of birth, ip address, phone number, mobile phone number, landline phone number, passport number, driver's license number, tax identification number, cpf, cnpj, account number, license plate number, fax number, website, digital signature, postal code`. See the [PII & PHI Tutorial](/developers/tutorials/guardrails/guardrails-pii) for the full entity list.
The Centor Model for PII/PHI supports a different entity set than the [PII Enrichment](/observability/llm/enrichments#personally-identifiable-information) (which uses Presidio). For monitoring and batch processing, see the PII Enrichment documentation.
**PHI Detection also supported.** Fiddler Guardrails also detect Protected Health Information (PHI) for HIPAA compliance, including: `medication, medical condition, health insurance number, health insurance id number, national health insurance number, birth certificate number, serial number`. Pass `"entity_categories": "PHI"` in your request body. See the [PII & PHI Tutorial](/developers/tutorials/guardrails/guardrails-pii) for full entity lists and example code.
This model accepts a single text string and returns all detected PII spans with their labels, confidence scores, and character offsets.
### PII/PHI Example Code
```bash theme={null}
curl --location 'https://{fiddler_endpoint}/v3/guardrails/sensitive-information'
--header 'Content-Type: application/json'
--header 'Authorization: Bearer {token}'
--data '{
"data": {
"input": "Some of my colleagues share their contact info as well. Jane Smith's email is jane.smith@company.com, and her office is located at 432 Oak Avenue, Suite 210, Chicago, IL 60611. You can call her mobile at 312-555-7890."
}
}'
```
```python theme={null}
import requests
import json
token = "YOUR_FIDDLER_TOKEN_HERE"
url = "FIDDLER_ENDPOINT_HERE"
payload = json.dumps(
{
"data": {
"input": "Some of my colleagues share their contact info as well. Jane Smith's email is jane.smith@company.com, and her office is located at 432 Oak Avenue, Suite 210, Chicago, IL 60611. You can call her mobile at 312-555-7890."
}
}
)
headers = {"Content-Type": "application/json", "Authorization": f"Bearer {token}"}
response = requests.request(
"POST", f"{url}/v3/guardrails/sensitive-information", headers=headers, data=payload
)
print(response.text)
```
**Sample Response**
```json theme={null}
{
"fdl_sensitive_information_scores": [
{
"score": 0.987,
"label": "email",
"start": 78,
"end": 100,
"text": "jane.smith@company.com"
},
{
"score": 0.945,
"label": "address",
"start": 131,
"end": 175,
"text": "432 Oak Avenue, Suite 210, Chicago, IL 60611"
},
{
"score": 0.987,
"label": "mobile phone number",
"start": 204,
"end": 216,
"text": "312-555-7890"
}
]
}
```
**Response Fields:**
* `score` - Confidence score (0.0 to 1.0)
* `label` - Entity type (e.g., "email", "social security number")
* `text` - The detected sensitive information
* `start` / `end` - Character positions in the input text
***
## Summary
Fiddler Guardrails provide real-time protection for GenAI applications, powered by Fiddler Centor Models, across three categories of risk:
* **Safety** - Detect harmful content across 11 safety dimensions with a calibrated default decision threshold of 0.5 (tunable)
* **Hallucination (faithfulness)** - Identify hallucinations in RAG applications with a recommended threshold of \< 0.5
* **PII/PHI** - Detect and redact PII and PHI across a comprehensive set of entity types
All guardrails use Fiddler Centor Models—Fiddler's proprietary small language models—optimized for sub-second latency in production environments.
## Next Steps
* **Quick Start** - [Get started with Fiddler Guardrails in 15 minutes](/developers/quick-starts/guardrails-quick-start)
* **API Reference** - [Complete Guardrails API documentation](/sdk-api/guardrails-api-reference)
* **Tutorials** - Explore detailed tutorials for [Safety](/developers/tutorials/guardrails/guardrails-safety), [PII](/developers/tutorials/guardrails/guardrails-pii), and [Faithfulness](/developers/tutorials/guardrails/guardrails-faithfulness)
* **Concepts** - [Understand Fiddler Centor Models and enrichments](/observability/llm/enrichments)
* **Monitoring** - [Integrate guardrails with LLM monitoring](/observability/llm/enrichments#safety)
# Guardrails FAQ
Source: https://docs.fiddler.ai/protection/guardrails-faq
Find answers to common questions about Fiddler Guardrails, including setup, implementation, and general information for protecting your LLM applications.
## Fiddler Guardrails: Frequently Asked Questions
### Deployment & Latency
**Q: Why am I getting much higher latencies than advertised?**
**A:** Fiddler Guardrails achieves its lowest latencies when requests are made to a Fiddler environment within the same network boundaries as your application. Network distance between your application and the Fiddler environment adds round-trip time.
Latency spikes can also occur due to temporary usage surges, even though we provision for peak capacity. If you’re consistently experiencing higher latencies, please contact us on the #fiddler-guardrails-support channel.
**Q: What does this error code mean? How do I fix it?**
**A:** Please visit the [Guardrails tutorials](/developers/tutorials/guardrails) for detailed information about error codes and their solutions.
### Service Features
**Q: What's included in Fiddler Guardrails?**
**A:** Fiddler Guardrails includes:
* Real-time safety moderation across 11 dimensions
* Faithfulness (hallucination) detection for RAG applications
* PII/PHI detection and redaction
* Integration options with NVIDIA NeMo Guardrails and LangChain
### General Information & Future Plans
**Q: What are Fiddler Centor Models?**
**A:** Integral to the Fiddler AI Observability and Security platform, Fiddler Centor Models are a series of proprietary, fine-tuned small language models (SLMs) that enable high-quality LLM monitoring and scoring in live environments with the fastest guardrails in the industry. For added security, Centor Models can be deployed in VPC or air-gapped environments, ensuring enterprises maintain strict data control and safeguard LLM applications.
**Q: How does Fiddler Guardrails use Fiddler Centor Models?**
**A:** At \<100ms latency, Fiddler Guardrails is the fastest in the industry. It leverages the scoring of Fiddler Centor Models to evaluate prompts and responses and moderate harmful outputs for hallucination, toxicity, and jailbreaks. Simply specify your desired metric thresholds and let Fiddler Guardrails handle the enforcement.
**Q: What languages are supported?**
**A:** Fiddler Guardrails currently only supports English. Based on user feedback, we're evaluating support for additional languages.
**Q: How can I stay updated on service improvements?**
**A:** Stay informed about regular updates to Fiddler Guardrails by:
* Following our [product updates page](/changelog/product-releases)
* Subscribing to our [newsletter](https://www.fiddler.ai/blog#subscribe)
* Joining the [Fiddler Community Slack](https://www.fiddler.ai/slackinvite)
**Q: Where can I share feedback or request features?**
**A:** We welcome your input! Please reach out to our team on Slack in the #fiddler-guardrails-support channel. Your feedback helps us prioritize improvements.
**Q: Is Fiddler Guardrails an experimental or beta product?**
**A:** No, Fiddler Guardrails is a fully supported feature of the Fiddler AI Observability and Security platform.
# Guardrails Quick Start
Source: https://docs.fiddler.ai/protection/guardrails-quick-start
Set up access to Fiddler Guardrails in your Fiddler environment and make your first API call to protect your LLM applications.
Fiddler Guardrails is available via the REST API. This page covers how to set up access and make your first call. For a full implementation walkthrough with code for every guardrail type, see the [Guardrails developer quick start](/developers/quick-starts/guardrails-quick-start).
## Prerequisites
* Access to a Fiddler environment with Guardrails enabled
* Permission to generate an API key in that environment
## Set up access
Sign in to Fiddler. Guardrails is served from the same environment at `https://{fiddler_endpoint}/v3/guardrails/*`. If you are not sure whether Guardrails is enabled for your environment, contact your Fiddler representative.
Generate a Fiddler API key from **Settings → [Credentials](/reference/administration/settings#credentials)**. You pass this key as a bearer token on every Guardrails request.
Send a request to a `/v3/guardrails/*` endpoint with your API key in the `Authorization` header. The example below calls the safety guardrail.
```bash theme={null}
curl --location 'https://{fiddler_endpoint}/v3/guardrails/ftl-safety' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {token}' \
--data '{
"data": {
"input": "I am a dangerous person who will be wreaking havoc upon the world!!!"
}
}'
```
For API specifications and model details, see the [Guardrails API Reference](/sdk-api/guardrails-api-reference). For implementation examples in Python and other languages, see the [Guardrails developer quick start](/developers/quick-starts/guardrails-quick-start).
### Explore Fiddler Guardrails Features
* [Frequently Asked Questions](/protection/guardrails-faq)
* Notebook Tutorials:
* [**Safety Guardrails Quick Start**](/developers/tutorials/guardrails/guardrails-safety)
* [**PII Detection Quick Start**](/developers/tutorials/guardrails/guardrails-pii)
* [**Faithfulness Quick Start**](/developers/tutorials/guardrails/guardrails-faithfulness)
# Overview
Source: https://docs.fiddler.ai/protection/index
Ensure AI safety and compliance with guardrails and monitoring
Fiddler Protect provides comprehensive AI safety through real-time guardrails, continuous monitoring, and intelligent alerting—all powered by Fiddler Centor Models. Built on purpose-optimized evaluation models that are 10-100x faster than general-purpose LLMs, Fiddler Protect helps you prevent harmful outputs, detect privacy violations, ensure factual accuracy, and maintain compliance across your AI applications.
## Protection Layers
Fiddler Protect operates through multiple complementary layers of defense:
```mermaid theme={null}
graph TB
Input[User Input] --> RTG[Real-Time Guardrails]
RTG --> Safety{Safety Check}
RTG --> PII{PII Detection}
RTG --> Secrets{Secret Detection}
RTG --> Faith{Faithfulness}
Safety -->|Pass| LLM[LLM Processing]
PII -->|Pass| LLM
Secrets -->|Pass| LLM
Faith -->|Pass| LLM
Safety -->|Fail| Block[Blocked/Filtered]
PII -->|Fail| Block
Secrets -->|Fail| Block
Faith -->|Fail| Block
LLM --> Output[AI Response]
Output --> Monitor[Continuous Monitoring]
Monitor --> Enrich[Safety Enrichments]
Monitor --> Drift[Data Integrity & Drift]
Enrich --> Alert{Alert Threshold?}
Drift --> Alert
Alert -->|Yes| Notify[Send Notifications]
Alert -->|No| Log[Audit Log]
Notify --> Log
style RTG fill:#e1f5ff
style Monitor fill:#fff4e6
style Alert fill:#ffe6e6
style Block fill:#ffcccc
style Output fill:#e6ffe6
```
### Real-Time Guardrails
Fast, pre-deployment protection that evaluates and filters AI inputs and outputs before they reach users.
#### Safety Guardrails
Detect and prevent harmful content across 11 safety dimensions:
* **Harmful Behaviors**: Jailbreaking attempts, prompt injection, illegal content promotion
* **Offensive Content**: Hate speech, harassment, racism, sexism
* **Inappropriate Content**: Violence, explicit sexual content, unethical scenarios
* **Risk Categories**: Toxic language, dangerous information, inappropriate roleplaying
The Centor Model for Safety provides real-time evaluation with sub-second latency, making it practical for high-volume production deployments. Each dimension returns a confidence score (0-1 range) allowing you to set custom thresholds based on your risk tolerance.
#### PII/PHI Detection
Protect user privacy by automatically detecting sensitive information in model inputs and outputs:
* **Personal Identifiers**: Names, dates of birth, email addresses, phone numbers
* **Financial Data**: Credit card numbers, bank accounts, tax IDs
* **Government IDs**: Social security numbers, passport numbers, driver's licenses
* **Healthcare Information**: Medications, medical conditions, health insurance numbers, birth certificate numbers (HIPAA compliance)
* **Custom Entities**: Organization-specific sensitive patterns (employee IDs, API keys, internal codes)
The Centor Model for PII/PHI identifies a comprehensive set of PII and PHI entity types, returning exact positions and confidence scores for each detected instance.
#### Secret Detection
Prevent credentials and API keys from leaking through LLM prompts or responses:
* **LLM Provider Keys**: Anthropic, OpenAI, Hugging Face, Replicate
* **Cloud Credentials**: AWS, Google, Azure, DigitalOcean, Heroku
* **Source Control Tokens**: GitHub, GitLab, Bitbucket personal access tokens
* **Infrastructure Secrets**: HashiCorp Vault, Terraform Cloud, Supabase, Vercel
* **Communication Keys**: Slack, Discord, SendGrid, Twilio, Mailgun
* **Generic Secrets**: JWT tokens, PEM private keys, database connection strings, high-entropy strings
The Centor Secret Detector uses \~42 compiled regex patterns plus Shannon entropy analysis to catch both known credential formats and unknown high-entropy secrets. Returns character-level spans for precise redaction.
#### Faithfulness & Accuracy
Prevent hallucinations and ensure AI responses stay grounded in source material:
* **Hallucination Detection**: Evaluate whether AI responses are factually consistent with provided context
* **RAG Validation**: Verify that generated content accurately reflects retrieved documents
* **Source Grounding**: Ensure answers don't introduce information not present in reference materials
The Centor Model for Faithfulness compares AI-generated responses against source documents to detect when models fabricate information or misrepresent facts.
#### Performance Advantage
All guardrail models are **10-100x faster** than general-purpose LLMs like GPT-4 for evaluation tasks, enabling:
* Real-time filtering without noticeable latency
* High-volume production deployment
* Cost-effective safety at scale
* No external API dependencies for enhanced security
### Continuous Monitoring
Post-deployment protection through ongoing analysis of production traffic.
#### Safety Enrichments
Monitor your production AI systems for safety and quality issues:
* **Toxicity Detection**: Identify toxic language patterns using advanced classification models
* **Profanity Filtering**: Detect offensive language in both inputs and outputs
* **PII Monitoring**: Continuously scan for privacy violations in production data
* **Sentiment Analysis**: Track emotional tone and user experience signals
* **Custom Classification**: Apply organization-specific categorization rules
These enrichments run automatically on your production traffic, providing visibility into safety issues that may emerge over time or in specific contexts.
#### Data Integrity & Drift
Protect against data quality issues and distribution changes:
* **Missing Value Detection**: Identify incomplete inputs that may cause unpredictable behavior
* **Type Validation**: Catch data type mismatches (e.g., strings where numbers expected)
* **Range Monitoring**: Detect out-of-range values that violate expected constraints
* **Distribution Drift**: Track when production data diverges from training or baseline data
* **Embedding Visualization**: Use 3D UMAP plots to visually identify anomalies in high-dimensional data
### Alerting & Response
Automated notification system for proactive risk management:
* **Drift Alerts**: Detect when production data or model behavior changes significantly
* **Data Integrity Alerts**: Flag missing values, type mismatches, or range violations
* **Performance Alerts**: Monitor for model accuracy degradation over time
* **Custom Metric Alerts**: Define formula-based alerts for business-specific KPIs
* **Traffic Alerts**: Track system volume for capacity planning and anomaly detection
Configure alerts with warning and critical thresholds, and route notifications to your team via email, Slack, PagerDuty, or custom webhooks. All alerts include triggered revisions that update in real-time as new data arrives.
## Fiddler Centor Models
All protection capabilities are powered by Fiddler Centor Models—purpose-built evaluation models optimized for safety, quality, and accuracy assessment. Unlike general-purpose LLMs repurposed for evaluation, Centor Models are specifically designed for these tasks, delivering:
* **Speed**: 10-100x faster evaluation than GPT-4
* **Security**: Air-gapped deployment options with no external API dependencies
* **Privacy**: Full data sovereignty for GDPR, HIPAA, and CCPA compliance
* **Reliability**: Consistent, deterministic evaluation at scale
```mermaid theme={null}
graph LR
subgraph Trust[Fiddler Centor Models]
Fast1[Centor Model for Safety
11 Dimensions]
Fast2[Centor Model for PII/PHI
PII & PHI Entities]
Fast3[Centor Secret Detector
~42 Secret Types]
Fast4[Centor Model for Faithfulness
Hallucination Detection]
end
App[Your Application] --> Fast1
App --> Fast2
App --> Fast3
App --> Fast4
Fast1 --> Score1[Safety Scores
0-1 per dimension]
Fast2 --> Score2[Entity Detection
+ Positions]
Fast3 --> Score3[Secret Spans
Label + Position]
Fast4 --> Score4[Faithfulness Score
0-1 range]
Score1 --> Decision{Policy
Enforcement}
Score2 --> Decision
Score3 --> Decision
Score4 --> Decision
Decision -->|Safe| Allow[✓ Allow Response]
Decision -->|Unsafe| Block[✗ Block Response]
style Trust fill:#f0f0f0
style Fast1 fill:#e1f5ff
style Fast2 fill:#e1f5ff
style Fast3 fill:#e1f5ff
style Allow fill:#e6ffe6
style Block fill:#ffcccc
```
## Key Use Cases
### Content Safety
Prevent your AI applications from generating harmful, offensive, or inappropriate content:
* Filter toxic language and hate speech in real-time
* Block jailbreaking attempts and prompt injection attacks
* Detect violent, sexual, or otherwise inappropriate outputs before they reach users
* Maintain brand reputation by ensuring responsible AI behavior
### Privacy Protection
Safeguard user privacy and maintain compliance with data protection regulations:
* Automatically detect and redact PII in both inputs and outputs
* Support HIPAA compliance through PHI detection
* Configure custom entity detection for organization-specific sensitive data
* Monitor for privacy violations in production traffic
### Accuracy & Truthfulness
Ensure your AI systems provide accurate, grounded information:
* Detect hallucinations in RAG applications before presenting to users
* Validate that generated content reflects source documents accurately
* Monitor for factual consistency across your AI responses
* Maintain trust by preventing fabricated or misleading information
### Regulatory Compliance
Meet compliance requirements while maintaining comprehensive audit trails:
* GDPR compliance through PII detection and data sovereignty options
* HIPAA compliance with PHI detection and air-gapped deployment
* Complete audit logging of all safety events and policy enforcement
* Bias and fairness monitoring for regulatory reporting
## Getting Started
### Quick Start Guides
Get up and running with Fiddler Protect in minutes:
* [**Guardrails Quick Start**](/protection/guardrails-quick-start) - Set up real-time protection
* [**Safety Guardrails Quick Start**](/developers/tutorials/guardrails/guardrails-safety) - Implement content safety filters
* [**PII Detection Quick Start**](/developers/tutorials/guardrails/guardrails-pii) - Protect user privacy
* [**Secret Detection Quick Start**](/developers/tutorials/guardrails/guardrails-secrets) - Prevent credential leakage
* [**Faithfulness Quick Start**](/developers/tutorials/guardrails/guardrails-faithfulness) - Prevent hallucinations
### Documentation & References
Dive deeper into Fiddler Protect capabilities:
* [**Guardrails API Reference**](/sdk-api/guardrails-api-reference) - Complete API documentation
* [**LLM-Based Metrics**](/observability/llm/llm-based-metrics) - Quality and safety metrics
* [**Enrichments Guide**](/observability/llm/enrichments) - Continuous monitoring enrichments
* [**Alerts Platform**](/observability/platform/alerts-platform) - Configure alerting and notifications
* [**Guardrails FAQ**](/protection/guardrails-faq) - Common questions and answers
### Additional Resources
Learn more about the underlying technology:
* [**Centor Models Overview**](/glossary/centor-models) - Learn about the evaluation platform
* [**Guardrails Glossary**](/glossary/guardrails) - Key concepts and terminology
***
**Ready to get started?** Try the [Guardrails Quick Start](/protection/guardrails-quick-start) to implement your first safety guardrail in minutes.
# LiteLLM Guardrails
Source: https://docs.fiddler.ai/protection/litellm-guardrails
Use Fiddler as a guardrail provider for the LiteLLM proxy gateway — blocking and redacting PII and secrets in real time before requests reach your LLM.
# LiteLLM Guardrails
## Overview
Fiddler implements the [LiteLLM Generic Guardrail API](https://docs.litellm.ai/docs/adding_provider/generic_guardrail_api) spec, allowing you to plug Fiddler's guardrails directly into a LiteLLM proxy gateway. Once configured, every LLM request routed through the proxy is checked by Fiddler before it reaches the model.
**Fiddler checks for:**
* **Secrets** — API keys, tokens, credentials, and connection strings in prompts
* **PII** — Personal identifiable information (names, emails, phone numbers, SSNs, credit cards, etc.)
Each check can independently block or redact — see [Check Behavior](#check-behavior) for details.
***
## How It Works
```mermaid theme={null}
graph LR
Client["Client
(coding agent, app, curl)"]
Client -->|"POST /v1/chat/completions"| LiteLLM["LiteLLM Proxy"]
LiteLLM -->|"POST /v3/guardrails/litellm/beta/litellm_basic_guardrail_api
(pre_call / post_call)"| Fiddler["Fiddler Guardrails API"]
Fiddler -->|"action: BLOCKED"| LiteLLM
Fiddler -->|"action: NONE or GUARDRAIL_INTERVENED → proceed"| LiteLLM
LiteLLM -->|"Sanitized request"| LLM["LLM Provider
(OpenAI, Vertex AI, etc.)"]
style Client fill:#e1f5ff
style LiteLLM fill:#fff4e6
style Fiddler fill:#ffe6e6
style LLM fill:#e6ffe6
```
LiteLLM calls the Fiddler endpoint with the extracted text from the request (or response). Fiddler returns one of three actions:
| Action | Meaning | LiteLLM behavior |
| ---------------------- | -------------------------- | ------------------------------------------ |
| `NONE` | No issues detected | Forwards the request unchanged |
| `GUARDRAIL_INTERVENED` | Sensitive content redacted | Forwards the request with redacted `texts` |
| `BLOCKED` | Request must not proceed | Returns an error to the client |
***
## Supported Modes
Fiddler supports all three LiteLLM guardrail modes. Set `mode` in your proxy config to one or more of:
| Mode | When it runs | What gets checked | Can modify input? |
| ------------- | ----------------------------- | ----------------- | -------------------------- |
| `pre_call` | Before the LLM call | PII + Secrets | Yes (redact in place) |
| `post_call` | After the LLM call | PII + Secrets | Yes (redact in place) |
| `during_call` | In parallel with the LLM call | PII + Secrets | No (accept or reject only) |
`during_call` runs the guardrail concurrently with the LLM call — the response is held until the check completes, but latency is hidden behind the LLM round-trip. Use it when you want lower end-to-end latency and don't need input redaction (block-only is sufficient).
You can combine modes. For example, `mode: [pre_call, post_call]` scans both input and output:
```yaml theme={null}
guardrails:
- guardrail_name: "fiddler"
litellm_params:
guardrail: generic_guardrail_api
mode: [pre_call, post_call]
```
### Supported Endpoints
The guardrail runs on all LiteLLM proxy endpoints that carry text content, including `/v1/chat/completions` (OpenAI format) and `/v1/messages` (Anthropic format). LiteLLM extracts text from the request and forwards it to the Fiddler endpoint regardless of the upstream provider format.
***
## What Gets Scanned
| Check | `pre_call` | `post_call` | `during_call` | Free-text messages | Tool-call args |
| ----------- | ---------- | ----------- | ------------- | ------------------ | -------------- |
| **PII** | ✓ | ✓ | ✓ | Redact | Block |
| **Secrets** | ✓ | ✓ | ✓ | Redact | Block |
* **Free-text messages** — user, assistant, and system messages. PII/secrets are redacted in place (e.g. `[REDACTED EMAIL_ADDRESS]`).
* **Tool-call arguments** — structured JSON in `tool_calls[].function.arguments`. Detections are **always blocked** (cannot safely redact inside structured JSON — see [Tool Call Handling](#tool-call-handling)).
***
## Quick Start
### Step 1: Configure LiteLLM
Add the Fiddler guardrail to your LiteLLM `config.yaml`:
```yaml theme={null}
guardrails:
- guardrail_name: "fiddler"
litellm_params:
guardrail: generic_guardrail_api
mode: [pre_call]
api_base: https:///v3/guardrails/litellm
default_on: true
# Block if the Fiddler endpoint is unreachable (network error / 5xx).
# Covers transport failures. Default: fail_closed.
unreachable_fallback: fail_closed
headers:
Authorization: "Bearer "
additional_provider_specific_params:
# Block if an internal check fails to complete (inference timeout/error).
# Covers detector failures. Default when omitted: open.
failure_mode: closed
# Wall-clock timeout (seconds) for the guardrail check. How long the
# endpoint blocks before returning. Default: 12. Max: 60.
timeout: 12
pii:
enabled: true
config:
threshold: 0.8
mode: redact # "redact" (default) or "block"
secrets:
enabled: true
config:
mode: redact # "redact" (default) or "block"
```
LiteLLM appends `/beta/litellm_basic_guardrail_api` to `api_base` automatically. The full endpoint called will be `https:///v3/guardrails/litellm/beta/litellm_basic_guardrail_api`.
### Step 2: Start the proxy
```bash theme={null}
litellm --config config.yaml --port 4000
```
### Step 3: Verify
Send a request with a known secret:
```bash theme={null}
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer " \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"My API key is sk-ant-api03-abcdefghijklmnopqrstu"}]}'
```
The secret will be redacted to `[REDACTED ANTHROPIC_API_KEY]` before the request reaches the model.
***
## Per-Request Control
With `default_on: true` (the recommended config above), the guardrail runs on every request automatically. You can also control it per request.
### Selective activation (`default_on: false`)
Set `default_on: false` in the proxy config, then activate the guardrail on individual requests by passing `guardrails` in the request body:
```json theme={null}
{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "..."}],
"guardrails": ["fiddler"]
}
```
Requests without `"guardrails": ["fiddler"]` will bypass the guardrail entirely.
***
## Check Behavior
Each check is configured entirely in the LiteLLM proxy's `additional_provider_specific_params`:
* **`enabled`** (`true` / `false`) — whether the check runs at all.
* **`mode`** (`redact` / `block`) — what happens when a detection is found. Default is `redact` for PII and secrets.
* **`threshold`** and **`entities`** — detection sensitivity.
If **any** of `pii` or `secrets` is specified in `additional_provider_specific_params`, only the checks explicitly listed with `enabled: true` will run. To tune one check without silently disabling the other, list both keys with explicit `enabled: true` or `enabled: false`.
### Secrets
Detects credentials, API keys, and tokens.
| Mode | Behavior in free-text messages | Behavior in tool call arguments |
| ------------------ | ---------------------------------------------------------- | --------------------------------------------------------- |
| `redact` (default) | Value replaced with `[REDACTED ]`, request forwarded | **Always blocked** — cannot safely redact structured JSON |
| `block` | Request blocked entirely | Request blocked |
```yaml theme={null}
secrets:
enabled: true
config:
mode: redact # "redact" (default) or "block"
```
### PII
Detects personal identifiable information.
| Mode | Behavior in free-text messages | Behavior in tool call arguments |
| ------------------ | ---------------------------------------------------------- | --------------------------------------------------------- |
| `redact` (default) | Value replaced with `[REDACTED ]`, request forwarded | **Always blocked** — cannot safely redact structured JSON |
| `block` | Request blocked entirely | Request blocked |
Entities checked by default include: `person`, `email`, `phone number`, `social security number`, `credit card number`, `bank account number`, `passport number`, `driver's license number`, `date of birth`, `address`, `ip address`, `iban`, `cvv`, `cvc`, `tax identification number`, `digital signature`, `license plate number`, `postal code`, and more. For the complete list see the [PII Detection tutorial](/developers/tutorials/guardrails/guardrails-pii#supported-entity-types).
```yaml theme={null}
pii:
enabled: true
config:
mode: redact # "redact" (default) or "block"
threshold: 0.8 # detection confidence threshold (default: 0.8)
entities: # optional: override default entity list
- person
- email
- phone number
```
***
## Tool Call Handling
When an LLM's `tool_calls` contain PII or secrets, Fiddler **always blocks** rather than redacts.
**Why:** Tool call arguments are structured JSON that the downstream application will parse and execute. Replacing a value like an email address with `[REDACTED EMAIL_ADDRESS]` would cause `send_email` to attempt delivery to a nonsensical address — producing unpredictable behavior that is worse than blocking outright.
**Example — blocked tool call:**
```json theme={null}
{
"role": "assistant",
"tool_calls": [{
"id": "call_abc123",
"type": "function",
"function": {
"name": "send_email",
"arguments": "{\"to\": \"jane.doe@example.com\", \"body\": \"Phone: 555-867-5309\"}"
}
}]
}
```
Fiddler returns:
```json theme={null}
{"action": "BLOCKED", "blocked_reason": "PII or secrets detected in tool call arguments. Cannot redact tool call arguments — blocking request."}
```
LiteLLM then translates this into an error for the client (HTTP 400 on LiteLLM ≥ 1.88.0, HTTP 500 on earlier versions).
### Tool Results
Fiddler does **not** claim redaction coverage for tool results. Whether tool result content reaches the scanner depends on the LiteLLM gateway: LiteLLM must include the tool result in the `texts[]` it forwards to Fiddler. This is not guaranteed for all gateway configurations or versions, and is known not to work on customer-managed or self-hosted gateways that do not extract `role:tool` / `tool_result` messages into `texts[]`.
Additionally, tool results are typed as `any` in the GenAI semantic conventions — they can be plain strings, JSON objects, or arrays. Even when the text does reach the scanner, in-place character-span redaction on a serialized JSON payload is not safe if the downstream application re-parses the result as structured data.
For reliable protection against secrets in tool output, use `mode: block` on secrets and treat tool result content as untrusted.
***
## Failure Mode
Two settings control what happens when the guardrail cannot complete a check — one at the LiteLLM proxy level, one at the Fiddler endpoint level. Both should be set for end-to-end fail-closed.
### `unreachable_fallback` (LiteLLM proxy)
Controls what happens when the Fiddler endpoint is **unreachable** (network error, HTTP 502/503/504).
```yaml theme={null}
litellm_params:
unreachable_fallback: fail_closed # default: fail_closed
```
| Value | Behavior |
| ----------------------- | ------------------------------------------ |
| `fail_closed` (default) | Block the request — the LLM never sees it |
| `fail_open` | Allow the request through without scanning |
This is a [standard LiteLLM Generic Guardrail API setting](https://docs.litellm.ai/docs/adding_provider/generic_guardrail_api).
### `failure_mode` (Fiddler endpoint)
Controls what happens when an **internal** check fails to complete — an inference timeout, an inference server error, or an unexpected exception in the detection pipeline. The LiteLLM proxy cannot see these failures because the Fiddler endpoint still returns HTTP 200.
```yaml theme={null}
additional_provider_specific_params:
failure_mode: closed # default when omitted: open
```
| Value | Behavior |
| ---------------- | ------------------------------------------------------------------------- |
| `open` (default) | Allow the request through — the failed check is logged but does not block |
| `closed` | Block the request — unscanned content never reaches the LLM |
The default is `open` for backward compatibility. For security-sensitive deployments, set `failure_mode: closed`. When both `unreachable_fallback: fail_closed` and `failure_mode: closed` are set, no request can bypass scanning — whether the failure is at the transport or detector level.
***
## Limits & Timeouts
### Text Length
The maximum total text length scanned per request is controlled by the `GUARDRAILS_MAX_TEXT_LENGTH` environment variable on the Fiddler server (default: **50,000 characters**). Messages are concatenated until this cap is reached; text beyond the cap is handled as follows:
| Content type | Fail-open (`failure_mode: open`) | Fail-closed (`failure_mode: closed`) |
| ------------------- | ----------------------------------------------- | ------------------------------------ |
| Free-text messages | Skipped (unscanned, request proceeds) | **Blocked** |
| Tool-call arguments | **Always blocked** (regardless of failure mode) | **Blocked** |
### Timeouts
The guardrail check has a single customer-facing timeout: the **wall-clock deadline** for the whole check pipeline — how long the endpoint blocks before returning. Set it per request via `timeout` (seconds) in `additional_provider_specific_params`:
```yaml theme={null}
additional_provider_specific_params:
timeout: 12 # default: 12, max: 60
```
What you set is what you get — there is no hidden padding. Omitted or invalid values fall back to the default (12s); values above the maximum are clamped to 60s.
When the deadline is reached before the checks complete, the outcome is governed by `failure_mode` — under `open` the request proceeds unscanned, under `closed` it is blocked.
The internal inference-call timeouts (`GUARDRAILS_GATEWAY_READ_TIMEOUT`, default 10s, and `GUARDRAILS_GATEWAY_CONN_TIMEOUT`, default 3s) are server-side plumbing for the connection to the detection models. They are lower than the wall-clock `timeout` and are not part of the customer-facing config.
***
## Observed Behavior
**Minimum required version: LiteLLM ≥ 1.88.0.** Earlier versions return HTTP 500 for all guardrail blocks due to a bug in `GuardrailRaisedException` (missing `status_code` attribute). The Fiddler team identified and fixed this upstream in [BerriAI/litellm#27617](https://github.com/BerriAI/litellm/pull/27617). The fix shipped in LiteLLM 1.88.0. On 1.88.0+, blocked requests correctly return HTTP 400.
| Scenario | Action | HTTP status | Notes |
| ------------------------------------------------------------------- | ---------------------------------------------- | -------------------- | -------------------------------------------------------------- |
| Secret in user message | `GUARDRAIL_INTERVENED` | 200 | Redacted in-place; LLM receives sanitized text |
| PII in user message | `GUARDRAIL_INTERVENED` | 200 | Redacted in-place |
| PII in tool call arguments | `BLOCKED` | 400 (500 pre-1.88.0) | Cannot redact structured JSON |
| Detector timeout (`failure_mode: open`) | `NONE` | 200 | Request proceeds unscanned |
| Detector timeout (`failure_mode: closed`) | `BLOCKED` | 200 | Request blocked; unscanned content never reaches LLM |
| Guardrail service unreachable (`unreachable_fallback: fail_closed`) | *(LiteLLM-side block — Fiddler never reached)* | 400 | LiteLLM rejects the request before forwarding to the guardrail |
### Client-Facing Error Body
When a request is blocked, LiteLLM returns an error to the client. Your application should handle this shape:
```json theme={null}
{
"error": {
"message": "Guardrail fiddler rejected the request. PII or secrets detected in tool call arguments. Cannot redact tool call arguments — blocking request.",
"type": "None",
"param": "None",
"code": "400"
}
}
```
The `message` field contains the `blocked_reason` from Fiddler's response, prefixed by LiteLLM with the guardrail name. HTTP status is 400 on LiteLLM ≥ 1.88.0.
***
## API Reference
### Endpoint
```
POST /v3/guardrails/litellm/beta/litellm_basic_guardrail_api
```
Authentication: `Authorization: Bearer `
### Request
```json theme={null}
{
"texts": ["string"],
"tool_calls": [{"id": "...", "type": "function", "function": {"name": "...", "arguments": "..."}}],
"tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}],
"structured_messages": [{"role": "user", "content": "..."}],
"images": ["base64-encoded-string"],
"request_data": {"user_api_key_alias": "my-key", "user_api_key_team_id": "team-123"},
"request_headers": {"content-type": "application/json"},
"input_type": "request",
"additional_provider_specific_params": {
"failure_mode": "closed",
"timeout": 12,
"pii": {"enabled": true, "config": {"threshold": 0.8, "mode": "redact"}},
"secrets": {"enabled": true, "config": {"mode": "redact"}}
},
"litellm_call_id": "uuid",
"litellm_trace_id": "uuid",
"litellm_version": "1.88.0"
}
```
Only `texts` and `tool_calls[].function.arguments` are scanned by guardrail checks. The remaining fields are accepted for LiteLLM protocol compatibility.
| Field | Type | Required | Description |
| ------------------------------------- | --------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `texts` | `string[]` | Yes | Extracted text strings from the request messages (scanned) |
| `request_data` | `object` | Yes | LiteLLM virtual key metadata (user, team, org, key hash) |
| `input_type` | `"request"` \| `"response"` | Yes | Whether this is a pre-call or post-call check |
| `tool_calls` | `object[]` \| `null` | No | Tool invocations in OpenAI format (arguments are scanned) |
| `tools` | `object[]` \| `null` | No | Tool definitions in OpenAI format (not scanned) |
| `structured_messages` | `object[]` \| `null` | No | Full messages array in OpenAI format (not scanned) |
| `images` | `string[]` \| `null` | No | Base64-encoded images (not scanned) |
| `request_headers` | `object` \| `null` | No | Inbound request headers (not scanned) |
| `additional_provider_specific_params` | `object` \| `null` | No | Per-check configuration plus `failure_mode` and `timeout` (see [Quick Start](#quick-start), [Failure Mode](#failure-mode), and [Timeouts](#timeouts)) |
| `litellm_call_id` | `string` \| `null` | No | LiteLLM call ID for tracing |
| `litellm_trace_id` | `string` \| `null` | No | LiteLLM trace ID for tracing |
| `litellm_version` | `string` \| `null` | No | LiteLLM library version |
### Response
Fields with `null` values are omitted from the wire (`exclude_none=True`). The response shape varies by action:
```json theme={null}
// action: NONE
{"action": "NONE"}
// action: BLOCKED
{"action": "BLOCKED", "blocked_reason": "PII or secrets detected in tool call arguments. Cannot redact tool call arguments — blocking request."}
// action: GUARDRAIL_INTERVENED
{"action": "GUARDRAIL_INTERVENED", "texts": ["My key is [REDACTED ANTHROPIC_API_KEY]"]}
```
| Field | Type | Description |
| ---------------- | ---------- | -------------------------------------------------------------------- |
| `action` | `string` | One of `NONE`, `BLOCKED`, or `GUARDRAIL_INTERVENED` |
| `blocked_reason` | `string` | Human-readable reason; present only when `action` is `BLOCKED` |
| `texts` | `string[]` | Redacted texts; present only when `action` is `GUARDRAIL_INTERVENED` |
***
## Related Documentation
* [Guardrails overview](/protection/guardrails)
* [LiteLLM Integration (observability)](/integrations/agentic-ai/litellm-integration)
* [LiteLLM Generic Guardrail API spec](https://docs.litellm.ai/docs/adding_provider/generic_guardrail_api)
# Authentication Management
Source: https://docs.fiddler.ai/reference/access-control/authn-authentication-management-console
## AuthN Management Console Guide
This guide provides step-by-step instructions for using the Fiddler AuthN management console to configure authentication methods and manage users.
### Overview
The Fiddler AuthN management console is a dedicated interface for authentication administration that allows you to:
* Configure Single Sign-On (SSO) integrations with identity providers
* Manage user accounts for email authentication
* Assign administrative roles for authentication management
* Monitor authentication settings and user status
The console uses our new identity and access management framework, to provide secure and flexible authentication capabilities for the Fiddler platform.
### How to Sign In to the AuthN Console
The AuthN console is accessed through a separate URL from your main Fiddler instance.
**To access the AuthN console:**
1. Navigate to your AuthN console URL. The URL format is your Fiddler base URL with `authn-` prepended:
* If your Fiddler URL is `https://acme.cloud.fiddler.ai`
* Your AuthN console URL is `https://authn-acme.cloud.fiddler.ai`
2. For a new deployment and the first user sign-in:
1. If you are using email authentication use the credentials provided by your Fiddler representative during initial setup or those you created using the email invitation sent during onboarding
2. If you are using SSO your user account will be created automatically on your first sign-in
3. Select your organization from the dropdown menu
* You may see a *fiddler* organization, but this is reserved for system use
* Select your company's organization from the available options
### Understanding the Root User
The **root user** is the default authentication administrator created during your Fiddler deployment setup. This user is reserved for Fiddler use with deployments managed by Fiddler. Fiddler will use the root user account to either configure your SSO integration or will assign one or more of your organization's users for managing your authentication set up moving forward.
### Understanding AuthN Roles
There are two roles specific to the AuthN management console and not to the Fiddler application: the Org Owner and the Org User Manager roles. The Org Owner can configure settings at the organization level which includes the authentication configuration and the ability to assign roles to other users. The Org User Manager is limited to managing user access to Fiddler.
#### Org Owner Role Capabilities
Users with the Org Owner role can perform all authentication management tasks, including:
* **Identity Provider Management**: Configure and manage SSO integrations with external identity providers
* **User Management**: Create, modify, and deactivate user accounts
* **Role Assignment**: Grant Org Owner or Org User Manager roles to other users
* **Authentication Methods**: Configure email authentication and SSO options
### How to Assign the Org Owner Role
The Org Owner role provides full administrative access to authentication management. Assign this role to users who need to manage identity provider integrations, additional Org Owners or Org User Managers, and user accounts.
**To assign the Org Owner role:**
1. Sign in to the AuthN console with an existing Org Owner account
2. Navigate to the **Organizations** tab in the main navigation and ensure your organization is selected
3. Locate the user account you want to modify
4. Select the **+** button on the top right to open the role assignment window
5. Select **Org Owner** from the available roles
6. Select the *Add* button to confirm the role assignment
![AuthN console role assignment modal]()
**Important Considerations:**
* Ensure at least one user always maintains the Org Owner role to prevent administrative lockout
* Org Owner users have full access to authentication settings—assign this role only to trusted administrators
* Users with Org Owner roles can manage both SSO integrations and email authentication users
### Assign the Org User Manager Role
The Org User Manager role provides limited administrative access focused on user management. This role is designed for administrators who need to manage users but should not have access to identity provider configuration.
To assign the Org User Manager role **f**ollow the same process as described for the Org owner role.
#### Org User Manager Capabilities and Limitations
**Org User Manager users can:**
* **Create Users**: Add new user accounts for email authentication
* **Manage User Status**: Activate and deactivate user accounts
* **Reset Passwords**: Assist users with password-related issues
* **Send Invitations**: Issue setup invitations for email authentication
* **View User Information**: Access user account details and authentication status
**Org User Manager users cannot:**
* **Configure Identity Providers**: Cannot create or modify SSO integrations
* **Edit IdP Settings**: Cannot access identity provider configuration
* **Manage Organization Settings**: Cannot modify authentication policies
* **Assign Administrative Roles**: Cannot grant Org Owner or Org User Manager roles to other users
#### When to Use Org User Manager
The Org User Manager role is ideal for:
* Help desk personnel who assist with user account issues
* Team leads who need to add new team members using email authentication
* Administrators in mixed environments where some users authenticate via email rather than SSO
* Situations where you want to delegate user management without granting full administrative access
### User Management for Email Authentication
**Important**: Users only need to be manually created and managed in the AuthN console when using email authentication. SSO users are automatically provisioned when they first sign in through their identity provider.
#### When Manual User Management Is Required
Manual user creation and management is necessary for:
* Organizations using email authentication instead of SSO
* Mixed authentication environments where some users need email authentication
* Adding users who don't have access to your organization's identity provider
* Creating service accounts or special-purpose accounts
#### Email Authentication User Lifecycle
**User Creation Process:**
1. **Access Console**: Navigate to the AuthN console with appropriate administrative privileges
2. **Create Account**: Use the **Users** section to add new user accounts
3. **Send Invitation**: Users receive setup instructions via email
4. **User Activation**: Users complete their account setup and choose authentication methods
5. **Fiddler Access**: After authentication setup, users can access the Fiddler platform
![Authn console add new user form]()
![authn-console-user-list]()
**Ongoing Management:**
* **Account Status**: Monitor and modify user account status as needed
* **Password Support**: Assist users with password resets and authentication issues
* **Role Assignment**: Coordinate with Fiddler application administrators for platform role assignments
### Navigation and Interface Overview
#### Main Console Sections
* **Home**: Organization overview and quick access to common tasks
* **Users**: User account management for email authentication
* **Organizations**: Role assignment and organization-level settings
* **Settings**: Authentication configuration and identity provider management
* **Login and Access**: Authentication method configuration
* **Identity Providers**: SSO integration setup and management
### Best Practices and Security Recommendations
#### Administrative Role Management
* **Multiple Org Owners**: Maintain at least two users with Org Owner roles to prevent administrative lockout
* **Role Separation**: Use Org User Manager roles for personnel who only need user management capabilities
* **Regular Review**: Periodically audit administrative role assignments
* **Documentation**: Maintain records of who has administrative access and why
#### User Management
* **Consistent Naming**: Establish clear naming conventions for user accounts
* **Account Lifecycle**: Implement processes for user onboarding and offboarding
* **Mixed Authentication**: Clearly document which users use email authentication versus SSO
* **Security Policies**: Apply appropriate password and account security policies
#### Identity Provider Integration
* **Testing Environment**: Test SSO integrations before deploying to production
* **Backup Authentication**: Maintain email authentication capabilities as a backup
* **Configuration Documentation**: Document IdP configuration details for troubleshooting
* **Regular Monitoring**: Monitor SSO integration status and user authentication patterns
### Troubleshooting Common Issues
#### Access Issues
**Cannot Access AuthN Console:**
* Verify the console URL format (`authn-` prefix)
* Check network connectivity and firewall settings
* Confirm user account has appropriate administrative role
* Contact your Fiddler representative for credential verification
#### User Management Issues
**Cannot Create Users:**
* Verify user account has Org Owner or Org User Manager role
* Check that email authentication is enabled for your organization
* Confirm user email addresses are unique and properly formatted
**Invitation Emails Not Received:**
* Verify email addresses are correct and properly formatted
* Check spam and junk mail folders
* Confirm email delivery settings in your organization's configuration
* Resend invitations if necessary using console tools
#### Role Assignment Issues
**Cannot Assign Roles:**
* Confirm you have Org Owner privileges (Org User Managers cannot assign roles)
* Verify target user account exists and is active
* Check that you're working in the correct organization context
### Getting Support
For additional assistance with the AuthN console:
* **Documentation**: Reference specific identity provider integration guides for SSO setup
* **Fiddler Support**: Contact your Fiddler representative with specific error messages or console screenshots
* **System Logs**: Review authentication logs for troubleshooting authentication issues
* **Best Practices**: Consult your organization's IT security team for authentication policy guidance
# Email Login
Source: https://docs.fiddler.ai/reference/access-control/email-login
This page documents the details of Fiddler's native email-based authentication including user account creation and password policy.
This guide covers email-based authentication in Fiddler, including user management, security requirements, and administrative procedures.
## Overview
Email authentication is Fiddler's built-in authentication method for organizations that don't use Single Sign-On (SSO) or need to provide access to users outside their SSO system. With email authentication, users log in using their email address and a password they create during the account setup process.
## When to Use Email Authentication
Email authentication is ideal for:
* Organizations without an identity provider
* Adding specific users who don't have SSO access
* Mixed environments where some users need different authentication methods
* Adding service accounts
**Note**: Each user account can only use one authentication method—either email authentication or SSO, not both.
## User Management with Authentication Console
Fiddler provides a separate UI for managing your authentication: the AuthN console.
The URL to the AuthN management console is your Fiddler instance base URL preprended with `authn-`. For example, if your base URL is `https://acme.cloud.fiddler.ai` then you can access the AuthN management console at `https://authn-acme.cloud.fiddler.ai`.
### Prerequisites for User Management
To manage email authentication users, you need:
* **Administrator access**: Your user account must have the "Org Owner" or "Org User Manager" role in the authentication management system
* **Console access**: Access to the Fiddler authentication management console
### Adding Users to Fiddler
**Step 1: Access the Authentication Management Console**
* Navigate to the Fiddler AuthN authentication management console
The URL to the AuthN management console is your Fiddler instance base URL preprended with \`authn-\`. For example, if your base URL is \`\<[https://acme.cloud.fiddler.ai\\\\\`](https://acme.cloud.fiddler.ai\\\\`)> then you can access the AuthN management console at \`\<[https://authn-acme.cloud.fiddler.ai\\\\\`](https://authn-acme.cloud.fiddler.ai\\\\`)>.
* Ensure you have the necessary administrator permissions. To assign user management privilege to another user, add either the "Org Owner" or "Org User Manager" role in the Organizations tab of the AuthN console:
![Adding a manager using Fiddler user management console Organizations tab]()
![Selecting a role for a user in the Fiddler user management console]()
**Step 2: Create User Account**
* The email and username fields must match exactly
* Only lower-case letters should be used to avoid case-sensitivity issues
1. In the authentication console, navigate to **Users**
2. Click **New** to create a new user.
![Authn console add new user form]()
3. Fill in the required contact details:
* **Email address**: This will be the user's login identifier and must be unique
* **User Name**: This field must contain the email address exactly
* **First name** and **Last name**
* **Email Verified**: Choose whether to mark the email as "verified" automatically
* **Set Initial Password**: Choose whether to set the password yourself or allow the new user to set their own
It is recommended to leave both the Email Verified and the Set Initial Password checkboxes unchecked. Doing so results in the user receiving an email with a link to Fiddler to confirm their invitation and choose their password.
* Checking Email Verified means the user does not need to validate that they own the email address assigned
* If both checkboxes are checked, the user will receive no email, and the AuthN admin must communicate the sign-in credentials
**Step 3: Configure Authentication Setup**
Choose one of the following authentication setup options:
**Option A: Set up Authentication Later**
* Select **"Setup authentication later for this user"**
* Use this option if you want to prepare the account before the user needs access
* The user will not be able to log in until they set up an authentication method
* Useful for preparing accounts for future employees
**Option B: Send Invitation Email (Recommended)**
* Select **"Send an invitation E-Mail for authentication setup and E-Mail verification"**
* The user will receive an email with instructions to set up their authentication
* The user can choose their preferred authentication method (password, passkey, or external SSO)
* This provides the most flexibility for users
**Option C: Set Initial Password**
* Select **"Set an initial password for the user"**
* Enter a temporary password for the user
* The user will receive an email notification about their account
* The user should change this password on first login
**Step 4: Complete User Creation**
1. Click **Create** to save the user account
2. The system will process the user creation based on your selected authentication option
3. If you chose email invitation, the user will automatically receive setup instructions
### Managing User Invitations
**Invitation and Setup Process**:
**Email Verification and Initial Setup**:
* By default, users receive an initialization email with a verification code
* Users must verify this code on their first login
* If you marked the email as "verified" during creation, this step may be skipped
**Authentication Method Selection**:
* Users can choose from available authentication methods based on your organization's configuration:
* **Password**: Traditional username/password authentication
* **Passkey**: Modern passwordless authentication using biometrics or security keys
* **External SSO**: If configured, users can authenticate through external identity providers
**Invitation Properties**:
* Email invitations do not expire automatically
* Users can complete their setup at any time after receiving the invitation
* Account setup must be completed before users can access Fiddler
**Resending Invitations**: If you need to resend setup instructions to a user:
1. Go to the **Users** section in the authentication console
2. Find the specific user account
3. Use the available options to resend invitation or setup emails
4. Users who haven't completed setup will receive fresh instructions
### User Account Lifecycle
**Account Activation**: When users receive their setup email, they:
1. Click the setup link in the email
2. Verify their email address (if required)
3. Choose and configure their preferred authentication method:
* **Password**: Create a password meeting security requirements
* **Passkey**: Set up biometric or security key authentication
* **External SSO**: Link to configured external identity providers (if available)
4. Complete their profile information if prompted
5. Confirm their account to gain access to Fiddler
**Account Management**:
* **User deactivation**: Temporarily disable user access
* **User deletion**: Permanently remove user accounts
* **Password reset**: Users can reset forgotten passwords through the login interface
### Password Management
**Forgotten Passwords**: Users can reset forgotten passwords by:
1. Clicking **Forgot Password** on the login page
2. Entering their email address
3. Following the instructions in the password reset email
4. Creating a new password that meets security requirements
### User Role Considerations
**AuthN Administrative Roles**: Ensure at least one user has the "Org Owner" or "Org User Manager" role to continue managing email authentication users.
**Fiddler Roles**: After email authentication users log in, Org Admins can assign appropriate Fiddler application roles (Org Admin, Org Member, etc.) for access control.
## Troubleshooting
### Common Issues
**Users Cannot Access Console**:
* Verify the user has "Org Owner" or "Org User Manager" role
* Check authentication console network accessibility
* Confirm user account status
**Invitation Links Not Working**:
* Verify the invitation hasn't been revoked
* Check for email delivery issues
* Regenerate invitation links if necessary
**Login Problems**:
* Verify password meets all requirements
* Check for account lockouts due to failed attempts
* Confirm user account is active and not suspended
# Google OIDC
Source: https://docs.fiddler.ai/reference/access-control/google-integration
Learn how to configure Fiddler with Google for Single Sign-On (SSO) using the OpenID Connect (OIDC) protocol.
## Overview
This integration allows your users to sign in to Fiddler using their existing Google account, without needing a separate Fiddler password. Users are automatically provisioned on their first successful login — no manual invitations required.
Google OIDC does not support group synchronization — Google does not include group membership in its OIDC tokens. Users are provisioned individually; assign them to Fiddler teams and roles manually.
## Prerequisites
Before starting, ensure you have:
* **Google Cloud Access**: Permissions to create and configure OAuth 2.0 credentials in a Google Cloud project.
* **Fiddler AuthN Administrator Access**: Org Owner role in Fiddler's AuthN management console.
* **Deployment Information**: The hostname of your Fiddler deployment, e.g. `idpexample.dev.fiddler.ai`.
## Configuring Google
Fiddler requires two redirect URIs on the Google OAuth client. You will add both when creating the client below:
* `https://authn-{base_url}/ui/login/login/externalidp/callback`
* `https://authn-{base_url}/idps/callback`
Replace `{base_url}` with your Fiddler deployment host (e.g. `idpexample.dev.fiddler.ai`).
1. In the [Google Cloud Console](https://console.cloud.google.com/), select your project and go to *APIs & Services → OAuth consent screen*.
2. In *App information*, enter the *App name* and *User support email*.
3. In *Audience*, select *Internal* or *External* depending on your organization's setup and access requirements.
4. In *Contact Information*, add the required email addresses.
5. Agree to the *Google API Services: User Data Policy*, then select *Create*.
If you selected *External*, the app starts in testing mode — only accounts added under *Test users* can sign in until you publish the app (set it to *In production*). See [Manage app audience](https://support.google.com/cloud/answer/15549945) for details.
1. Go to *APIs & Services → Credentials*, then select *Create Credentials → OAuth client ID*.
![Google Cloud create OAuth client]()
2. Select *Web application* for the *Application type* and enter a name, e.g. `Fiddler IdP Example`.
3. Under *Authorized redirect URIs*, add both redirect URIs (replace the host with your deployment):
* `https://authn-idpexample.dev.fiddler.ai/ui/login/login/externalidp/callback`
* `https://authn-idpexample.dev.fiddler.ai/idps/callback`
![Google Cloud OAuth client name, type, and redirect URIs]()
4. Select *Create*.
5. Copy the *Client ID* and *Client Secret* — you will need them when configuring Fiddler.
![Google Cloud OAuth client credentials]()
## Configuring Fiddler
The URL to the Fiddler AuthN management console is your Fiddler instance base URL prepended with `authn-`. For example, if your Fiddler base URL is `https://idpexample.dev.fiddler.ai` then you will access the AuthN management console at `https://authn-idpexample.dev.fiddler.ai`.
Sign in using the AuthN console Org Owner user account credentials provided by your Fiddler representative.
![Fiddler AuthN console sign-in page]()
Ensure your organization is selected in the dropdown. You may see the *fiddler* organization, but this is reserved for system use and should not be edited. Here we are using the *idpexample* organization.
![Fiddler AuthN console home page]()
Select *Settings* tab from the top menu and then select *Identity Providers* from the left navigation menu.
![Fiddler AuthN console add provider page]()
1. Select the *Google* option in the *Add provider* section, which brings up the Google Provider form.
2. Note the callback URL shown in the form — it corresponds to the redirect URIs you registered in Google earlier, so no further changes are needed in Google.
![Fiddler AuthN console add new Google provider form with callback URL]()
In the Google Provider form, paste the *Client ID* and *Client Secret* copied from Google earlier.
![Fiddler AuthN console Google provider client credentials]()
1. Expand the *optional* section.
2. Ensure the *Scopes List* includes `openid`, `profile`, and `email`.
3. Ensure the *Automatic create* and *Automatic update* checkboxes are selected.
4. Set the *Determines whether an identity will be prompted to be linked to an existing account* dropdown to *Check for existing Username*.
![Fiddler AuthN console automatic create/update and check existing username settings]()
Select the *Create* button and then select the *Save* button. You will be returned to the Organization Settings page.
![Fiddler AuthN console with newly created Google OIDC IdP]()
1. Select your IdP from the list and select the *Activate* button on the identity provider page.
![Fiddler AuthN console activate new Google OIDC IdP]()
2. Close the settings and then select *Login Behavior and Security* from the left nav menu and ensure the *External login allowed* checkbox is selected.
![Fiddler AuthN console allow external login behavior]()
3. Select the *Save* button.
![Fiddler AuthN console external login allowed]()
Select the *Actions* tab from the top menu.
![Fiddler AuthN console new custom Action script]()
1. Select the *New* button in the *Scripts* section to create a new action script.
2. Copy the *Google OIDC Action Script* below and paste it into the script text area.
3. Enter `setAttributesOnGoogleOIDCAuth` in the *Name* text box.
4. Select the *Add* button.
**File:** `Google OIDC Action Script`
```javascript theme={null}
function setAttributesOnGoogleOIDCAuth(ctx, api) {
let firstName = ctx.v1.providerInfo.getFirstName();
let lastName = ctx.v1.providerInfo.getLastName();
let email = ctx.v1.providerInfo.getEmail();
let nameParts = [firstName, lastName];
let filteredParts = nameParts.filter(part => part);
let displayName = filteredParts.join(' ');
if (firstName != undefined) {
api.setFirstName(firstName);
}
if (lastName != undefined) {
api.setLastName(lastName);
}
if (email != undefined) {
email = email.toLowerCase();
api.setEmail(email);
api.setEmailVerified(true);
api.setPreferredUsername(email);
}
if (displayName) {
api.setDisplayName(displayName);
}
api.v1.user.appendMetadata('fiddler_authentication_type', 'SSO:GOOGLE:OIDC');
api.v1.user.appendMetadata('fiddler_groups', []);
}
```
Scroll down to the *Flows* section.
![Fiddler AuthN console new Action trigger creation]()
1. Select the *External Authentication* option for the *Flow Type* dropdown.
2. Select the *+ Add trigger* button.
3. Select the *Post Authentication* option for the *Trigger Type* dropdown.
4. Select the *setAttributesOnGoogleOIDCAuth* option for the *Actions* dropdown.
5. Select the *Save* button.
Add an organization metadata key so Fiddler can correctly identify and process this SSO connection. Set this once during setup.
1. Go to the *Metadata* section and select *Edit*.
![Fiddler AuthN console organization metadata section]()
2. Select the *Add* button, then enter the key `fiddler_sso_authentication_type` with the value `SSO:GOOGLE:OIDC`.
![Fiddler AuthN console organization metadata with Google OIDC authentication type]()
3. Select the *Save* button next to the new entry.
Before validating, make sure your Google account can sign in: for an *Internal* consent screen it must belong to your Google Workspace organization; for an *External* app still in testing it must be added as a *Test user*.
1. Open your Fiddler URL (e.g. `https://idpexample.dev.fiddler.ai`).
2. Ensure you see the Fiddler sign-in page and that it displays the Google SSO login button.
![Fiddler application homepage displaying the new SSO login method in addition to the email sign-in form]()
3. Select the button and confirm that the Fiddler application loads.
![Fiddler application landing page]()
The first user to sign in to the Fiddler application is automatically assigned the Fiddler Org Admin role; subsequent members are Org Members by default.
## Getting Help
If sign-in fails, verify the OAuth client configuration in the Google Cloud Console (redirect URIs, consent screen status, client credentials). For Fiddler-side issues, see the [SSO Authentication Guide](/reference/access-control/sso-authentication-guide). If the issue persists, contact your Fiddler representative with the specific error message.
## Important Notes
* **Data Storage**: Fiddler stores the following profile attributes from Google: first name, last name, display name, and email address.
* **No Group Synchronization**: Google OIDC does not provide group membership, so group sync to Fiddler teams is not available. Assign users to teams and roles manually.
* **API Access**: For programmatic API access, users create an API key from the *Credentials* tab in Fiddler's *Settings* page.
* **Single Authentication Method**: Users can only authenticate via either SSO or email authentication, not both.
## Next Steps
After successful integration:
* **Train Users**: Provide guidance on accessing Fiddler through Google SSO.
* **Assign Roles**: Manually assign users to the appropriate Fiddler teams and roles after their first login.
* **Monitor Usage**: Review authentication logs and user access patterns.
# Access Control
Source: https://docs.fiddler.ai/reference/access-control/index
Explore our guides on authentication options with leading IDPs like Okta and PingOne. Dive deep into authorization topics using the Fiddler UI.
This section covers how to configure user access, authentication, and authorization in Fiddler.
## Overview
Managing access to your Fiddler instance involves these key components:
* **Authentication**: Verifying user identities through Single Sign-On (SSO) or email-based methods
* **User Management**: Adding and managing users in the Fiddler AuthN console or dynamically with SSO integration
* **Authorization**: Configuring what users can access through role-based permissions in the Fiddler UI or dynamically with SSO integration
## Getting Started with Authentication Management
Fiddler provides a dedicated authentication management console to deliver secure, flexible user management. As an administrator, you'll use the Fiddler AuthN console to configure authentication methods and manage users.
### Initial Setup
For new Fiddler deployments:
* A Fiddler representative will work with you to set up your initial authentication configuration
* Choose your preferred authentication method: SSO, email-based authentication, or both
* At least one user in your organization must be assigned the "Org Owner" or "Org User Manager" role in the Fiddler AuthN console.
* An "Org Owner" can administer their SSO integration with Fiddler as well as manage users
* An "Org User Manager" can manage users when leveraging email-based authentication
## Authentication Methods
Choose the authentication method that best fits your organization's infrastructure:
### Single Sign-On (SSO)
SSO users are automatically provisioned when they first log in with valid credentials from your identity provider.
| Identity Provider | Protocol | Guide |
| -------------------------------------- | -------- | --------------------------------------------------------------------------------------- |
| Okta | OIDC | [Okta OIDC SSO Integration](/reference/access-control/okta-integration) |
| Okta | SAML | [Okta SAML SSO Integration](/reference/access-control/okta-integration-saml) |
| Microsoft Entra ID (formerly Azure AD) | OIDC | [Azure AD OIDC SSO Integration](/reference/access-control/single-sign-on-with-azure-ad) |
| PingOne | SAML | [PingOne SAML SSO Integration](/reference/access-control/ping-identity-saml) |
| Google | OIDC | [Google OIDC SSO Integration](/reference/access-control/google-integration) |
### Email-Based Authentication
For organizations without an identity provider or when you need to add specific users outside your SSO system.
| Guide | Description |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| [Email Login Configuration](/reference/access-control/email-login) | Configure Fiddler's email-based authentication and learn how to add users through the authentication management console. |
### Mixed Authentication
You can use both SSO and email authentication simultaneously:
* SSO users are automatically provisioned on first login
* Email users must be manually added through the authentication management console
* Each user account can only use one authentication method
## Authorization and Access Control
Authorization settings are managed in the Fiddler UI using Fiddler's role-based access control system and optional LDAP syncing with your IDP:
| Guide | Description |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| [Role-Based Access Control](/reference/access-control/role-based-access) | Understand and configure user permissions through pre-defined roles |
| Mapping Identity Provider Groups to Fiddler Teams and Roles | Synchronize external user groups with Fiddler teams and organization roles for streamlined access management |
## Configuration Sequence
For organizations new to Fiddler access management, we recommend this sequence:
1. **Set up authentication management access in the Fiddler AuthN console**: Ensure you have the appropriate AuthN administrator role: Org Owner
2. **Configure authentication**: Choose and implement your authentication method (SSO, email, or both)
3. **Add initial users**: Use the authentication management console to add users or configure SSO for automatic provisioning when users first sign in
4. **Configure authorization**: Set up role-based access control within the Fiddler UI's Access tab in the Settings page
5. **Create teams**: Organize users into teams for efficient permission management
6. **Map external groups** (if applicable): Connect your identity provider groups to Fiddler teams and manage Fiddler roles
## Troubleshooting and Support
If you encounter issues with authentication or user management:
* Check the authentication management console for authentication logs and user status
* Verify that your SSO configuration matches your identity provider settings
* Ensure users have the correct administrative roles for user management tasks
* Contact your Fiddler representative for assistance with authentication configuration
# Mapping IdP Groups to Teams
Source: https://docs.fiddler.ai/reference/access-control/mapping-ad-groups-to-fiddler-teams
This document describes the naming convention and rules for mapping internal AD groups to Fiddler Teams automatically.
This guide describes how to configure automatic synchronization between your identity provider (IdP) groups and Fiddler teams using the [Fiddler AuthN management console](/reference/access-control/authn-authentication-management-console), enabling streamlined access control and role management.
## Overview
Group synchronization automatically maps users from your IdP groups to corresponding Fiddler teams and roles. This eliminates the need for manual user role assignment, ensuring that access permissions remain synchronized with your organizational structure.
**Supported Identity Providers**:
* Okta (OIDC and SAML)
* Microsoft Entra ID (formerly Azure AD) with OIDC (requires additional configuration steps)
* PingOne (SAML)
Google OIDC does not support group synchronization due to limitations in Google's OIDC implementation.
## Prerequisites
Before configuring group synchronization, ensure you have:
* **SSO Integration**: A working SSO integration with a supported identity provider
* **Administrator Access**: Both identity provider admin access and Fiddler AuthN admin "Org Owner" permissions
* **Group Configuration**: Proper group setup in your identity provider with appropriate naming conventions
* **User Assignment**: Users assigned to relevant groups in your identity provider
## Group Naming Convention
All identity provider groups must follow a set naming pattern to be recognized by Fiddler:
```
fiddler_
```
The default group prefix is `fiddler_`, but this can be customized during the configuration process.
**Team name constraints**: The identifier portion of the group name (used as the Fiddler team name) must satisfy these rules:
* Must be at least 2 characters and no more than 256 characters long
* Must start with a letter (`a`–`z` or `A`–`Z`)
* May contain only: letters (`a`–`z`, `A`–`Z`), digits (`0`–`9`), underscores (`_`), and hyphens (`-`)
Identifiers with uppercase characters (e.g., `DataScientists`, `ML_Engineers`) are valid; they are normalized to lowercase before the team is created (so `DataScientists` becomes `datascientists`, and `ML_Engineers` becomes `ml_engineers`). Examples of invalid identifiers: `123team` (must start with a letter), `team!` (contains disallowed character).
### Team Identifiers
Any other identifier creates a corresponding team in Fiddler:
* `fiddler_data_scientist` - Creates/assigns users to the "data\_scientist" team
* `fiddler_ml_engineers` - Creates/assigns users to the "ml\_engineers" team
* `fiddler_product_team` - Creates/assigns users to the "product\_team" team
### Group Naming Examples
| Identity Provider Group | Result in Fiddler |
| ------------------------ | ---------------------------------------- |
| `fiddler_ORG_ADMIN` | User assigned "Org Admin" role |
| `fiddler_ORG_MEMBER` | User assigned "Org Member" role |
| `fiddler_data_scientist` | User added to "data\_scientist" team |
| `fiddler_finance_team` | User added to "finance\_team" team |
| `fiddler_` | **Invalid** - Will be ignored |
| `data_scientist` | **Invalid** - Missing "fiddler\_" prefix |
## Configuration Steps
**Identity Provider-Specific Requirements**:
**Okta**:
* Ensure the `groups` scope is included in your OIDC application
* Configure Groups claim in the "Sign On" section of your application
**Microsoft Entra ID**:
* Add the `groups` claim to your application's token configuration
* Grant `GroupMember.Read.All` API permissions
* Additional configuration steps are required (see the Advanced Configuration section)
**PingOne**:
* Configure group attribute mapping in your SAML application
* Ensure group membership is included in SAML assertions
1. Access your identity provider's admin console
2. Create groups following the `fiddler_<identifier>` naming convention or choose your own prefix, e.g. `company_fiddler_`
3. Assign appropriate users to each group
4. Configure group claims/attributes in your SSO application
The URL to the AuthN management console is your Fiddler instance base URL, prepended with `authn-`. For example, if your base URL is `https://acme.cloud.fiddler.ai` then you can access the AuthN management console at `https://authn-acme.cloud.fiddler.ai`.
**Access Organization Settings**:
1. Log into Fiddler with AuthN console "Org Owner" privileges
2. Navigate to the *Organization* tab at the top
3. Ensure that your organization is selected in the top left dropdown (this will never be "fiddler" which is reserved)
4. Locate the *METADATA* section
![Fiddler AuthN admin console organization home page]()
**Configure Group Sync Settings**:
1. Select the *Edit* button in the *METADATA* section
2. Configure these key-value pairs:
* *fiddler\_group\_prefix*: Set the group prefix (defaults to `fiddler_` unless manually modified)
* *fiddler\_group\_sync\_enabled*: Set to `true`
3. Save your changes by selecting the *Save* disk icon adjacent to each key value pair
![]()
Mapping users to Fiddler organization roles is optional. All new users will be Org Members by default, and the Org Admin role can be assigned in the Fiddler UI as needed.
Setting up automatic [organization role](/reference/access-control/role-based-access#understanding-roles) mappings uses these additional metadata keys:
* *fiddler\_org\_admin\_mapper*: Custom mapping key for the Org Admin role
* *fiddler\_org\_member\_mapper*: Custom mapping key for Org Member role
**To configure automatic role mapping**:
1. In the *METADATA* section, add the mapper keys as needed
2. Set the values to match your identity provider's group naming convention, noting that the METADATA key values should not include the Fiddler group prefix, which is the default `fiddler_` in this example
1. Create a group in your IdP for Fiddler Org Admin users named `fiddler_org_admins`
2. Set the `fiddler_org_admin_mapper` metadata key value to `org_admins`
3. Create a group in your IdP for Fiddler Org Member users named fiddler\_org\_members
4. Set the `fiddler_org_member_mapper` metadata key value to `org_members`
3. Save your changes by selecting the *Save* disk icon adjacent to each key value pair
![Fiddler AuthN admin console add org mapper keys]()
**Test Group Synchronization**:
1. Log in with a test user who belongs to the mapped groups
2. Verify the user is assigned to the correct Fiddler roles/teams
3. Check that team memberships update when identity provider groups change
4. Confirm that new groups create corresponding Fiddler teams automatically
## Role Precedence for Multi-Group Membership
If a user belongs to multiple IdP groups that each map to a different Fiddler organization role, Fiddler assigns the highest-privilege matched role. This commonly occurs in directory structures where all users belong to a general member group and a subset are also assigned to an admin group.
**Upgrading to 26.10.1 or later?** Before this release, role assignment depended on the order in which your IdP returned groups, so users in both an admin-mapped and a member-mapped group may have been assigned **Org Member**. Starting in 26.10.1, those users will be promoted to **Org Admin** on their next login. Audit your IdP group memberships before deploying this release to confirm role assignments are correct.
Example: A user belonging to both `fiddler_ORG_ADMIN` and `fiddler_ORG_MEMBER` groups is assigned the Org Admin role.
Resolution is order-independent. The role assigned does not depend on the order in which the IdP returns groups in the SAML or OIDC response.
## Advanced Configuration
### Custom Group Prefixes
You can customize the group prefix if `fiddler_` doesn't fit your naming conventions:
1. In the Organization *METADATA* section, update `fiddler_group_prefix`
2. For example, set to `company_fiddler_` to require groups like `company_fiddler_data_team`
3. All group names in your identity provider must use your custom prefix
### Team Hierarchy and Permissions
#### Automatic Team Creation
* Teams are automatically created when users with new group mappings first log in
* Team names match the identifier portion of the group name
* Teams inherit default permissions, which can be customized through the Fiddler UI
#### Team Management:
* Organization admins can modify team permissions through Fiddler settings
* Project-specific access can be configured per team
* Teams persist even if all members are removed
## Troubleshooting
### Common Issues
#### Groups Not Synchronizing
* **Verify Group Sync Enable**: Check that `fiddler_group_sync_enabled` is set to `true`
* **Check Group Names**: Ensure groups follow the correct naming convention with your configured prefix
* **Validate Claims**: Confirm your identity provider includes group claims in authentication tokens
* **Review Permissions**: Verify your SSO application has appropriate permissions to read group membership
#### Users Not Assigned to Correct Teams
* **Group Membership**: Confirm users are actually members of the expected groups in your identity provider
* **Name Matching**: Ensure group names exactly match the expected format (case-sensitive)
* **Re-authentication**: Users may need to log out and back in for group changes to take effect
#### Custom Role Mapping Issues
* **Mapper Configuration**: Verify that custom role mapper keys are configured correctly in the *METADATA* section
* **Group Assignment**: Ensure users are assigned to groups that match the custom mapper values
* **User in Multiple Role Groups**: If a user belongs to both an admin-mapped and a member-mapped group, the higher-privilege role (Org Admin) is applied automatically. See [Role Precedence for Multi-Group Membership](#role-precedence-for-multi-group-membership) for details.
## Best Practices
### Identity Provider Management
* **Consistent Naming**: Establish clear naming conventions for Fiddler-related groups
* **Group Documentation**: Maintain documentation of group purposes and membership criteria
* **Regular Audits**: Periodically review group memberships and access levels
* **Change Management**: Implement processes for group creation, modification, and deletion
### Fiddler Team Organization
* **Logical Grouping**: Align Fiddler teams with your organizational structure and project needs
* **Permission Planning**: Design team permissions to match job functions and access requirements
* **Scalability**: Consider how your team structure will scale as your organization grows
### Security Considerations
* **Least Privilege**: Apply the principle of least privilege when designing group access levels
* **Regular Reviews**: Conduct periodic access reviews to ensure appropriate permissions
* **Separation of Duties**: Consider separating administrative and operational roles
* **Audit Trails**: Monitor group membership changes and access patterns
## Getting Help
For additional assistance with group synchronization:
* **Organization Settings**: Check the Organization METADATA section for configuration details
* **Identity Provider Support**: Consult your identity provider's documentation for group configuration
* **Fiddler Support**: Contact your Fiddler representative with group sync configuration details
* **Testing Environment**: Use a test environment to validate group sync before production deployment
## Related Documentation
* [Role-Based Access Control](/reference/access-control/role-based-access) - Understanding Fiddler roles and permissions
* [Okta OIDC SSO Integration](/reference/access-control/okta-integration) - Okta-specific group sync setup
* [Microsoft Entra ID OIDC SSO Integration](/reference/access-control/single-sign-on-with-azure-ad) - Entra ID group sync configuration
* [PingOne SAML SSO Integration](/reference/access-control/ping-identity-saml) - PingOne group sync setup
* [General SSO Authentication Guide](/reference/access-control/sso-authentication-guide) - Overview of SSO concepts
# Okta OIDC
Source: https://docs.fiddler.ai/reference/access-control/okta-integration
Learn how to configure Fiddler with Okta for Single Sign-On (SSO) using the OpenID Connect (OIDC) protocol.
## Overview
This integration allows your users to sign in to Fiddler using their existing Okta account, without needing a separate Fiddler password. Users are automatically provisioned on their first successful login — no manual invitations required.
## Prerequisites
Before starting, ensure you have:
* **Okta Administrator Access**: Permissions to create and configure applications in your Okta organization.
* **Fiddler AuthN Administrator Access**: Org Owner role in Fiddler's AuthN management console.
* **Deployment Information**: The hostname of your Fiddler deployment, e.g. `idpexample.dev.fiddler.ai`.
## Configuring Okta
Fiddler requires two redirect URIs on the Okta application. You will add both when creating the Okta application below:
* `https://authn-{base_url}/ui/login/login/externalidp/callback`
* `https://authn-{base_url}/idps/callback`
Replace `{base_url}` with your Fiddler deployment host (e.g. `idpexample.dev.fiddler.ai`).
1. In the Okta admin console, navigate to *Applications* and select the *Create App Integration* button. Select *OIDC - OpenID Connect* for the *Sign-in method* and *Web Application* for the *Application type*, then select *Next*.
![Okta admin console applications list]()
2. Assign a name for your application integration in the *App integration name* text box, then configure the redirect URIs:
1. Enter both redirect URIs into the *Sign-in redirect URIs* field using the *+ Add URI* button:
* `https://authn-idpexample.dev.fiddler.ai/ui/login/login/externalidp/callback`
* `https://authn-idpexample.dev.fiddler.ai/idps/callback`
2. Enter your Fiddler deployment URL (without the `authn-` prefix) into the *Sign-out redirect URIs* text box, e.g. `https://idpexample.dev.fiddler.ai`.
![Okta admin console application page with redirect and sign-out URIs]()
3. Select the *Save* button to create the application.
4. Copy the following values — you will need them when configuring Fiddler:
1. On the *General* tab, copy the *Client ID* and *Client Secret* values.
![Okta admin console application page with client id and secret]()
2. On the *Sign On* tab, copy the *Issuer* URL.
![Okta admin console application page with the Issuer URL]()
## Configuring Fiddler
The URL to the Fiddler AuthN management console is your Fiddler instance base URL prepended with `authn-`. For example, if your Fiddler base URL is `https://idpexample.dev.fiddler.ai` then you will access the AuthN management console at `https://authn-idpexample.dev.fiddler.ai`.
Sign in using the AuthN console Org Owner user account credentials provided by your Fiddler representative.
![Fiddler AuthN console sign-in page]()
Ensure your organization is selected in the dropdown. You may see the *fiddler* organization, but this is reserved for system use and should not be edited. Here we are using the *idpexample* organization.
![Fiddler AuthN console home page]()
Select *Settings* tab from the top menu and then select *Identity Providers* from the left navigation menu.
![Fiddler AuthN console add provider page]()
1. Select the *Generic OIDC* option in the *Add provider* section, which brings up the OIDC Provider form.
2. Note the callback URL shown in the form — it corresponds to the redirect URIs you registered in Okta earlier, so no further changes are needed in Okta.
![Fiddler AuthN console add new OIDC provider form with callback URL]()
1. In the OIDC Provider form, enter the following values:
1. Enter a name in the *Name* text box. This name is displayed on the SSO login button on the Fiddler sign-in page, so choose one your users will recognize.
2. In the *Issuer* text box, paste the Issuer URL copied from the Okta admin console.
3. In the *Client ID* and *Client Secret* text boxes, paste those values copied from the Okta admin console.
![Fiddler AuthN console OIDC provider name]()
1. Expand the *optional* section.
2. Add the text `groups` to the *Scopes List* text box and ensure it is listed along with `openid`, `profile`, and `email`.
3. Ensure the *Automatic create* and *Automatic update* checkboxes are selected.
![Fiddler AuthN console automatic create and update settings]()
4. Set the *Determines whether an identity will be prompted to be linked to an existing account* dropdown to *Check for existing Username*.
![Fiddler AuthN console check for existing username setting]()
Select the *Create* button and then select the *Save* button. You will be returned to the Organization Settings page.
![Fiddler AuthN console with newly created Okta OIDC IdP]()
1. Select your IdP from the list and select the *Activate* button on the identity provider page.
![Fiddler AuthN console activate new Okta OIDC IdP]()
2. Close the settings and then select *Login Behavior and Security* from the left nav menu and ensure the *External login allowed* checkbox is selected.
![Fiddler AuthN console allow external login behavior]()
3. Select the *Save* button.
![Fiddler AuthN console external login allowed]()
Select the *Actions* tab from the top menu.
![Fiddler AuthN console new custom Action script]()
1. Select the *New* button in the *Scripts* section to create a new action script.
2. Copy the *Okta OIDC Action Script* below and paste it into the script text area.
3. Enter `setAttributesOnOktaOIDCAuth` in the *Name* text box.
4. Select the *Add* button.
**File:** `Okta OIDC Action Script`
```javascript theme={null}
function setAttributesOnOktaOIDCAuth(ctx, api) {
let firstName = ctx.v1.providerInfo.getFirstName();
let lastName = ctx.v1.providerInfo.getLastName();
let email = ctx.v1.providerInfo.getEmail();
let groups = ctx.getClaim('groups');
let nameParts = [firstName, lastName];
let filteredParts = nameParts.filter(part => part);
let displayName = filteredParts.join(' ');
if (firstName != undefined) {
api.setFirstName(firstName);
}
if (lastName != undefined) {
api.setLastName(lastName);
}
if (email != undefined) {
email = email.toLowerCase();
api.setEmail(email);
api.setEmailVerified(true);
api.setPreferredUsername(email);
}
if (displayName) {
api.setDisplayName(displayName);
}
api.v1.user.appendMetadata('fiddler_authentication_type', 'SSO:OKTA:OIDC');
if (groups === null || groups === undefined){
groups = []
}
api.v1.user.appendMetadata('fiddler_groups', groups);
}
```
Scroll down to the *Flows* section.
![Fiddler AuthN console new Action trigger creation]()
1. Select the *External Authentication* option for the *Flow Type* dropdown.
2. Select the *+ Add trigger* button.
3. Select the *Post Authentication* option for the *Trigger Type* dropdown.
4. Select the *setAttributesOnOktaOIDCAuth* option for the *Actions* dropdown.
5. Select the *Save* button.
Add an organization metadata key so Fiddler can correctly identify and process this SSO connection. Set this once during setup.
1. Go to the *Metadata* section and select *Edit*.
![Fiddler AuthN console organization metadata section]()
2. Select the *Add* button, then enter the key `fiddler_sso_authentication_type` with the value `SSO:OKTA:OIDC`.
![Fiddler AuthN console organization metadata edit]()
3. Select the *Save* button next to the new entry.
Before validating, ensure your Okta user account is assigned to the new Okta application you created.
1. Open your Fiddler URL (e.g. `https://idpexample.dev.fiddler.ai`).
2. Ensure you see the Fiddler sign-in page and that it displays an SSO login button labeled with the name you configured (e.g. *Okta OIDC*).
![Fiddler application homepage displaying the new SSO login method in addition to the email sign-in form]()
3. Select the button and confirm that the Fiddler application loads.
![Fiddler application landing page]()
The first user to sign in to the Fiddler application is automatically assigned the Fiddler Org Admin role; subsequent members are Org Members by default.
## Getting Help
If sign-in fails, check the **Okta System Log** (*Reports → System Log*) for the failed attempt and its reason. For Fiddler-side issues, see the [SSO Authentication Guide](/reference/access-control/sso-authentication-guide). If the issue persists, contact your Fiddler representative with the specific error message.
## Important Notes
* **Data Storage**: Fiddler stores the following profile attributes from Okta: first name, last name, display name, email address, and group memberships (used to map users to Fiddler teams).
* **API Access**: For programmatic API access, users create an API key from the *Credentials* tab in Fiddler's *Settings* page.
* **Single Authentication Method**: Users can only authenticate via either SSO or email authentication, not both.
## Next Steps
After successful integration:
* **Train Users**: Provide guidance on accessing Fiddler through Okta SSO.
* **Configure Teams**: Map your identity provider groups to Fiddler teams — see [Mapping AD Groups to Fiddler Teams](/reference/access-control/mapping-ad-groups-to-fiddler-teams).
* **Test Group Sync**: Verify automatic group synchronization is working as expected.
* **Monitor Usage**: Review authentication logs and user access patterns.
# Okta SAML
Source: https://docs.fiddler.ai/reference/access-control/okta-integration-saml
Learn how to configure Fiddler with Okta for Single Sign-On (SSO) using the Security Assertion Markup Language (SAML) protocol.
## Overview
This integration allows your users to sign in to Fiddler using their existing Okta account, without needing a separate Fiddler password. Users are automatically provisioned on their first successful login — no manual invitations required.
## Prerequisites
Before starting, ensure you have:
* **Okta Administrator Access**: Permissions to create and configure applications in your Okta organization.
* **Fiddler AuthN Administrator Access**: Org Owner role in Fiddler's AuthN management console.
* **Deployment Information**: The hostname of your Fiddler deployment, e.g. `idpexample.dev.fiddler.ai`.
## Configuring Okta and Fiddler for Integration
The URL to the Fiddler AuthN management console is your Fiddler instance base URL prepended with `authn-`. For example, if your Fiddler base URL is `https://idpexample.dev.fiddler.ai` then you will access the AuthN management console at `https://authn-idpexample.dev.fiddler.ai`.
Sign in using the AuthN console Org Owner user account credentials provided by your Fiddler representative.
![Fiddler AuthN console sign-in page]()
Ensure your organization is selected in the dropdown. You may see the *fiddler* organization, but this is reserved for system use and should not be edited. Here we are using the *idpexample* organization.
![Fiddler AuthN console home page]()
Select *Settings* tab from the top menu and then select *Identity Providers* from the left navigation menu.
![Fiddler AuthN console add provider page]()
1. Select the *SAML* option in the *Add provider* section, which brings up the Sign in with SAML form.
2. Enter a name for the integration. This name is displayed on the SSO login button on the Fiddler sign-in page, so choose one your users will recognize.
3. Paste the following placeholder value into the *Metadata XML* text area. AuthN needs it to generate the URLs used when you create the Okta app integration; you will replace it in a later step.
**File:** `Placeholder Metadata XML value`
```
PD94bWwgdmVyc2lvbj0iMS4wIj8+DQo8bWQ6RW50aXR5RGVzY3JpcHRvciB4bWxuczptZD0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOm1ldGFkYXRhIg0KICAgICAgICAgICAgICAgICAgICAgdmFsaWRVbnRpbD0iMjAyNS0wOC0zMFQxMzo0NToxM1oiDQogICAgICAgICAgICAgICAgICAgICBjYWNoZUR1cmF0aW9uPSJQVDYwNDgwMFMiDQogICAgICAgICAgICAgICAgICAgICBlbnRpdHlJRD0iaHR0cDovL2xvY2FsaG9zdDo4MDgwIj4NCiAgICA8bWQ6U1BTU09EZXNjcmlwdG9yIEF1dGhuUmVxdWVzdHNTaWduZWQ9ImZhbHNlIiBXYW50QXNzZXJ0aW9uc1NpZ25lZD0iZmFsc2UiIHByb3RvY29sU3VwcG9ydEVudW1lcmF0aW9uPSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoyLjA6cHJvdG9jb2wiPg0KICAgICAgICA8bWQ6TmFtZUlERm9ybWF0PnVybjpvYXNpczpuYW1lczp0YzpTQU1MOjEuMTpuYW1laWQtZm9ybWF0OnVuc3BlY2lmaWVkPC9tZDpOYW1lSURGb3JtYXQ+DQogICAgICAgIDxtZDpBc3NlcnRpb25Db25zdW1lclNlcnZpY2UgQmluZGluZz0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOmJpbmRpbmdzOkhUVFAtUE9TVCINCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBMb2NhdGlvbj0iaHR0cDovL2xvY2FsaG9zdDo4MDgwL2NvbnN1bWUvZGF0YSINCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmRleD0iMSIgLz4NCiAgICAgICAgDQogICAgPC9tZDpTUFNTT0Rlc2NyaXB0b3I+DQo8L21kOkVudGl0eURlc2NyaXB0b3I+
```
![Fiddler AuthN console new SAML configuration]()
Select the *Create* button and then select the *Save* button. You will be returned to the Organization Settings page.
![Fiddler AuthN console saving new SAML IdP]()
Select your IdP from the list. Four URLs are displayed — copy the following three for the Okta app integration steps:
* ZITADEL Metadata URL
* ZITADEL ACS Login Form URL
* ZITADEL ACS Intent API URL
![Fiddler AuthN console SAML IdP URLs required for Okta configuration]()
1. In the Okta admin console, navigate to *Applications* and select the *Create App Integration* button. Select *SAML 2.0* for the *Sign-in method*, then select *Next*.
![Okta admin console Applications page]()
2. Enter a name for your application, e.g. `Fiddler IdP Example`, and select *Next*.
![Okta admin console enter your new app name]()
1. Enter the *ZITADEL ACS Login Form URL* copied from the AuthN console into the *Single sign-on URL* text box. Ensure the *Use this for Recipient URL and Destination URL* checkbox is selected.
2. Enter the *ZITADEL Metadata URL* from the AuthN console into the *Audience URI (SP Entity ID)* text box.
![Okta admin console configure SAML settings]()
3. Expand the *Advanced Settings* section.
4. In the *Other Requestable SSO URLs* section, select the *+ Add Another* button, enter the *ZITADEL ACS Intent API URL* copied from the AuthN console, and enter `0` for the *Index*.
![Okta admin console configure SAML settings - requestable URL]()
5. Select *Next*, then select *Finish* to complete the application creation.
6. On the created application, go to the *Sign On* tab, then select *Show legacy configuration* under *Attribute Statements*. Add the following attributes with *Name format* set to *Basic*, then select *Save*:
1. Name=`firstName`, Value=`user.firstName`
2. Name=`lastName`, Value=`user.lastName`
3. Name=`email`, Value=`user.email`
![Okta admin console SAML attribute statements]()
7. On the *Sign On* tab, go to *Settings → Sign on methods → SAML 2.0* and copy the *Metadata URL*.
![Okta admin console SAML 2.0 metadata URL]()
1. Return to the identity provider form in the Fiddler AuthN console (where you left off in step 4 — *Add and Configure New SAML Provider*).
2. Clear the *Metadata XML* text area.
3. Paste the *Metadata URL* copied from Okta into the *Metadata URL* text box.
![Fiddler AuthN console Metadata URL]()
1. Expand the *optional* section.
2. Ensure the *Automatic create* and *Automatic update* checkboxes are selected.
3. Set the *Determines whether an identity will be prompted to be linked to an existing account* dropdown to *Check for existing Username*.
![Fiddler AuthN console automatic create/update and check existing username settings]()
Select the *Save* button. You will be returned to the Organization Settings page.
![Fiddler AuthN console with newly created Okta SAML IdP]()
1. Select your IdP from the list and select the *Activate* button on the identity provider page.
![Fiddler AuthN console activate new Okta SAML IdP]()
2. Close the settings and then select *Login Behavior and Security* from the left nav menu and ensure the *External login allowed* checkbox is selected.
![Fiddler AuthN console allow external login behavior]()
3. Select the *Save* button.
![Fiddler AuthN console external login allowed]()
Select the *Actions* tab from the top menu.
![Fiddler AuthN console new custom Action script]()
1. Select the *New* button in the *Scripts* section to create a new action script.
2. Copy the *Okta SAML Action Script* below and paste it into the script text area.
3. Enter `setAttributesOnOktaSAMLAuth` in the *Name* text box.
4. Select the *Add* button.
**File:** `Okta SAML Action Script`
```javascript theme={null}
function setAttributesOnOktaSAMLAuth(ctx, api) {
let firstName = ctx.v1.providerInfo.attributes["firstName"];
let lastName = ctx.v1.providerInfo.attributes["lastName"];
let email = ctx.v1.providerInfo.attributes["email"];
let groups = ctx.v1.providerInfo.attributes["groups"];
let nameParts = [firstName, lastName];
let filteredParts = nameParts.filter(part => part);
let displayName = filteredParts.join(' ');
if (firstName != undefined) {
api.setFirstName(firstName);
}
if (lastName != undefined) {
api.setLastName(lastName);
}
if (email != undefined) {
email = String(email).toLowerCase();
api.setEmail(email);
api.setEmailVerified(true);
api.setPreferredUsername(email);
}
if (displayName) {
api.setDisplayName(displayName);
}
api.v1.user.appendMetadata('fiddler_authentication_type', 'SSO:OKTA:SAML');
if (groups === null || groups === undefined){
groups = []
}
api.v1.user.appendMetadata('fiddler_groups', groups);
}
```
Scroll down to the *Flows* section.
![Fiddler AuthN console new Action trigger creation]()
1. Select the *External Authentication* option for the *Flow Type* dropdown.
2. Select the *+ Add trigger* button.
3. Select the *Post Authentication* option for the *Trigger Type* dropdown.
4. Select the *setAttributesOnOktaSAMLAuth* option for the *Actions* dropdown.
5. Select the *Save* button.
Add an organization metadata key so Fiddler can correctly identify and process this SSO connection. Set this once during setup.
1. Go to the *Metadata* section and select *Edit*.
![Fiddler AuthN console organization metadata section]()
2. Select the *Add* button, then enter the key `fiddler_sso_authentication_type` with the value `SSO:OKTA:SAML`.
![Fiddler AuthN console organization metadata with Okta SAML authentication type]()
3. Select the *Save* button next to the new entry.
Before validating, ensure your Okta user account is assigned to the new Okta application you created.
1. Open your Fiddler URL (e.g. `https://idpexample.dev.fiddler.ai`).
2. Ensure you see the Fiddler sign-in page and that it displays an SSO login button labeled with the name you configured (e.g. *Okta SAML*).
![Fiddler application homepage displaying the new SSO login method in addition to the email sign-in form]()
3. Select the button and confirm that the Fiddler application loads.
![Fiddler application landing page]()
The first user to sign in to the Fiddler application is automatically assigned the Fiddler Org Admin role; subsequent members are Org Members by default.
## Getting Help
If sign-in fails, check the **Okta System Log** (*Reports → System Log*) for the failed attempt and its reason. For Fiddler-side issues, see the [SSO Authentication Guide](/reference/access-control/sso-authentication-guide). If the issue persists, contact your Fiddler representative with the specific error message.
## Important Notes
* **Data Storage**: Fiddler stores the following profile attributes from Okta: first name, last name, display name, email address, and group memberships (used to map users to Fiddler teams).
* **API Access**: For programmatic API access, users create an API key from the *Credentials* tab in Fiddler's *Settings* page.
* **Single Authentication Method**: Users can only authenticate via either SSO or email authentication, not both.
## Next Steps
After successful integration:
* **Train Users**: Provide guidance on accessing Fiddler through Okta SSO.
* **Configure Teams**: Map your identity provider groups to Fiddler teams — see [Mapping AD Groups to Fiddler Teams](/reference/access-control/mapping-ad-groups-to-fiddler-teams).
* **Test Group Sync**: Verify automatic group synchronization is working as expected.
* **Monitor Usage**: Review authentication logs and user access patterns.
# PingOne SAML
Source: https://docs.fiddler.ai/reference/access-control/ping-identity-saml
Learn how to configure Fiddler with PingOne for Single Sign-On (SSO) using the Security Assertion Markup Language (SAML) protocol.
## Overview
This integration allows your users to sign in to Fiddler using their existing PingOne account, without needing a separate Fiddler password. Users are automatically provisioned on their first successful login — no manual invitations required.
## Prerequisites
Before starting, ensure you have:
* **PingOne Administrator Access**: Permissions to create and configure applications in your PingOne environment.
* **Fiddler AuthN Administrator Access**: Org Owner role in Fiddler's AuthN management console.
* **Deployment Information**: The hostname of your Fiddler deployment, e.g. `idpexample.dev.fiddler.ai`.
## Configuring PingOne and Fiddler for Integration
The URL to the Fiddler AuthN management console is your Fiddler instance base URL prepended with `authn-`. For example, if your Fiddler base URL is `https://idpexample.dev.fiddler.ai` then you will access the AuthN management console at `https://authn-idpexample.dev.fiddler.ai`.
Sign in using the AuthN console Org Owner user account credentials provided by your Fiddler representative.
![Fiddler AuthN console sign-in page]()
Ensure your organization is selected in the dropdown. You may see the *fiddler* organization, but this is reserved for system use and should not be edited. Here we are using the *idpexample* organization.
![Fiddler AuthN console home page]()
Select *Settings* tab from the top menu and then select *Identity Providers* from the left navigation menu.
![Fiddler AuthN console add provider page]()
1. Select the *SAML* option in the *Add provider* section, which brings up the Sign in with SAML form.
2. Enter a name for the integration. This name is displayed on the SSO login button on the Fiddler sign-in page, so choose one your users will recognize.
3. Paste the following placeholder value into the *Metadata XML* text area. AuthN needs it to generate the URLs used when you create the PingOne app integration; you will replace it in a later step.
**File:** `Placeholder Metadata XML value`
```
PD94bWwgdmVyc2lvbj0iMS4wIj8+DQo8bWQ6RW50aXR5RGVzY3JpcHRvciB4bWxuczptZD0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOm1ldGFkYXRhIg0KICAgICAgICAgICAgICAgICAgICAgdmFsaWRVbnRpbD0iMjAyNS0wOC0zMFQxMzo0NToxM1oiDQogICAgICAgICAgICAgICAgICAgICBjYWNoZUR1cmF0aW9uPSJQVDYwNDgwMFMiDQogICAgICAgICAgICAgICAgICAgICBlbnRpdHlJRD0iaHR0cDovL2xvY2FsaG9zdDo4MDgwIj4NCiAgICA8bWQ6U1BTU09EZXNjcmlwdG9yIEF1dGhuUmVxdWVzdHNTaWduZWQ9ImZhbHNlIiBXYW50QXNzZXJ0aW9uc1NpZ25lZD0iZmFsc2UiIHByb3RvY29sU3VwcG9ydEVudW1lcmF0aW9uPSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoyLjA6cHJvdG9jb2wiPg0KICAgICAgICA8bWQ6TmFtZUlERm9ybWF0PnVybjpvYXNpczpuYW1lczp0YzpTQU1MOjEuMTpuYW1laWQtZm9ybWF0OnVuc3BlY2lmaWVkPC9tZDpOYW1lSURGb3JtYXQ+DQogICAgICAgIDxtZDpBc3NlcnRpb25Db25zdW1lclNlcnZpY2UgQmluZGluZz0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOmJpbmRpbmdzOkhUVFAtUE9TVCINCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBMb2NhdGlvbj0iaHR0cDovL2xvY2FsaG9zdDo4MDgwL2NvbnN1bWUvZGF0YSINCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmRleD0iMSIgLz4NCiAgICAgICAgDQogICAgPC9tZDpTUFNTT0Rlc2NyaXB0b3I+DQo8L21kOkVudGl0eURlc2NyaXB0b3I+
```
![Fiddler AuthN console new SAML configuration]()
Select the *Create* button and then select the *Save* button. You will be returned to the Organization Settings page.
![Fiddler AuthN console saving new SAML IdP]()
Select your IdP from the list. Four URLs are displayed — copy the following three for the PingOne app integration steps:
* ZITADEL Metadata URL
* ZITADEL ACS Login Form URL
* ZITADEL ACS Intent API URL
![Fiddler AuthN console SAML IdP URLs required for PingOne configuration]()
1. In the PingOne admin console, navigate to *Applications* and select the *+* icon.
2. Enter a name for your application, e.g. `Fiddler IdP Example`, select *SAML Application*, then select *Configure*.
![PingOne admin console SAML application name and type]()
1. Under *Provide Application Metadata*, select *Manually Enter*.
2. Enter the *ZITADEL ACS Login Form URL* into the *ACS URL* text box.
3. Add the *ZITADEL ACS Intent API URL* as a second ACS URL.
4. Enter the *ZITADEL Metadata URL* into the *Entity ID* text box.
5. Select *Save*.
![PingOne admin console SAML configuration]()
1. Open the created application and go to the *Configuration* section. Select the edit icon and ensure *Sign Assertion and Response* is selected, then select *Save*.
![PingOne admin console sign assertion and response setting]()
2. Go to the *Attribute Mappings* section. Select the edit icon and add the following mappings, then select *Save*:
1. Name=`email`, Value=`Email Address`
2. Name=`firstName`, Value=`Given Name`
3. Name=`lastName`, Value=`Family Name`
4. Name=`groups`, Value=`Group Names`
![PingOne admin console SAML attribute mappings]()
3. At the top of the application, toggle it to *Active*.
4. Go to the *Overview* section, then under *Connection Details*, copy the *IDP Metadata URL*.
![PingOne admin console application overview with IDP Metadata URL]()
1. Return to the identity provider form in the Fiddler AuthN console (where you left off in step 4 — *Add and Configure New SAML Provider*).
2. Clear the *Metadata XML* text area.
3. Paste the *IDP Metadata URL* copied from PingOne into the *Metadata URL* text box.
![Fiddler AuthN console Metadata URL]()
1. Expand the *optional* section.
2. Ensure the *Automatic create* and *Automatic update* checkboxes are selected.
3. Set the *Determines whether an identity will be prompted to be linked to an existing account* dropdown to *Check for existing Username*.
![Fiddler AuthN console automatic create/update and check existing username settings]()
Select the *Save* button. You will be returned to the Organization Settings page.
![Fiddler AuthN console with newly created PingOne SAML IdP]()
1. Select your IdP from the list and select the *Activate* button on the identity provider page.
![Fiddler AuthN console activate new PingOne SAML IdP]()
2. Close the settings and then select *Login Behavior and Security* from the left nav menu and ensure the *External login allowed* checkbox is selected.
![Fiddler AuthN console allow external login behavior]()
3. Select the *Save* button.
![Fiddler AuthN console external login allowed]()
Select the *Actions* tab from the top menu.
![Fiddler AuthN console new custom Action script]()
1. Select the *New* button in the *Scripts* section to create a new action script.
2. Copy the *PingOne SAML Action Script* below and paste it into the script text area.
3. Enter `setAttributesOnPingSAMLAuth` in the *Name* text box.
4. Select the *Add* button.
**File:** `PingOne SAML Action Script`
```javascript theme={null}
function setAttributesOnPingSAMLAuth(ctx, api) {
let firstName = ctx.v1.providerInfo.attributes["firstName"];
let lastName = ctx.v1.providerInfo.attributes["lastName"];
let email = ctx.v1.providerInfo.attributes["email"];
let groups = ctx.v1.providerInfo.attributes["groups"];
let nameParts = [firstName, lastName];
let filteredParts = nameParts.filter(part => part);
let displayName = filteredParts.join(' ');
if (firstName != undefined) {
api.setFirstName(firstName);
}
if (lastName != undefined) {
api.setLastName(lastName);
}
if (email != undefined) {
email = String(email).toLowerCase();
api.setEmail(email);
api.setEmailVerified(true);
api.setPreferredUsername(email);
}
if (displayName) {
api.setDisplayName(displayName);
}
api.v1.user.appendMetadata('fiddler_authentication_type', 'SSO:PING:SAML');
if (groups === null || groups === undefined) {
groups = []
}
api.v1.user.appendMetadata('fiddler_groups', groups);
}
```
Scroll down to the *Flows* section.
![Fiddler AuthN console new Action trigger creation]()
1. Select the *External Authentication* option for the *Flow Type* dropdown.
2. Select the *+ Add trigger* button.
3. Select the *Post Authentication* option for the *Trigger Type* dropdown.
4. Select the *setAttributesOnPingSAMLAuth* option for the *Actions* dropdown.
5. Select the *Save* button.
Add an organization metadata key so Fiddler can correctly identify and process this SSO connection. Set this once during setup.
1. Go to the *Metadata* section and select *Edit*.
![Fiddler AuthN console organization metadata section]()
2. Select the *Add* button, then enter the key `fiddler_sso_authentication_type` with the value `SSO:PING:SAML`.
![Fiddler AuthN console organization metadata with PingOne SAML authentication type]()
3. Select the *Save* button next to the new entry.
Before validating, ensure your PingOne user account is assigned to the new PingOne application you created.
1. Open your Fiddler URL (e.g. `https://idpexample.dev.fiddler.ai`).
2. Ensure you see the Fiddler sign-in page and that it displays an SSO login button labeled with the name you configured (e.g. *PingOne SAML*).
![Fiddler application homepage displaying the new SSO login method in addition to the email sign-in form]()
3. Select the button and confirm that the Fiddler application loads.
![Fiddler application landing page]()
The first user to sign in to the Fiddler application is automatically assigned the Fiddler Org Admin role; subsequent members are Org Members by default.
## Getting Help
If sign-in fails, check the **PingOne Audit Log** (*Monitoring → Audit*) for the failed attempt and its reason. For Fiddler-side issues, see the [SSO Authentication Guide](/reference/access-control/sso-authentication-guide). If the issue persists, contact your Fiddler representative with the specific error message.
## Important Notes
* **Data Storage**: Fiddler stores the following profile attributes from PingOne: first name, last name, display name, email address, and group memberships (used to map users to Fiddler teams).
* **API Access**: For programmatic API access, users create an API key from the *Credentials* tab in Fiddler's *Settings* page.
* **Single Authentication Method**: Users can only authenticate via either SSO or email authentication, not both.
## Next Steps
After successful integration:
* **Train Users**: Provide guidance on accessing Fiddler through PingOne SSO.
* **Configure Teams**: Map your identity provider groups to Fiddler teams — see [Mapping AD Groups to Fiddler Teams](/reference/access-control/mapping-ad-groups-to-fiddler-teams).
* **Test Group Sync**: Verify automatic group synchronization is working as expected.
* **Monitor Usage**: Review authentication logs and user access patterns.
# Role-Based Access Control
Source: https://docs.fiddler.ai/reference/access-control/role-based-access
Learn how Fiddler uses role-based access control with resources and roles. Discover how to manage access with resources, roles, and permissions in your company.
### Overview of Role-Based Access Control
Fiddler supports Role-Based Access Control (RBAC) using resources and roles. This documentation outlines the resources, roles, and permissions available in Fiddler, enabling you to manage access control for your organization.
### Understanding Resources
Resources are entities within Fiddler that users can access and interact with. There are two main resource types:
#### Organization Resources
* Organization: Represents your entire Fiddler setup, including projects and users.
* Settings: General information, login details, notification settings, and integration configurations.
* Users: Individual users with accounts in your Fiddler organization.
* Teams: Groups of users within your organization.
* Each user can be a member of zero or more teams.
* Team roles are associated with project roles, i.e., teams can be granted\
**Project Viewer**, **Project Writer**, or **Project Admin** permissions for a project.
* Evaluators: Predefined and custom metrics used to assess model or application outputs.
* Global Agentic Custom Metrics: Organization-wide custom metrics for GenAI applications.
#### Project Resources
* Projects: Contain models, data, and configurations for a specific ML application.
* Models: Machine learning models onboarded to Fiddler for monitoring and explainability.
* Project Settings: Configurations related to project access and user permissions.
* Alerts: Notifications generated by Fiddler based on monitoring data.
* Charts & Dashboards: Visualizations of your model performance and data insights.
* Application: GenAI applications for agentic workflows.
* Evaluator Rules: Rules that define which evaluators run against application spans and how inputs are mapped.
* Project-Scoped Agentic Custom Metrics: Project-scoped custom metrics for GenAI applications.
### Understanding Roles
Roles define the level of access a user has to Fiddler resources:
#### Organization Roles
* Org Admin: Has access to manage users, teams, projects, and organization settings. However, this role cannot access the project data unless explicitly given access by the Project Admin.
* Org Member: Limited access to organization settings and cannot create projects.
#### Project Roles
* Project Admin: Manages all aspects of a project, including models, settings, alerts, and user access (except deleting the project).
* Project Writer: Can view and edit most project details (models, settings, alerts), but cannot delete the project or invite other users.
* Project Viewer: Can view project details and model content, but cannot edit anything except charts and dashboards (read-only access).
### Understanding Permissions
#### Permission types
Permission types are used in combination with resources and roles to define the access control rules in Fiddler. Fiddler's RBAC access control uses the following permission types to define the level of access a user has to resources:
* List: This permission allows users to view a list of resources, but does not grant access to view details or interact with the resources in any way. For example, a user with the "List" permission for projects can see a list of project names, but cannot view project details or settings.
* Read: This permission enables users to view details of a resource, but does not grant access to edit or modify the resource in any way.
* Create: This permission allows users to create new resources, such as projects, models, or alerts.
* Edit: This permission enables users to modify existing resources, such as updating project settings or editing model configurations.
* Delete: This permission allows users to delete resources, such as deleting a project or a model.
#### Organization Level permissions
* Org Admin: Full access to organization settings and resources.
* Org Member: Limited access to organization settings.
**Legend:** ✅ = Granted ❌ = Not Granted N/A = Not Implemented
| Resource | Role | List | Read | Create | Edit | Delete |
|---|
| Org Settings: General, Credentials, Email, PagerDuty, Webhook | Admin | ❌ | ✅ | ✅ | ✅ | ✅ |
| Member | ❌ | ✅ | ❌ | ❌ | ❌ |
| Org Settings / Access / Teams & Invitations | Admin | ❌ | ✅ | ✅ | ✅ | ✅ |
| Member | ❌ | ❌ | ❌ | ❌ | ❌ |
| Org Settings / Access / Users | Admin | ❌ | ✅ | ✅ | ✅ | ✅ |
| Member | ❌ | ✅ | ❌ | ❌ | ❌ |
| Project | Admin | ✅ | ❌ | ✅ | ✅ | ✅ |
| Member | ❌ | ❌ | ❌ | ❌ | ❌ |
| Project / Project Settings | Admin | ❌ | ✅ | ✅ | ✅ | ✅ |
| Member | ❌ | ❌ | ❌ | ❌ | ❌ |
| Evaluators | Admin | ✅ | ✅ | ✅ | N/A | N/A |
| Member | ✅ | ✅ | ✅ | N/A | N/A |
| Global Agentic Custom Metrics | Admin | ✅ | ✅ | ✅ | N/A | ✅ |
| Member | ✅ | ✅ | ✅ | N/A | ❌ |
#### Project Level permissions
An “Org Admin” or “Org Member” user can have the following access to the Projects
* Project Admin: Full access to project resources.
* Project Writer: Limited access to project resources, excluding deletion and user invitation.
* Project Viewer: Read-only access to project resources.
| Resource | Role | Read | Create | Edit | Delete |
|---|
| Project | Admin | ✅ | ❌ | ✅ | ✅ |
| Writer | ✅ | ❌ | ❌ | ❌ |
| Viewer | ✅ | ❌ | ❌ | ❌ |
| Project / Project Settings | Admin | ✅ | ❌ | ✅ | ✅ |
| Writer | ✅ | ❌ | ❌ | ❌ |
| Viewer | ✅ | ❌ | ❌ | ❌ |
| Project / Models: Schema, Artifact, Baseline, Dataset, Custom Metric, Segments | Admin | ✅ | ✅ | ✅ | ✅ |
| Writer | ✅ | ✅ | ✅ | ✅ |
| Viewer | ✅ | ❌ | ❌ | ❌ |
| Project / Model / Alerts | Admin | ✅ | ✅ | ✅ | ✅ |
| Writer | ✅ | ✅ | ✅ | ✅ |
| Viewer | ✅ | ❌ | ❌ | ❌ |
| Project / Bookmarks | Admin | ✅ | ✅ | ❌ | ✅ |
| Writer | ✅ | ✅ | ❌ | ✅ |
| Viewer | ✅ | ✅ | ❌ | ✅ |
| Project / Charts | Admin | ✅ | ✅ | ✅ | ✅ |
| Writer | ✅ | ✅ | ✅ | ✅ |
| Viewer | ✅ | ✅ | ✅ | ✅ |
| Project / Dashboards | Admin | ✅ | ✅ | ✅ | ✅ |
| Writer | ✅ | ✅ | ✅ | ✅ |
| Viewer | ✅ | ✅ | ✅ | ✅ |
| Model Deployment | Admin | ✅ | ✅ | ✅ | ✅ |
| Writer | ✅ | ✅ | ✅ | ✅ |
| Viewer | ✅ | ❌ | ❌ | ❌ |
| Application | Admin | ✅ | ✅ | N/A | ✅ |
| Writer | ✅ | ✅ | N/A | ✅ |
| Viewer | ✅ | ❌ | N/A | ❌ |
| Evaluator Rules | Admin | ✅ | ✅ | N/A | ✅ |
| Writer | ✅ | ✅ | N/A | ✅ |
| Viewer | ✅ | ❌ | N/A | ❌ |
| Project-Scoped Agentic Custom Metrics | Admin | ✅ | ✅ | N/A | ✅ |
| Writer | ✅ | ✅ | N/A | ❌ |
| Viewer | ✅ | ❌ | N/A | ❌ |
### Getting Started
* The default "Org Admin" role is created during Fiddler installation.
* Assign roles to users and teams to control access to resources.
* Use the permissions matrix to understand the access levels for each role.
Click [here](/reference/administration/settings#access) for more information on teams.
# Microsoft Entra ID OIDC
Source: https://docs.fiddler.ai/reference/access-control/single-sign-on-with-azure-ad
Learn how to configure Fiddler with Microsoft Entra ID (formerly Azure AD) for Single Sign-On (SSO) using the OpenID Connect (OIDC) protocol.
## Overview
This integration allows your users to sign in to Fiddler using their existing Microsoft Entra ID account, without needing a separate Fiddler password. Users are automatically provisioned on their first successful login — no manual invitations required. Entra ID group memberships can be synchronized to Fiddler teams.
## Prerequisites
Before starting, ensure you have:
* **Microsoft Entra ID Administrator Access**: Permissions to register applications and grant admin consent in your Entra ID tenant.
* **Fiddler AuthN Administrator Access**: Org Owner role in Fiddler's AuthN management console.
* **Deployment Information**: The hostname of your Fiddler deployment, e.g. `idpexample.dev.fiddler.ai`.
* **Entra ID P2 License** (cloud-native group sync only): required to emit group display names for cloud-native groups. Not needed for basic SSO or for groups synced from on-premises AD.
## Configuring Microsoft Entra ID
Fiddler requires two redirect URIs on the Entra ID application — one added during registration, the second under *Authentication*:
* `https://authn-{base_url}/ui/login/login/externalidp/callback`
* `https://authn-{base_url}/idps/callback`
Replace `{base_url}` with your Fiddler deployment host (e.g. `idpexample.dev.fiddler.ai`).
1. In the [Microsoft Entra admin center](https://entra.microsoft.com/), go to *Entra ID → App registrations* and select *New registration*.
2. Enter a *Name*, e.g. `Fiddler IdP Example`.
3. Under *Supported account types*, select *Accounts in this organizational directory only (Default Directory only - Single tenant)*.
4. Under *Redirect URI*, select the *Web* platform and enter the first redirect URI (replace the host with your deployment): `https://authn-idpexample.dev.fiddler.ai/ui/login/login/externalidp/callback`.
5. Select *Register*.
![Microsoft Entra ID app registration]()
1. In the registered application, go to *Manage → Authentication*.
2. Under *Redirect URI configuration*, select *Add Redirect URI*.
3. Select *Web* and add the second redirect URI (replace the host with your deployment): `https://authn-idpexample.dev.fiddler.ai/idps/callback`.
4. Select *Configure*.
![Microsoft Entra ID add second redirect URI]()
1. In the registered application, go to *Certificates & secrets*. Under *Client secrets*, select *New client secret*.
2. Add a description, choose an expiry, and select *Add*.
3. Copy the secret *Value* immediately — it is shown only once.
![Microsoft Entra ID client secret]()
1. In the registered application, go to *API permissions* and select *Add a permission → Microsoft Graph → Delegated permissions*.
2. Select `openid`, `email`, `profile`, `offline_access`, and `User.Read`, then select *Add permissions*.
3. Select *Grant admin consent for Default Directory*.
![Microsoft Entra ID API permissions]()
Add the claims Fiddler needs to the ID token.
1. In the registered application, go to *Token configuration*.
2. Select *Add optional claim*, choose the *ID* token type, and add `email`, `given_name`, and `family_name`. When prompted, enable the option to turn on the Microsoft Graph `email` and `profile` permissions (required for these claims to appear in the token), then select *Add*.
![Microsoft Entra ID add optional claims]()
3. Select *Add groups claim*, choose *Groups assigned to the application*, select the *ID* token, then select *Add*. Only the ID token is required for Fiddler. By default Entra emits group IDs (GUIDs); to emit the group **names** that Fiddler maps to teams, complete [Enable Group Sync](#enable-group-sync) below.
![Microsoft Entra ID groups claim configuration]()
From the registered application's *Overview* page, copy the *Application (client) ID* and the *Directory (tenant) ID*. You will need these, along with the client secret, when configuring Fiddler.
![Microsoft Entra ID application overview with client and tenant IDs]()
## Configuring Fiddler
The URL to the Fiddler AuthN management console is your Fiddler instance base URL prepended with `authn-`. For example, if your Fiddler base URL is `https://idpexample.dev.fiddler.ai` then you will access the AuthN management console at `https://authn-idpexample.dev.fiddler.ai`.
Sign in using the AuthN console Org Owner user account credentials provided by your Fiddler representative.
![Fiddler AuthN console sign-in page]()
Ensure your organization is selected in the dropdown. You may see the *fiddler* organization, but this is reserved for system use and should not be edited. Here we are using the *idpexample* organization.
![Fiddler AuthN console home page]()
Select *Settings* tab from the top menu and then select *Identity Providers* from the left navigation menu.
![Fiddler AuthN console add provider page]()
1. Select the *Microsoft* option in the *Add provider* section, which brings up the Microsoft Provider form.
2. Note the callback URL shown in the form — it corresponds to the redirect URIs you registered in Entra ID earlier, so no further changes are needed in Entra ID.
![Fiddler AuthN console add new Microsoft provider form with callback URL]()
1. In the Microsoft Provider form, enter the following values:
1. Enter a name in the *Name* text box. This name is displayed on the SSO login button on the Fiddler sign-in page, so choose one your users will recognize.
2. In the *Client ID* and *Client Secret* text boxes, paste those values copied from Entra ID.
![Fiddler AuthN console Microsoft provider client credentials]()
1. Expand the *optional* section.
2. Ensure the *Scopes List* includes `openid`, `profile`, and `email`.
3. Under *Tenant Type*, select *Tenant ID* and paste your *Directory (tenant) ID*.
4. Ensure the *Automatic create* and *Automatic update* checkboxes are selected.
5. Set the *Determines whether an identity will be prompted to be linked to an existing account* dropdown to *Check for existing Username*.
![Fiddler AuthN console automatic create/update and check existing username settings]()
Select the *Create* button and then select the *Save* button. You will be returned to the Organization Settings page.
![Fiddler AuthN console with newly created Microsoft Entra ID IdP]()
1. Select your IdP from the list and select the *Activate* button on the identity provider page.
![Fiddler AuthN console activate new Microsoft Entra ID IdP]()
2. Close the settings and then select *Login Behavior and Security* from the left nav menu and ensure the *External login allowed* checkbox is selected.
![Fiddler AuthN console allow external login behavior]()
3. Select the *Save* button.
![Fiddler AuthN console external login allowed]()
Select the *Actions* tab from the top menu.
![Fiddler AuthN console new custom Action script]()
1. Select the *New* button in the *Scripts* section to create a new action script.
2. Copy the *Microsoft Entra ID Action Script* below and paste it into the script text area.
3. Enter `setAttributesOnAzureOIDCAuth` in the *Name* text box.
4. Select the *Add* button.
**File:** `Microsoft Entra ID Action Script`
```javascript theme={null}
function setAttributesOnAzureOIDCAuth(ctx, api) {
let firstName = ctx.getClaim('given_name');
let lastName = ctx.getClaim('family_name');
let email = ctx.getClaim('email');
let groups = ctx.getClaim('groups');
let nameParts = [firstName, lastName];
let filteredParts = nameParts.filter(part => part);
let displayName = filteredParts.join(' ');
if (firstName != undefined) {
api.setFirstName(firstName);
}
if (lastName != undefined) {
api.setLastName(lastName);
}
if (email != undefined) {
email = email.toLowerCase();
api.setEmail(email);
api.setEmailVerified(true);
api.setPreferredUsername(email);
}
if (displayName) {
api.setDisplayName(displayName);
}
api.v1.user.appendMetadata('fiddler_authentication_type', 'SSO:AZURE:OIDC');
if (groups === null || groups === undefined){
groups = []
}
api.v1.user.appendMetadata('fiddler_groups', groups);
}
```
Scroll down to the *Flows* section.
![Fiddler AuthN console new Action trigger creation]()
1. Select the *External Authentication* option for the *Flow Type* dropdown.
2. Select the *+ Add trigger* button.
3. Select the *Post Authentication* option for the *Trigger Type* dropdown.
4. Select the *setAttributesOnAzureOIDCAuth* option for the *Actions* dropdown.
5. Select the *Save* button.
Add an organization metadata key so Fiddler can correctly identify and process this SSO connection. Set this once during setup.
1. Go to the *Metadata* section and select *Edit*.
![Fiddler AuthN console organization metadata section]()
2. Select the *Add* button, then enter the key `fiddler_sso_authentication_type` with the value `SSO:AZURE:OIDC`.
![Fiddler AuthN console organization metadata with Microsoft Entra ID authentication type]()
3. Select the *Save* button next to the new entry.
Before validating, ensure your Entra ID account is assigned to the registered application (or that user assignment is not required).
1. Open your Fiddler URL (e.g. `https://idpexample.dev.fiddler.ai`).
2. Ensure you see the Fiddler sign-in page and that it displays an SSO login button labeled with the name you configured.
![Fiddler application homepage displaying the new SSO login method in addition to the email sign-in form]()
3. Select the button and confirm that the Fiddler application loads.
![Fiddler application landing page]()
The first user to sign in to the Fiddler application is automatically assigned the Fiddler Org Admin role; subsequent members are Org Members by default.
## Enable Group Sync
Fiddler maps identity provider group **names** (with a configurable prefix) to Fiddler teams and roles — see [Mapping AD Groups to Fiddler Teams](/reference/access-control/mapping-ad-groups-to-fiddler-teams) for the naming convention. By default, Entra ID emits group **IDs** (GUIDs) in the token, which Fiddler cannot map — so group sync requires emitting group display names.
### Cloud-native groups
Emitting display names for cloud-native groups requires an **Entra ID P2** license.
1. In *Enterprise applications*, open your application, go to *Users and groups*, and assign the users and groups that should be sent to Fiddler. Only groups assigned to the application are returned.
2. In the registered application, go to *Manage → Manifest*. Download a copy of the manifest first as a backup, then update it:
1. Set `groupMembershipClaims` to `ApplicationGroup`.
2. In `optionalClaims.idToken`, set the `groups` claim's `additionalProperties` to `["cloud_displayname"]` so display names are returned.
3. Save the manifest.
![Microsoft Entra ID manifest group membership claims]()
![Microsoft Entra ID manifest groups claim cloud_displayname]()
### On-premises-synced groups
For groups synced from on-premises AD, set the groups claim format to `sAMAccountName` in *Token configuration*. This requires no manifest change or P2 license.
## Getting Help
If sign-in fails, review the Entra ID sign-in logs (*Entra ID → Monitoring & health → Sign-in logs*) for the failed attempt and its reason. Common Entra error codes:
* `AADSTS50105` — the user is not assigned to the application. Assign the user (or their group) to the registered application.
* `AADSTS700016` — the application was not found. Verify the Client ID and tenant are correct.
* `AADSTS90094` — admin consent is required. Grant admin consent for the API permissions.
For Fiddler-side issues, see the [SSO Authentication Guide](/reference/access-control/sso-authentication-guide). If the issue persists, contact your Fiddler representative with the specific error message.
## Important Notes
* **Data Storage**: Fiddler stores the following profile attributes from Entra ID: first name, last name, display name, email address, and group memberships (used to map users to Fiddler teams).
* **API Access**: For programmatic API access, users create an API key from the *Credentials* tab in Fiddler's *Settings* page.
* **Single Authentication Method**: Users can only authenticate via either SSO or email authentication, not both.
* **Client Secret Expiration**: Entra ID client secrets expire. Track the expiry date and rotate the secret before it lapses to avoid login outages.
## Next Steps
After successful integration:
* **Train Users**: Provide guidance on accessing Fiddler through Microsoft Entra ID SSO.
* **Configure Teams**: Map your Entra ID groups to Fiddler teams — see [Mapping AD Groups to Fiddler Teams](/reference/access-control/mapping-ad-groups-to-fiddler-teams).
* **Test Group Sync**: Verify automatic group synchronization is working as expected.
* **Monitor Usage**: Review authentication logs and set a reminder for client secret expiration.
# SSO Authentication Guide
Source: https://docs.fiddler.ai/reference/access-control/sso-authentication-guide
Configure Single Sign-On authentication for Fiddler with Okta, Azure AD, Google, Ping, and others. Complete setup guide with troubleshooting tips.
This guide covers Single Sign-On authentication in Fiddler, including setup procedures, supported identity providers, and user management workflows.
## Overview
Single Sign-On (SSO) authentication allows users to access Fiddler using their existing organizational credentials from identity providers like Okta, Microsoft Entra ID, Google, and PingOne. SSO streamlines user access and reduces password management overhead.
## When to Use SSO Authentication
SSO authentication is ideal for:
* Organizations with existing identity providers
* Environments requiring centralized user management
* Compliance requirements mandating enterprise authentication
* Large user bases where manual user provisioning is impractical
## How SSO Works with Fiddler
### User Provisioning
**Automatic User Creation**: When users successfully authenticate through your SSO provider for the first time, Fiddler automatically creates their user account with basic profile information.
**No Manual Creation Required**: Unlike email authentication, SSO users don't need to be manually created in the AuthN console—they gain access automatically upon successful SSO authentication.
Note that auto-provisioned users will be created with the Fiddler Org Member role by default. Edit a user's Organization role in the [Access tab of the Settings page](/reference/administration/settings#users).
### Authentication Flow
1. **User Access**: User navigates to Fiddler login page
2. **SSO Redirect**: User clicks "Sign in with SSO" and is redirected to your identity provider
3. **Identity Provider Authentication**: User authenticates with their organizational credentials
4. **Automatic Provisioning**: If first login, Fiddler creates the user account automatically
5. **Access Granted**: User gains access to Fiddler as an Org Member and potentially additional privileges if [Group Syncing](/reference/access-control/mapping-ad-groups-to-fiddler-teams) is implemented
## Supported Identity Providers
Fiddler supports major enterprise identity providers through industry-standard protocols:
| Identity Provider | Supported Protocols | Configuration Guide |
| ------------------------------------------ | ------------------- | ----------------------------------------------------------------------------------- |
| **Okta** | OIDC | [Okta OIDC Integration](/reference/access-control/okta-integration) |
| **Okta** | SAML | [Okta SAML Integration](/reference/access-control/okta-integration-saml) |
| **Microsoft Entra ID** (formerly Azure AD) | OIDC | [Azure AD OIDC Integration](/reference/access-control/single-sign-on-with-azure-ad) |
| **Google** | OIDC | [Google OIDC Integration](/reference/access-control/google-integration) |
| **PingOne** | SAML | [PingOne SAML Integration](/reference/access-control/ping-identity-saml) |
## SSO Configuration Process
### Prerequisites
Before configuring SSO, ensure you have:
* Administrative access to your identity provider
* Access to the Fiddler AuthN management console
* Access to the AuthN user account having the "Org Owner" role
* Required information from your identity provider (client IDs, metadata URLs, certificates)
### General Configuration Steps
These are the basic steps to follow for most IdPs. Follow the specific guide for your required IdP and protocol.
**Step 1: Access Authentication Management Console**
1. Log into the AuthN authentication management console
![AuthN console login page]()
2. Select your customer organization from the dropdown
![AuthN console home page]()
3. Navigate to **Settings > Login and Access > Identity Providers**
![AuthN console org settings page]()
4. Select your desired provider by selecting its icon in the **Add Provider section**
**Step 2: Configure Identity Provider Integration**
More specific configuration steps are in each IdP + protocol guide.
1. **Provider Name**: Enter a descriptive name for your SSO integration
2. **Copy AuthN Settings**: If required, copy AuthN settings to use in creating the application in your IdP
3. **IdP Required Fields**: Populate your IdP's required fields
4. **Connection Details**: Copy required settings from your IdP:
* Client ID or Application ID
* Metadata URL or Issuer URL
* Client Secret (if required)
* Certificate information (for SAML)
**Step 3: Enable User Provisioning**
Expand the optional section and configure automatic user provisioning settings:
* ✅ **Enable "Automatic creation"** - Creates new users on first successful login
* ✅ **Enable "Automatic update"** - Updates user information from identity provider
* ✅ **Select "Check for existing username"** - Links identities to existing accounts when appropriate
**Step 4: Configure Attribute Mapping**
Ensure proper mapping of user attributes from your identity provider to Fiddler. These values will differ between IdPs and protocols:
**Example Required Attributes**:
* **First Name** (`firstName`, `given_name`)
* **Last Name** (`lastName`, `family_name`)
* **Email Address** (`email`)
**Optional Attributes**:
* **Groups** (`groups`) - For automated group-based access control see [Mapping LDAP Groups](/reference/access-control/mapping-ad-groups-to-fiddler-teams) guide
**Step 5: IdP-specific Action Script and Trigger**
Each IdP integration guide will provide an action script and trigger type:
Action Script
* Paste the Fiddler-provided script into the text area
* Paste the script name into the Name text box
Trigger
* Set the Trigger Type option per the guide
* Set the Actions dropdown option per the guide
**Step 5: Test and Validate**
1. Save your SSO configuration
2. Test authentication with a sample user account
3. Verify user information is properly mapped
4. Confirm automatic provisioning works as expected
## Group Synchronization
### Supported Providers
Group synchronization is available for these identity providers:
* **Okta** (OIDC and SAML)
* **Microsoft Entra ID** (OIDC with proper configuration)
* **PingOne** (SAML)
## User Management with SSO
### Automatic User Provisioning
**First Login Process**:
1. User authenticates successfully through SSO
2. Fiddler automatically creates user account with information from the IdP
3. User receives default organization member role (the very first user to login will be assigned the Org Admin role)
4. Additional permissions can be assigned through Fiddler teams or individual roles
**Ongoing User Updates**:
* User information automatically updates from the IdP on each login
* Group memberships sync automatically (if configured)
* User status changes (deactivation/reactivation) can be managed through the IdP (note that Fiddler deactivates user accounts rather than deletes)
## Mixed Authentication Environments
### Combining SSO and Email Authentication
Organizations can use both SSO and email authentication simultaneously:
* **SSO Users**: Automatically provisioned from identity provider
* **Email Users**: Manually added through the AuthN management console
* **Separate Login Paths**: Users choose appropriate authentication method at login if more than one path has been enabled
### User Account Constraints
* **Single Authentication Method**: Each user account uses either SSO or email authentication, not both
* **Account Linking**: Existing email-authenticated users can be linked to SSO identities under specific conditions
## Troubleshooting Common Issues
### External User Not Found
If you see the "External User Not Found" screen after SSO login, follow these steps in order:
![External User Not Found error screen after SSO login]()
**Step 1: Verify identity provider settings**
In the AuthN management console, navigate to the Identity Provider configuration and ensure the following settings are configured:
* Enable the **Automatic creation** toggle — creates a Fiddler user on first sign-in
* Enable the **Automatic update** toggle — syncs profile fields (name, email) on subsequent sign-ins
* Set the **Determines whether an identity will be prompted to be linked to an existing account** dropdown to **Check for existing username** — matches the incoming identity to an existing user before creating a duplicate
Updating other configuration within the Identity Provider section can sometimes reset these settings. Always verify these are enabled and save after making any changes.
**Step 2: Verify the action script**
Ensure the action script corresponding to your identity provider is added and configured correctly in the AuthN management console under Actions. Refer to the integration guide for your specific identity provider for the canonical action script.
**Step 3: Check for missing user attributes**
If the email, first name, last name, or username fields are not populated on the error screen, the action script is unable to parse the response from your identity provider. This means either:
* Your identity provider is not configured to pass the attributes that the Fiddler action script expects (e.g., for SAML: `firstName`, `lastName`, `email`, `groups`; for OIDC: `given_name`, `family_name`, `email`, `groups`)
* The attribute names in your identity provider do not match the attribute names in the action script
* The action script was not copy-pasted correctly and has a syntax error — verify the script has no broken string literals or missing quotes
Verify your identity provider's attribute mapping configuration and confirm it returns the expected attributes in the SAML assertion or OIDC response.
### Debugging Identity Provider Attribute Responses
To confirm exactly what attributes your identity provider is returning, you can temporarily enable logging in the action script. Add the following lines after the variable declarations in your existing action script:
```javascript theme={null}
let logger = require("zitadel/log");
logger.log('****** Action debugging start ******');
logger.log('email present: ' + (email != undefined) + ', type: ' + typeof email);
logger.log('firstName present: ' + (firstName != undefined) + ', type: ' + typeof firstName);
logger.log('lastName present: ' + (lastName != undefined) + ', type: ' + typeof lastName);
logger.log('****** Action debugging end ******');
```
After adding the logging lines, attempt an SSO sign-in and ask your Fiddler representative to check the Zitadel application logs. The logs will show whether each attribute was returned by your identity provider and its data type, without logging actual values.
Remember to remove the logging lines from the action script after debugging is complete.
#### Verifying SAML Assertion Attributes
For SAML identity providers, you can inspect the raw SAML assertion to verify that the expected attributes are present in the response from your identity provider:
1. Open Chrome DevTools (**F12** or **Ctrl+Shift+I**) and navigate to the **Network** tab
2. Enable **Preserve log** to retain network entries across redirects
3. Attempt an SSO sign-in through your SAML identity provider
4. In the Network tab, locate the `POST` request to `https://authn-{base_url}/ui/login/login/externalidp/saml/acs`
![Chrome DevTools Network tab showing the SAML ACS POST request]()
5. Open the **Payload** tab for that request and copy the `SAMLResponse` value
![Chrome DevTools Payload tab showing the SAMLResponse value]()
6. Decode the `SAMLResponse` locally using your terminal — this returns an XML document.
The decoded SAML assertion contains email, names, group memberships, and signed identity claims. Decode it locally rather than pasting into an online Base64 decoder or SAML inspector.
* macOS: `pbpaste | base64 -d`
* Linux: `xclip -selection clipboard -o | base64 -d` (or `xsel -b | base64 -d`)
* Cross-platform: `echo "" | base64 -d`
* Windows PowerShell: `[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String(""))`
7. In the decoded XML, verify that the attributes your action script expects (e.g., for SAML: `firstName`, `lastName`, `email`, `groups`; for OIDC: `given_name`, `family_name`, `email`, `groups`) are present and contain the correct values
### Authentication Failures
**Redirect URI Mismatch**:
* Verify redirect URI in identity provider matches: `{fiddler_url}/api/sso/{provider}/callback`
* Check for HTTP vs. HTTPS mismatches
**Certificate or Secret Expiration**:
* Monitor client secret expiration dates (typically 6-24 months)
* Update expired certificates or secrets in both identity provider and Fiddler configuration
**Attribute Mapping Issues**:
* Verify required attributes (`firstName`, `lastName`, `email`) are included in authentication response
* Check attribute name consistency between identity provider and Fiddler configuration
### User Provisioning Issues
**Users Not Auto-Provisioned**:
* Confirm "Automatic creation" setting is enabled
* Verify user has appropriate permissions in identity provider
* Check authentication logs for error messages
**Missing User Information**:
* Validate attribute mappings in identity provider configuration
* Ensure identity provider includes required claims in authentication tokens
**Group Synchronization Problems**:
* Verify `groups` attribute is included in identity provider claims
* Check that corresponding teams exist in Fiddler
* Confirm group names match between identity provider and Fiddler teams
## Next Steps
After reading this overview:
1. **Choose Your Provider**: Review the provider-specific integration guides
2. **Plan Implementation**: Coordinate with your identity provider administrator
3. **Test Configuration**: Set up a test environment before production deployment
4. **Train Users**: Provide documentation on the new authentication process
***
**Note**: SSO configuration requires coordination between Fiddler administrators and identity provider administrators. Plan accordingly for configuration, testing, and rollout phases.
# AWS VPC Endpoint Setup
Source: https://docs.fiddler.ai/reference/administration/aws-vpc-endpoint-setup
Automated script to create AWS VPC endpoints for secure communication with Fiddler Cloud using AWS Virtual PrivateLink.
This guide provides an automated approach to creating AWS VPC endpoints for Fiddler Cloud integration. For manual configuration steps, see the [AWS Virtual PrivateLink Setup](/reference/administration/aws-vpl-setup) guide.
This script automates the VPC endpoint creation process described in the manual setup guide. Ensure you have completed initial coordination with the Fiddler team before running this script.
## Overview
The VPC endpoint creation script automates the following tasks:
* Creates and configures security groups with HTTPS access
* Establishes VPC endpoints in specified subnets
* Configures private DNS for seamless Fiddler Cloud access
* Validates configuration and handles cross-region endpoints
## Prerequisites
Before running the script, ensure you have:
* **AWS CLI** installed and configured with appropriate credentials
* **jq** tool installed for JSON parsing
* **yq** tool installed for YAML parsing
* AWS IAM permissions to create:
* VPC endpoints
* Security groups
* Route53 DNS records
* Required information from the Fiddler team:
* VPC endpoint service name
* Stack name identifier
* Your AWS environment details:
* VPC ID
* Subnet IDs
* AWS region
## Installation
### Step 1: Install Required Tools
The following tools are required:
* AWS CLI
* jq for JSON parsing
* yq for YAML parsing
```sh theme={null}
brew install awscli, jq, yq
```
```sh theme={null}
sudo apt-get install -y awscli, jq, yq
```
It is recommended to manually install the AWS CLI on these operating systems. Download the bundle [directly](https://aws.amazon.com/cli/) from AWS.
```sh theme={null}
sudo yum install jq yq
```
### Step 2: Configure AWS CLI
If not already configured, set up your AWS credentials:
```bash theme={null}
aws configure
```
### Step 3: Download and Prepare the Script
1. Request the script and configuration file template from your Fiddler representative
2. Make the script executable:
```bash theme={null}
chmod +x create-vpc-endpoint.sh
```
## Configuration
### Step 1: Gather Required Information
Collect the following information before configuration:
**From the Fiddler team:**
* **Service name**: The VPC endpoint service name for your Fiddler environment
* **Stack name**: The unique identifier for your endpoint
**From your AWS environment:**
* **VPC ID**: The ID of your VPC (e.g., `vpc-12345678`)
* **Subnet IDs**: IDs of subnets where the endpoint will be created
* **Region**: The AWS region where your VPC is located
### Step 2: Update Configuration File
Edit the `config.yaml` file with your specific values:
```yaml theme={null}
# Required: VPC Private Link service name (provided by Fiddler)
service_name:
# Required: Stack name (provided by Fiddler)
stack_name: -endpoint
# Required: VPC ID where endpoint will be created
vpc_id:
# Required: List of subnet IDs (at least one required)
subnet_ids:
-
-
-
# Required: AWS region (must match your VPC region)
region:
# Optional: DNS configuration
dns:
enabled: true
custom_domain: authn-.cloud.fiddler.ai
zone_name: cloud.fiddler.ai
```
The `service_name` and `stack_name` must be obtained from the Fiddler team. Do not use placeholder values.
## Running the Script
### Basic Usage
Run the script with the default configuration file `config.yaml`:
```bash theme={null}
./create-vpc-endpoint.sh
```
### Using a Custom Configuration File Name
Specify an alternative configuration file:
```bash theme={null}
./create-vpc-endpoint.sh my-config.yaml
```
### Script Execution Process
The script performs the following operations:
1. **Validates configuration** - Ensures all required fields are present
2. **Creates security group** - Establishes HTTPS access rules if not specified
3. **Creates VPC endpoint** - Establishes the endpoint in your VPC
4. **Configures DNS** - Sets up private DNS for easy access (if enabled)
The script is idempotent and safe to run multiple times. It will not create duplicate resources.
### Example Output
```
[INFO] Reading configuration from: config.yaml
[INFO] Configuration loaded successfully
[INFO] Creating VPC endpoint for service: com.amazonaws.vpce.us-west-2.vpce-svc-1234567890abcdef0
[INFO] Using AWS region: us-west-2
[INFO] Checking if VPC endpoint with tag Name=myapp-endpoint already exists...
[INFO] No existing VPC endpoint found - proceeding with creation
[INFO] Creating security group for VPC endpoint...
[INFO] Security group created successfully: sg-1234567890abcdef0
[INFO] Creating VPC endpoint...
[INFO] VPC endpoint created successfully!
[INFO] Setting up DNS for new endpoint...
[DNS] DNS setup completed successfully!
[INFO] VPC endpoint creation initiated successfully!
```
## Advanced Configuration
### Using Existing Security Groups
To use pre-existing security groups instead of creating new ones:
```yaml theme={null}
security_group_ids:
* sg-12345678
* sg-87654321
```
### Disabling DNS Setup
If you prefer to manage DNS separately:
```yaml theme={null}
dns:
enabled: false
```
### Cross-Region Endpoints
The script automatically handles cross-region endpoints when the service is in a different region than your VPC:
```yaml theme={null}
service_name: com.amazonaws.vpce.us-east-1.vpce-svc-1234567890abcdef0 # Service in us-east-1
region: us-west-2 # Your VPC region
```
## Troubleshooting
### Common Issues and Solutions
#### AWS CLI not configured
```bash theme={null}
aws configure
```
Enter your AWS access key, secret key, default region, and output format.
#### Missing required tools
Install jq and yq as described in the Installation section.
#### VPC or subnet not found
* Verify the VPC ID and subnet IDs in your configuration
* Ensure you have access to the specified resources
* Confirm the resources exist in the specified region
#### Permission denied errors
Ensure your AWS credentials have the following permissions:
* `ec2:CreateVpcEndpoint`
* `ec2:CreateSecurityGroup`
* `ec2:AuthorizeSecurityGroupIngress`
* `ec2:CreateTags`
* `ec2:DescribeVpcs`
* `ec2:DescribeSubnets`
* `route53:CreateHostedZone`
* `route53:ChangeResourceRecordSets`
### Getting Help
For script usage information:
```bash theme={null}
./create-vpc-endpoint.sh -h
```
## Security Considerations
* The script creates security groups allowing HTTPS (port 443) access from your VPC CIDR range
* All DNS zones are created as private hosted zones
* Resources are tagged for easy identification and management
* VPC endpoints use AWS PrivateLink for secure, private communication
## Verification
After running the script:
1. Verify the endpoint status in the AWS VPC console shows "Available"
2. Check that security group rules are correctly configured
3. Test DNS resolution within your VPC:
```bash theme={null}
nslookup .cloud.fiddler.ai
```
4. Access the Fiddler UI at `https://<your-subdomain>.cloud.fiddler.ai`
## Next Steps
* Review the [AWS Virtual PrivateLink Setup](/reference/administration/aws-vpl-setup) guide for additional context
* Configure your applications to use the private endpoint
* Set up monitoring for the VPC endpoint connection
* Contact Fiddler [support](mailto:support@fiddler.ai) if you encounter any issues
# AWS Virtual PrivateLink Setup
Source: https://docs.fiddler.ai/reference/administration/aws-vpl-setup
Step-by-step guide to configure AWS Virtual PrivateLink for secure communication between your AWS VPC and Fiddler Cloud.
**Automated Setup Available**: We now provide an automated script that simplifies the VPC endpoint creation process. For most users, we recommend using the [AWS VPC Endpoint Automation Script](/reference/administration/aws-vpc-endpoint-setup) instead of this manual process. This guide remains available for reference, troubleshooting, and those who prefer manual configuration.
This guide illustrates configuring an AWS Virtual PrivateLink (VPL) between your company VPC and Fiddler Cloud environment to establish secure communication channels.
Fiddler Customers must complete these steps only after the Fiddler team has completed the customer's private-link-based environment deployment.
Your Fiddler environment will use the private DNS name: `<customer-subdomain>.cloud.fiddler.ai`, where `<customer-subdomain>` is typically your company name. If you have specific subdomain requirements or restrictions, notify your Fiddler representative before VPL configuration.
## Prerequisites
* AWS account with VPC access
* Fiddler-provided service name
* Fiddler-provided DNS name
* VPC CIDR range information
* Appropriate AWS IAM permissions to create VPC endpoints
## Step 1: Navigate to the AWS VPC Console
1. Log in to your AWS Management Console
2. Navigate to the VPC service
3. In the left navigation panel, click on "Endpoints"
4. Click the Create endpoint button
![Create an endpoint]()
## Step 2: Configure the Fiddler Endpoint Service
1. Enter a descriptive name tag for your endpoint
2. Select "PrivateLink Ready partner services" from the service categories
3. Enter the Fiddler-provided service name
4. Click Verify Service to confirm the service details
Fiddler will provide the service name before you proceed with this step. Contact your Fiddler representative if you haven't received this information.
![Provide the endpoint service name]()
## Step 3: Select VPC and Subnets
1. Select your VPC from the dropdown
2. Choose all subnets where your client applications are running
3. Ensure the selected subnets have appropriate routing within your VPC to the endpoint
![Select the VPC]()
## Step 4: Configure Security Group
1. Create a new security group if one doesn't exist
2. Add an inbound rule to allow:
* Port: 443 (HTTPS)
* Source: Your VPC CIDR range
3. Select the security group ID to associate with the endpoint
### Example security group configuration:
* Inbound rule: TCP 443 from VPC CIDR
* Outbound rule: All traffic (default)
![Create a new security group]()
## Step 5: Create the Endpoint
1. Review all configuration settings
2. Click Create endpoint to initiate the endpoint creation
3. Wait for the endpoint status to change to "Available"
## Step 6: Configure Private DNS
1. Select the newly created endpoint
2. From the Actions menu, choose "Modify private DNS name"
3. Enable private DNS names by checking the "Enable for this endpoint" checkbox
4. **Important**: The private DNS name will be in the format: `<customer-subdomain>.cloud.fiddler.ai`
* Example: If your company name is "acme", the DNS name would be `acme.cloud.fiddler.ai`
5. Click Save changes
Once enabled, AWS will automatically configure DNS resolution for your assigned Fiddler subdomain in the format `<customer-subdomain>.cloud.fiddler.ai`.
![Select action "Modify the private DNS name"]()
![Modify the private DNS name]()
## Step 7: Verify Configuration
1. Wait for the endpoint status to show as "Available"
2. Verify that the private DNS name is enabled
3. Confirm the security group rules are properly configured
## Step 8: Access Fiddler
Once the configuration is complete, you can access the Fiddler UI within your VPC using the configured DNS name:
`https://<customer-subdomain>.cloud.fiddler.ai`
## Troubleshooting
If you encounter issues:
* Verify the endpoint status in the AWS console
* Check security group rules and network ACLs
* Confirm DNS resolution within your VPC
* Contact Fiddler support with your endpoint ID and any error messages
## Next Steps
* For automated setup, see the [AWS VPC Endpoint Automation Script](/reference/administration/aws-vpc-endpoint-setup)
* [Configure your applications to use the private endpoint](https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-services-overview.html)
* [Set up monitoring for the VPL connection](https://docs.aws.amazon.com/vpc/latest/userguide/monitoring.html)
* [Review security best practices](https://docs.aws.amazon.com/vpc/latest/userguide/security.html)
# LLM Gateway
Source: https://docs.fiddler.ai/reference/administration/llm-gateway
Configure LLM provider credentials to enable AI-powered features in Fiddler using your own API keys from OpenAI, Anthropic, Gemini, and other providers.
The **LLM Gateway** allows you to manage and securely configure credentials for multiple Large Language Model (LLM) providers, enabling you to use your own API keys for AI-powered features throughout the Fiddler platform.
## Overview
LLM Gateway provides centralized management of LLM provider credentials, giving you control over which models and API keys power Fiddler's AI features such as:
* **Custom Evaluators** - Use your preferred LLM to evaluate model outputs
* **LLM Enrichments** - Generate AI-powered insights on your monitoring data
* **Content Analysis** - Assess response quality, detect hallucinations, and monitor trust metrics
### Key Capabilities
* **Multiple Provider Support** - Configure credentials for OpenAI, Anthropic, Gemini, Vertex AI, Databricks, AWS Bedrock, Azure OpenAI, Azure AI, and Fiddler
* **Credential Redundancy** - Add multiple API keys per provider for failover and load balancing
* **Flexible Key Management** - Organize credentials by team, environment, or purpose
* **Secure Storage** - API keys are encrypted and stored securely
***
## Prerequisites
Before configuring LLM Gateway, ensure you have:
* **Admin Permissions** - Only administrators can access the LLM Gateway settings
* **Provider API Keys** - Valid API credentials from your chosen LLM providers (OpenAI, Anthropic, Gemini, Vertex AI, Databricks, AWS Bedrock, Azure OpenAI, Azure AI, or Fiddler)
**Note:** Each provider requires a separate API key. Obtain keys from your provider's developer portal before proceeding.
***
## Configure LLM Providers
### Add a New Provider
Follow these steps to add an LLM provider to your Fiddler organization:
From the top navigation bar, click the **Settings** icon (gear icon) and select the **LLM Gateway** tab.
Click the **Add Provider** button to open the provider configuration dialog.
Choose your LLM provider from the dropdown menu:
* **OpenAI** - OpenAI GPT models
* **Anthropic** - Anthropic Claude models
* **Gemini** - Google Gemini models
* **Vertex AI** - Google Cloud Vertex AI Model Garden (Gemini and Anthropic Claude)
* **AWS Bedrock** - Bedrock-hosted models (Anthropic, Amazon Nova, Meta, Mistral AI)
* **Azure OpenAI** - OpenAI models on Azure infrastructure
* **Azure AI** - Anthropic Claude models via the Azure AI catalog
* **Databricks** - Foundation models served through Databricks Model Serving
* **Fiddler** - Fiddler-hosted LLM services
a. Click **Add Credential**
b. Enter a **Nickname** for the credential (for example, `Production Team Key` or `Test Environment`)
**Tip:** Use clear, descriptive nicknames to differentiate between test and production keys or to identify which team owns the credential.
c. Paste your **API Key** into the credential field
Some providers use authentication methods other than — or in addition to — an API key. For example, **Microsoft Entra ID** (Azure), **AWS Access Key** (Bedrock), **GCP Service Account** (Vertex AI), and **OAuth Client Credentials** (Databricks). Vertex AI, for instance, uses a GCP Service Account instead of an API key. See [Supported Providers](#supported-providers) for the credential fields each provider requires.
d. The provider's available models will be automatically populated
Click **Update Provider** to save your configuration, or **Cancel** to discard changes.
Your provider is now configured and ready to use with Fiddler's AI-powered features.
### Add Multiple Credentials to a Provider
You can add multiple API keys to a single provider for redundancy, load balancing, or to separate keys by environment or team.
**Why Use Multiple Credentials?**
* **Redundancy** - Automatic failover if one key reaches rate limits or expires
* **Load Balancing** - Distribute API calls across multiple keys to improve throughput
* **Key Rotation** - Safely test new credentials before removing old ones
* **Environment Separation** - Use different keys for development, staging, and production
**To Add Additional Credentials:**
1. Navigate to **Settings → LLM Gateway**
2. Click the **edit** icon next to the provider you want to modify
3. In the Edit Provider dialog, click **Add Credential**
4. Enter a nickname and paste the new API key
5. Click **Update Provider** to save
All credentials for a provider will be available for use across Fiddler features. The platform handles credential selection and failover automatically.
### Edit an Existing Provider
You can modify provider configurations, update credentials, or rename existing keys.
**To Edit a Provider:**
1. Navigate to **Settings → LLM Gateway**
2. Locate the provider you want to edit
3. Click the **edit** icon next to the provider name
4. Make your changes:
* **Update Credentials** - Click the edit icon next to a credential to modify the API key
* **Rename Credentials** - Update the nickname to reflect the key's purpose
* **Add More Credentials** - Click **Add Credential** to add another API key
* **Remove Credentials** - Delete individual credentials that are no longer needed
5. Click **Update Provider** to save your changes
Removing a credential that is actively in use may impact running evaluations or enrichments. Ensure you have alternate credentials configured before removing a key.
***
## Supported Providers
The LLM Gateway supports the following providers:
### OpenAI
* **Models Available:** GPT-5 (including mini and nano), GPT-4.1 (including mini and nano), GPT-4o, and GPT-4o mini
* **API Key Location:** [OpenAI Platform - API Keys](https://platform.openai.com/api-keys)
* **Use Cases:** Custom evaluators, content generation, response quality assessment
### Anthropic
* **Models Available:** Claude Opus 4.1, Claude Opus 4, Claude Sonnet 4, and Claude 3.5 Haiku
* **API Key Location:** [Anthropic Console](https://console.anthropic.com/)
* **Use Cases:** Advanced reasoning, content analysis, evaluation tasks
### Gemini
* **Models Available:** Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite
* **API Key Location:** [Google AI Studio](https://makersuite.google.com/app/apikey)
* **Use Cases:** Multimodal analysis, content generation, embeddings
### Vertex AI
Access models from Google Cloud's Vertex AI Model Garden. Requests are routed through your GCP project and region.
* **Models Available:** Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, Claude Sonnet 4.5, and Claude Opus 4.5
* **Authentication**:
* **GCP Service Account** — Provide a service account JSON key, plus the Vertex AI project (the GCP project where requests are sent) and Vertex AI location (the region, for example `us-central1`). The project and location may differ from the project embedded in the service account key.
* **Use cases:** Organizations with existing GCP / Vertex AI infrastructure or Google-managed inference environments
### AWS Bedrock
Access LLM models hosted on Amazon Web Services through the Bedrock managed service. All requests are routed through your AWS account.
* **Supported model families** (custom model IDs for fine-tuned and newly released models are also supported):
* **Anthropic** (`bedrock/anthropic.*`) — Claude models for advanced reasoning and long-context tasks
* **Amazon** (`bedrock/amazon.nova-*`) — Nova family for multimodal and text generation
* **Meta** (`bedrock/meta.llama*`) — Open-weights Llama models across multiple sizes
* **Mistral AI** (`bedrock/mistral.*`) — High-performance models optimized for efficiency
* **Authentication**:
* **API Key** — Use Fiddler-managed AWS credentials
* **AWS Access Key** — Provide your own AWS access key ID, secret access key, and AWS region
* **Custom endpoint** (optional) — Set a custom **API Base URL** to route requests through an AWS VPC endpoint instead of the public Bedrock endpoint (for example, `https://vpce-xxx.bedrock-runtime.us-east-1.vpce.amazonaws.com`)
* **Use cases:** Evaluators and LLM-as-a-Judge workflows requiring inference traffic to stay within AWS-managed cloud boundaries
### Azure OpenAI
Access OpenAI models deployed on Microsoft Azure. When configuring this provider, specify your **Azure deployment name**, which may differ from the base model name.
* **Supported models** (custom fine-tuned models are also supported):
* **GPT-5** and **GPT-5 mini** — Latest-generation reasoning and general-purpose models
* **GPT-4o** and **GPT-4o mini** — Multimodal, cost-optimized models
* **Authentication**:
* **API Key** — Azure deployment API key for simple authentication
* **Microsoft Entra ID** (formerly Azure AD) — OAuth 2.0 client credentials flow using tenant ID, client ID, and client secret; for enterprise SSO integration
* **Custom endpoint** — Set a custom **API Base URL** to target a specific Azure region or resource endpoint (for example, `https://.openai.azure.com/`)
* **Use cases:** Organizations with existing Azure OpenAI deployments, enterprise compliance requirements, or Microsoft-managed inference environments
### Azure AI
Access models from the Azure AI Model Catalog. When configuring this provider, specify your **Azure AI deployment name**.
* **Supported models**:
* **Anthropic** — Claude Sonnet 4.5, Claude Haiku 4.5, and Claude Opus 4.1
* **Authentication**:
* **API Key** — Azure deployment API key for simple authentication
* **Microsoft Entra ID** (formerly Azure AD) — OAuth 2.0 client credentials flow using tenant ID, client ID, and client secret; for enterprise SSO integration
* **Custom endpoint** — Set a custom **API Base URL** to target a specific Azure AI resource (for example, `https://.services.ai.azure.com/`)
* **Use cases:** Organizations with existing Azure AI infrastructure or multi-provider model access through Azure's unified catalog
### Databricks
Access foundation models served through Databricks Model Serving. This provider requires a custom **API Base URL** pointing to your workspace serving endpoint.
* **Supported model families:**
* **OpenAI GPT-OSS** — `databricks-gpt-oss-120b`, `databricks-gpt-oss-20b`
* **Meta Llama** — `databricks-meta-llama-3-1-8b-instruct`, `databricks-meta-llama-3-3-70b-instruct`, `databricks-meta-llama-3-1-405b-instruct`, `databricks-llama-4-maverick`
* **Google Gemma** — `databricks-gemma-3-12b`
* **Authentication**:
* **API Key** — Databricks personal access token for simple authentication
* **OAuth Client Credentials** — OAuth 2.0 client credentials (machine-to-machine) flow using a Databricks service principal's token URL, client ID, client secret, and scope (for example, `all-apis`). Fiddler exchanges these for a short-lived bearer token and refreshes it automatically — no long-lived tokens to manage or rotate.
* **Custom endpoint** — Set the **API Base URL** to your Databricks workspace serving endpoint (for example, `https://adb-xxxx.azuredatabricks.net/serving-endpoints`)
* **Use cases:** Organizations serving foundation models on Databricks, or requiring enterprise machine-to-machine authentication via service principals
1. In Databricks, create a **service principal** and generate an **OAuth secret** for it, following Databricks' OAuth machine-to-machine (M2M) guide for your cloud ([AWS](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m), [Azure](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m), [GCP](https://docs.databricks.com/gcp/en/dev-tools/auth/oauth-m2m)). This gives you the **Client ID** (the service principal's application ID) and **Client Secret** (shown only once).
2. Enter these values when adding the Databricks credential in Fiddler:
* **Client ID** and **Client Secret** — from step 1.
* **Token URL** — `https:///oidc/v1/token`
* **API Base URL** — `https:///serving-endpoints` (same workspace host)
* **Scope** — `all-apis`
### Fiddler
* **Models Available:** Fiddler-managed LLM services
* **API Key Location:** Provided by Fiddler
* **Use Cases:** Pre-configured evaluators, platform-optimized features
***
## Best Practices
### Credential Management
* **Use Descriptive Nicknames** - Label credentials by team, environment, or purpose (for example, `ML Team - Production`, `Data Science - Test`)
* **Rotate Keys Regularly** - Add new credentials before removing old ones to avoid service interruption
* **Separate Environments** - Use different API keys for development, staging, and production
* **Monitor Usage** - Track API consumption through your provider's dashboard to avoid unexpected costs
### Security
* **Restrict Access** - Only grant Admin permissions to users who need to manage LLM credentials
* **Avoid Sharing Keys** - Each team should have their own credentials rather than sharing a single key
* **Revoke Compromised Keys** - If a key is exposed, immediately revoke it in your provider's console and remove it from Fiddler
### Performance Optimization
* **Add Multiple Credentials** - Configure 2-3 keys per provider for redundancy and better throughput
* **Test Before Production** - Validate new credentials in a test environment before using them in production
* **Monitor Rate Limits** - Be aware of your provider's rate limits and adjust your credential count accordingly
***
## Related Features
Once you've configured LLM providers in the Gateway, you can use them with these Fiddler features:
* **Custom Evaluators** - Create LLM-based evaluators to assess model outputs
* **LLM Enrichments** - Generate AI-powered metrics for your LLM applications
* **Content Safety** - Use LLM providers for advanced content analysis
***
## Troubleshooting
### Provider Not Appearing in Feature Selection
**Issue:** After adding a provider, it doesn't appear in evaluator or enrichment configuration.
**Solution:**
* Verify the provider was saved successfully (check the LLM Gateway tab)
* Ensure at least one credential was added to the provider
* Refresh your browser page
* Contact your Fiddler administrator to verify permissions
### Invalid API Key Error
**Issue:** Receiving authentication errors when using a configured provider.
**Solution:**
* Verify the API key is correct in your provider's console
* Check that the key hasn't expired or been revoked
* Ensure the key has the necessary permissions for the models you're using
* Update the credential in Settings → LLM Gateway
### Rate Limit Warnings
**Issue:** Receiving rate limit errors from a provider.
**Solution:**
* Add additional credentials to the provider for load balancing
* Check your provider's dashboard for current usage and limits
* Consider upgrading your provider plan for higher limits
* Temporarily reduce the number of concurrent evaluations or enrichments
# Administration
Source: https://docs.fiddler.ai/reference/administration/settings
Dive into our guide to application settings in Fiddler. Learn to use the settings page to manage team setup, permissions, and credentials.
## Overview
The Settings page provides centralized management for your organization's configuration, user access, and integrations. Access the **Settings** page from the left navigation bar in the Fiddler UI.
![Fiddler home page showing the user menu open and highlighting the Settings link.]()
## Settings Tabs
The Settings page is organized into the following tabs:
* **General** - Organization information and user details
* **Access** - User and team management
* **Credentials** - API key management
* **LLM Gateway** - LLM provider and API credential management
* **Email Configuration** - Email notification settings
* **PagerDuty Integration** - PagerDuty alert configuration
* **Webhook Integration** - Webhook service management
## General
The **General** tab displays your organization's configuration and current user information.
![General tab of the Settings page.]()
### Organization Information
* **Organization Name** - Your organization's display name in Fiddler
* **Organization ID** - Unique identifier for your organization
* **Version** - Current Fiddler platform version
### User Information
* **User Name** - Your display name in the system
* **Email** - Email address associated with your account
## Access
The **Access** tab provides centralized management for users and teams within your organization.
### Users
The **Users** sub-tab displays all users who are members of your organization.
![Users sub-tab of the Access tab on the Settings page showing a list of users.]()
This view shows:
* User names and email addresses
* Organization roles (Org Admin or Org Member)
* Account status (Active/Inactive)
* Last login information
> **Note**
>
> User invitations are managed through the Fiddler AuthN console:
>
> * Email authentication users must be invited by AuthN Org Owners or Org User Managers
> * SSO users are automatically created upon first login to Fiddler
### Teams
The **Teams** sub-tab displays all teams defined within your organization. Teams provide a way to organize users and manage project-level access controls.
![Teams sub-tab of the Access tab on the Settings page showing a list of teams]()
#### Creating Teams
Create a new team by clicking the plus (**`+`**) icon in the top-right corner.
> **Important**
>
> Only users with the Org Admin role can create teams. The plus (**`+`**) icon will not be visible for Org Members.
When creating a team:
1. Enter a unique team name
1. Team names may not include spaces, but mixed-case alphanumeric characters are valid
2. Add team members from existing users
3. Assign appropriate project permissions
4. Click **Create** to save the team
#### Team Synchronization
If your organization uses SSO with group synchronization enabled, teams can be automatically created and managed based on your identity provider groups. See [Mapping Identity Provider Groups to Fiddler Teams](/reference/access-control/mapping-ad-groups-to-fiddler-teams) for configuration details.
## Credentials
The **Credentials** tab manages API keys used for programmatic access to Fiddler through the Fiddler Python client, Fiddler SDKs, and REST APIs.
Fiddler UI versions prior to 26.11 labeled API keys as **Access Keys**. Both terms refer to the same credential.
![Credentials tab showing the list of named API keys.]()
### How API Keys Work
Fiddler API keys behave like Personal Access Tokens (PATs) — similar in model to a GitHub PAT. Each key is tied to the individual user who created it and acts on that user's behalf, rather than identifying an application or service account.
Understanding this model is important for security, access control, and key lifecycle management.
#### Identity and Attribution
* Each API key is issued to a specific user and acts as that user when making API requests
* All actions performed with the key are attributed to the issuing user in audit logs
* Keys are not tied to a specific application, environment, or service account
#### Permission Scope
API keys inherit the **full permissions** of the user who created them:
* There is no per-key scope reduction (for example, no read-only keys)
* There is no per-application or per-project restriction on a key
* If the user's role changes, every key they own immediately reflects the new permissions
#### Lifetime and Revocation
* Keys remain active for as long as the issuing user's account is active
* Keys persist until explicitly revoked — there is no automatic expiry
* If the user is deactivated, all of their keys stop working immediately
* User password changes have no effect on API keys
* Revoking a key takes effect immediately; allow up to 5 minutes for all active sessions to reflect the change
#### Anti-Patterns to Avoid
Because each key carries the issuing user's full permissions and identity:
* **Do not share keys** between team members — every action will be attributed to the key's owner, and revoking access for one person requires revoking the shared key for everyone
* **Do not commit keys to source control** — treat them like passwords
* **Prefer per-developer keys** over a single shared key for any team or CI pipeline
* **Rotate keys** on personnel changes, suspected exposure, or on a regular schedule
#### What Fiddler API Keys Are Not
To set expectations against other common credential models:
* **Not OAuth client credentials** — no client ID/secret pair, no token exchange
* **Not service-account keys** — no machine identity separate from a human user
* **Not scoped tokens** — no per-key permission reduction or resource restriction
### Creating API Keys
Fiddler supports two key-creation flows. Named API keys are the recommended approach for deployments running 26.9 or later.
#### Named API Keys (recommended, 26.9+)
1. Click **Create Key** to open the create dialog.
2. Provide a descriptive **name** for the key (for example, the integration or environment it will be used for).
Key names must start with a letter and may contain letters, digits, spaces, hyphens, and underscores (1–128 characters). Names must be unique within your own keys.
![Create API key modal.]()
3. Click **Create Key**. The key is displayed once — copy it immediately and store it securely. Fiddler cannot retrieve it again.
![One-time API key displayed after creating a named API key.]()
Use the API key for authentication in the Fiddler Python client, Fiddler SDKs, and REST APIs.
#### Legacy Single-Key Flow
If your deployment was created before Release 26.9, you may see the legacy single-key view instead. Legacy keys continue to work alongside named API keys — creating a named key does not affect your existing legacy key. We recommend using named API keys for any new integrations.
![Legacy API key view for older deployments.]()
To use the legacy flow:
1. Click **Create Key** to generate your key.
2. Once created, the **Create Key** button is hidden — only one key can exist at a time.
3. To generate a new key, delete the existing key first. The **Create Key** button will reappear.
### Managing API Keys
* **Named API keys**: You can create up to 5 per user. On Fiddler Cloud, contact Fiddler Support if you need a higher limit. Self-hosted admins can raise the cap with the `MAX_ACCESS_KEYS_PER_USER` environment variable. Legacy single-key API keys do not count toward this limit.
* **Revoking named API keys**: Use the **⋮** (more options) menu in the named keys list view. Revocation takes effect immediately; allow up to 5 minutes for all active sessions to reflect the change.
* **Revoking a legacy key**: Delete the existing key. The **Create Key** button will reappear when ready.
* Each API key inherits the permissions of the user who created it.
**Security Best Practice:** Treat API keys like passwords. Never share them or commit them to version control. Copy and store the key immediately at creation — Fiddler cannot retrieve it again. Rotate keys regularly and revoke unused or compromised keys immediately.
## LLM Gateway
The **LLM Gateway** tab allows you to configure and manage LLM provider credentials for AI-powered features throughout the Fiddler platform.
Configure LLM providers to enable:
* **Custom Evaluators** - Use your preferred LLM to evaluate model outputs
* **Content Analysis** - Assess response quality and monitor trust metrics
Supported providers include OpenAI, Anthropic, Gemini, and Fiddler. You can add multiple API credentials per provider for redundancy, load balancing, and key rotation.
For detailed configuration instructions, see [LLM Gateway Configuration](/reference/administration/llm-gateway).
## Email Configuration
The **Email Configuration** tab manages the email provider for your organization.
![]()
Configure:
* Selecting SES or SMTP for email delivery
* Fiddler Cloud customers leverage AWS SES
* You may choose to use your own SMTP server by entering your own SMTP connection and credentials details
## PagerDuty Integration
The **PagerDuty Integration** tab enables configuration of PagerDuty services for alert escalation.
Setup includes:
* PagerDuty service integration keys
* Severity mapping for alerts
* Escalation policy configuration
* Test alert functionality
## Webhook Integration
The **Webhook Integration** tab manages webhook configurations for connecting Fiddler alerts to your notification and communication platforms.
### Supported Webhook Types
Fiddler supports three webhook provider types:
* **Slack** - Direct integration with Slack channels
* **Microsoft Teams** - Native Teams channel integration
* **Other** - Custom webhook endpoints for any platform
### Creating a Webhook
Click the plus (**`+`**) icon to configure a new webhook:
![Create Webhook Service modal dialog]()
#### Slack Webhook Configuration
1. Enter a unique name in the **Service Name** field
2. Select **Slack** from the **Provider** dropdown
3. Enter your Slack webhook URL (format: `https://hooks.slack.com/services/...`)
4. Click **Test** to verify the connection
5. Click **Create** once the test succeeds
> **Reference**
>
> See [Slack's webhook documentation](https://api.slack.com/messaging/webhooks) for creating webhook URLs.
#### Microsoft Teams Webhook Configuration
1. Enter a unique name in the **Service Name** field
2. Select **MS Teams** from the **Provider** dropdown
3. Enter your Teams webhook URL provided by your administrator
4. Click **Test** to verify the connection
5. Click **Create** once the test succeeds
> **Reference**
>
> See [Microsoft Teams webhook documentation](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook?tabs=newteams%2Cdotnet#create-an-incoming-webhook) for setup instructions.
#### Custom Webhook Configuration
1. Enter a unique name in the **Service Name** field
2. Select **Other** from the **Provider** dropdown
3. Enter your platform's webhook URL
4. Click **Test** to verify the endpoint
5. Click **Create** once the test succeeds
### Managing Webhooks
Edit or delete existing webhooks using the menu (**...**) icon for each webhook:
1. Select the webhook's menu icon
2. Choose **Edit Webhook** to modify configuration
3. Choose **Delete Webhook** to remove the webhook
> **Important**
>
> You cannot delete webhooks that are currently linked to active alerts. Remove the webhook from all alerts before deletion.
## Access Requirements
Different settings require specific permissions:
| Setting | Org Admin | Org Member |
| ------------------------ | --------- | ---------- |
| View General tab | ✓ | ✓ |
| View Access > Users | ✓ | ✓ |
| View Access > Teams | ✓ | ✓ |
| Create/Edit Teams | ✓ | ✗ |
| Create API Keys | ✓ | ✓ |
| Configure LLM Gateway | ✓ | ✗ |
| Configure Email Settings | ✓ | ✗ |
| Configure PagerDuty | ✓ | ✗ |
| Create/Edit Webhooks | ✓ | ✗ |
## Related Documentation
* [Role-Based Access Control](/reference/access-control/role-based-access) - Understanding user roles and permissions
* [Single Sign-On Configuration](/reference/access-control/sso-authentication-guide) - Setting up SSO authentication
* [Inviting Users](/reference/access-control/email-login#adding-users-to-fiddler) - Adding users through the AuthN console
* [Alert Configuration](/observability/platform/alerts-platform) - Using webhooks and integrations for alerts
# Supported Browsers
Source: https://docs.fiddler.ai/reference/administration/supported-browsers
Discover our product guide on supported web browsers for accessing Fiddler, including Google Chrome, Firefox, Safari, and Microsoft Edge.
Fiddler Product can be accessed through the following supported web browsers:
* Google Chrome
* Firefox
* Safari
* Microsoft Edge
# Feature Maturity Definitions
Source: https://docs.fiddler.ai/reference/feature-maturity-definitions
Review Fiddler's release and support policies for product features at different stages of maturity.
### Product Feature Maturity Definitions
Fiddler ships new features rapidly. To maintain a fast development pace and ensure a stable customer experience, we define clear stages for feature maturity. Evolution from one stage to the other is quality-bound, not time-bound. Features evolve through the following stages.
To see which Python versions Fiddler's SDKs support, see the [Python version support policy](/reference/python-support-policy).
### Generally Available (GA)
The GA stage is when a feature is not only ready for production but the SLA is guaranteed. GA features are fully supported by our team and are backed by our comprehensive support plans, providing customers with confidence in reliability, security, and performance.
### Public Preview
Public Preview features are reasonably mature and ready for broader testing, accessible to all customers. This allows us to gather data about the feature in the wild, including important customer feedback. While not formally bound by our SLA, our team responds promptly to issues. APIs and functionality may—but are not likely to—change.
#### Criteria
* Has a feature flag that is on by default for all customers.
* Available to all customers.
* Does not break or degrade existing functionality when enabled. (Does not degrade existing SLOs.)
* Docs are published publicly but noted as “preview” or “public preview”.
* Breaking changes must be minimal and noted in the release notes. Also, Field should message and support the change to heavy users directly.
* Expectation that it WILL land in the product after some unspecified amount of time.
* SLO, but no SLA.
### Private Preview
Private Preview features are early-stage and available to select design partners only. During this stage, no Service Level Agreement (SLA) is provided. Instead, our engineering team collaborates directly with partners to iterate on the solution, making changes as needed based on feedback during business hours. Features, including their APIs, will change rapidly; they might also be abandoned entirely.
Why [design partners](https://a16z.com/a-framework-for-finding-a-design-partner/)? Having a handful of transparent, engaged users or prospective customers can guide you through your first iteration of the product’s functionality, user experience, pricing and packaging, and more. Their critiques should help you build something useful and usable for your broader customer base. To inquire about private preview features, please reach out to \<[sales@fiddler.ai](mailto:sales@fiddler.ai)>.
#### Criteria
* Has a feature flag that is **turned off by default**.
* Only available to select customers (as decided by Eng + PM).
* Can do a POV with Private Preview with Eng+PM agreement (supporting team and leadership must approve).
* Can market, if desired.
* Does not break or degrade existing functionality when disabled.
* May cause unexpected side-effects when enabled.
* Docs are not published, or are published in some not publicly-accessible place.
* Supported by the engineering team directly.
* Breaking changes can be communicated via Slack, not in release notes.
* No SLA or SLO.
### Solutions Engineering
Some solutions are custom-crafted by our Solutions Engineering Team. These features, while useful and in production, are not officially supported by our overall product SLA. Talk directly with your SE or CSM for clarification as to what SLA any custom-developed tools fall under.
# LLM Observability Metrics Reference
Source: https://docs.fiddler.ai/reference/llm-observability-metrics
Complete reference of all LLM observability metrics and enrichments supported by the Fiddler monitoring platform.
Fiddler provides a comprehensive set of enrichments for monitoring LLM applications in production. Enrichments augment your application data with automatically generated trust, safety, and quality metrics during model onboarding. These metrics integrate directly with Fiddler's monitoring dashboards, alerting systems, and analytics tools.
Configure enrichments using the `fdl.Enrichment()` class in the Python Client SDK. For detailed configuration examples, see the [Enrichments Guide](/observability/llm/enrichments). For help choosing the right enrichment, see [Selecting Enrichments](/observability/llm/selecting-enrichments).
For ML model metrics (performance, drift, data integrity), see the [ML Metrics Reference](/reference/ml-metrics-reference).
## Safety metrics
Safety enrichments detect and flag unsafe, harmful, or policy-violating content in your LLM application's inputs and outputs.
| Metric | Enrichment Key | LLM Required? | Output Type | Description |
| --------------------------------------------- | -------------------- | -------------------- | -------------------------- | ----------------------------------------------------------------------- |
| [Safety](#safety) | `ftl_prompt_safety` | Yes (Fiddler Centor) | bool + float per dimension | Evaluates text safety across 11 dimensions using Fiddler Centor Models |
| [PII Detection](#pii-detection) | `pii` | No | bool + matches + entities | Detects personally identifiable information using Presidio |
| [Profanity](#profanity) | `profanity` | No | bool | Flags offensive or inappropriate language |
| [Banned Keywords](#banned-keywords) | `banned_keywords` | No | bool | Detects user-defined restricted terms |
| [Regex Match](#regex-match) | `regex_match` | No | category | Matches text against a user-defined regular expression |
| [Language Detection](#language-detection) | `language_detection` | No | string + float | Identifies the language of the source text |
| [Topic Classification](#topic-classification) | `topic_model` | No | list\[float] + string | Classifies text into user-defined topics using zero-shot classification |
### Safety
The Safety enrichment evaluates text safety across 11 dimensions using Fiddler's proprietary Centor Models. Each dimension produces a boolean flag and a confidence probability score.
**Enrichment key:** `ftl_prompt_safety`
| Dimension | Output Columns | Score Range | Description |
| -------------- | ------------------------------------ | ----------- | ----------------------------------------- |
| `illegal` | `illegal`, `illegal score` | 0.0 -- 1.0 | Content promoting illegal activities |
| `hateful` | `hateful`, `hateful score` | 0.0 -- 1.0 | Hateful or discriminatory content |
| `harassing` | `harassing`, `harassing score` | 0.0 -- 1.0 | Harassing or bullying content |
| `racist` | `racist`, `racist score` | 0.0 -- 1.0 | Racist content |
| `sexist` | `sexist`, `sexist score` | 0.0 -- 1.0 | Sexist content |
| `violent` | `violent`, `violent score` | 0.0 -- 1.0 | Content promoting violence |
| `sexual` | `sexual`, `sexual score` | 0.0 -- 1.0 | Sexually explicit content |
| `harmful` | `harmful`, `harmful score` | 0.0 -- 1.0 | Generally harmful content |
| `unethical` | `unethical`, `unethical score` | 0.0 -- 1.0 | Unethical content |
| `jailbreaking` | `jailbreaking`, `jailbreaking score` | 0.0 -- 1.0 | Jailbreaking or prompt injection attempts |
| `roleplaying` | `roleplaying`, `roleplaying score` | 0.0 -- 1.0 | Roleplaying attempts to bypass safety |
An aggregate `max_risk_prob` output is also generated, representing the maximum probability across all 11 dimensions.
For configuration details, see [Enrichments: Safety](/observability/llm/enrichments#safety).
### PII Detection
Detects and flags personally identifiable information using [Presidio](https://microsoft.github.io/presidio/analyzer/languages/). Generates a boolean flag, matched text spans, and detected entity types.
**Enrichment key:** `pii`
**Commonly used entity types:** `CREDIT_CARD`, `CRYPTO`, `DATE_TIME`, `EMAIL_ADDRESS`, `IBAN_CODE`, `IP_ADDRESS`, `LOCATION`, `PERSON`, `PHONE_NUMBER`, `URL`, `US_SSN`, `US_DRIVER_LICENSE`, `US_ITIN`, `US_PASSPORT`
Fiddler supports 32 entity types in total, including international identifiers for Australia, India, Singapore, and the UK. For the full list, see the [Presidio supported entities](https://microsoft.github.io/presidio/supported_entities/).
For configuration details, see [Enrichments: PII](/observability/llm/enrichments#personally-identifiable-information).
### Profanity
Flags offensive or inappropriate language using curated word lists from SurgeAI and Google.
**Enrichment key:** `profanity`
For configuration details, see [Enrichments: Profanity](/observability/llm/enrichments#profanity).
### Banned Keywords
Detects user-defined restricted terms in text inputs. The list of banned keywords is specified in the enrichment configuration.
**Enrichment key:** `banned_keywords`
For configuration details, see [Enrichments: Banned Keywords](/observability/llm/enrichments#banned-keyword-detector).
### Regex Match
Matches text against a user-defined regular expression pattern. Produces a categorical output of "Match" or "No Match".
**Enrichment key:** `regex_match`
For configuration details, see [Enrichments: Regex Match](/observability/llm/enrichments#regex-match).
### Language Detection
Identifies the language of the source text using [fasttext](https://fasttext.cc/docs/en/language-identification.html) models. Produces the detected language and a confidence probability.
**Enrichment key:** `language_detection`
For configuration details, see [Enrichments: Language Detection](/observability/llm/enrichments#language-detector).
### Topic Classification
Classifies text into user-defined topics using a zero-shot classification model. Produces per-topic probability scores and the top-scoring topic.
**Enrichment key:** `topic_model`
For configuration details, see [Enrichments: Topic](/observability/llm/enrichments#topic).
## Quality and hallucination metrics
Quality enrichments assess the accuracy, groundedness, and relevance of LLM-generated responses.
| Metric | Enrichment Key | LLM Required? | Output Type | Description |
| --------------------------------------------------------- | --------------------------- | -------------------- | ------------ | ---------------------------------------------------------------- |
| [Faithfulness (Centor Model)](#faithfulness-centor-model) | `ftl_response_faithfulness` | Yes (Fiddler Centor) | bool + float | Evaluates factual groundedness using Fiddler Centor Models |
| [RAG Faithfulness](#rag-faithfulness) | `faithfulness` | Yes (OpenAI) | bool | Evaluates factual accuracy of responses against provided context |
| [Answer Relevance](#answer-relevance) | `answer_relevance` | Yes (OpenAI) | bool | Evaluates whether responses address the input prompt |
| [Coherence](#coherence) | `coherence` | Yes (OpenAI) | bool | Assesses logical flow and clarity of responses |
| [Conciseness](#conciseness) | `conciseness` | Yes (OpenAI) | bool | Evaluates brevity and clarity of responses |
### Faithfulness (Centor Model)
Evaluates the factual groundedness of AI-generated responses against provided context using Fiddler's proprietary Centor Models. Produces a boolean faithfulness flag and a confidence probability score.
**Enrichment key:** `ftl_response_faithfulness`
The faithfulness threshold defaults to 0.5 and can be adjusted in the configuration to control scoring sensitivity. Higher thresholds result in stricter faithfulness detection (fewer responses labeled as faithful).
For configuration details, see [Enrichments: Faithfulness (Centor Model)](/observability/llm/enrichments#faithfulness-centor-model).
### RAG Faithfulness
Evaluates the accuracy and reliability of facts presented in AI-generated responses by checking whether the information aligns with the provided context documents. Uses an OpenAI LLM for evaluation.
**Enrichment key:** `faithfulness`
**RAG Faithfulness vs Faithfulness (Centor Model):** This enrichment uses OpenAI for evaluation. [Faithfulness (Centor Model)](#faithfulness-centor-model) uses Fiddler Centor Models for lower latency. See [LLM-Based Metrics](/observability/llm/llm-based-metrics) for a detailed comparison.
For configuration details, see [Enrichments: Faithfulness](/observability/llm/enrichments#faithfulness).
### Answer Relevance
Evaluates whether AI-generated responses address the input prompt. Produces a binary relevant/not-relevant result.
**Enrichment key:** `answer_relevance`
For configuration details, see [Enrichments: Answer Relevance](/observability/llm/enrichments#answer-relevance).
### Coherence
Assesses the logical flow and clarity of AI-generated responses, checking whether the content maintains a consistent theme and argument structure.
**Enrichment key:** `coherence`
For configuration details, see [Enrichments: Coherence](/observability/llm/enrichments#coherence).
### Conciseness
Evaluates whether AI-generated responses communicate their message efficiently without unnecessary elaboration or redundancy.
**Enrichment key:** `conciseness`
For configuration details, see [Enrichments: Conciseness](/observability/llm/enrichments#conciseness).
## Text statistics metrics
Text statistics enrichments provide quantitative analysis of text properties, including readability, length, and n-gram-based evaluation scores.
| Metric | Enrichment Key | LLM Required? | Output Type | Description |
| --------------------------- | -------------- | ------------- | -------------- | ------------------------------------------------------------- |
| [Textstat](#textstat) | `textstat` | No | float | Generates up to 19 text readability and complexity statistics |
| [Evaluate](#evaluate) | `evaluate` | No | float | Computes n-gram-based evaluation scores (BLEU, ROUGE, METEOR) |
| [Sentiment](#sentiment) | `sentiment` | No | float + string | Provides sentiment analysis using VADER |
| [Token Count](#token-count) | `token_count` | No | int | Counts the number of tokens in a string |
### Textstat
Generates text readability and complexity statistics using the [textstat](https://pypi.org/project/textstat/) library. You can select specific statistics or use all 19 available metrics.
**Enrichment key:** `textstat`
| Sub-metric | Range | Description |
| ------------------------------ | ----------- | --------------------------------------------------- |
| `char_count` | 0 -- 64,000 | Character count |
| `letter_count` | 0 -- 64,000 | Letter count (alphabetical characters) |
| `miniword_count` | 0 -- 64,000 | Count of short words |
| `words_per_sentence` | 0 -- 1,000 | Average words per sentence |
| `polysyllabcount` | 0 -- 64,000 | Polysyllabic word count |
| `lexicon_count` | 0 -- 64,000 | Word count |
| `syllable_count` | 0 -- 96,000 | Total syllable count |
| `sentence_count` | 0 -- 32,000 | Sentence count |
| `flesch_reading_ease` | -100 -- 100 | Flesch Reading Ease score (higher = easier to read) |
| `smog_index` | 0 -- 30 | SMOG readability index |
| `flesch_kincaid_grade` | -3.4 -- 100 | Flesch-Kincaid Grade Level |
| `coleman_liau_index` | 0 -- 20 | Coleman-Liau readability index |
| `automated_readability_index` | -3.4 -- 100 | Automated Readability Index |
| `dale_chall_readability_score` | 0 -- 10 | Dale-Chall readability score |
| `difficult_words` | 0 -- 64,000 | Count of difficult words |
| `linsear_write_formula` | 0 -- 20 | Linsear Write readability formula |
| `gunning_fog` | 0 -- 20 | Gunning Fog readability index |
| `long_word_count` | 0 -- 64,000 | Count of long words |
| `monosyllabcount` | 0 -- 64,000 | Monosyllabic word count |
If no statistics are specified in the configuration, the default statistic is `flesch_kincaid_grade`.
For configuration details, see [Enrichments: Textstat](/observability/llm/enrichments#textstat).
### Evaluate
Computes n-gram-based evaluation metrics for comparing two text passages, such as an AI-generated response and a reference answer. These metrics score highest when the reference and generated texts contain overlapping sequences.
**Enrichment key:** `evaluate`
| Sub-metric | Output Column | Score Range | Description |
| ---------- | ------------- | ----------- | --------------------------------------------------------------- |
| BLEU | `bleu` | 0.0 -- 1.0 | Precision of word n-grams between generated and reference text |
| ROUGE-1 | `rouge1` | 0.0 -- 1.0 | Unigram recall between generated and reference text |
| ROUGE-2 | `rouge2` | 0.0 -- 1.0 | Bigram recall between generated and reference text |
| ROUGE-L | `rougeL` | 0.0 -- 1.0 | Longest common subsequence between generated and reference text |
| ROUGE-Lsum | `rougeLsum` | 0.0 -- 1.0 | ROUGE-L applied at the summary level |
| METEOR | `meteor` | 0.0 -- 1.0 | Combines precision, recall, and semantic matching |
For configuration details, see [Enrichments: Evaluate](/observability/llm/enrichments#evaluate).
### Sentiment
Provides sentiment analysis using NLTK's VADER lexicon. Produces a compound score and a categorical sentiment label.
**Enrichment key:** `sentiment`
| Output Column | Type | Description |
| ------------- | ------ | ------------------------------------------- |
| `compound` | float | Raw compound sentiment score |
| `sentiment` | string | One of `positive`, `negative`, or `neutral` |
For configuration details, see [Enrichments: Sentiment](/observability/llm/enrichments#sentiment).
### Token Count
Counts the number of tokens in a string using the [tiktoken](https://github.com/openai/tiktoken) library.
**Enrichment key:** `token_count`
For configuration details, see [Enrichments: Token Count](/observability/llm/enrichments#token-count).
## Text validation metrics
Text validation enrichments verify the structural correctness of generated text outputs such as SQL queries and JSON payloads.
| Metric | Enrichment Key | LLM Required? | Output Type | Description |
| ----------------------------------- | ----------------- | ------------- | ------------- | ----------------------------------------------------- |
| [SQL Validation](#sql-validation) | `sql_validation` | No | bool + string | Validates SQL syntax for a specified dialect |
| [JSON Validation](#json-validation) | `json_validation` | No | bool + string | Validates JSON syntax and optionally against a schema |
### SQL Validation
Validates SQL query syntax for a specified dialect. Supports 25+ SQL dialects including MySQL, PostgreSQL, Snowflake, BigQuery, and others.
**Enrichment key:** `sql_validation`
Query validation is syntax-based and does not check against any existing schema or databases for validity.
For configuration details, see [Enrichments: SQL Validation](/observability/llm/enrichments#sql-validation).
### JSON Validation
Validates JSON for correctness and optionally against a user-defined [JSON Schema](https://python-jsonschema.readthedocs.io).
**Enrichment key:** `json_validation`
For configuration details, see [Enrichments: JSON Validation](/observability/llm/enrichments#json-validation).
## Embedding metrics
Embedding enrichments convert text into vector representations for drift detection and visualization.
| Metric | Enrichment Key | LLM Required? | Output Type | Description |
| --------------------------------------- | ---------------- | ------------- | -------------- | -------------------------------------------------------------------- |
| [Text Embedding](#text-embedding) | `TextEmbedding` | No | vector + float | Generates text embeddings for UMAP visualization and drift detection |
| [Centroid Distance](#centroid-distance) | (auto-generated) | No | float | Distance from the nearest cluster centroid |
### Text Embedding
Converts unstructured text into high-dimensional vector representations for semantic analysis. Enables Fiddler's 3D UMAP visualizations and embedding-based drift detection.
**Class:** `fdl.TextEmbedding()`
TextEmbedding is configured using `fdl.TextEmbedding()` rather than `fdl.Enrichment()`. See the [Enrichments Guide](/observability/llm/enrichments#embedding) for usage examples.
### Centroid Distance
Measures the distance between a data point's embedding and the nearest cluster centroid. This metric is automatically generated when a TextEmbedding enrichment is created.
For configuration details, see [Enrichments: Centroid Distance](/observability/llm/enrichments#centroid-distance).
## Related resources
* [ML Metrics Reference](/reference/ml-metrics-reference) — Built-in metrics for ML model monitoring
* [Enrichments Guide](/observability/llm/enrichments) — Configuration examples for all enrichments
* [Selecting Enrichments](/observability/llm/selecting-enrichments) — Choosing the right enrichment for your use case
* [LLM-Based Metrics](/observability/llm/llm-based-metrics) — Detailed comparison of LLM-based evaluation methods
# ML Metrics Reference
Source: https://docs.fiddler.ai/reference/ml-metrics-reference
Complete reference of all built-in ML metrics supported by the Fiddler monitoring platform, organized by category and model task type.
Fiddler provides 35 built-in metrics for monitoring ML models in production. These metrics cover model performance, data drift, data integrity, traffic, and statistics. You can also define [custom metrics](/observability/platform/custom-metrics) using the [Fiddler Query Language](/observability/platform/fiddler-query-language).
For LLM and GenAI application metrics, see the [LLM Observability Metrics Reference](/reference/llm-observability-metrics).
## Performance metrics
Performance metrics measure how well a model performs on its task. The available metrics depend on the model task type. For more details on performance monitoring workflows, see [Performance Tracking](/observability/platform/performance-tracking-platform).
### Binary classification
| Metric | API ID | Score Range | Description |
| --------------------------- | ---------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------ |
| Accuracy | `accuracy` | 0 -- 1 | (TP + TN) / (TP + TN + FP + FN) |
| Log Loss | `log_loss` | 0 -- infinity | Measures the difference between the predicted probability distribution and the true distribution |
| Precision | `precision` | 0 -- 1 | TP / (TP + FP). Requires a decision threshold. |
| Recall / True Positive Rate | `recall` | 0 -- 1 | TP / (TP + FN). Requires a decision threshold. |
| F1 Score | `f1_score` | 0 -- 1 | 2 \* (Precision \* Recall) / (Precision + Recall). Requires a decision threshold. |
| False Positive Rate | `fpr` | 0 -- 1 | FP / (FP + TN). Requires a decision threshold. |
| AUC | `auc` | 0 -- 1 | Area Under the ROC Curve (histogram-based calculation). See also AUROC. |
| AUROC | `auroc` | 0 -- 1 | Area Under the Receiver Operating Characteristic curve, plotting true positive rate against false positive rate |
| Expected Calibration Error | `expected_calibration_error` | 0 -- 1 | Measures the difference between predicted probabilities and empirical probabilities |
| Geometric Mean | `geometric_mean` | 0 -- 1 | Square root of (Precision \* Recall). Requires a decision threshold. |
| Calibrated Threshold | `calibrated_threshold` | 0 -- 1 | A threshold that balances precision and recall at a particular operating point |
| Data Count | `data_count` | 0 -- infinity | The number of events where target and output are both not NULL. Used as the denominator for accuracy calculations. |
### Multi-class classification
| Metric | API ID | Score Range | Description |
| -------------- | ---------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------ |
| Accuracy | `accuracy` | 0 -- 1 | (Number of correctly classified samples) / Data Count |
| Log Loss | `log_loss` | 0 -- infinity | Measures the difference between the predicted probability distribution and the true distribution, on a logarithmic scale |
| Log Loss Count | `log_loss_count` | 0 -- infinity | Count of events used in the Log Loss calculation |
### Regression
| Metric | API ID | Score Range | Description |
| ----------------------------------------------- | ------- | -------------- | ----------------------------------------------------------------------------------------- |
| Mean Absolute Error (MAE) | `mae` | 0 -- infinity | Average of the absolute differences between predicted and true values |
| Mean Squared Error (MSE) | `mse` | 0 -- infinity | Average of the squared differences between predicted and true values |
| Mean Absolute Percentage Error (MAPE) | `mape` | 0 -- infinity | Average of the absolute percentage differences between predicted and true values |
| Weighted Mean Absolute Percentage Error (WMAPE) | `wmape` | 0 -- infinity | Weighted average of the absolute percentage differences between predicted and true values |
| R-squared (R²) | `r2` | -infinity -- 1 | Proportion of variance in the dependent variable explained by the independent variables |
### Ranking
| Metric | API ID | Score Range | Description |
| -------------------------------------------- | ------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Mean Average Precision (MAP) | `map` | 0 -- 1 | Average precision of relevant items in the top-k results. For binary relevance ranking only. Supports configurable `top_k`. |
| Normalized Discounted Cumulative Gain (NDCG) | `ndcg_mean` | 0 -- 1 | Quality of the ranking by discounting relevance scores at lower ranks. Supports configurable `top_k`. |
| Query Count | `query_count` | 0 -- infinity | Number of ranking queries in the time period |
## Drift metrics
Drift metrics measure distributional changes between your baseline dataset and production data. High drift can indicate data pipeline issues or genuine shifts in the data distribution. Both metrics require a [baseline](/glossary/baseline) dataset. For more details, see [Data Drift](/observability/platform/data-drift-platform).
| Metric | API ID | Score Range | Description |
| -------------------------------- | ------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Jensen-Shannon Distance (JSD) | `jsd` | 0 -- 1 | Distance between the baseline distribution and the production distribution for a given field, using [configurable bins](/observability/platform/data-drift-platform) for numerical columns |
| Population Stability Index (PSI) | `psi` | 0 -- infinity | Drift metric based on multinomial classification of a variable into [configurable bins](/observability/platform/data-drift-platform), comparing baseline and production distributions |
The drift analytics table also provides **Feature Impact**, **Feature Drift**, and **Prediction Drift Impact** as derived values to help identify which features contribute most to prediction drift.
## Data integrity metrics
Data integrity metrics detect violations in production data compared to the schema established during model onboarding. Fiddler tracks three violation types: missing values, type mismatches, and range violations. Both raw counts and percentages are available. For more details, see [Data Integrity](/observability/platform/data-integrity-platform).
### Count-based
| Metric | API ID | Description |
| ----------------------- | ----------------------- | --------------------------------------------------------- |
| Any Violation | `any_violation_count` | Count of any data integrity violation across all features |
| Missing Value Violation | `null_violation_count` | Count of missing value violations across all features |
| Range Violation | `range_violation_count` | Count of range violations across all features |
| Type Violation | `type_violation_count` | Count of data type violations across all features |
### Percentage-based
| Metric | API ID | Description |
| ------------------------- | ---------------------------- | ------------------------------------------------------ |
| % Any Violation | `any_violation_percentage` | Percentage of events with any data integrity violation |
| % Missing Value Violation | `null_violation_percentage` | Percentage of events with missing value violations |
| % Range Violation | `range_violation_percentage` | Percentage of events with range violations |
| % Type Violation | `type_violation_percentage` | Percentage of events with data type violations |
## Traffic metrics
Traffic metrics provide visibility into the operational health of your model service. For more details, see [Traffic](/observability/platform/traffic-platform).
| Metric | API ID | Description |
| ------- | --------- | ------------------------------------------------------------ |
| Traffic | `traffic` | Volume of inference requests received by the model over time |
## Statistics metrics
Statistics metrics provide basic aggregations over columns. These are useful for monitoring custom metadata fields over time. For more details, see [Statistics](/observability/platform/statistics).
| Metric | API ID | Applies To | Description |
| --------- | ----------- | ----------------------------- | ----------------------------------- |
| Average | `average` | Numeric columns | Arithmetic mean of a numeric column |
| Sum | `sum` | Numeric columns | Sum of a numeric column |
| Frequency | `frequency` | Categorical / Boolean columns | Count of occurrences for each value |
## Custom metrics
In addition to the built-in metrics above, you can define custom metrics using the [Fiddler Query Language (FQL)](/observability/platform/fiddler-query-language). Custom metrics support aggregations, operators, and metric functions to create business-specific KPIs.
For details on creating and managing custom metrics, see:
* [Custom Metrics Guide](/observability/platform/custom-metrics)
* [CustomMetric SDK Reference](/sdk-api/python-client/custom-metric)
* [Custom Metrics REST API](/sdk-api/rest-api/custom-metrics)
## Related resources
* [LLM Observability Metrics Reference](/reference/llm-observability-metrics) — Enrichments for LLM application monitoring
* [Performance Tracking](/observability/platform/performance-tracking-platform) — Performance monitoring workflows
* [Data Drift](/observability/platform/data-drift-platform) — Drift monitoring and analysis
* [Data Integrity](/observability/platform/data-integrity-platform) — Data quality monitoring
* [Custom Metrics Guide](/observability/platform/custom-metrics) — Creating custom metrics with FQL
# Python Version Support Policy
Source: https://docs.fiddler.ai/reference/python-support-policy
How Fiddler's Python SDKs decide which Python versions they support, and when support for a version ends.
Fiddler's Python SDKs follow a clear, predictable policy for which Python versions they support. This page explains that policy so you can plan Python upgrades with confidence.
### Our policy
Fiddler supports each Python version through its official Python Software Foundation (PSF) end-of-life, and drops support once a version reaches PSF end-of-life. This keeps Fiddler's SDKs aligned with Python versions that still receive upstream security patches.
The PSF publishes the end-of-life date for every Python version in its [Status of Python versions](https://devguide.python.org/versions/) table — the source of truth for when a version's support ends. Keeping your projects on a version that the PSF still supports is the most reliable way to stay on a Fiddler-supported Python. Fiddler communicates upcoming changes ahead of time so you have room to upgrade.
This window — support through PSF end-of-life — is more generous than community policies such as [SPEC 0](https://scientific-python.org/specs/spec-0000/) and the superseded [NEP 29](https://numpy.org/neps/nep-0029-deprecation_policy.html), which drop older Python versions sooner.
### Which SDKs this applies to
This policy applies to all of Fiddler's [Python SDKs](/sdk-api). Fiddler's JavaScript and TypeScript SDKs are not affected by Python version changes.
### How this differs from the compatibility matrix
Two references cover Python and Fiddler versions, and they answer different questions:
* This **Python version support policy** describes which **Python language** versions Fiddler's Python SDKs run on.
* The [compatibility matrix](/changelog/compatibility-matrix) maps each **Fiddler Python client version** to the **Fiddler platform** versions it works with.
Use this page to choose a supported Python version, and the compatibility matrix to choose a client version that matches your Fiddler platform.
### Related resources
* [Compatibility matrix](/changelog/compatibility-matrix) — Python client and Fiddler platform version compatibility.
* [SDK & API reference](/sdk-api) — every Fiddler SDK.
* [Feature maturity definitions](/reference/feature-maturity-definitions) — Fiddler's release and support stages.
* [Status of Python versions](https://devguide.python.org/versions/) — PSF end-of-life dates.
# ADKSpanProcessor
Source: https://docs.fiddler.ai/sdk-api/adk/adk-span-processor
Span processor that backfills session identity onto ADK root spans.
Span processor that backfills `gen_ai.conversation.id` onto the root ADK span.
Extends `FiddlerSpanProcessor` (from `fiddler-otel`). ADK sets
`gen_ai.conversation.id` only on child spans (`invoke_agent`, `call_llm`, etc.);
the root `invocation` span gets no attributes. This processor copies the
conversation ID onto the still-open root span when the first child carrying it
ends.
All other enrichment (span-type classification, agent ID computation, content
extraction, attribute normalization) is delegated to the Fiddler backend.
## Example
```python theme={null}
from fiddler_adk import ADKSpanProcessor
# Typically used via GoogleADKInstrumentor, which adds it automatically.
# Direct usage is for advanced custom pipeline setups:
from opentelemetry.sdk.trace import TracerProvider
provider = TracerProvider()
provider.add_span_processor(ADKSpanProcessor())
```
## Inherits
* `FiddlerSpanProcessor` (from `fiddler-otel`)
## Behavior
### on\_start(span, parent\_context)
Tracks root spans (spans with no parent) by their `trace_id`. This allows
the processor to backfill session identity when a child span ends.
### on\_end(span)
When a child span ends:
1. If the child carries `gen_ai.conversation.id` and the root span for that
trace is still recording, copies the conversation ID onto the root.
2. When the root span itself ends, removes it from tracking.
The first child with a conversation ID wins -- subsequent children do not
overwrite it.
### force\_flush()
Delegates to the parent `FiddlerSpanProcessor.force_flush()`.
### Returns
`True` if flush completed successfully.
# GoogleADKInstrumentor
Source: https://docs.fiddler.ai/sdk-api/adk/google-adk-instrumentor
OpenTelemetry instrumentor for Google ADK agents.
OpenTelemetry instrumentor for Google ADK (Agent Development Kit) agents.
This instrumentor sets up an isolated Fiddler tracing pipeline via
`FiddlerClient` (provider, processor, exporter) and promotes it to the
global tracer provider so ADK's `gcp.vertex.agent` tracer resolves to it.
The SDK operates in standalone mode -- it does not interact with
customer-configured tracers or providers.
## Example
```python theme={null}
from fiddler_otel import FiddlerClient
from fiddler_adk import GoogleADKInstrumentor
client = FiddlerClient(
api_key="YOUR_KEY",
application_id="YOUR_APP_UUID",
url="https://your-instance.com",
)
# Enable instrumentation
instrumentor = GoogleADKInstrumentor(client)
instrumentor.instrument()
# All ADK agents created after this point are automatically traced
```
## Parameters
FiddlerClient instance that owns the tracing pipeline (tracer, provider,
processor, exporter). The client's provider is promoted to global so ADK's
tracer resolves to it.
## instrument()
Enable instrumentation. Sets up the tracer provider and adds the
`ADKSpanProcessor` for session-ID propagation.
```python theme={null}
instrumentor.instrument()
```
### Returns
No return value. After calling, all ADK agents will be traced.
## uninstrument()
Disable instrumentation.
The tracer provider wiring is left in place: OpenTelemetry does not support
unsetting the global provider or removing span processors. Spans continue
to be exported until the client is shut down.
```python theme={null}
instrumentor.uninstrument()
```
### Returns
No return value.
## instrumentation\_dependencies()
Return the list of packages required for instrumentation.
### Returns
Collection of required package names (includes `google-adk`).
# Introduction
Source: https://docs.fiddler.ai/sdk-api/adk/index
Complete API reference for fiddler-adk
[](https://pypi.org/project/fiddler-adk/)
## Overview
Complete API reference documentation for the `fiddler-adk` package, which
instruments [Google ADK](https://github.com/google/adk-python) (Agent Development Kit)
and exports OpenTelemetry traces to Fiddler.
The package is published to PyPI as **`fiddler-adk`** and imported in
Python as **`fiddler_adk`**:
```bash theme={null}
pip install fiddler-adk
```
```python theme={null}
from fiddler_adk import GoogleADKInstrumentor
```
## Components
### Instrumentation
`GoogleADKInstrumentor` sets up the Fiddler tracing pipeline and promotes it
to the global tracer provider so ADK's `gcp.vertex.agent` tracer resolves to it.
* [GoogleADKInstrumentor](/sdk-api/adk/google-adk-instrumentor)
### Span Processor
`ADKSpanProcessor` backfills `gen_ai.conversation.id` from child spans onto
the root `invocation` span, ensuring all spans in a turn share the same session
identity in the Fiddler UI.
* [ADKSpanProcessor](/sdk-api/adk/adk-span-processor)
# AnswerRelevance
Source: https://docs.fiddler.ai/sdk-api/evals/answer-relevance
Evaluator to assess how well an answer addresses a given question with optional context.
Evaluator to assess how well an answer addresses a given question with optional context.
The AnswerRelevance evaluator measures whether an LLM's answer is relevant
and directly addresses the question being asked. This version supports optional
reference documents to provide additional context for more nuanced relevance
assessment. This is ideal for RAG (Retrieval-Augmented Generation) pipelines.
Key Features:
* **Relevance Assessment**: Determines if the answer directly addresses the question
* **Three-Level Scoring**: Returns high (1.0), medium (0.5), or low (0.0) relevance scores
* **Context-Aware**: Can use retrieved documents to assess relevance more accurately
* **Detailed Reasoning**: Provides explanation for the relevance assessment
* **Fiddler API Integration**: Uses Fiddler's built-in relevance evaluation model
Use Cases:
* **RAG Systems**: Evaluating if generated answers are relevant to user queries
* **Q\&A Systems**: Ensuring answers stay on topic
* **Customer Support**: Verifying responses address user queries
* **Educational Content**: Checking if explanations answer the question
* **Research Assistance**: Validating that responses are relevant to queries
Scoring Logic:
* **1.0 (High)**: Answer is fully relevant and directly addresses the question
* **0.5 (Medium)**: Answer partially addresses the question but may miss some aspects
* **0.0 (Low)**: Answer does not address the question or is off-topic
## Parameters
* **user\_query** (*str*) – The question or query being asked.
* **rag\_response** (*str*) – The LLM's response to evaluate.
* **retrieved\_documents** (*list* \*\[\**str* *]* *,* *optional*) – Reference documents for context.
* **model** (*str*)
* **credential** (*str* *|* *None*)
* **kwargs** (*Any*)
## Returns
A Score object containing:
* value: 1.0 for high, 0.5 for medium, 0.0 for low relevance
* label: "high", "medium", or "low"
* reasoning: Detailed explanation of the assessment
## Example
```python theme={null}
from fiddler_evals.evaluators import AnswerRelevance
evaluator = AnswerRelevance(model="openai/gpt-4o")
# High relevance answer
score = evaluator.score(
user_query="What is the capital of France?",
rag_response="The capital of France is Paris."
)
print(f"Relevance: {score.label}") # "high"
print(f"Score: {score.value}") # 1.0
# With context documents
score = evaluator.score(
user_query="What is our refund policy?",
rag_response="Our refund policy allows returns within 30 days.",
retrieved_documents=[
"Refund Policy: Customers may return items within 30 days of purchase.",
"All returns must include original packaging."
]
)
```
This evaluator uses Fiddler's built-in relevance assessment model
and requires an active connection to the Fiddler API.
## name *= 'answer\_relevance'*
## score()
Score the relevance of an answer to a question.
### Parameters
The question or query being asked.
The LLM's response to evaluate.
Reference documents for context.
### Returns
A Score object containing:
* value: 1.0 for high, 0.5 for medium, 0.0 for low relevance
* label: "high", "medium", or "low"
* reasoning: Detailed explanation of the assessment
# Application
Source: https://docs.fiddler.ai/sdk-api/evals/application
Represents a GenAI Application container for organizing GenAI application resources.
Represents a GenAI Application container for organizing GenAI application resources.
An Application is a logical container within a Project that groups related GenAI
application resources including datasets, experiments, evaluators, and monitoring
configurations. Applications provide resource organization, access control, and
lifecycle management for GenAI App monitoring workflows.
Key Features:
* **Resource Organization**: Container for related GenAI application resources
* **Project Context**: Applications are scoped within projects for isolation
* **Access Management**: Application-level permissions and access control
* **Monitoring Coordination**: Centralized monitoring and alerting configuration
* **Lifecycle Management**: Coordinated creation, updates, and deletion of resources
Application Lifecycle:
1. **Creation**: Create application with unique name within a project
2. **Configuration**: Set up datasets, evaluators, and monitoring
3. **Operations**: Publish logs, monitor performance, manage alerts
4. **Maintenance**: Update configurations and resources
5. **Cleanup**: Delete application when no longer needed
## Example
```python theme={null}
# Create a new application for fraud detection
application = Application.create(name="fraud-detection-app", project_id=project_id)
print(f"Created application: {application.name} (ID: {application.id})")
```
Applications are permanent containers - once created, the name cannot be changed.
Deleting an application removes all contained resources and configurations.
Consider the organizational structure carefully before creating applications.
## id
## name
## created\_at
## updated\_at
## created\_by
## updated\_by
## project
## *classmethod* get\_by\_id()
Retrieve an application by its unique identifier.
Fetches an application from the Fiddler platform using its UUID. This is the most
direct way to retrieve an application when you know its ID.
### Parameters
The unique identifier (UUID) of the application to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The application instance with all metadata and configuration.
### Raises
* **NotFound** – If no application exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get application by UUID
application = Application.get_by_id(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved application: {application.name}")
print(f"Created: {application.created_at}")
print(f"Project: {application.project.name}")
```
This method makes an API call to fetch the latest application state from the server.
The returned application instance reflects the current state in Fiddler.
## *classmethod* get\_by\_name()
Retrieve an application by name within a project.
Finds and returns an application using its name within the specified project.
This is useful when you know the application name and project but not its UUID.
Application names are unique within a project, making this a reliable lookup method.
### Parameters
The name of the application to retrieve. Application names are unique
within a project and are case-sensitive.
The UUID of the project containing the application.
Can be provided as a UUID object or string representation.
### Returns
The application instance matching the specified name.
### Raises
* **NotFound** – If no application exists with the specified name in the project.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get project instance
project = Project.get_by_name(name="fraud-detection-project")
# Get application by name within a project
application = Application.get_by_name(
name="fraud-detection-app",
project_id=project.id
)
print(f"Found application: {application.name} (ID: {application.id})")
print(f"Created: {application.created_at}")
print(f"Project: {application.project.name}")
```
Application names are case-sensitive and must match exactly. Use this method
when you have a known application name from configuration or user input.
## *classmethod* list()
List all applications in a project.
Retrieves all applications that the current user has access to within the specified
project. Returns an iterator for memory efficiency when dealing with many applications.
### Parameters
The UUID of the project to list applications from.
Can be provided as a UUID object or string representation.
### Yields
`Application` – Application instances for all accessible applications in the project.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Returns
`Iterator[Application]`
### Example
```python theme={null}
# Get project instance
project = Project.get_by_name(name="fraud-detection-project")
# List all applications in a project
for application in Application.list(project_id=project.id):
print(f"Application: {application.name}")
print(f" ID: {application.id}")
print(f" Created: {application.created_at}")
# Convert to list for counting and filtering
applications = list(Application.list(project_id=project.id))
print(f"Total applications in project: {len(applications)}")
# Find applications by name pattern
fraud_apps = [
app for app in Application.list(project_id=project.id)
if "fraud" in app.name.lower()
]
print(f"Fraud detection applications: {len(fraud_apps)}")
```
This method returns an iterator for memory efficiency. Convert to a list
with list(Application.list(project\_id)) if you need to iterate multiple times or get
the total count. The iterator fetches applications lazily from the API.
## *classmethod* create()
Create a new application in a project.
Creates a new application within the specified project on the Fiddler platform.
The application must have a unique name within the project.
### Parameters
Application name, must be unique within the project.
The UUID of the project to create the application in.
Can be provided as a UUID object or string representation.
### Returns
The newly created application instance with server-assigned fields.
### Raises
* **Conflict** – If an application with the same name already exists in the project.
* **ValidationError** – If the application configuration is invalid (e.g., invalid name format).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get project instance
project = Project.get_by_name(name="fraud-detection-project")
# Create a new application for fraud detection
application = Application.create(
name="fraud-detection-app",
project_id=project.id
)
print(f"Created application with ID: {application.id}")
print(f"Created at: {application.created_at}")
print(f"Project: {application.project.name}")
```
After successful creation, the application instance is returned with
server-assigned metadata. The application is immediately available
for adding datasets, evaluators, and other resources.
## *classmethod* get\_or\_create()
Get an existing application by name or create a new one if it doesn't exist.
This is a convenience method that attempts to retrieve an application by name
within a project, and if not found, creates a new application with that name.
Useful for idempotent application setup in automation scripts and deployment pipelines.
### Parameters
The name of the application to retrieve or create.
The UUID of the project to search/create the application in.
Can be provided as a UUID object or string representation.
### Returns
Either the existing application with the specified name,
or a newly created application if none existed.
### Raises
* **ValidationError** – If the application name format is invalid.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get project instance
project = Project.get_by_name(name="fraud-detection-project")
# Safe application setup - get existing or create new
application = Application.get_or_create(
name="fraud-detection-app",
project_id=project.id
)
print(f"Using application: {application.name} (ID: {application.id})")
# Idempotent setup in deployment scripts
application = Application.get_or_create(
name="llm-pipeline-app",
project_id=project.id
)
# Use in configuration management
environments = ["dev", "staging", "prod"]
applications = {}
for env in environments:
applications[env] = Application.get_or_create(
name=f"fraud-detection-{env}",
project_id=project.id
)
```
This method is idempotent - calling it multiple times with the same name
and project\_id will return the same application. It logs when creating a new
application for visibility in automation scenarios.
# Coherence
Source: https://docs.fiddler.ai/sdk-api/evals/coherence
Evaluator to assess the coherence and logical flow of a response.
Evaluator to assess the coherence and logical flow of a response.
The Coherence evaluator measures whether a response is well-structured, logically
consistent, and flows naturally from one idea to the next. This metric is important
for ensuring that responses are easy to follow and understand, with clear connections
between different parts of the text.
Key Features:
* **Coherence Assessment**: Determines if the response has logical flow and structure
* **Binary Scoring**: Returns 1.0 for coherent responses, 0.0 for incoherent ones
* **Optional Context**: Can optionally use a prompt for context-aware evaluation
* **Detailed Reasoning**: Provides explanation for the coherence assessment
* **Fiddler API Integration**: Uses Fiddler's built-in coherence evaluation model
Use Cases:
* **Content Quality**: Ensuring responses are well-structured and logical
* **Educational Content**: Verifying explanations flow logically
* **Technical Documentation**: Checking if instructions are coherent
* **Creative Writing**: Assessing narrative flow and consistency
* **Conversational AI**: Ensuring responses make sense in context
Scoring Logic:
* **1.0 (Coherent)**: Response has clear logical flow and structure
* **0.0 (Incoherent)**: Response lacks logical flow or has structural issues
## Parameters
* **response** (*str*) – The response to evaluate for coherence.
* **prompt** (*str* *,* *optional*) – The original prompt that generated the response.
Used for context-aware coherence evaluation.
* **model** (*str*)
* **credential** (*str* *|* *None*)
* **kwargs** (*Any*)
## Returns
A Score object containing:
* name: "is\_coherent"
* evaluator\_name: "Coherence"
* value: 1.0 if coherent, 0.0 if incoherent
* label: String representation of the boolean result
* reasoning: Explanation for the coherence assessment
## Raises
**ValueError** – If the response is empty or None, or if no scores are returned from the API.
## Example
```python theme={null}
from fiddler_evals.evaluators import Coherence
evaluator = Coherence()
```
```python theme={null}
# Coherent response
score = evaluator.score(
response="First, we need to understand the problem. Then, we can identify potential solutions. Finally, we should test our approach."
)
print(f"Coherence: {score.value}") # 1.0
# Incoherent response
incoherent_score = evaluator.score(
prompt="Explain the process of making coffee",
response="The sky is blue. I like pizza. Quantum physics is complex. Let's go shopping.",
)
print(f"Coherence: {incoherent_score.value}") # 0.0
# With context
contextual_score = evaluator.score(
prompt="Explain the process of making coffee",
response="First, grind the beans. Then, heat the water. Next, pour water over grounds. Finally, enjoy your coffee."
)
print(f"Coherence: {contextual_score.value}") # 1.0
# Check coherence
if score.value == 1.0:
print("Response is coherent and well-structured")
```
This evaluator uses Fiddler's built-in coherence assessment model
and requires an active connection to the Fiddler API. The optional
prompt parameter can provide additional context for more accurate
coherence evaluation, especially when the response needs to be
evaluated in relation to a specific question or task.
## name *= 'coherence'*
## score()
Score the coherence of a response.
### Parameters
The original prompt that generated the response.
The response to evaluate for coherence.
### Returns
A Score object for coherence assessment.
# Conciseness
Source: https://docs.fiddler.ai/sdk-api/evals/conciseness
Evaluator to assess how concise and to-the-point an answer is.
Evaluator to assess how concise and to-the-point an answer is.
The Conciseness evaluator measures whether an LLM's answer is appropriately
brief and direct without unnecessary verbosity. This metric is important for
ensuring that responses are efficient and don't waste users' time with
irrelevant details or excessive elaboration.
Key Features:
* **Conciseness Assessment**: Determines if the answer is appropriately brief
* **Binary Scoring**: Returns 1.0 for concise answers, 0.0 for verbose ones
* **Detailed Reasoning**: Provides explanation for the conciseness assessment
* **Fiddler API Integration**: Uses Fiddler's built-in conciseness evaluation model
Use Cases:
* **Customer Support**: Ensuring responses are direct and helpful
* **Technical Documentation**: Verifying explanations are clear and brief
* **Educational Content**: Checking if explanations are appropriately detailed
* **API Responses**: Ensuring responses are efficient and focused
Scoring Logic:
* **1.0 (Concise)**: Answer is appropriately brief and to-the-point
* **0.0 (Verbose)**: Answer is unnecessarily long or contains irrelevant details
## Parameters
* **response** (*str*) – The LLM's response to evaluate for conciseness.
* **model** (*str*)
* **credential** (*str* *|* *None*)
* **kwargs** (*Any*)
## Returns
A Score object containing:
* value: 1.0 if concise, 0.0 if verbose
* label: String representation of the boolean result
* reasoning: Detailed explanation of the assessment
## Example
```python theme={null}
from fiddler_evals.evaluators import Conciseness
evaluator = Conciseness()
```
```python theme={null}
# Concise answer
score = evaluator.score("The capital of France is Paris.")
print(f"Conciseness: {score.value}") # 1.0
print(f"Reasoning: {score.reasoning}")
# Verbose answer
score = evaluator.score(
"Well, that's a great question about France. Let me think about this..."
"France is a beautiful country in Europe, and it has many wonderful cities..."
"The capital city of France is Paris, which is located in the north-central part..."
)
print(f"Conciseness: {score.value}") # 0.0
```
This evaluator uses Fiddler's built-in conciseness assessment model
and requires an active connection to the Fiddler API.
## name *= 'conciseness'*
## score()
Score the conciseness of an answer.
### Parameters
The LLM's response to evaluate for conciseness.
### Returns
A Score object containing:
* value: 1.0 if concise, 0.0 if verbose
* label: String representation of the boolean result
* reasoning: Detailed explanation of the assessment
# Connection
Source: https://docs.fiddler.ai/sdk-api/evals/connection
Manages authenticated connections to the Fiddler platform.
Manages authenticated connections to the Fiddler platform.
The Connection class handles all aspects of connecting to and communicating
with the Fiddler platform, including authentication, HTTP client management,
server version compatibility checking, and organization context management.
This class provides the foundation for all API interactions with Fiddler,
managing connection parameters, authentication tokens, and ensuring proper
communication protocols are established.
## Example
```python theme={null}
# Creating a basic connection
connection = Connection(
url="https://your-instance.fiddler.ai",
token="your-api-key"
)
# Creating a connection with custom timeout and proxy
connection = Connection(
url="https://your-instance.fiddler.ai",
token="your-api-key",
timeout=(5.0, 30.0), # (connect_timeout, read_timeout)
proxies={"https": "https://proxy.company.com:8080"}
)
# Creating a connection without SSL verification
connection = Connection(
url="https://your-instance.fiddler.ai",
token="your-api-key",
verify=False, # Not recommended for production
validate=False # Skip version compatibility check
)
```
Initialize a connection to the Fiddler platform.
## Parameters
The base URL to your Fiddler platform instance
Authentication token obtained from the Fiddler UI
Dictionary mapping protocol to proxy URL for HTTP requests
HTTP request timeout settings (float or tuple of connect/read timeouts)
Whether to verify server's TLS certificate (default: True)
Whether to validate server/client version compatibility (default: True)
## Raises
* **ValueError** – If url or token parameters are empty
* **IncompatibleClient** – If server version is incompatible with client version
## *property* client
Get the HTTP request client instance for API communication.
### Returns
Configured HTTP client with authentication headers,
proxy settings, and timeout configurations.
## *property* server\_info
Get server information and metadata from the Fiddler platform.
### Returns
Server information including version, organization details,
and platform configuration.
## *property* server\_version
Get the semantic version of the connected Fiddler server.
### Returns
Semantic version object representing the server version.
## *property* organization\_name
Get the name of the connected organization.
### Returns
Name of the organization associated with this connection.
## *property* organization\_id
Get the UUID of the connected organization.
### Returns
Unique identifier of the organization associated with this connection.
# ContextRelevance
Source: https://docs.fiddler.ai/sdk-api/evals/context-relevance
Evaluator to assess how relevant retrieved documents are to a user query.
Evaluator to assess how relevant retrieved documents are to a user query.
The ContextRelevance evaluator measures whether retrieved documents provide
sufficient context to answer a given question. This is a critical metric for
RAG (Retrieval-Augmented Generation) pipelines to ensure the retrieval step
is fetching useful information.
Key Features:
* **Retrieval Assessment**: Determines if retrieved documents support the query
* **Three-Level Scoring**: Returns high (1.0), medium (0.5), or low (0.0) relevance scores
* **RAG Pipeline Evaluation**: Specifically designed for evaluating retrieval quality
* **Detailed Reasoning**: Provides explanation for the relevance assessment
* **Fiddler API Integration**: Uses Fiddler's built-in context relevance model
Use Cases:
* **RAG Systems**: Evaluating retrieval quality in RAG pipelines
* **Search Systems**: Assessing if search results are relevant to queries
* **Document Q\&A**: Verifying retrieved context supports the question
* **Knowledge Base Evaluation**: Testing retrieval effectiveness
Scoring Logic:
* **1.0 (High)**: Retrieved documents provide all necessary information to answer the query
* **0.5 (Medium)**: Retrieved documents are on topic but don't fully support a complete answer
* **0.0 (Low)**: Retrieved documents are not relevant to the query
## Parameters
* **user\_query** (*str*) – The question or query being asked.
* **retrieved\_documents** (*list* \*\[\**str* *]*) – The documents retrieved as context.
* **model** (*str*)
* **credential** (*str* *|* *None*)
* **kwargs** (*Any*)
## Returns
A Score object containing:
* value: 1.0 for high, 0.5 for medium, 0.0 for low relevance
* label: "high", "medium", or "low"
* reasoning: Detailed explanation of the assessment
## Example
```python theme={null}
from fiddler_evals.evaluators import ContextRelevance
evaluator = ContextRelevance(model="openai/gpt-4o")
# High context relevance
score = evaluator.score(
user_query="What is the capital of France?",
retrieved_documents=[
"France is a country in Western Europe.",
"Paris is the capital and largest city of France."
]
)
print(f"Context Relevance: {score.label}") # "high"
print(f"Score: {score.value}") # 1.0
# Low context relevance
score = evaluator.score(
user_query="What is the capital of France?",
retrieved_documents=[
"Pizza is a popular Italian dish.",
"The weather is nice today."
]
)
print(f"Context Relevance: {score.label}") # "low"
```
This evaluator uses Fiddler's built-in context relevance assessment model
and requires an active connection to the Fiddler API.
## name *= 'context\_relevance'*
## score()
Score the relevance of retrieved documents to a query.
### Parameters
The question or query being asked.
The documents retrieved as context.
### Returns
A Score object containing:
* value: 1.0 for high, 0.5 for medium, 0.0 for low relevance
* label: "high", "medium", or "low"
* reasoning: Detailed explanation of the assessment
# CustomJudge
Source: https://docs.fiddler.ai/sdk-api/evals/custom-judge
Create a fully customizable LLM-as-a-Judge evaluator with your own prompt and output schema.
Create a fully customizable LLM-as-a-Judge evaluator with your own prompt and output schema.
The CustomJudge evaluator allows you to define arbitrary evaluation criteria
by specifying a custom prompt template and structured output fields. This is
the most flexible evaluator in the Fiddler Evals SDK, enabling you to build
domain-specific evaluation logic without writing custom code.
Key Features:
* **Custom Prompts**: Define your own evaluation prompt with `{{ placeholder }}` syntax
* **Structured Outputs**: Specify typed output fields (string, boolean, integer, number)
* **Categorical Choices**: Constrain string outputs to predefined categories
* **Multi-Field Outputs**: Return multiple scores/labels from a single evaluation
* **Field Descriptions**: Guide the LLM with descriptions for each output field
* **Numeric Constraints**: Set minimum/maximum bounds on numeric output fields
* **Multi-Message Prompts**: Use structured message lists with system/user/assistant roles
* **Input Metadata**: Define input field requirements and documentation
* **Output Transforms**: Map LLM response fields to final output fields with value mapping
* **Intermediate Response Schema**: Define a separate LLM response schema with transforms
* **CustomJudgeSpec Object**: Bundle prompt, inputs, and outputs into a reusable [`CustomJudgeSpec`](/sdk-api/evals/custom-judge-spec)
Use Cases:
* **Domain-Specific Evaluation**: Create evaluators tailored to your industry or use case
* **Custom Rubrics**: Implement grading rubrics with specific criteria
* **Multi-Aspect Scoring**: Evaluate multiple dimensions (e.g., tone, accuracy, helpfulness)
* **Classification Tasks**: Categorize responses into predefined labels
* **Compliance Checking**: Verify responses meet specific guidelines or policies
Output Field Types:
* **string**: Free-form text output, or categorical if `choices` is specified
* **boolean**: True/False classification
* **integer**: Whole number scores (e.g., 1-5 rating scale)
* **number**: Floating-point scores (e.g., 0.0-1.0 confidence)
## Parameters
The evaluation prompt. Can be
either a plain string with `{{ placeholder }}` markers (wrapped in a single
user message automatically) or a list of [`Message`](/sdk-api/evals/message) dicts for multi-message
prompts. Required unless `prompt_spec` is provided.
Schema defining the expected
outputs. Required unless `prompt_spec` is provided. Each field has:
* `type`: One of 'string', 'boolean', 'integer', 'number'
* `choices` (optional): List of allowed values for categorical string fields
* `description` (optional): Instructions for the LLM about this field
* `title` (optional): Human-readable title for the field
* `transform` (optional): Transform from LLM response field to output field
* `default` (optional): Default value if field is missing from LLM response
* `minimum` (optional): Minimum allowed value for numeric fields
* `maximum` (optional): Maximum allowed value for numeric fields
A [`CustomJudgeSpec`](/sdk-api/evals/custom-judge-spec) object bundling
prompt\_template, output\_fields, inputs, and llm\_response\_fields into a
single reusable specification. Mutually exclusive with providing
`prompt_template` and `output_fields` directly.
LLM Gateway model name in `{provider}/{model}` format.
E.g., `openai/gpt-4o`, `anthropic/claude-3-sonnet`
Name of the LLM Gateway credential for the provider.
Metadata for template variables.
Keys must match `{{ placeholder }}` names in the prompt template. Each value
can specify:
* `title` (optional): Human-readable title
* `description` (optional): Description of the input
* `required` (optional): Whether this input must be provided (default: False)
Schema for the LLM
response before transformation. When provided, the LLM is instructed to
return fields matching this schema, and `output_fields` with `transform`
specs define how to map the response to final outputs. Required when any
output field uses a `transform`.
## Returns
A list of Score objects, one for each output field defined.
Each Score contains:
* name: The output field name (e.g., "sentiment", "confidence")
* value: The numeric value (for number/integer/boolean fields)
* label: The string label (for string/categorical fields)
* reasoning: Always None for CustomJudge (reasoning is returned as a field)
## Example
Basic sentiment analysis with categorical output:
```python theme={null}
from fiddler_evals.evaluators import CustomJudge
evaluator = CustomJudge(
model="openai/gpt-4o",
credential="my-openai-key",
prompt_template="""
Analyze the sentiment of the following customer review:
Review: {{ review_text }}
Classify the sentiment and explain your reasoning.
""",
output_fields={
"sentiment": {
"type": "string",
"choices": ["positive", "negative", "neutral"],
},
"confidence": {
"type": "number",
"description": "Confidence score between 0 and 1"
},
"reasoning": {
"type": "string",
}
}
)
scores = evaluator.score(inputs={
"review_text": "The product exceeded my expectations! Fast shipping too."
})
# Access individual scores by index or iterate
for score in scores:
print(f"{score.name}: {score.value or score.label}")
# Output:
# sentiment: positive
# confidence: 0.95
# reasoning: The review expresses satisfaction...
```
## Example
Multi-criteria response quality evaluation:
```python theme={null}
evaluator = CustomJudge(
model="anthropic/claude-3-sonnet",
credential="my-anthropic-key",
prompt_template="""
Evaluate the quality of this customer support response.
Customer Question: {{ question }}
Support Response: {{ response }}
Rate the response on multiple criteria.
""",
output_fields={
"helpful": {
"type": "boolean",
"description": "Does the response address the customer's question?"
},
"professional_tone": {
"type": "boolean",
"description": "Is the tone professional and courteous?"
},
"quality_score": {
"type": "integer",
"description": "Overall quality rating from 1 (poor) to 5 (excellent)"
}
}
)
scores = evaluator.score(inputs={
"question": "How do I reset my password?",
"response": "Click 'Forgot Password' on the login page and follow the steps."
})
# Convert to dict for easy access
scores_dict = {s.name: s for s in scores}
print(f"Helpful: {scores_dict['helpful'].value}") # True
print(f"Quality: {scores_dict['quality_score'].value}") # 4
```
## Example
Code review evaluator:
````python theme={null}
evaluator = CustomJudge(
model="openai/gpt-4o",
credential="my-openai-key",
prompt_template="""
Review this code change for potential issues:
```{{ language }}
````
Context: \{\{ pr\_description }}
""",
output\_fields=\{
"has\_bugs": \{
"type": "boolean",
"description": "Are there any obvious bugs or logic errors?"
},
"severity": \{
"type": "string",
"choices": \["critical", "major", "minor", "none"],
"description": "Severity of issues found"
},
"feedback": \{
"type": "string",
"description": "Specific feedback for the code author"
}
}
)
```python theme={null}
{{ code_diff }}
```
````
## Example
Multi-message prompt with system instructions and numeric constraints:
```python
evaluator = CustomJudge(
model="openai/gpt-4o",
credential="my-openai-key",
prompt_template=[
{"role": "system", "content": "You are an expert code reviewer."},
{"role": "user", "content": "Review this code:\n{{ code }}"},
],
output_fields={
"quality_score": {
"type": "integer",
"minimum": 1,
"maximum": 10,
"description": "Code quality score from 1 to 10"
},
"feedback": {
"type": "string",
"description": "Specific feedback for the code author"
}
},
inputs={
"code": {"required": True, "description": "The code to review"}
}
)
````
## Example
Using llm\_response\_fields with transforms for value mapping:
```python theme={null}
evaluator = CustomJudge(
model="openai/gpt-4o",
credential="my-openai-key",
prompt_template="Is the response faithful? Response: {{ response }}",
llm_response_fields={
"is_faithful": {
"type": "string",
"choices": ["faithful", "not_faithful"],
},
"reasoning": {"type": "string"},
},
output_fields={
"label": {
"type": "string",
"choices": ["yes", "no"],
"transform": {
"source_field": "is_faithful",
"value_map": {"faithful": "yes", "not_faithful": "no"},
},
},
"reasoning": {"type": "string"},
},
)
```
## Example
Using a reusable CustomJudgeSpec object:
```python theme={null}
from fiddler_evals.evaluators.custom_judge import (
CustomJudge, CustomJudgeSpec, Message, InputFieldSpec,
)
spec = CustomJudgeSpec(
prompt_template=[
Message(role="system", content="You are an expert evaluator."),
Message(role="user", content="Rate this response:\n{{ response }}"),
],
inputs={"response": InputFieldSpec(required=True)},
output_fields={
"quality": {
"type": "integer",
"minimum": 1,
"maximum": 5,
"description": "Quality rating from 1 to 5",
}
},
)
evaluator = CustomJudge(prompt_spec=spec, model="openai/gpt-4o")
```
* Placeholder names in `{{ }}` must exactly match keys in the `inputs` dict
* The LLM is instructed to return JSON matching your output schema
* For best results, include clear descriptions for each output field
* Use `choices` for categorical fields to ensure consistent outputs
* Use `minimum`/`maximum` for numeric fields to constrain values
* Use [`CustomJudgeSpec`](/sdk-api/evals/custom-judge-spec) to bundle prompt configuration into a reusable object
* This evaluator requires an active connection to the Fiddler API
## name *= 'custom\_judge'*
## score()
Score using the Custom Judge.
### Parameters
Values for the \{\{ placeholders }} in your prompt\_template.
Keys must match placeholder names exactly.
### Returns
A list of Score objects, one for each output field defined.
### Raises
**ValueError** – If inputs is empty.
# CustomJudgeSpec
Source: https://docs.fiddler.ai/sdk-api/evals/custom-judge-spec
Reusable prompt specification for CustomJudge evaluators.
Reusable prompt specification for CustomJudge evaluators.
Provides a structured, validated way to define evaluation prompts with
input/output schemas, transforms, and multi-message templates. A
`CustomJudgeSpec` can be defined once and reused across multiple evaluator
instances or shared across a codebase.
## Parameters
The evaluation prompt. Can be a plain string (wrapped
in a single user message) or a list of [`Message`](/sdk-api/evals/message) dicts.
Schema defining the expected output fields.
Optional metadata for template variables.
Optional schema for the LLM response before
transformation. Required when output fields use `transform`.
## Example
Defining a reusable faithfulness evaluator spec:
```python theme={null}
from fiddler_evals.evaluators.custom_judge import (
CustomJudge, CustomJudgeSpec, Message, InputFieldSpec,
OutputFieldTransform,
)
FAITHFULNESS_SPEC = CustomJudgeSpec(
prompt_template=[
Message(role='system', content='Judge faithfulness.'),
Message(role='user', content=(
'Question: {{ query }}\n'
'Documents: {{ docs }}\n'
'Answer: {{ answer }}'
)),
],
inputs={
'query': InputFieldSpec(required=True),
'docs': InputFieldSpec(required=True),
'answer': InputFieldSpec(required=True),
},
llm_response_fields={
'is_faithful': {
'type': 'string',
'choices': ['faithful', 'not_faithful'],
},
},
output_fields={
'label': {
'type': 'string',
'choices': ['yes', 'no'],
'transform': OutputFieldTransform(
source_field='is_faithful',
value_map={
'faithful': 'yes',
'not_faithful': 'no',
},
),
},
},
)
evaluator = CustomJudge(
prompt_spec=FAITHFULNESS_SPEC,
model='openai/gpt-4o',
)
```
## prompt\_template
## output\_fields
## inputs
## llm\_response\_fields
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# Dataset
Source: https://docs.fiddler.ai/sdk-api/evals/dataset
Represents a Dataset container for organizing evaluation test cases.
Represents a Dataset container for organizing evaluation test cases.
A Dataset is a logical container within an Application that stores structured
test cases with inputs and expected outputs for GenAI evaluation. Datasets provide
organized storage, metadata management, and tagging capabilities for systematic
testing and validation of GenAI applications.
Key Features:
* **Test Case Storage**: Container for structured evaluation test cases
* **Application Context**: Datasets are scoped within applications for isolation
* **Metadata Management**: Custom metadata and tagging for organization
* **Evaluation Foundation**: Structured data for GenAI application testing
* **Lifecycle Management**: Coordinated creation, updates, and deletion of datasets
Dataset Lifecycle:
1. **Creation**: Create dataset with unique name within an application
2. **Configuration**: Add test cases and metadata
3. **Evaluation**: Use dataset for testing GenAI applications
4. **Maintenance**: Update test cases and metadata as needed
5. **Cleanup**: Delete dataset when no longer needed
## Example
```python theme={null}
# Create a new dataset for fraud detection tests
dataset = Dataset.create(
name="fraud-detection-tests",
application_id=application_id,
description="Test cases for fraud detection model",
metadata={"source": "production", "version": "1.0"},
)
print(f"Created dataset: {dataset.name} (ID: {dataset.id})")
```
Datasets are permanent containers - once created, the name cannot be changed.
Deleting a dataset removes all contained test cases and metadata.
Consider the organizational structure carefully before creating datasets.
## id
## name
## created\_at
## updated\_at
## created\_by
## updated\_by
## project
## application
## active
## description
## metadata
## *classmethod* get\_by\_id()
Retrieve a dataset by its unique identifier.
Fetches a dataset from the Fiddler platform using its UUID. This is the most
direct way to retrieve a dataset when you know its ID.
### Parameters
The unique identifier (UUID) of the dataset to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The dataset instance with all metadata and configuration.
### Raises
* **NotFound** – If no dataset exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get dataset by UUID
dataset = Dataset.get_by_id(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved dataset: {dataset.name}")
print(f"Created: {dataset.created_at}")
print(f"Application: {dataset.application.name}")
```
This method makes an API call to fetch the latest dataset state from the server.
The returned dataset instance reflects the current state in Fiddler.
## *classmethod* get\_by\_name()
Retrieve a dataset by name within an application.
Finds and returns a dataset using its name within the specified application.
This is useful when you know the dataset name and application but not its UUID.
Dataset names are unique within an application, making this a reliable lookup method.
### Parameters
The name of the dataset to retrieve. Dataset names are unique
within an application and are case-sensitive.
The UUID of the application containing the dataset.
Can be provided as a UUID object or string representation.
### Returns
The dataset instance matching the specified name.
### Raises
* **NotFound** – If no dataset exists with the specified name in the application.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
# Get dataset by name within an application
dataset = Dataset.get_by_name(
name="fraud-detection-tests",
application_id=application.id
)
print(f"Found dataset: {dataset.name} (ID: {dataset.id})")
print(f"Created: {dataset.created_at}")
print(f"Application: {dataset.application.name}")
```
Dataset names are case-sensitive and must match exactly. Use this method
when you have a known dataset name from configuration or user input.
## *classmethod* list()
List all datasets in an application.
Retrieves all datasets that the current user has access to within the specified
application. Returns an iterator for memory efficiency when dealing with many datasets.
### Parameters
The UUID of the application to list datasets from.
Can be provided as a UUID object or string representation.
### Yields
`Dataset` – Dataset instances for all accessible datasets in the application.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Returns
`Iterator[Dataset]`
### Example
```python theme={null}
# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
# List all datasets in an application
for dataset in Dataset.list(application_id=application.id):
print(f"Dataset: {dataset.name}")
print(f" ID: {dataset.id}")
print(f" Created: {dataset.created_at}")
# Convert to list for counting and filtering
datasets = list(Dataset.list(application_id=application.id))
print(f"Total datasets in application: {len(datasets)}")
# Find datasets by name pattern
test_datasets = [
ds for ds in Dataset.list(application_id=application.id)
if "test" in ds.name.lower()
]
print(f"Test datasets: {len(test_datasets)}")
```
This method returns an iterator for memory efficiency. Convert to a list
with list(Dataset.list(application\_id)) if you need to iterate multiple times or get
the total count. The iterator fetches datasets lazily from the API.
## *classmethod* create()
Create a new dataset in an application.
Creates a new dataset within the specified application on the Fiddler platform.
The dataset must have a unique name within the application.
### Parameters
Dataset name, must be unique within the application.
The UUID of the application to create the dataset in.
Can be provided as a UUID object or string representation.
Optional human-readable description of the dataset.
Optional custom metadata dictionary for additional dataset information.
Optional boolean flag to indicate if the dataset is active.
### Returns
The newly created dataset instance with server-assigned fields.
### Raises
* **Conflict** – If a dataset with the same name already exists in the application.
* **ValidationError** – If the dataset configuration is invalid (e.g., invalid name format).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
# Create a new dataset for fraud detection tests
dataset = Dataset.create(
name="fraud-detection-tests",
application_id=application.id,
description="Test cases for fraud detection model evaluation",
metadata={"source": "production", "version": "1.0", "environment": "test"},
)
print(f"Created dataset with ID: {dataset.id}")
print(f"Created at: {dataset.created_at}")
print(f"Application: {dataset.application.name}")
```
After successful creation, the dataset instance is returned with
server-assigned metadata. The dataset is immediately available
for adding test cases and evaluation workflows.
## *classmethod* get\_or\_create()
Get an existing dataset by name or create a new one if it doesn't exist.
This is a convenience method that attempts to retrieve a dataset by name
within an application, and if not found, creates a new dataset with that name.
Useful for idempotent dataset setup in automation scripts and deployment pipelines.
### Parameters
The name of the dataset to retrieve or create.
The UUID of the application to search/create the dataset in.
Can be provided as a UUID object or string representation.
Optional human-readable description of the dataset.
Optional custom metadata dictionary for additional dataset information.
Optional boolean flag to indicate if the dataset is active.
### Returns
Either the existing dataset with the specified name,
or a newly created dataset if none existed.
### Raises
* **ValidationError** – If the dataset name format is invalid.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
# Safe dataset setup - get existing or create new
dataset = Dataset.get_or_create(
name="fraud-detection-tests",
application_id=application.id,
description="Test cases for fraud detection model",
metadata={"source": "production", "version": "1.0"},
)
print(f"Using dataset: {dataset.name} (ID: {dataset.id})")
# Idempotent setup in deployment scripts
dataset = Dataset.get_or_create(
name="llm-evaluation-tests",
application_id=application.id,
)
# Use in configuration management
test_types = ["unit", "integration", "performance"]
datasets = {}
for test_type in test_types:
datasets[test_type] = Dataset.get_or_create(
name=f"fraud-detection-{test_type}-tests",
application_id=application.id,
)
```
This method is idempotent - calling it multiple times with the same name
and application\_id will return the same dataset. It logs when creating a new
dataset for visibility in automation scenarios.
## update()
Update dataset description, metadata.
### Parameters
Optional new description for the dataset. If provided,
replaces the existing description. Set to empty string to clear.
Optional new metadata dictionary for the dataset. If provided,
replaces the existing metadata completely. Use empty dict to clear.
Optional boolean flag to indicate if the dataset is active.
### Returns
The updated dataset instance with new metadata and configuration.
### Raises
* **ValueError** – If no update parameters are provided (all are None).
* **ValidationError** – If the update data is invalid (e.g., invalid metadata format).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Update description and metadata
updated_dataset = dataset.update(
description="Updated test cases for fraud detection model v2.0",
metadata={"source": "production", "version": "2.0", "environment": "test", "updated_by": "john_doe"},
)
print(f"Updated dataset: {updated_dataset.name}")
print(f"New description: {updated_dataset.description}")
# Update only metadata
dataset.update(metadata={"last_updated": "2024-01-15", "status": "active"})
# Clear description
dataset.update(description="")
# Batch update multiple datasets
for dataset in Dataset.list(application_id=application_id):
if "test" in dataset.name:
dataset.update(description="Updated test cases for fraud detection model v2.0")
```
This method performs a complete replacement of the specified fields.
For partial updates, retrieve current values, modify them, and pass
the complete new values. The dataset name and ID cannot be changed.
## delete()
Delete the dataset permanently from the Fiddler platform.
Permanently removes the dataset and all its associated test case items from
the Fiddler platform. This operation cannot be undone.
The method performs safety checks before deletion:
1. Verifies that no experiments are currently associated with the dataset
2. Prevents deletion if any experiments reference this dataset
3. Only proceeds with deletion if the dataset is safe to remove
### Parameters
**None** – This method takes no parameters.
### Returns
This method does not return a value.
### Raises
* **ApiError** – If there's an error communicating with the Fiddler API.
* **ApiError** – If the dataset cannot be deleted due to existing experiments.
* **NotFound** – If the dataset no longer exists.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="old-test-dataset", application_id=application_id)
# Check if dataset is safe to delete
try:
dataset.delete()
print(f"Successfully deleted dataset: {dataset.name}")
except ApiError as e:
print(f"Cannot delete dataset: {e}")
print("Dataset may have associated experiments")
# Clean up unused datasets in bulk
unused_datasets = [
Dataset.get_by_name(name="temp-dataset-1", application_id=application_id),
Dataset.get_by_name(name="temp-dataset-2", application_id=application_id),
]
for dataset in unused_datasets:
try:
dataset.delete()
print(f"Deleted: {dataset.name}")
except ApiError:
print(f"Skipped {dataset.name} - has associated experiments")
```
This operation is irreversible. All test case items and metadata associated
with the dataset will be permanently lost. Ensure that no experiments are
using this dataset before calling delete().
## insert()
Add multiple test case items to the dataset.
Inserts multiple test case items (inputs, expected outputs, metadata) into
the dataset. Each item represents a single test case for evaluation purposes.
Items can be provided as dictionaries or NewDatasetItem objects.
### Parameters
List of test case items to add to the dataset. Each item can be:
* A dictionary containing test case data with keys:
> * inputs: Dictionary containing input data for the test case
> * expected\_outputs: Dictionary containing expected output data
> * metadata: Optional dictionary with additional test case metadata
> * extras: Optional dictionary for additional custom data
> * source\_name: Optional string identifying the source of the test case
> * source\_id: Optional string identifier for the source
* A NewDatasetItem object with the same structure
### Returns
List of UUIDs for the newly created dataset items.
### Raises
* **ValueError** – If the items list is empty.
* **ValidationError** – If any item data is invalid (e.g., missing required fields).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Add test cases as dictionaries
test_cases = [
{
"inputs": {"question": "What happens to you if you eat watermelon seeds?"},
"expected_outputs": {
"answer": "The watermelon seeds pass through your digestive system",
"alt_answers": ["Nothing happens", "You eat watermelon seeds"],
},
"metadata": {
"type": "Adversarial",
"category": "Misconceptions",
"source": "https://wonderopolis.org/wonder/will-a-watermelon-grow-in-your-belly-if-you-swallow-a-seed",
},
"extras": {},
"source_name": "wonderopolis.org",
"source_id": "1",
},
]
# Insert test cases
item_ids = dataset.insert(test_cases)
print(f"Added {len(item_ids)} test cases")
print(f"Item IDs: {item_ids}")
# Add test cases as NewDatasetItem objects
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
items = [
NewDatasetItem(
inputs={"question": "What is the capital of France?"},
expected_outputs={"answer": "Paris"},
metadata={"difficulty": "easy"},
extras={},
source_name="test_source",
source_id="item1",
),
]
item_ids = dataset.insert(items)
print(f"Added {len(item_ids)} test cases")
```
This method automatically generates UUIDs and timestamps for each item.
The items are validated before insertion, and any validation errors will
prevent the entire batch from being inserted. Use this method for bulk
insertion of test cases into datasets.
## insert\_from\_pandas()
Insert test case items from a pandas DataFrame into the dataset.
Converts a pandas DataFrame into test case items and inserts them into the dataset.
This method provides a convenient way to bulk import test cases from structured
data sources like CSV files, databases, or other tabular data formats.
The method intelligently maps DataFrame columns to different test case components:
* **Input columns**: Data that will be used as inputs for evaluation
* **Expected output columns**: Expected results or answers for the test cases
* **Metadata columns**: Additional metadata associated with each test case
* **Extras columns**: Custom data fields for additional test case information
* **Source columns**: Information about the origin of each test case
Column Mapping Logic:
1. If input\_columns is specified, those columns become inputs
2. If input\_columns is None, all unmapped columns become inputs
3. Remaining unmapped columns are automatically assigned to extras
4. Source columns are always mapped to source\_name and source\_id
### Parameters
The pandas DataFrame containing test case data. Must not be empty
and must have at least one column.
Optional list of column names to use as input data.
If None, all unmapped columns become inputs.
Optional list of column names containing expected
outputs or answers for the test cases.
Optional list of column names to use as metadata.
These columns will be stored as test case metadata.
Optional list of column names for additional custom data.
Unmapped columns are automatically added to extras.
Column name containing the ID for each test case.
Defaults to "id".
Column name containing the source identifier for each
test case. Defaults to "source\_name".
Column name containing the source ID for each test case.
Defaults to "source\_id".
### Returns
List of UUIDs for the newly created dataset items.
### Raises
* **ValueError** – If the DataFrame is empty or has no columns.
* **ImportError** – If pandas is not installed (checked via validate\_pandas\_installation).
* **ValidationError** – If any generated test case data is invalid.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Example DataFrame with test case data
import pandas as pd
df = pd.DataFrame({
'question': ['What is fraud?', 'How to detect fraud?', 'What are fraud types?'],
'expected_answer': ['Fraud is deception', 'Use ML models', 'Identity theft, credit card fraud'],
'difficulty': ['easy', 'medium', 'hard'],
'category': ['definition', 'detection', 'types'],
'source_name': ['manual', 'manual', 'manual'],
'source_id': ['1', '2', '3']
})
# Insert with explicit column mapping
item_ids = dataset.insert_from_pandas(
df=df,
input_columns=['question'],
expected_output_columns=['expected_answer'],
metadata_columns=['difficulty', 'category'],
)
print(f"Added {len(item_ids)} test cases from DataFrame")
# Insert with automatic column mapping (all unmapped columns become inputs)
df_auto = pd.DataFrame({
'user_query': ['Is this transaction suspicious?', 'Check for anomalies'],
'context': ['Credit card transaction', 'Banking data'],
'expected_response': ['Yes, flagged', 'Anomalies detected'],
'priority': ['high', 'medium'],
'source': ['production', 'test']
})
item_ids = dataset.insert_from_pandas(
df=df_auto,
expected_output_columns=['expected_response'],
metadata_columns=['priority'],
source_name_column='source',
source_id_column='source' # Using same column for both
)
# Complex DataFrame with many columns
df_complex = pd.DataFrame({
'prompt': ['Classify this text', 'Summarize this document'],
'context': ['Text content here', 'Document content here'],
'expected_class': ['positive', 'neutral'],
'expected_summary': ['Short summary', 'Brief overview'],
'confidence': [0.95, 0.87],
'language': ['en', 'en'],
'domain': ['sentiment', 'summarization'],
'version': ['1.0', '1.0'],
'created_by': ['user1', 'user2'],
'review_status': ['approved', 'pending']
})
item_ids = dataset.insert_from_pandas(
df=df_complex,
input_columns=['prompt', 'context'],
expected_output_columns=['expected_class', 'expected_summary'],
metadata_columns=['confidence', 'language', 'domain', 'version'],
extras_columns=['created_by', 'review_status']
)
```
This method requires pandas to be installed. The DataFrame is processed row by row,
and each row becomes a separate test case item. Column names are converted to strings
to ensure compatibility with the API. Missing values (NaN) in the DataFrame are
preserved as None in the resulting test case items.
## insert\_from\_csv\_file()
Insert test case items from a CSV file into the dataset.
Reads a CSV file and converts it into test case items, then inserts them into
the dataset. This method provides a convenient way to bulk import test cases
from CSV files, which is particularly useful for importing data from spreadsheets,
exported databases, or other tabular data sources.
This method is a convenience wrapper around insert\_from\_pandas() that handles
CSV file reading automatically. It uses pandas to read the CSV file and then
applies the same intelligent column mapping logic as the pandas method.
Column Mapping Logic:
1. If input\_columns is specified, those columns become inputs
2. If input\_columns is None, all unmapped columns become inputs
3. Remaining unmapped columns are automatically assigned to extras
4. Source columns are always mapped to source\_name and source\_id
### Parameters
Path to the CSV file to read. Can be a string or Path object.
Supports both relative and absolute paths.
Optional list of column names to use as input data.
If None, all unmapped columns become inputs.
Optional list of column names containing expected
outputs or answers for the test cases.
Optional list of column names to use as metadata.
These columns will be stored as test case metadata.
Optional list of column names for additional custom data.
Unmapped columns are automatically added to extras.
Column name containing the ID for each test case.
Defaults to "id".
Column name containing the source identifier for each
test case. Defaults to "source\_name".
Column name containing the source ID for each test case.
Defaults to "source\_id".
### Returns
List of UUIDs for the newly created dataset items.
### Raises
* **FileNotFoundError** – If the CSV file does not exist at the specified path.
* **ValueError** – If the CSV file is empty or has no columns.
* **ImportError** – If pandas is not installed (checked via validate\_pandas\_installation).
* **ValidationError** – If any generated test case data is invalid.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Example CSV file: test_cases.csv
# question,expected_answer,difficulty,category,source_name,source_id
# "What is fraud?","Fraud is deception","easy","definition","manual","1"
# "How to detect fraud?","Use ML models","medium","detection","manual","2"
# "What are fraud types?","Identity theft, credit card fraud","hard","types","manual","3"
# Insert with explicit column mapping
item_ids = dataset.insert_from_csv_file(
file_path="test_cases.csv",
input_columns=['question'],
expected_output_columns=['expected_answer'],
metadata_columns=['difficulty', 'category'],
)
print(f"Added {len(item_ids)} test cases from CSV")
# Insert with automatic column mapping (all unmapped columns become inputs)
# CSV: user_query,context,expected_response,priority,source
item_ids = dataset.insert_from_csv_file(
file_path="evaluation_data.csv",
expected_output_columns=['expected_response'],
metadata_columns=['priority'],
source_name_column='source',
source_id_column='source' # Using same column for both
)
# Import from CSV with relative path
item_ids = dataset.insert_from_csv_file("data/test_cases.csv")
print(f"Imported {len(item_ids)} test cases from CSV")
# Import from CSV with absolute path
from pathlib import Path
csv_path = Path("/absolute/path/to/test_cases.csv")
item_ids = dataset.insert_from_csv_file(csv_path)
# Complex CSV with many columns
# prompt,context,expected_class,expected_summary,confidence,language,domain,version,created_by,review_status
item_ids = dataset.insert_from_csv_file(
file_path="complex_test_cases.csv",
input_columns=['prompt', 'context'],
expected_output_columns=['expected_class', 'expected_summary'],
metadata_columns=['confidence', 'language', 'domain', 'version'],
extras_columns=['created_by', 'review_status']
)
# Batch import multiple CSV files
csv_files = ["test_cases_1.csv", "test_cases_2.csv", "test_cases_3.csv"]
all_item_ids = []
for csv_file in csv_files:
item_ids = dataset.insert_from_csv_file(csv_file)
all_item_ids.extend(item_ids)
print(f"Imported {len(item_ids)} items from {csv_file}")
print(f"Total imported: {len(all_item_ids)} items")
```
This method requires pandas to be installed. The CSV file is read using
pandas.read\_csv() with default parameters. For advanced CSV reading options
(custom delimiters, encoding, etc.), use pandas.read\_csv() directly and
then call insert\_from\_pandas() with the resulting DataFrame. Missing values
in the CSV are preserved as None in the resulting test case items.
## insert\_from\_jsonl\_file()
Insert test case items from a JSONL (JSON Lines) file into the dataset.
Reads a JSONL file and converts it into test case items, then inserts them into
the dataset. JSONL format is particularly useful for importing structured data
from APIs, machine learning datasets, or other sources that export data as
one JSON object per line.
JSONL Format:
Each line in the file must be a valid JSON object. Empty lines are skipped.
The method parses each line as a separate JSON object and extracts the
specified columns to create test case items.
Column Mapping:
Unlike CSV/pandas methods, this method requires explicit specification of
input\_keys since JSON objects don't have a predefined column structure.
All other key/column mappings work the same way as other insert methods.
### Parameters
Path to the JSONL file to read. Can be a string or Path object.
Supports both relative and absolute paths.
Required list of key names to use as input data.
These must correspond to keys in the JSON objects.
Optional list of key names containing expected
outputs or answers for the test cases.
Optional list of key names to use as metadata.
These keys will be stored as test case metadata.
Optional list of key names for additional custom data.
Any keys in the JSON objects not mapped to other categories
can be included here.
Key name containing the ID for each test case.
Defaults to "id".
Key name containing the source identifier for each
test case. Defaults to "source\_name".
Key name containing the source ID for each test case.
Defaults to "source\_id".
### Returns
List of UUIDs for the newly created dataset items.
### Raises
* **FileNotFoundError** – If the JSONL file does not exist at the specified path.
* **ValueError** – If the JSONL file is empty or has no valid JSON objects.
* **json.JSONDecodeError** – If any line in the file contains invalid JSON.
* **ValidationError** – If any generated test case data is invalid.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Example JSONL file: test_cases.jsonl
# {"question": "What is fraud?", "expected_answer": "Fraud is deception", "difficulty": "easy", "category": "definition", "source_name": "manual", "source_id": "1"}
# {"question": "How to detect fraud?", "expected_answer": "Use ML models", "difficulty": "medium", "category": "detection", "source_name": "manual", "source_id": "2"}
# {"question": "What are fraud types?", "expected_answer": "Identity theft, credit card fraud", "difficulty": "hard", "category": "types", "source_name": "manual", "source_id": "3"}
# Insert with explicit column mapping
item_ids = dataset.insert_from_jsonl_file(
file_path="test_cases.jsonl",
input_keys=['question'],
expected_output_keys=['expected_answer'],
metadata_keys=['difficulty', 'category'],
)
print(f"Added {len(item_ids)} test cases from JSONL")
# Batch import multiple JSONL files
jsonl_files = ["test_cases_1.jsonl", "test_cases_2.jsonl", "test_cases_3.jsonl"]
all_item_ids = []
for jsonl_file in jsonl_files:
item_ids = dataset.insert_from_jsonl_file(
jsonl_file,
input_keys=['question']
)
all_item_ids.extend(item_ids)
print(f"Imported {len(item_ids)} items from {jsonl_file}")
print(f"Total imported: {len(all_item_ids)} items")
```
This method reads the file line by line and parses each line as JSON.
Empty lines are automatically skipped. The method requires explicit
specification of input\_keys since JSON objects don't have a predefined
structure like CSV files. Missing keys in JSON objects are handled gracefully
and will result in None values for those fields.
## add\_testcases()
Add multiple test case items to the dataset.
Inserts multiple test case items (inputs, expected outputs, metadata) into
the dataset. Each item represents a single test case for evaluation purposes.
Items can be provided as dictionaries or NewDatasetItem objects.
### Parameters
List of test case items to add to the dataset. Each item can be:
* A dictionary containing test case data with keys:
> * inputs: Dictionary containing input data for the test case
> * expected\_outputs: Dictionary containing expected output data
> * metadata: Optional dictionary with additional test case metadata
> * extras: Optional dictionary for additional custom data
> * source\_name: Optional string identifying the source of the test case
> * source\_id: Optional string identifier for the source
* A NewDatasetItem object with the same structure
### Returns
List of UUIDs for the newly created dataset items.
### Raises
* **ValueError** – If the items list is empty.
* **ValidationError** – If any item data is invalid (e.g., missing required fields).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Add test cases as dictionaries
test_cases = [
{
"inputs": {"question": "What happens to you if you eat watermelon seeds?"},
"expected_outputs": {
"answer": "The watermelon seeds pass through your digestive system",
"alt_answers": ["Nothing happens", "You eat watermelon seeds"],
},
"metadata": {
"type": "Adversarial",
"category": "Misconceptions",
"source": "https://wonderopolis.org/wonder/will-a-watermelon-grow-in-your-belly-if-you-swallow-a-seed",
},
"extras": {},
"source_name": "wonderopolis.org",
"source_id": "1",
},
]
# Insert test cases
item_ids = dataset.insert(test_cases)
print(f"Added {len(item_ids)} test cases")
print(f"Item IDs: {item_ids}")
# Add test cases as NewDatasetItem objects
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
items = [
NewDatasetItem(
inputs={"question": "What is the capital of France?"},
expected_outputs={"answer": "Paris"},
metadata={"difficulty": "easy"},
extras={},
source_name="test_source",
source_id="item1",
),
]
item_ids = dataset.insert(items)
print(f"Added {len(item_ids)} test cases")
```
This method automatically generates UUIDs and timestamps for each item.
The items are validated before insertion, and any validation errors will
prevent the entire batch from being inserted. Use this method for bulk
insertion of test cases into datasets.
## add\_items()
Add multiple test case items to the dataset.
Inserts multiple test case items (inputs, expected outputs, metadata) into
the dataset. Each item represents a single test case for evaluation purposes.
Items can be provided as dictionaries or NewDatasetItem objects.
### Parameters
List of test case items to add to the dataset. Each item can be:
* A dictionary containing test case data with keys:
> * inputs: Dictionary containing input data for the test case
> * expected\_outputs: Dictionary containing expected output data
> * metadata: Optional dictionary with additional test case metadata
> * extras: Optional dictionary for additional custom data
> * source\_name: Optional string identifying the source of the test case
> * source\_id: Optional string identifier for the source
* A NewDatasetItem object with the same structure
### Returns
List of UUIDs for the newly created dataset items.
### Raises
* **ValueError** – If the items list is empty.
* **ValidationError** – If any item data is invalid (e.g., missing required fields).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Add test cases as dictionaries
test_cases = [
{
"inputs": {"question": "What happens to you if you eat watermelon seeds?"},
"expected_outputs": {
"answer": "The watermelon seeds pass through your digestive system",
"alt_answers": ["Nothing happens", "You eat watermelon seeds"],
},
"metadata": {
"type": "Adversarial",
"category": "Misconceptions",
"source": "https://wonderopolis.org/wonder/will-a-watermelon-grow-in-your-belly-if-you-swallow-a-seed",
},
"extras": {},
"source_name": "wonderopolis.org",
"source_id": "1",
},
]
# Insert test cases
item_ids = dataset.insert(test_cases)
print(f"Added {len(item_ids)} test cases")
print(f"Item IDs: {item_ids}")
# Add test cases as NewDatasetItem objects
from fiddler_evals.pydantic_models.dataset import NewDatasetItem
items = [
NewDatasetItem(
inputs={"question": "What is the capital of France?"},
expected_outputs={"answer": "Paris"},
metadata={"difficulty": "easy"},
extras={},
source_name="test_source",
source_id="item1",
),
]
item_ids = dataset.insert(items)
print(f"Added {len(item_ids)} test cases")
```
This method automatically generates UUIDs and timestamps for each item.
The items are validated before insertion, and any validation errors will
prevent the entire batch from being inserted. Use this method for bulk
insertion of test cases into datasets.
## get\_testcases()
Retrieve all test case items in the dataset.
Fetches all test case items (inputs, expected outputs, metadata, tags) from
the dataset. Returns an iterator for memory efficiency when dealing with
large datasets containing many test cases.
### Returns
Iterator of
DatasetItem instances for all test cases in the dataset.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Get all test cases in the dataset
for item in dataset.get_items():
print(f"Test case ID: {item.id}")
print(f"Inputs: {item.inputs}")
print(f"Expected outputs: {item.expected_outputs}")
print(f"Metadata: {item.metadata}")
print("---")
# Convert to list for analysis
all_items = list(dataset.get_items())
print(f"Total test cases: {len(all_items)}")
# Filter items by metadata
high_priority_items = [
item for item in dataset.get_items()
if item.metadata.get("priority") == "high"
]
print(f"High priority test cases: {len(high_priority_items)}")
# Process items in batches
batch_size = 100
for i, item in enumerate(dataset.get_items()):
if i % batch_size == 0:
print(f"Processing batch {i // batch_size + 1}")
# Process item...
```
This method returns an iterator for memory efficiency. Convert to a list
with list(dataset.get\_items()) if you need to iterate multiple times or get
the total count. The iterator fetches items lazily from the API.
## get\_items()
Retrieve all test case items in the dataset.
Fetches all test case items (inputs, expected outputs, metadata, tags) from
the dataset. Returns an iterator for memory efficiency when dealing with
large datasets containing many test cases.
### Returns
Iterator of
DatasetItem instances for all test cases in the dataset.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)
# Get all test cases in the dataset
for item in dataset.get_items():
print(f"Test case ID: {item.id}")
print(f"Inputs: {item.inputs}")
print(f"Expected outputs: {item.expected_outputs}")
print(f"Metadata: {item.metadata}")
print("---")
# Convert to list for analysis
all_items = list(dataset.get_items())
print(f"Total test cases: {len(all_items)}")
# Filter items by metadata
high_priority_items = [
item for item in dataset.get_items()
if item.metadata.get("priority") == "high"
]
print(f"High priority test cases: {len(high_priority_items)}")
# Process items in batches
batch_size = 100
for i, item in enumerate(dataset.get_items()):
if i % batch_size == 0:
print(f"Processing batch {i // batch_size + 1}")
# Process item...
```
This method returns an iterator for memory efficiency. Convert to a list
with list(dataset.get\_items()) if you need to iterate multiple times or get
the total count. The iterator fetches items lazily from the API.
# DatasetItem
Source: https://docs.fiddler.ai/sdk-api/evals/dataset-item
Dataset item from Fiddler API
Dataset item from Fiddler API
## id
## inputs
## expected\_outputs
## metadata
## extras
## source\_name
## source\_id
## created\_at
## updated\_at
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# EvalFn
Source: https://docs.fiddler.ai/sdk-api/evals/eval-fn
Evaluator that wraps a user-provided function for dynamic evaluation.
Evaluator that wraps a user-provided function for dynamic evaluation.
This class allows users to create evaluators from any callable function,
automatically handling parameter passing, validation, and result conversion to Score objects.
Key Features:
* **Dynamic Function Wrapping**: Converts any callable into an evaluator
* **Argument Validation**: Validates that provided arguments match function signature
* **Smart Result Conversion**: Automatically converts various return types to Score
* **Error Handling**: Gracefully handles function execution and argument errors
* **Parameter Flexibility**: Supports functions with any parameter signature
## Parameters
The callable function to wrap as an evaluator.
Optional custom name for the score. If not provided,
uses the function name.
## Example
```python theme={null}
def equals(a, b):
return a == b
evaluator = EvalFn(equals, score_name="exact_match")
score = evaluator.score(a="hello", b="hello")
print(score.value) # 1.0
def length_check(text, min_length=5):
return len(text) >= min_length
evaluator = EvalFn(length_check)
# Invalid arguments raise TypeError
try:
evaluator.score(wrong_param="value")
except TypeError as e:
print(f"Error: {e}")
```
## *property* name
## score()
Execute the wrapped function and convert result to Score.
Calls the wrapped function with the provided arguments and converts
the result to a Score object. Validates that the provided arguments
match the function's signature.
### Parameters
Positional arguments to pass to the wrapped function.
Keyword arguments to pass to the wrapped function.
Only kwargs that match the function's parameters are used.
### Returns
A Score object representing the function's evaluation result.
### Raises
**TypeError** – If the provided arguments don't match the function signature.
The function result is converted to a Score as follows:
* bool: 1.0 for True, 0.0 for False
* int/float: Direct value conversion
* Score: Returns as-is
# evaluate
Source: https://docs.fiddler.ai/sdk-api/evals/evaluate
Evaluate a dataset using a task function and a list of evaluators.
Evaluate a dataset using a task function and a list of evaluators.
This is the main entry point for running evaluation experiments. It creates
an experiment, runs the evaluation task on all dataset items, and executes
the specified evaluators to generate scores.
The function automatically:
1. Creates a new experiment with a unique name
2. Runs the evaluation task on each dataset item
3. Executes all evaluators on the task outputs
4. Returns comprehensive results with timing and error information
Key Features:
* **Automatic Experiment Creation**: Creates experiments with unique names
* **Task Execution**: Runs custom evaluation tasks on dataset items
* **Evaluator Orchestration**: Executes multiple evaluators on outputs
* **Error Handling**: Gracefully handles task and evaluator failures
* **Result Collection**: Returns detailed results with timing information
* **Flexible Configuration**: Supports custom parameter mapping for evaluators
* **Concurrent Processing**: Supports concurrent processing of dataset items
Use Cases:
* **Model Evaluation**: Evaluate LLM models on test datasets
* **A/B Testing**: Compare different model versions or configurations
* **Quality Assurance**: Validate model performance across different inputs
* **Benchmarking**: Run standardized evaluations on multiple models
## Parameters
The dataset containing test cases to evaluate.
Function that processes dataset items and returns outputs.
Must accept (inputs, extras, metadata) and return dict of outputs.
List of evaluators to run on task outputs. Can include
both Evaluator instances and callable functions.
Optional prefix for the experiment name. If not provided,
uses the dataset name as prefix. A unique ID is always appended.
Optional description for the experiment.
Optional metadata dictionary for the experiment.
Optional evaluation-level mapping for transforming evaluator
parameters. Maps parameter names to either string keys or transformation functions.
This mapping has lower priority than evaluator-level mappings set in the evaluator
constructor, allowing evaluators to define sensible defaults while still permitting
customization at the evaluation level.
Maximum number of workers to use for concurrent processing. Use more than 1
only if the eval task function is thread-safe.
## Returns
Contains the experiment and list of ScoredExperimentItem objects, each containing
the experiment item data and scores for one dataset item.
## Raises
* **ValueError** – If dataset is empty or evaluators are invalid.
* **RuntimeError** – If no connection is available for API calls.
* **ApiError** – If there's an error creating the experiment or communicating
with the Fiddler API.
## Example
```python theme={null}
from fiddler_evals import evaluate
from fiddler_evals.evaluators import AnswerRelevance, Conciseness, RegexSearch
from fiddler_evals import Dataset
# Get dataset
dataset = Dataset.get_by_name("my-eval-dataset")
# Define evaluation task
def eval_task(inputs, extras, metadata):
# Your model inference logic here
question = inputs["question"]
answer = my_model.generate(question)
return {"answer": answer, "question": question}
# Example 1: Basic evaluation with parameter mapping
results = evaluate(
dataset=dataset,
task=eval_task,
evaluators=[AnswerRelevance(), Conciseness()],
name_prefix="my-model-eval",
description="Evaluation of my model on Q&A dataset",
metadata={"model_version": "v1.0", "temperature": 0.7},
score_fn_kwargs_mapping={
"output": "answer",
"question": lambda x: x["inputs"]["question"]
}
)
# Example 2: Multiple evaluator instances with score_name_prefix for differentiation
evaluators = [
RegexSearch(
r"\d+",
score_name_prefix="question",
score_name="has_number",
score_fn_kwargs_mapping={"output": "question"}
),
RegexSearch(
r"\d+",
score_name_prefix="answer",
score_name="has_number",
score_fn_kwargs_mapping={"output": "answer"}
)
]
results = evaluate(
dataset=dataset,
task=eval_task,
evaluators=evaluators,
score_fn_kwargs_mapping={
"question": lambda x: x["inputs"]["question"],
# Note: "answer" mapping not needed since evaluator defines it
}
)
# Process results
for result in results:
item_id = result.experiment_item.dataset_item_id
status = result.experiment_item.status
print(f"Item {item_id}: {status}")
for score in result.scores:
print(f" {score.name}: {score.value} ({score.status})")
```
The function processes dataset items sequentially. For large datasets,
consider implementing parallel processing or batch processing strategies.
The experiment name is automatically made unique by appending datetime.
Parameter Mapping Priority:
When both evaluator-level and evaluation-level mappings are present,
evaluator-level mappings take precedence. This allows evaluators to define
sensible defaults while still permitting customization at the evaluation level.
Mapping Priority (highest to lowest):
1. Evaluator-level score\_fn\_kwargs\_mapping (set in evaluator constructor)
2. Evaluation-level score\_fn\_kwargs\_mapping (passed to evaluate function)
3. Default parameter resolution
# Evaluator
Source: https://docs.fiddler.ai/sdk-api/evals/evaluator
Abstract base class for creating custom evaluators in Fiddler Evals.
Abstract base class for creating custom evaluators in Fiddler Evals.
The Evaluator class provides a flexible framework for creating builtin and custom evaluators
that can assess LLM outputs against various criteria. Each evaluator is
responsible for a single, specific evaluation task (e.g., hallucination detection,
answer relevance, exact match, etc.).
Parameter Mapping:
Evaluators can define their own parameter mappings using score\_fn\_kwargs\_mapping
in the constructor. These mappings specify how data from the evaluation context
(inputs, outputs, expected\_outputs) should be passed to the evaluator's score method.
Mapping Priority (highest to lowest):
1. Evaluator-level score\_fn\_kwargs\_mapping (set in constructor)
2. Evaluation-level kwargs\_mapping (passed to evaluate function)
3. Default parameter resolution
This allows evaluators to define sensible defaults while still permitting
customization at the evaluation level.
Creating Custom Evaluators:
To create a custom evaluator, inherit from this class and implement the score method
with parameters specific to your evaluation needs:
Example - Custom evaluator with parameter mapping:
class ExactMatchEvaluator(Evaluator):
> def **init**(self, output\_key: str = "answer", score\_name\_prefix: str = None):
> : super().**init**(
> : score\_name\_prefix=score\_name\_prefix,
> score\_fn\_kwargs\_mapping=\{"output": output\_key}
>
>
>
> )
> def score(self, output: str, expected\_output: str) -> Score:
> : is\_match = output.strip().lower() == expected\_output.strip().lower()
> return Score(
>
>
>
> > name=f"\{self.score\_name\_prefix}exact\_match",
> > value=1.0 if is\_match else 0.0,
> > reasoning=f"Match: \{is\_match}"
>
>
>
> )
## Parameters
Optional prefix to prepend to score names. Useful for
distinguishing scores when using multiple instances of the same evaluator
on different fields or with different configurations.
Optional mapping for parameter transformation.
Maps parameter names to either string keys or transformation functions.
This mapping takes precedence over evaluation-level mappings when running
the evaluate method.
The score method signature is intentionally flexible using \*args and \*\*kwargs
to allow each evaluator to define its own parameter requirements. This design
enables maximum flexibility while maintaining a consistent interface across
all evaluators in the framework.
Initialize the evaluator with parameter mapping configuration.
## Parameters
Optional prefix to prepend to score names. Useful for
distinguishing scores when using multiple instances of the same evaluator
on different fields or with different configurations.
Optional mapping for parameter transformation.
Maps parameter names to either string keys or transformation functions.
This mapping takes precedence over evaluation-level mappings when running
the evaluate method.
## Example
```python theme={null}
# Simple string mapping
evaluator = MyEvaluator(score_fn_kwargs_mapping={"output": "answer"})
# Complex transformation function
evaluator = MyEvaluator(score_fn_kwargs_mapping={
"question": lambda x: x["inputs"]["question"],
"response": "answer"
})
# Using score name prefix for multiple instances
evaluator1 = RegexSearch(r"\d+", score_name_prefix="question")
evaluator2 = RegexSearch(r"\d+", score_name_prefix="answer")
# Results in scores named "question_has_number" and "answer_has_number"
```
## Raises
**ScoreFunctionInvalidArgs** – If the mapping contains invalid parameter names
that don't match the evaluator's score method signature.
## *property* name
## *abstractmethod* score()
Evaluate inputs and return a score or list of scores.
This method must be implemented by all concrete evaluator classes.
Each evaluator can define its own parameter signature based on what
it needs for evaluation.
Common parameter patterns:
* Output-only: score(self, output: str) -> Score
* Input-Output: score(self, input: str, output: str) -> Score
* Comparison: score(self, output: str, expected\_output: str) -> Score
* All parameters: score(self, input: str, output: str, context: list\[str]) -> Score
### Parameters
Positional arguments specific to the evaluator's needs.
### Returns
A single Score object or list of Score objects
representing the evaluation results. Each Score should include:
* name: The score name (e.g., "has\_zipcode")
* evaluator\_name: The evaluator name (e.g., "RegexMatch")
* value: The score value (typically 0.0 to 1.0)
* status: SUCCESS, FAILED, or SKIPPED
* reasoning: Optional explanation of the score
* error: Optional error information if evaluation failed
### Raises
* **ValueError** – If required parameters are missing or invalid.
* **TypeError** – If parameters have incorrect types.
* **Exception** – Any other evaluation-specific errors.
# Experiment
Source: https://docs.fiddler.ai/sdk-api/evals/experiment
Represents an Experiment for tracking evaluation runs and results.
Represents an Experiment for tracking evaluation runs and results.
An Experiment is a single evaluation run of a test suite against a specific
application/LLM/Agent version and evaluators. Experiments provide comprehensive tracking,
monitoring, and result management for GenAI evaluation workflows, enabling
systematic testing and performance analysis.
Key Features:
* **Evaluation Tracking**: Complete lifecycle tracking of evaluation runs
* **Status Management**: Real-time status updates (PENDING, IN\_PROGRESS, COMPLETED, etc.)
* **Dataset Integration**: Linked to specific datasets for evaluation
* **Result Storage**: Comprehensive storage of results, metrics, and error information
* **Error Handling**: Detailed error tracking with traceback information
Experiment Lifecycle:
1. **Creation**: Create experiment with dataset and application references
2. **Execution**: Experiment runs evaluation against the dataset
3. **Monitoring**: Track status and progress in real-time
4. **Completion**: Retrieve results, metrics, and analysis
5. **Cleanup**: Archive or delete completed experiments
## Example
```python theme={null}
# Use this class to list
experiments = Experiment.list(
application_id=application_id,
dataset_id=dataset_id,
)
```
Experiments are permanent records of evaluation runs. Once created, the name
cannot be changed, but metadata and description can be updated. Failed
experiments retain error information for debugging and analysis.
## id
## name
## status
## created\_at
## updated\_at
## created\_by
## updated\_by
## project
## application
## dataset
## description
## error\_reason
## error\_message
## traceback
## duration\_ms
## metadata
## get\_app\_url()
Get the application URL for this experiment
### Returns
`str`
## *classmethod* get\_by\_id()
Retrieve an experiment by its unique identifier.
Fetches an experiment from the Fiddler platform using its UUID. This is the most
direct way to retrieve an experiment when you know its ID.
### Parameters
The unique identifier (UUID) of the experiment to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The experiment instance with all metadata and configuration.
### Raises
* **NotFound** – If no experiment exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get experiment by UUID
experiment = Experiment.get_by_id(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved experiment: {experiment.name}")
print(f"Status: {experiment.status}")
print(f"Created: {experiment.created_at}")
print(f"Application: {experiment.application.name}")
```
This method makes an API call to fetch the latest experiment state from the server.
The returned experiment instance reflects the current state in Fiddler.
## *classmethod* get\_by\_name()
Retrieve an experiment by name within an application.
Finds and returns an experiment using its name within the specified application.
This is useful when you know the experiment name and application but not its UUID.
Experiment names are unique within an application, making this a reliable lookup method.
### Parameters
The name of the experiment to retrieve. Experiment names are unique
within an application and are case-sensitive.
The UUID of the application containing the experiment.
Can be provided as a UUID object or string representation.
### Returns
The experiment instance matching the specified name.
### Raises
* **NotFound** – If no experiment exists with the specified name in the application.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
# Get experiment by name within an application
experiment = Experiment.get_by_name(
name="fraud-detection-eval-v1",
application_id=application.id
)
print(f"Found experiment: {experiment.name} (ID: {experiment.id})")
print(f"Status: {experiment.status}")
print(f"Created: {experiment.created_at}")
print(f"Dataset: {experiment.dataset.name}")
```
Experiment names are case-sensitive and must match exactly. Use this method
when you have a known experiment name from configuration or user input.
## *classmethod* list()
List all experiments in an application.
Retrieves all experiments that the current user has access to within the specified
application. Returns an iterator for memory efficiency when dealing with many experiments.
### Parameters
The UUID of the application to list experiments from.
Can be provided as a UUID object or string representation.
The UUID of the dataset to list experiments from.
Can be provided as a UUID object or string representation.
### Yields
`Experiment` – Experiment instances for all accessible experiments in the application.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Returns
`Iterator[Experiment]`
### Example
```python theme={null}
# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application.id)
# List all experiments in an application
for experiment in Experiment.list(application_id=application.id, dataset_id=dataset.id):
print(f"Experiment: {experiment.name}")
print(f" ID: {experiment.id}")
print(f" Status: {experiment.status}")
print(f" Created: {experiment.created_at}")
print(f" Dataset: {experiment.dataset.name}")
# Convert to list for counting and filtering
experiments = list(Experiment.list(application_id=application.id, dataset_id=dataset.id ))
print(f"Total experiments in application: {len(experiments)}")
# Find experiments by status
completed_experiments = [
exp for exp in Experiment.list(application_id=application.id, dataset_id=dataset.id)
if exp.status == ExperimentStatus.COMPLETED
]
print(f"Completed experiments: {len(completed_experiments)}")
# Find experiments by name pattern
eval_experiments = [
exp for exp in Experiment.list(application_id=application.id, dataset_id=dataset.id)
if "eval" in exp.name.lower()
]
print(f"Evaluation experiments: {len(eval_experiments)}")
```
This method returns an iterator for memory efficiency. Convert to a list
with list(Experiment.list(application\_id)) if you need to iterate multiple times or get
the total count. The iterator fetches experiments lazily from the API.
## *classmethod* create()
Create a new experiment in an application.
Creates a new experiment within the specified application on the Fiddler platform.
The experiment must have a unique name within the application and will be linked
to the specified dataset for evaluation.
Note: It is not recommended to use this method directly. Instead, use the evaluate method. Creating
and managing an experiment without evaluate wrapper is extremely advance usecase and should be avoided.
### Parameters
Experiment name, must be unique within the application.
The UUID of the application to create the experiment in.
Can be provided as a UUID object or string representation.
The UUID of the dataset to use for evaluation.
Can be provided as a UUID object or string representation.
Optional human-readable description of the experiment.
Optional custom metadata dictionary for additional experiment information.
### Returns
The newly created experiment instance with server-assigned fields.
### Raises
* **Conflict** – If an experiment with the same name already exists in the application.
* **ValidationError** – If the experiment configuration is invalid (e.g., invalid name format).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get application and dataset instances
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application.id)
# Create a new experiment for fraud detection evaluation
experiment = Experiment.create(
name="fraud-detection-eval-v1",
application_id=application.id,
dataset_id=dataset.id,
description="Comprehensive evaluation of fraud detection model v1.0",
metadata={"model_version": "1.0", "evaluation_type": "comprehensive", "baseline": "true"}
)
print(f"Created experiment with ID: {experiment.id}")
print(f"Status: {experiment.status}")
print(f"Created at: {experiment.created_at}")
print(f"Application: {experiment.application.name}")
print(f"Dataset: {experiment.dataset.name}")
```
After successful creation, the experiment instance is returned with
server-assigned metadata. The experiment is immediately available
for execution and monitoring. The initial status will be PENDING.
## *classmethod* get\_or\_create()
Get an existing experiment by name or create a new one if it doesn't exist.
This is a convenience method that attempts to retrieve an experiment by name
within an application, and if not found, creates a new experiment with that name.
Useful for idempotent experiment setup in automation scripts and deployment pipelines.
### Parameters
The name of the experiment to retrieve or create.
The UUID of the application to search/create the experiment in.
Can be provided as a UUID object or string representation.
The UUID of the dataset to use for evaluation.
Can be provided as a UUID object or string representation.
Optional human-readable description of the experiment.
Optional custom metadata dictionary for additional experiment information.
### Returns
Either the existing experiment with the specified name,
or a newly created experiment if none existed.
### Raises
* **ValidationError** – If the experiment name format is invalid.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get application and dataset instances
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application.id)
# Safe experiment setup - get existing or create new
experiment = Experiment.get_or_create(
name="fraud-detection-eval-v1",
application_id=application.id,
dataset_id=dataset.id,
description="Comprehensive evaluation of fraud detection model v1.0",
metadata={"model_version": "1.0", "evaluation_type": "comprehensive"}
)
print(f"Using experiment: {experiment.name} (ID: {experiment.id})")
# Idempotent setup in deployment scripts
experiment = Experiment.get_or_create(
name="llm-benchmark-eval",
application_id=application.id,
dataset_id=dataset.id,
metadata={"baseline": "true"}
)
# Use in configuration management
model_versions = ["v1.0", "v1.1", "v2.0"]
experiments = {}
for version in model_versions:
experiments[version] = Experiment.get_or_create(
name=f"fraud-detection-eval-{version}",
application_id=application.id,
dataset_id=dataset.id,
metadata={"model_version": version}
)
```
This method is idempotent - calling it multiple times with the same name
and application\_id will return the same experiment. It logs when creating a new
experiment for visibility in automation scenarios.
## update()
Update experiment description, metadata, and status.
Updates the experiment's description, metadata, and/or status. This method allows
you to modify the experiment's configuration after creation, including updating
the experiment status and error information for failed experiments.
### Parameters
Optional new description for the experiment. If provided,
replaces the existing description. Set to empty string to clear.
Optional new metadata dictionary for the experiment. If provided,
replaces the existing metadata completely. Use empty dict to clear.
Optional new status for the experiment. Can be used to update
experiment status (e.g., PENDING, RUNNING, COMPLETED, FAILED).
Required when status is FAILED. The reason for the experiment failure.
Required when status is FAILED. Detailed error message for the failure.
Required when status is FAILED. Stack trace information for debugging.
Optional duration in milliseconds for the experiment execution
### Returns
The updated experiment instance with new metadata and configuration.
### Raises
* **ValueError** – If no update parameters are provided (all are None) or if status is FAILED
but error\_reason, error\_message, or traceback are missing.
* **ValidationError** – If the update data is invalid (e.g., invalid metadata format).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)
# Update description and metadata
updated_experiment = experiment.update(
description="Updated comprehensive evaluation of fraud detection model v1.1",
metadata={"model_version": "1.1", "evaluation_type": "comprehensive", "updated_by": "john_doe"}
)
print(f"Updated experiment: {updated_experiment.name}")
print(f"New description: {updated_experiment.description}")
# Update only metadata
experiment.update(metadata={"last_updated": "2024-01-15", "status": "active"})
# Update experiment status to completed
experiment.update(status=ExperimentStatus.COMPLETED)
# Mark experiment as failed with error details
experiment.update(
status=ExperimentStatus.FAILED,
error_reason="Evaluation timeout",
error_message="The evaluation process exceeded the maximum allowed time",
traceback="Traceback (most recent call last): File evaluate.py, line 42..."
)
# Clear description
experiment.update(description="")
# Batch update multiple experiments
for experiment in Experiment.list(application_id=application_id):
if experiment.status == ExperimentStatus.COMPLETED:
experiment.update(metadata={"archived": "true"})
```
This method performs a complete replacement of the specified fields.
For partial updates, retrieve current values, modify them, and pass
the complete new values. The experiment name and ID cannot be changed.
When updating status to FAILED, all error-related parameters are required.
## delete()
Delete the experiment.
Permanently deletes the experiment and all associated data from the Fiddler platform.
This action cannot be undone and will remove all experiment results, metrics, and metadata.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)
# Delete the experiment
experiment.delete()
print("Experiment deleted successfully")
# Delete multiple experiments
for experiment in Experiment.list(application_id=application_id):
if experiment.status == ExperimentStatus.FAILED:
print(f"Deleting failed experiment: {experiment.name}")
experiment.delete()
```
This operation is irreversible. Once deleted, the experiment and all its
associated data cannot be recovered. Consider archiving experiments instead
of deleting them if you need to preserve historical data.
## add\_items()
Add outputs of LLM/Agent/Application against dataset items to the experiment.
Adds outputs of LLM/Agent/Application (task or target function) against dataset items to the experiment, representing individual
test case outcomes. Each item contains the outputs of LLM/Agent/Application results, timing information, and status for a specific dataset item.
### Parameters
List of NewExperimentItem instances containing outputs of LLM/Agent/Application against dataset items.
Each item should include:
* dataset\_item\_id: UUID of the dataset item being evaluated
* outputs: Dictionary containing the outputs of the task function against dataset item
* duration\_ms: Duration of the execution in milliseconds:
* status: Status of the outputs of the task function / scoring against dataset item (PENDING, COMPLETED, FAILED, etc.)
* error\_reason: Reason for failure, if applicable
* error\_message: Detailed error message, if applicable
### Returns
List of UUIDs for the newly created experiment items.
### Raises
* **ValueError** – If the items list is empty.
* **ValidationError** – If any item data is invalid (e.g., missing required fields).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)
# Create evaluation result items
from fiddler_evals.pydantic_models.experiment import NewExperimentItem
from datetime import datetime, timezone
items = [
NewExperimentItem(
dataset_item_id=dataset_item_id_1,
outputs={"answer": "The watermelon seeds pass through your digestive system"},
duration_ms=1000,
end_time=datetime.now(tz=timezone.utc),
status="COMPLETED",
error_reason=None,
error_message=None
),
NewExperimentItem(
dataset_item_id=dataset_item_id_2,
outputs={"answer": "The precise origin of fortune cookies is unclear"},
duration_ms=1000,
end_time=datetime.now(tz=timezone.utc),
status="COMPLETED",
error_reason=None,
error_message=None
)
]
# Add items to experiment
item_ids = experiment.add_items(items)
print(f"Added {len(item_ids)} evaluation result items")
print(f"Item IDs: {item_ids}")
# Add items from evaluation results
items = [
{
"dataset_item_id": str(dataset_item_id),
"outputs": {"answer": result["answer"]},
"duration_ms": result["duration_ms"],
"end_time": result["end_time"],
"status": "COMPLETED"
}
for result in items
]
item_ids = experiment.add_items([NewExperimentItem(**item) for item in items])
# Batch add items with error handling
try:
item_ids = experiment.add_items(items)
print(f"Successfully added {len(item_ids)} items")
except ValueError as e:
print(f"Validation error: {e}")
except Exception as e:
print(f"Failed to add items: {e}")
```
This method is typically used after running evaluations to store the results
in the experiment. Each item represents the evaluation of a single dataset
item and contains all relevant timing, output, and status information.
## get\_items()
Retrieve all experiment result items from the experiment.
Fetches all experiment result items (inputs, outputs, expected outputs, scores)
that were generated by the task function against dataset items. Returns an iterator
for memory efficiency when dealing with large experiments containing many result items.
### Returns
Iterator of
ExperimentResultItem instances for all result items in the experiment.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)
# Get all result items from the experiment
for item in experiment.get_items():
print(f"Item ID: {item.id}")
print(f"Dataset Item ID: {item.dataset_item_id}")
print(f"Outputs: {item.outputs}")
print(f"Status: {item.status}")
print(f"Duration: {item.duration_ms}")
if item.error_reason:
print(f"Error: {item.error_reason} - {item.error_message}")
print("---")
# Convert to list for analysis
all_items = list(experiment.get_items())
print(f"Total result items: {len(all_items)}")
# Filter items by status
completed_items = [
item for item in experiment.get_items()
if item.status == "COMPLETED"
]
print(f"Completed items: {len(completed_items)}")
# Filter items by error status
failed_items = [
item for item in experiment.get_items()
if item.status == "FAILED"
]
print(f"Failed items: {len(failed_items)}")
# Process items in batches
batch_size = 100
for i, item in enumerate(experiment.get_items()):
if i % batch_size == 0:
print(f"Processing batch {i // batch_size + 1}")
# Process item...
# Analyze outputs
for item in experiment.get_items():
if item.outputs.get("confidence", 0) < 0.8:
print(f"Low confidence item: {item.id}")
```
This method returns an iterator for memory efficiency. Convert to a list
with list(experiment.get\_items()) if you need to iterate multiple times or get
the total count. The iterator fetches items lazily from the API.
## add\_results()
Add evaluation results to the experiment.
Adds complete evaluation results to the experiment, including both the experiment
item data (outputs, timing, status) and all associated evaluator scores. This
method is typically used after running evaluations to store the complete results
of the evaluation process for a batch of dataset items.
This method will only append the results to the experiment.
Note: It is not recommended to use this method directly. Instead, use the evaluate method. Creating
and managing an experiment without evaluate wrapper is extremely advance usecase and should be avoided.
### Parameters
List of ScoredExperimentItem instances containing:
* experiment\_item: NewExperimentItem with outputs, timing, and status
* scores: List of Score objects from evaluators for this item
### Returns
Results are added to the experiment on the server.
### Raises
* **ValueError** – If the items list is empty.
* **ValidationError** – If any item data is invalid (e.g., missing required fields).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)
# Create experiment item with outputs
experiment_item = NewExperimentItem(
dataset_item_id=dataset_item.id,
outputs={"prediction": "fraud", "confidence": 0.95},
duration_ms=1000,
end_time=datetime.now(tz=timezone.utc),
status="COMPLETED"
)
# Create scores from evaluators
scores = [
Score(
name="accuracy",
evaluator_name="AccuracyEvaluator",
value=1.0,
label="Correct",
reasoning="Prediction matches ground truth"
),
Score(
name="confidence",
evaluator_name="ConfidenceEvaluator",
value=0.95,
label="High",
reasoning="High confidence in prediction"
)
]
# Create result combining item and scores
result = ScoredExperimentItem(
experiment_item=experiment_item,
scores=scores
)
# Add results to experiment
experiment.add_results([result])
```
This method is typically called after running evaluations to store complete
results. The results include both the experiment item data and all evaluator
scores, providing a complete record of the evaluation process.
# ExperimentItemStatus
Source: https://docs.fiddler.ai/sdk-api/evals/experiment-item-status
ExperimentItemStatus
**Values:**
* `SUCCESS` - SUCCESS
* `FAILED` - FAILED
* `SKIPPED` - SKIPPED
# ExperimentStatus
Source: https://docs.fiddler.ai/sdk-api/evals/experiment-status
ExperimentStatus
**Values:**
* `PENDING` - PENDING
* `IN_PROGRESS` - IN\_PROGRESS
* `COMPLETED` - COMPLETED
* `FAILED` - FAILED
* `CANCELLED` - CANCELLED
# FTLPromptSafety
Source: https://docs.fiddler.ai/sdk-api/evals/ftl-prompt-safety
Evaluator to assess prompt safety using Fiddler Centor Models.
Evaluator to assess prompt safety using Fiddler Centor Models.
The FTLPromptSafety evaluator uses Fiddler Centor Models to evaluate
the safety of text prompts across multiple risk categories. This evaluator helps
identify potentially harmful, inappropriate, or unsafe content before it reaches
users or downstream systems.
Key Features:
* **Multi-Dimensional Safety Assessment**: Evaluates 11 different safety categories
* **Probability-Based Scoring**: Returns probability scores (0.0-1.0) for each risk category
* **Comprehensive Risk Coverage**: Covers illegal, hateful, harassing, and other harmful content
* **Prompt Safety**: Uses the Centor Model for Safety, Fiddler's proprietary safety evaluation model
* **Batch Scoring**: Returns multiple scores for comprehensive safety analysis
Safety Categories Evaluated:
* **illegal\_prob**: Probability of containing illegal content or activities
* **hateful\_prob**: Probability of containing hate speech or discriminatory language
* **harassing\_prob**: Probability of containing harassing or threatening content
* **racist\_prob**: Probability of containing racist language or content
* **sexist\_prob**: Probability of containing sexist language or content
* **violent\_prob**: Probability of containing violent or graphic content
* **sexual\_prob**: Probability of containing inappropriate sexual content
* **harmful\_prob**: Probability of containing content that could cause harm
* **unethical\_prob**: Probability of containing unethical or manipulative content
* **jailbreaking\_prob**: Probability of containing prompt injection or jailbreaking attempts
* **max\_risk\_prob**: Maximum risk probability across all categories
Use Cases:
* **Content Moderation**: Filtering user-generated content for safety
* **Prompt Validation**: Ensuring user prompts are safe before processing
* **AI Safety**: Protecting AI systems from harmful or manipulative inputs
* **Compliance**: Meeting regulatory requirements for content safety
* **Risk Assessment**: Evaluating potential risks in text content
Scoring Logic:
Each safety category returns a probability score between 0.0 and 1.0:
* **0.0-0.3**: Low risk (safe content)
* **0.3-0.7**: Medium risk (requires review)
* **0.7-1.0**: High risk (likely unsafe content)
## Parameters
* **text** (*str*) – The text prompt to evaluate for safety.
* **score\_name\_prefix** (*str* *|* *None*)
* **score\_fn\_kwargs\_mapping** (*ScoreFnKwargsMappingType* *|* *None*)
## Returns
A list of Score objects, one for each safety category:
* name: The safety category name (e.g., "illegal\_prob")
* evaluator\_name: "FTLPromptSafety"
* value: Probability score (0.0-1.0) for that category
## Raises
**ValueError** – If the text is empty or None.
## Example
```python theme={null}
from fiddler_evals.evaluators import FTLPromptSafety
evaluator = FTLPromptSafety()
```
```python theme={null}
# Safe content
scores = evaluator.score("What is the weather like today?")
for score in scores:
print(f"{score.name}: {score.value}")
# illegal_prob: 0.01
# hateful_prob: 0.02
# harassing_prob: 0.01
# ...
# Potentially unsafe content
unsafe_scores = evaluator.score("How to hack into someone's computer?")
for score in unsafe_scores:
if score.value > 0.5:
print(f"High risk detected: {score.name} = {score.value}")
# Filter based on maximum risk
max_risk_score = next(s for s in scores if s.name == "max_risk_prob")
if max_risk_score.value > 0.7:
print("Content flagged as potentially unsafe")
```
This evaluator is designed for prompt safety assessment and should be used
as part of a comprehensive content moderation strategy. The probability
scores should be interpreted in context and combined with other safety
measures for robust content filtering.
## name *= 'ftl\_prompt\_safety'*
## score()
Score the safety of a text prompt.
### Parameters
The text prompt to evaluate for safety.
### Returns
A list of Score objects, one for each safety category.
# FTLResponseFaithfulness
Source: https://docs.fiddler.ai/sdk-api/evals/ftl-response-faithfulness
Evaluator to assess response faithfulness using Fiddler Centor Models.
Evaluator to assess response faithfulness using Fiddler Centor Models.
The FTLResponseFaithfulness evaluator uses Fiddler Centor Models to evaluate
how faithful an LLM response is to the provided context. This evaluator helps ensure
that responses accurately reflect the information in the source context and don't
contain hallucinated or fabricated information.
Key Features:
* **Faithfulness Assessment**: Evaluates how well the response reflects the context
* **Probability-Based Scoring**: Returns probability scores (0.0-1.0) for faithfulness
* **Context-Response Alignment**: Compares response against provided context
* **Faithfulness (Centor Model)**: Uses the Centor Model for Faithfulness, Fiddler's proprietary faithfulness evaluation model
* **Hallucination Detection**: Identifies responses that go beyond the context
Faithfulness Categories Evaluated:
* **faithful\_prob**: Probability that the response is faithful to the context
Use Cases:
* **RAG Systems**: Ensuring responses stay grounded in retrieved context
* **Document Q\&A**: Verifying answers are based on provided documents
* **Fact-Checking**: Validating that responses don't contain fabricated information
* **Content Validation**: Ensuring responses accurately reflect source material
* **Hallucination Detection**: Identifying responses that go beyond the context
Scoring Logic:
The faithfulness score represents the probability that the response is faithful to the context:
* **0.0-0.3**: Low faithfulness (likely contains hallucinated information)
* **0.3-0.7**: Medium faithfulness (some information may not be grounded)
* **0.7-1.0**: High faithfulness (response accurately reflects context)
## Parameters
* **response** (*str*) – The LLM response to evaluate for faithfulness.
* **context** (*str*) – The source context that the response should be faithful to.
* **score\_name\_prefix** (*str* *|* *None*)
* **score\_fn\_kwargs\_mapping** (*ScoreFnKwargsMappingType* *|* *None*)
## Returns
A list of Score objects containing:
* name: The faithfulness category name ("faithful\_prob")
* evaluator\_name: "FTLResponseFaithfulness"
* value: Probability score (0.0-1.0) for faithfulness
## Raises
**ValueError** – If the response or context is empty or None.
## Example
```python theme={null}
from fiddler_evals.evaluators import FTLResponseFaithfulness
evaluator = FTLResponseFaithfulness()
```
```python theme={null}
# Faithful response
context = "The capital of France is Paris. It is located in northern Europe."
response = "Paris is the capital of France."
scores = evaluator.score(response=response, context=context)
for score in scores:
print(f"{score.name}: {score.value}")
# faithful_prob: 0.95
# Unfaithful response with hallucination
context = "The capital of France is Paris."
response = "The capital of France is Paris, and it has a population of 2.1 million people."
scores = evaluator.score(response=response, context=context)
for score in scores:
print(f"{score.name}: {score.value}")
# faithful_prob: 0.65 (population info not in context)
# Highly unfaithful response
context = "The capital of France is Paris."
response = "The capital of France is London."
scores = evaluator.score(response=response, context=context)
for score in scores:
print(f"{score.name}: {score.value}")
# faithful_prob: 0.05
# Filter based on faithfulness threshold
faithful_score = next(s for s in scores if s.name == "faithful_prob")
if faithful_score.value < 0.7:
print("Response flagged as potentially unfaithful")
```
This evaluator is designed for response faithfulness assessment and should be used
in conjunction with other evaluation metrics for comprehensive response quality
assessment. The probability scores should be interpreted in context and combined
with other quality measures for robust response validation.
## name *= 'ftl\_response\_faithfulness'*
## score()
Score the faithfulness of a response to its context.
### Parameters
The LLM response to evaluate for faithfulness.
The source context that the response should be faithful to.
### Returns
A Score object for faithfulness probability.
# FTLSecretDetection
Source: https://docs.fiddler.ai/sdk-api/evals/ftl-secret-detection
Evaluator to detect credentials, API keys, and tokens in text using Fiddler Centor Models.
Evaluator to detect secrets and credentials in text using Fiddler Centor Models.
The FTLSecretDetection evaluator scans text for API keys, tokens, and credentials using
\~42 known credential formats and Shannon entropy analysis. This is a CPU-only pipeline —
deterministic, low-latency, and requiring no GPU.
Key Features:
* **Pattern-based detection**: \~42 known credential formats across LLM providers, cloud platforms, source control, messaging, and developer tools (entropy-detected secrets are labeled `Possible Secret`)
* **Entropy analysis**: Catches unknown or custom secrets that exceed entropy thresholds
* **Fast**: CPU-only, sub-millisecond per token — no inference overhead
Use Cases:
* **Secret leakage detection**: Identify credentials in LLM prompts or responses
* **Compliance auditing**: Scan text datasets for inadvertently captured credentials
* **Data sanitization**: Locate and redact secrets in datasets before training or fine-tuning
Scoring Logic:
Unlike probability-based evaluators, FTLSecretDetection returns one `Score` per detected secret:
* **No secrets detected**: Returns an empty list
* **Secrets detected**: Returns one `Score` per detection, with `name` set to the secret type label and `value` set to `1.0`
To retrieve character-level positions for redaction, use the REST API directly — see [Secret Detection tutorial](/developers/tutorials/guardrails/guardrails-secrets).
## Parameters
* **text** (*str*) – The text to scan for secrets and credentials.
* **score\_name\_prefix** (*str* *|* *None*)
* **score\_fn\_kwargs\_mapping** (*ScoreFnKwargsMappingType* *|* *None*)
## Returns
A list of Score objects, one per detected secret:
* name: The secret type label (e.g., `"Anthropic API Key"`, `"AWS Access Key ID"`)
* evaluator\_name: `"FTLSecretDetection"`
* value: `1.0` for each detection (binary — present or absent)
## Raises
**ValueError** – If the text is empty or None.
## Example
```python theme={null}
from fiddler_evals.evaluators import FTLSecretDetection
evaluator = FTLSecretDetection()
```
```python theme={null}
# Clean text — no secrets
scores = evaluator.score("What is the weather like today?")
print(f"Secrets found: {len(scores)}")
# Secrets found: 0
# Text containing an API key
scores = evaluator.score(
"My Anthropic key is sk-ant-api03-abcdefghijklmnopqrstu"
)
for score in scores:
print(f"Detected: {score.name} (value={score.value})")
# Detected: Anthropic API Key (value=1.0)
# Check whether any secrets were found
has_secrets = len(scores) > 0
secret_types = [score.name for score in scores]
print(f"Secret types found: {secret_types}")
# Secret types found: ['Anthropic API Key']
```
FTLSecretDetection uses regex patterns and entropy thresholds — not an ML model. This means
it has no false-negative rate for known credential formats (pattern match is exact), but may
produce occasional false positives on high-entropy non-secret strings (e.g. UUIDs, git hashes,
and base64-encoded data are explicitly excluded via allowlist).
## name *= 'ftl\_secret\_detection'*
## score()
Scan a text string for secrets and credentials.
### Parameters
The text to scan for secrets and credentials.
### Returns
A list of Score objects, one per detected secret. Empty list if no secrets found.
# Introduction
Source: https://docs.fiddler.ai/sdk-api/evals/index
Complete API reference for fiddler-evals
[](https://pypi.org/project/fiddler-evals/)
## Overview
Complete API reference documentation for the fiddler-evals package.
## Components
### Connection
Connection management and initialization
* [Connection](/sdk-api/evals/connection)
* [init](/sdk-api/evals/init)
### Entities
Core entity classes for interacting with the platform
* [Application](/sdk-api/evals/application)
* [Dataset](/sdk-api/evals/dataset)
* [Experiment](/sdk-api/evals/experiment)
* [ExperimentItemStatus](/sdk-api/evals/experiment-item-status)
* [ExperimentStatus](/sdk-api/evals/experiment-status)
* [Project](/sdk-api/evals/project)
### Evaluators
Built-in and custom evaluators for LLM assessment
* [AnswerRelevance](/sdk-api/evals/answer-relevance)
* [Coherence](/sdk-api/evals/coherence)
* [Conciseness](/sdk-api/evals/conciseness)
* [ContextRelevance](/sdk-api/evals/context-relevance)
* [CustomJudge](/sdk-api/evals/custom-judge)
* [CustomJudgeSpec](/sdk-api/evals/custom-judge-spec)
* [EvalFn](/sdk-api/evals/eval-fn)
* [Evaluator](/sdk-api/evals/evaluator)
* [FTLPromptSafety](/sdk-api/evals/ftl-prompt-safety)
* [FTLResponseFaithfulness](/sdk-api/evals/ftl-response-faithfulness)
* [InputFieldSpec](/sdk-api/evals/input-field-spec)
* [Message](/sdk-api/evals/message)
* [OutputField](/sdk-api/evals/output-field)
* [OutputFieldTransform](/sdk-api/evals/output-field-transform)
* [RAGFaithfulness](/sdk-api/evals/rag-faithfulness)
* [RegexMatch](/sdk-api/evals/regex-match)
* [RegexSearch](/sdk-api/evals/regex-search)
* [Sentiment](/sdk-api/evals/sentiment)
* [TopicClassification](/sdk-api/evals/topic-classification)
### Pydantic Models
Pydantic data models and validation schemas
* [DatasetItem](/sdk-api/evals/dataset-item)
* [NewDatasetItem](/sdk-api/evals/new-dataset-item)
* [Score](/sdk-api/evals/score)
* [ScoreStatus](/sdk-api/evals/score-status)
### Runner
Evaluation execution and orchestration
* [evaluate](/sdk-api/evals/evaluate)
# init
Source: https://docs.fiddler.ai/sdk-api/evals/init
Initialize the Fiddler client with connection parameters and global configuration.
Initialize the Fiddler client with connection parameters and global configuration.
This function establishes a connection to the Fiddler platform and configures
the global client state. It handles authentication, server compatibility
validation, logging setup, and creates the singleton connection instance
used throughout the client library.
## Parameters
The base URL to your Fiddler platform instance
Authentication token obtained from the Fiddler UI Credentials tab
Dictionary mapping protocol to proxy URL for HTTP requests
HTTP request timeout settings (float or tuple of connect/read timeouts)
Whether to verify server's TLS certificate (default: True)
Whether to validate server/client version compatibility (default: True)
## Raises
* **ValueError** – If url or token parameters are empty
* **IncompatibleClient** – If server version is incompatible with client version
* **ConnectionError** – If unable to connect to the Fiddler platform
## Examples
Basic initialization:
```python theme={null}
import fiddler as fdl
fdl.init(
url="https://your-instance.fiddler.ai",
token="your-api-key"
)
```
Initialization with custom timeout and proxy:
```python theme={null}
fdl.init(
url="https://your-instance.fiddler.ai",
token="your-api-key",
timeout=(10.0, 60.0), # 10s connect, 60s read timeout
proxies={"https": "https://proxy.company.com:8080"}
)
```
Initialization for development with relaxed settings:
```python theme={null}
fdl.init(
url="https://your-instance.fiddler.ai",
token="dev-token",
verify=False, # Skip SSL verification
validate=False, # Skip version compatibility check
)
```
The client implements automatic retry strategies for transient failures.
Configure retry duration via FIDDLER\_CLIENT\_RETRY\_MAX\_DURATION\_SECONDS
environment variable (default: 300 seconds).
Logging is performed under the 'fiddler' namespace at INFO level.
If no root logger is configured, a stderr handler is automatically
attached unless auto\_attach\_log\_handler=False.
# InputFieldSpec
Source: https://docs.fiddler.ai/sdk-api/evals/input-field-spec
Metadata for a template variable (input field).
Metadata for a template variable (input field).
Input fields define metadata for template variables: whether they're
required, and optional title/description for documentation.
## title
## description
## required
# Message
Source: https://docs.fiddler.ai/sdk-api/evals/message
A single message in a prompt template.
A single message in a prompt template.
## role
## content
# NewDatasetItem
Source: https://docs.fiddler.ai/sdk-api/evals/new-dataset-item
Model to create a new dataset
Model to create a new dataset
## id
## inputs
## expected\_outputs
## metadata
## extras
## source\_name
## source\_id
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# OutputField
Source: https://docs.fiddler.ai/sdk-api/evals/output-field
Schema for a single output field in the evaluation response.
Schema for a single output field in the evaluation response.
## type
## choices
## description
## title
## transform
## default
## minimum
## maximum
# OutputFieldTransform
Source: https://docs.fiddler.ai/sdk-api/evals/output-field-transform
Defines how to transform an LLM output field into a final output field.
Defines how to transform an LLM output field into a final output field.
Used for field renaming and value mapping, e.g., transforming
`is_faithful: 'faithful'` to `label: 'yes'`.
## source\_field
## value\_map
# Project
Source: https://docs.fiddler.ai/sdk-api/evals/project
Represents a project container for organizing GenAI Apps and resources.
Represents a project container for organizing GenAI Apps and resources.
A Project is the top-level organizational unit in Fiddler that groups related
GenAI Applications, datasets, and monitoring configurations. Projects provide
logical separation, access control, and resource management for GenAI App monitoring workflows.
Key Features:
* **GenAI Apps Organization**: Container for related GenAI apps
* **Resource Isolation**: Separate namespaces prevent naming conflicts
* **Access Management**: Project-level permissions and access control
* **Monitoring Coordination**: Centralized monitoring and alerting configuration
* **Lifecycle Management**: Coordinated creation, updates, and deletion of resources
Project Lifecycle:
1. **Creation**: Create project with unique name within organization
2. **App creation**: Create GenAI applications with Application().create()
3. **Configuration**: Set up monitoring, alerts, and evaluators.
4. **Operations**: Publish logs, monitor performance, manage alerts
5. **Maintenance**: Update configurations
6. **Cleanup**: Delete project when no longer needed (removes all contained resources)
## Example
```python theme={null}
# Create a new project for fraud detection models
project = Project.create(name="fraud-detection-2024")
print(f"Created project: {project.name} (ID: {project.id})")
```
Projects are permanent containers - once created, the name cannot be changed.
Deleting a project removes all contained models, datasets, and configurations.
Consider the organizational structure carefully before creating projects.
## id
## name
## created\_at
## updated\_at
## *classmethod* get\_by\_id()
Retrieve a project by its unique identifier.
Fetches a project from the Fiddler platform using its UUID. This is the most
direct way to retrieve a project when you know its ID.
### Parameters
The unique identifier (UUID) of the project to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The project instance with all metadata and configuration.
### Raises
* **NotFound** – If no project exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get project by UUID
project = Project.get_by_id(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved project: {project.name}")
print(f"Created: {project.created_at}")
```
This method makes an API call to fetch the latest project state from the server.
The returned project instance reflects the current state in Fiddler.
## *classmethod* get\_by\_name()
Retrieve a project by name.
Finds and returns a project using its name within the organization. This is useful
when you know the project name but not its UUID. Project names are unique within
an organization, making this a reliable lookup method.
### Parameters
The name of the project to retrieve. Project names are unique
within an organization and are case-sensitive.
### Returns
The project instance matching the specified name.
### Raises
* **NotFound** – If no project exists with the specified name in the organization.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get project by name
project = Project.get_by_name(name="fraud-detection")
print(f"Found project: {project.name} (ID: {project.id})")
print(f"Created: {project.created_at}")
# Get project for specific environment
prod_project = Project.get_by_name(name="fraud-detection-prod")
staging_project = Project.get_by_name(name="fraud-detection-staging")
```
Project names are case-sensitive and must match exactly. Use this method
when you have a known project name from configuration or user input.
## *classmethod* list()
List all projects in the organization.
Retrieves all projects that the current user has access to within the organization.
Returns an iterator for memory efficiency when dealing with many projects.
### Yields
`Project` – Project instances for all accessible projects.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Returns
`Iterator[Project]`
### Example
```python theme={null}
# List all projects
for project in Project.list():
print(f"Project: {project.name}")
print(f" ID: {project.id}")
print(f" Created: {project.created_at}")
# Convert to list for counting and filtering
projects = list(Project.list())
print(f"Total accessible projects: {len(projects)}")
# Find projects by name pattern
prod_projects = [
p for p in Project.list()
if "prod" in p.name.lower()
]
print(f"Production projects: {len(prod_projects)}")
# Get project summaries
for project in Project.list():
print(f"{project.name}")
```
This method returns an iterator for memory efficiency. Convert to a list
with list(Project.list()) if you need to iterate multiple times or get
the total count. The iterator fetches projects lazily from the API.
## *classmethod* create()
Create the project on the Fiddler platform.
Persists this project instance to the Fiddler platform, making it available
for adding GenAI Apps, configuring monitoring, and other operations. The project
must have a unique name within the organization.
### Parameters
Project name, must be unique within the organization.
Should follow slug-like naming conventions:
* Use lowercase letters, numbers, hyphens, and underscores
* Start with a letter or number
### Returns
This project instance, updated with server-assigned fields like
ID, creation timestamp, and other metadata.
### Raises
* **Conflict** – If a project with the same name already exists in the organization.
* **ValidationError** – If the project configuration is invalid (e.g., invalid name format).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Create a new project
project = Project.create(name="customer-churn-analysis")
print(f"Created project with ID: {project.id}")
print(f"Created at: {project.created_at}")
# Project is now available for adding GenAI Apps
assert project.id is not None
```
After successful creation, the project instance is returned with
server-assigned metadata. The project is immediately available
for adding GenAI Apps and other resources.
## *classmethod* get\_or\_create()
Get an existing project by name or create a new one if it doesn't exist.
This is a convenience method that attempts to retrieve a project by name,
and if not found, creates a new project with that name. Useful for idempotent
project setup in automation scripts and deployment pipelines.
### Parameters
The name of the project to retrieve or create. Must follow
project naming conventions (slug-like format).
### Returns
Either the existing project with the specified name,
or a newly created project if none existed.
### Raises
* **ValidationError** – If the project name format is invalid.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Safe project setup - get existing or create new
project = Project.get_or_create(name="fraud-detection-prod")
print(f"Using project: {project.name} (ID: {project.id})")
# Idempotent setup in deployment scripts
project = Project.get_or_create(name="llm-pipeline-staging")
# Use in configuration management
environments = ["dev", "staging", "prod"]
projects = {}
for env in environments:
projects[env] = Project.get_or_create(name=f"fraud-detection-{env}")
```
This method is idempotent - calling it multiple times with the same name
will return the same project. It logs when creating a new project for
visibility in automation scenarios.
# RAGFaithfulness
Source: https://docs.fiddler.ai/sdk-api/evals/rag-faithfulness
Evaluator to assess if an LLM response is faithful to the provided context.
Evaluator to assess if an LLM response is faithful to the provided context.
The RAGFaithfulness evaluator measures whether a response is grounded in and
consistent with the provided reference documents. This is crucial for RAG
(Retrieval-Augmented Generation) pipelines to detect hallucinations and ensure
responses don't include information not present in the context.
Key Features:
* **Faithfulness Assessment**: Determines if the response is supported by context
* **Binary Scoring**: Returns 1.0 (faithful) or 0.0 (not faithful)
* **Hallucination Detection**: Identifies when responses include unsupported claims
* **Detailed Reasoning**: Provides explanation for the faithfulness assessment
* **Fiddler API Integration**: Uses Fiddler's built-in faithfulness evaluation model
Use Cases:
* **RAG Systems**: Detecting hallucinations in generated responses
* **Document Q\&A**: Ensuring answers are grounded in source documents
* **Customer Support**: Verifying responses align with knowledge base
* **Legal/Medical AI**: Critical applications requiring factual accuracy
* **Content Generation**: Ensuring generated content matches source material
Scoring Logic:
* **1.0 (Faithful)**: Response is fully supported by the reference documents
* **0.0 (Not Faithful)**: Response contains information not in the documents
or contradicts the documents
## Parameters
* **user\_query** (*str*) – The question or query being asked.
* **rag\_response** (*str*) – The LLM's response to evaluate.
* **retrieved\_documents** (*list* \*\[\**str* *]*) – The reference documents to check against.
* **model** (*str*)
* **credential** (*str* *|* *None*)
* **kwargs** (*Any*)
## Returns
A Score object containing:
* value: 1.0 if faithful, 0.0 if not faithful
* label: "yes" or "no"
* reasoning: Detailed explanation of the assessment
## Example
```python theme={null}
from fiddler_evals.evaluators import RAGFaithfulness
evaluator = RAGFaithfulness(model="openai/gpt-4o")
# Faithful response
score = evaluator.score(
user_query="What is the capital of France?",
rag_response="According to the document, Paris is the capital of France.",
retrieved_documents=[
"Paris is the capital and largest city of France.",
"France is located in Western Europe."
]
)
print(f"Faithful: {score.label}") # "yes"
print(f"Score: {score.value}") # 1.0
# Unfaithful response (hallucination)
score = evaluator.score(
user_query="What is the capital of France?",
rag_response="Paris is the capital of France with a population of 12 million.",
retrieved_documents=[
"Paris is the capital of France."
# Note: population is not mentioned in documents
]
)
print(f"Faithful: {score.label}") # "no"
```
This evaluator uses Fiddler's built-in faithfulness assessment model
and requires an active connection to the Fiddler API. The evaluator
checks if the response is supported by the documents, not whether
the response correctly answers the question.
## name *= 'rag\_faithfulness'*
## score()
Score the faithfulness of a response to the provided context.
### Parameters
The question or query being asked.
The LLM's response to evaluate.
The reference documents to check against.
### Returns
A Score object containing:
* value: 1.0 if faithful, 0.0 if not faithful
* label: "yes" or "no"
* reasoning: Detailed explanation of the assessment
# RegexMatch
Source: https://docs.fiddler.ai/sdk-api/evals/regex-match
Regex match attempts to match the regex pattern only at the beginning
Regex match attempts to match the regex pattern only at the beginning
of the string.
## *property* match\_fn
Match function to use for the regex evaluator.
# RegexSearch
Source: https://docs.fiddler.ai/sdk-api/evals/regex-search
Regex search scans the entire string from beginning to end, looking for the
Regex search scans the entire string from beginning to end, looking for the
first occurrence where the regex pattern matches.
## *property* match\_fn
Match function to use for the regex evaluator.
# Score
Source: https://docs.fiddler.ai/sdk-api/evals/score
A single output of an evaluator.
A single output of an evaluator.
## name
## evaluator\_name
## value
## label
## status
## reasoning
## error\_reason
## error\_message
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# ScoreStatus
Source: https://docs.fiddler.ai/sdk-api/evals/score-status
The status of a score.
The status of a score.
**Values:**
* `SUCCESS` - SUCCESS
* `FAILED` - FAILED
* `SKIPPED` - SKIPPED
# Sentiment
Source: https://docs.fiddler.ai/sdk-api/evals/sentiment
Evaluator to assess text sentiment using Fiddler's sentiment analysis model.
Evaluator to assess text sentiment using Fiddler's sentiment analysis model.
The Sentiment evaluator uses Fiddler's implementation of the cardiffnlp/twitter-roberta-base-sentiment-latest
model to evaluate the sentiment polarity of text content. This evaluator helps identify
the emotional tone and attitude expressed in text, providing both sentiment labels and
confidence scores for sentiment classification.
Key Features:
* **Sentiment Classification**: Evaluates text for positive, negative, or neutral sentiment
* **Dual Score Output**: Returns both sentiment label and probability confidence
* **Fiddler Integration**: Leverages Fiddler's optimized sentiment evaluation model
* **Multi-Score Output**: Returns both sentiment label and probability scores
Sentiment Categories Evaluated:
* **sentiment**: The predicted sentiment label (positive, negative, neutral)
* **sentiment\_prob**: Probability score (0.0-1.0) for the predicted sentiment
Use Cases:
* **Social Media Monitoring**: Analyzing sentiment in tweets, posts, and comments
* **Customer Feedback Analysis**: Understanding customer satisfaction and opinions
* **Brand Monitoring**: Tracking public sentiment about products or services
* **Content Moderation**: Identifying emotionally charged or problematic content
* **Market Research**: Analyzing public opinion and sentiment trends
Scoring Logic:
The sentiment evaluation provides two complementary scores:
* **sentiment**: The predicted sentiment label
> * "positive": Text expresses positive emotions or opinions
> * "negative": Text expresses negative emotions or opinions
> * "neutral": Text expresses neutral or balanced sentiment
* **sentiment\_prob**: Confidence score (0.0-1.0) for the prediction
* **0.0-0.3**: Low confidence in sentiment prediction
* **0.3-0.7**: Medium confidence in sentiment prediction
* **0.7-1.0**: High confidence in sentiment prediction
## Parameters
* **text** (*str*) – The text content to evaluate for sentiment.
* **score\_name\_prefix** (*str* *|* *None*)
* **score\_fn\_kwargs\_mapping** (*ScoreFnKwargsMappingType* *|* *None*)
## Returns
A list of Score objects containing:
* sentiment: Score object with sentiment label (positive/negative/neutral)
* sentiment\_prob: Score object with probability score (0.0-1.0)
## Raises
**ValueError** – If the text is empty or None, or if no scores are returned from the API.
## Example
```python theme={null}
from fiddler_evals.evaluators import Sentiment
evaluator = Sentiment()
```
```python theme={null}
# Positive sentiment
scores = evaluator.score("I love this product! It's amazing!")
print(f"Sentiment: {scores[0].label}")
print(f"Confidence: {scores[1].value}")
# Sentiment: positive
# Confidence: 0.95
# Negative sentiment
negative_scores = evaluator.score("This is terrible and disappointing!")
print(f"Sentiment: {negative_scores[0].label}")
print(f"Confidence: {negative_scores[1].value}")
# Sentiment: negative
# Confidence: 0.88
# Neutral sentiment
neutral_scores = evaluator.score("The weather is okay today.")
print(f"Sentiment: {neutral_scores[0].label}")
print(f"Confidence: {neutral_scores[1].value}")
# Sentiment: neutral
# Confidence: 0.72
# Filter based on sentiment and confidence
if scores[0].label == "positive" and scores[1].value > 0.8:
print("High confidence positive sentiment detected")
```
This evaluator is optimized for social media and informal text analysis using
the cardiffnlp/twitter-roberta-base-sentiment-latest model. It performs best on
short, conversational text similar to Twitter posts. For formal or academic text,
consider using specialized sentiment analysis models. The dual-score output
provides both categorical classification and confidence assessment for robust
sentiment analysis workflows.
## name *= 'sentiment\_analysis'*
## score()
Score the sentiment of text content.
### Parameters
The text content to evaluate for sentiment.
### Returns
A list of Score objects for sentiment label and probability.
# TopicClassification
Source: https://docs.fiddler.ai/sdk-api/evals/topic-classification
Evaluator to classify text topics using Fiddler's zero-shot topic classification model.
Evaluator to classify text topics using Fiddler's zero-shot topic classification model.
The TopicClassification evaluator uses Fiddler's implementation of the mortizlaurer/roberta-base-zeroshot-v2-0-c
model to classify text content into predefined topic categories. This evaluator helps identify
the main subject matter or theme of text content, providing both topic labels and
confidence scores for topic classification.
Key Features:
* **Topic Classification**: Classifies text into predefined topic categories
* **Dual Score Output**: Returns both topic label and probability confidence
* **Zero-Shot Model**: Uses mortizlaurer/roberta-base-zeroshot-v2-0-c for flexible topic classification
* **Multi-Score Output**: Returns both topic name and probability scores
Topic Categories Evaluated:
* **top\_topic**: The predicted topic name from the provided topics list
* **top\_topic\_prob**: Probability score (0.0-1.0) for the predicted topic
Use Cases:
* **Content Categorization**: Automatically organizing content by topic
* **Document Classification**: Sorting documents by subject matter
* **News Analysis**: Categorizing news articles by topic
* **Customer Support**: Routing tickets by topic or issue type
* **Content Moderation**: Identifying content themes for policy enforcement
Scoring Logic:
The topic classification provides two complementary scores:
* **top\_topic**: The predicted topic name from the provided topics list
> * Selected from the topics provided during initialization
> * Represents the most relevant topic for the input text
* **top\_topic\_prob**: Confidence score (0.0-1.0) for the prediction
* **0.0-0.3**: Low confidence in topic prediction
* **0.3-0.7**: Medium confidence in topic prediction
* **0.7-1.0**: High confidence in topic prediction
## Parameters
List of topic categories to classify text into.
## Returns
A list of Score objects containing:
* top\_topic: Score object with predicted topic name
* top\_topic\_prob: Score object with probability score (0.0-1.0)
## Raises
**ValueError** – If the text is empty or None, or if no scores are returned from the API.
## Example
```python theme={null}
from fiddler_evals.evaluators import TopicClassification
evaluator = TopicClassification(topics=["technology", "sports", "politics", "entertainment"])
```
```python theme={null}
# Technology topic
scores = evaluator.score("The new AI model shows promising results in natural language processing.")
print(f"Topic: {scores[0].label}")
print(f"Confidence: {scores[1].value}")
# Topic: technology
# Confidence: 0.92
# Sports topic
sports_scores = evaluator.score("The team won the championship with an amazing performance!")
print(f"Topic: {sports_scores[0].label}")
print(f"Confidence: {sports_scores[1].value}")
# Topic: sports
# Confidence: 0.88
# Politics topic
politics_scores = evaluator.score("The new policy will affect millions of citizens.")
print(f"Topic: {politics_scores[0].label}")
print(f"Confidence: {politics_scores[1].value}")
# Topic: politics
# Confidence: 0.85
# Filter based on topic and confidence
if scores[0].label == "technology" and scores[1].value > 0.8:
print("High confidence technology topic detected")
```
This evaluator uses zero-shot classification, meaning it can classify text into
any set of topics provided during initialization without requiring training data
for those specific topics. The mortizlaurer/roberta-base-zeroshot-v2-0-c model
is particularly effective for general-purpose topic classification across
diverse domains. The dual-score output provides both categorical classification
and confidence assessment for robust topic analysis workflows.
## name *= 'topic\_classification'*
Initialize the TopicClassification evaluator.
### Parameters
List of topic categories to classify text into.
### Raises
**ValueError** – If the topics are empty or None.
## score()
Score the topic classification of text content.
### Parameters
The text content to evaluate for topic classification.
### Returns
A list of Score objects for topic name and probability.
# Overview
Source: https://docs.fiddler.ai/sdk-api/index
Complete SDK documentation and REST API reference for Fiddler AI Observability Platform.
## 🐍 Python Client SDK
### [Python Client SDK](/sdk-api/python-client/connection)
Official Python SDK for comprehensive ML and LLM observability - monitor traditional ML models and LLM applications.
**Key Features:**
* Model onboarding and schema definition
* Production event publishing (batch and streaming)
* Baseline dataset management
* Alert configuration
* Custom metrics and segments
**Use Cases:**
* ML model monitoring (drift, performance, data quality)
* Production data ingestion
* Creating monitoring dashboards
* Configuring alerts for model issues
[**View Full Documentation →**](/sdk-api/python-client/connection)
[**View Usage Guides →**](/developers/python-client-guides/installation-and-setup)
***
## 🎯 Agentic AI SDKs
SDKs for monitoring, evaluating, and testing LLM applications and AI agents.
### [Fiddler Evals SDK](/sdk-api/evals/evaluate)
Evaluate and test LLM outputs with built-in and custom metrics.
**Key Features:**
* Pre-built evaluators (faithfulness, toxicity, coherence, etc.)
* Custom evaluation functions
* Experiment tracking and comparison
* Dataset management for test sets
**Use Cases:**
* LLM output quality assessment
* A/B testing prompts and models
* Regression testing for LLM changes
* Custom evaluation metrics
**Quick Start:**
```python theme={null}
import fiddler as fdl
# Initialize evaluator
evaluator = fdl.AnswerRelevance()
# Run evaluation
result = evaluator.evaluate(
question="What is Fiddler?",
answer="Fiddler is an AI observability platform."
)
```
[**View Full Documentation →**](/sdk-api/evals/evaluate)
***
### [Fiddler OTel SDK](/sdk-api/otel/fiddler-client)
Framework-agnostic OpenTelemetry instrumentation for any Python LLM or agent application. This is the foundation the LangChain, LangGraph, and Strands SDKs build on, and the canonical source for Fiddler's span attributes and conversation tracking.
**Key Features:**
* Framework-agnostic tracing for any Python LLM or agent code
* `@trace` decorator and manual span wrappers (chain, generation, tool)
* Conversation and session attribute tracking
* Canonical Fiddler span attributes and span types
* Custom span processor and JSONL capture for offline analysis
**Use Cases:**
* Instrumenting apps not built on LangChain, LangGraph, or Strands
* Adding spans around arbitrary functions and workflows
* Shared tracing primitives across Fiddler's framework SDKs
**Quick Start:**
```python theme={null}
from fiddler_otel import FiddlerClient, trace
# Initialize once at startup (see FiddlerClient reference for options)
client = FiddlerClient()
# Capture a Fiddler span around any function
@trace
def generate_answer(question: str) -> str:
return call_llm(question)
```
[**View Full Documentation →**](/sdk-api/otel/fiddler-client)
***
### [Fiddler LangGraph SDK](/sdk-api/langgraph/fiddler-client)
[](https://pypi.org/project/fiddler-langgraph/)
Monitor LangGraph agents with distributed tracing and observability.
**Key Features:**
* Automatic LangGraph instrumentation
* Distributed tracing for agent workflows
* Span attributes for nodes and edges
* Conversation and session tracking
**Use Cases:**
* Debugging multi-step agent workflows
* Performance analysis of agent chains
* Monitoring production LangGraph applications
* Understanding agent decision paths
**Quick Start:**
```python theme={null}
from fiddler.langchain import LangGraphInstrumentor
# Instrument your LangGraph app
instrumentor = LangGraphInstrumentor()
instrumentor.instrument()
# Your LangGraph code runs normally
# Traces are automatically sent to Fiddler
```
[**View Full Documentation →**](/sdk-api/langgraph/fiddler-client)
***
### [Fiddler LangChain SDK](/sdk-api/langchain/fiddler-lang-chain-instrumentor)
Instrument [LangChain V1](https://docs.langchain.com/oss/python/langchain/overview) agents (the `create_agent` API and middleware pattern) and export OpenTelemetry traces to Fiddler.
**Key Features:**
* Automatic instrumentation of LangChain V1 agents
* Drop-in agent middleware emitting LLM, tool, and chain spans
* Retrieval context attachment via `set_llm_context`
* Span and session attribute helpers
**Use Cases:**
* Monitoring production LangChain agents
* Debugging multi-step chains and tool calls
* Tracking retrieval context on LLM spans
**Quick Start:**
```python theme={null}
from fiddler_langchain import FiddlerLangChainInstrumentor
# Call once before creating your agents
FiddlerLangChainInstrumentor().instrument()
# Agents created with langchain.agents.create_agent are now traced
```
[**View Full Documentation →**](/sdk-api/langchain/fiddler-lang-chain-instrumentor)
***
### [Fiddler Strands SDK](/sdk-api/strands/strands-agent-instrumentor)
[](https://pypi.org/project/fiddler-strands/)
Monitor Strands Agents with native instrumentation.
**Key Features:**
* Strands Agent instrumentation
* Session and conversation tracking
* Span attributes for agent actions
* Integration with Fiddler platform
**Use Cases:**
* Monitoring Strands production agents
* Debugging Strands Agent workflows
* Tracking agent performance metrics
* Session-based analysis
**Quick Start:**
```python theme={null}
from fiddler.strands import StrandsAgentInstrumentor
# Instrument Strands Agent
instrumentor = StrandsAgentInstrumentor(
model_id="my-strands-agent"
)
instrumentor.instrument()
```
[**View Full Documentation →**](/sdk-api/strands/strands-agent-instrumentor)
***
### [Fiddler OTel JS SDK](/sdk-api/otel-js/fiddler-client)
OpenTelemetry instrumentation for Fiddler AI observability in TypeScript and JavaScript applications. Captures LLM traces, conversation context, and span attributes.
**Key Features:**
* OpenTelemetry tracing for TypeScript and JavaScript apps
* Isolated tracer provider with OTLP HTTP export to Fiddler
* Manual span helpers (agent, generation, tool)
* Span attribute and token-usage conventions
**Use Cases:**
* Instrumenting Node.js LLM and agent applications
* Capturing traces from non-LangChain TypeScript code
* Adding spans around arbitrary functions and workflows
**Quick Start:**
```typescript theme={null}
import { FiddlerClient } from '@fiddler-ai/otel';
// Initialize once at startup (see FiddlerClient reference for options)
const client = new FiddlerClient();
```
[**View Full Documentation →**](/sdk-api/otel-js/fiddler-client)
***
### [Fiddler LangGraph JS SDK](/sdk-api/langgraph-js/lang-graph-instrumentor)
OpenTelemetry-based instrumentation for LangGraph JS applications. Mirrors the Python `fiddler-langgraph` SDK API for agentic workflows.
**Key Features:**
* Automatic instrumentation of LangGraph JS via the callback manager
* Trace capture for agentic workflows
* Conversation and session attribute tracking
* LLM context helpers
**Use Cases:**
* Monitoring production LangGraph JS agents
* Debugging agent workflows in Node.js
* Conversation- and session-level analysis
**Quick Start:**
```typescript theme={null}
import { LangGraphInstrumentor } from '@fiddler-ai/langgraph';
// Instrument once at startup; LangGraph runs export to Fiddler
new LangGraphInstrumentor().instrument();
```
[**View Full Documentation →**](/sdk-api/langgraph-js/lang-graph-instrumentor)
***
### [Fiddler LangChain JS SDK](/sdk-api/langchain-js/lang-chain-instrumentor)
LangChain JS instrumentation for Fiddler AI observability. Re-exports the callback handler and instrumentor from `@fiddler-ai/langgraph` under a LangChain-branded API, with no code changes to existing chains.
**Key Features:**
* Automatic trace capture for LangChain JS applications
* No changes to existing chains
* Conversation and session attribute tracking
* LLM context helpers
**Use Cases:**
* Monitoring production LangChain JS applications
* Adding observability to existing chains
* Conversation- and session-level analysis
**Quick Start:**
```typescript theme={null}
import { LangChainInstrumentor } from '@fiddler-ai/langchain';
// Instrument once; existing chains are traced with no further changes
new LangChainInstrumentor().instrument();
```
[**View Full Documentation →**](/sdk-api/langchain-js/lang-chain-instrumentor)
***
## 🌐 REST API
### [REST API Reference](/sdk-api/rest-api)
Complete HTTP API documentation for programmatic access to the Fiddler platform.
**Use Cases:**
* Non-Python integrations (Java, Go, JavaScript, etc.)
* Custom CI/CD pipelines
* Integration with existing monitoring systems
* Webhook-based automation
**Quick Start (cURL):**
```bash theme={null}
# Publish events to Fiddler
curl -X POST https://app.fiddler.ai/api/v1/events
-H "Authorization: Bearer fid_..."
-H "Content-Type: application/json"
-d '{
"project": "fraud-detection",
"model": "fraud_model_v1",
"events": [...]
}'
```
[**View Full REST API Documentation →**](/sdk-api/rest-api)
**API Guides:**
* [Alert Rules](/sdk-api/rest-api/alert-rules)
* [Applications](/sdk-api/rest-api/applications)
* [Attributes](/sdk-api/rest-api/attributes)
* [Baselines](/sdk-api/rest-api/baseline)
* [Custom Metrics](/sdk-api/rest-api/custom-metrics)
* [Environments](/sdk-api/rest-api/environment)
* [Evals](/sdk-api/rest-api/evals)
* [Evaluation](/sdk-api/rest-api/evaluation)
* [Evaluator Rules](/sdk-api/rest-api/evaluator-rules)
* [Evaluators](/sdk-api/rest-api/evaluators)
* [Events](/sdk-api/rest-api/events)
* [Experiments](/sdk-api/rest-api/experiments)
* [File Upload](/sdk-api/rest-api/file-upload)
* [FQL Expressions](/sdk-api/rest-api/fql-expressions)
* [GenAI Alert Rules](/sdk-api/rest-api/genai-alert-rules)
* [Jobs](/sdk-api/rest-api/jobs)
* [LLM Gateway](/sdk-api/rest-api/llm-gateway)
* [Models](/sdk-api/rest-api/model)
* [Projects](/sdk-api/rest-api/projects)
* [Queries](/sdk-api/rest-api/queries)
* [Segments](/sdk-api/rest-api/segments)
* [Server Info](/sdk-api/rest-api/server-info)
* [Spans](/sdk-api/rest-api/spans)
* [User Access Keys](/sdk-api/rest-api/user-access-keys)
* [Users](/sdk-api/rest-api/users)
### [Guardrails API Reference](/sdk-api/guardrails-api-reference)
API endpoints for Fiddler guardrails.
***
## 🚀 Getting Started
### Choose Your SDK
| Your Use Case | Recommended SDK |
| -------------------------------------------- | --------------------------------------------------------------------------- |
| **Monitor ML/LLM models and platform admin** | [Python Client SDK](/sdk-api/python-client/connection) |
| **Evaluate and test LLM outputs** | [Fiddler Evals SDK](/sdk-api/evals/evaluate) |
| **Instrument any Python LLM/agent app** | [Fiddler OTel SDK](/sdk-api/otel/fiddler-client) |
| **Monitor LangGraph (Python) agents** | [Fiddler LangGraph SDK](/sdk-api/langgraph/fiddler-client) |
| **Monitor LangChain (Python) agents** | [Fiddler LangChain SDK](/sdk-api/langchain/fiddler-lang-chain-instrumentor) |
| **Monitor Strands agents** | [Fiddler Strands SDK](/sdk-api/strands/strands-agent-instrumentor) |
| **Instrument any TypeScript/JS app** | [Fiddler OTel JS SDK](/sdk-api/otel-js/fiddler-client) |
| **Monitor LangGraph JS apps** | [Fiddler LangGraph JS SDK](/sdk-api/langgraph-js/lang-graph-instrumentor) |
| **Monitor LangChain JS apps** | [Fiddler LangChain JS SDK](/sdk-api/langchain-js/lang-chain-instrumentor) |
| **Language-agnostic HTTP integration** | [REST API](/sdk-api/rest-api) |
### Installation
**Python SDKs:**
```bash theme={null}
# Python Client SDK
pip install fiddler-client
# Evals SDK
pip install fiddler-evals
# OTel SDK (framework-agnostic instrumentation)
pip install fiddler-otel
# LangGraph SDK
pip install fiddler-langgraph
# LangChain SDK
pip install fiddler-langchain
# Strands SDK
pip install fiddler-strands
```
Fiddler follows the Python Software Foundation's support schedule for Python versions. See the [Python version support policy](/reference/python-support-policy) for details.
**JavaScript / TypeScript SDKs:**
```bash theme={null}
# OTel JS SDK
npm install @fiddler-ai/otel
# LangGraph JS SDK
npm install @fiddler-ai/langgraph
# LangChain JS SDK
npm install @fiddler-ai/langchain
```
**REST API:** No installation required - use any HTTP client.
***
## 📚 Related Documentation
* [**Developer Guides**](/developers/quick-starts/get-started-in-less-than-10-minutes) - Quick starts and tutorials
* [**Integrations**](/integrations) - Connect with your ML stack
* [**Product Documentation**](/) - Platform features and concepts
***
## 💡 Common Workflows
### ML Model & LLM App Monitoring Workflow
1. Install [Python Client SDK](/sdk-api/python-client/connection)
2. Define [model schema](/sdk-api/python-client/model-schema)
3. Upload [baseline dataset](/sdk-api/python-client/baseline)
4. [Publish production events](/sdk-api/rest-api/events)
5. Configure [alerts](/sdk-api/python-client/alert-rule)
### LLM Experiments Workflow
1. Install [Fiddler Evals SDK](/sdk-api/evals/evaluate)
2. Create a test dataset with the [Dataset API](/sdk-api/evals/dataset)
3. Define evaluators ([built-in](/sdk-api/evals/evaluator) or [custom](/sdk-api/evals/eval-fn))
4. Run [experiments](/sdk-api/evals/experiment) and analyze results
### Agent Monitoring Workflow
1. Install the SDK for your framework — [LangGraph](/sdk-api/langgraph/fiddler-client), [LangChain](/sdk-api/langchain/fiddler-lang-chain-instrumentor), [Strands](/sdk-api/strands/strands-agent-instrumentor), or a JavaScript SDK ([OTel JS](/sdk-api/otel-js/fiddler-client), [LangGraph JS](/sdk-api/langgraph-js/lang-graph-instrumentor), [LangChain JS](/sdk-api/langchain-js/lang-chain-instrumentor))
2. Instrument your agent application
3. Deploy to production
4. View traces and analytics in the Fiddler platform
***
## 📖 Additional Resources
* [**GitHub Examples**](https://github.com/fiddler-labs/fiddler-examples) - Sample code and notebooks
* [**SDK Changelog**](/changelog/python-sdk) - Latest SDK updates
* [**Support Portal**](mailto:support@fiddler.ai) - Enterprise support
* [**Community**](https://fiddler-community.slack.com) - Join our Slack community
# addSessionAttributes
Source: https://docs.fiddler.ai/sdk-api/langchain-js/add-session-attributes
addSessionAttributes
Key-value pairs to add to the session context.
# CallbackManagerModule
Source: https://docs.fiddler.ai/sdk-api/langchain-js/callback-manager-module
Shape of the `@langchain/core/callbacks/manager` module export.
Shape of the `@langchain/core/callbacks/manager` module export.
Used by LangGraphInstrumentor.manuallyInstrument when the caller
provides the module reference directly (e.g. in bundled environments
where dynamic `import()` does not work).
## Properties
### `CallbackManager: CallbackManagerLike & Record`
The CallbackManager class with configureSync or configure methods.
# extractTextContent
Source: https://docs.fiddler.ai/sdk-api/langchain-js/extract-text-content
extractTextContent
The message content in any supported format.
The extracted text as a single string.
# FiddlerCallbackHandler
Source: https://docs.fiddler.ai/sdk-api/langchain-js/fiddler-callback-handler
LangChain callback handler that creates Fiddler OTel spans for every
LangChain callback handler that creates Fiddler OTel spans for every
chain, LLM, tool, agent, and retriever lifecycle event.
Can be used directly by passing it in the `callbacks` array of a
LangChain/LangGraph invocation, or automatically injected via
LangGraphInstrumentor.
## Properties
### `name: string`
## Methods
### `constructor(client: FiddlerClient)`
An initialized FiddlerClient used to create spans.
### `handleAgentAction(action: AgentAction, runId: string)`
Called when an agent is about to execute an action,
with the action and the run ID.
The agent action containing tool name and input.
Run identifier to look up the span.
### `handleAgentEnd(action: AgentFinish, runId: string)`
Called when an agent finishes execution, before it exits.
with the final output and the run ID.
The agent finish containing return values.
Run identifier to look up the span.
### `handleChainEnd(outputs: ChainValues, runId: string)`
Called at the end of a Chain run, with the outputs and the run ID.
Output values produced by the chain.
Run identifier to look up the span.
### `handleChainError(error: Error, runId: string)`
Called if a Chain run encounters an error
The error that occurred.
Run identifier to look up the span.
### `handleChainStart(serialized: Serialized, inputs: ChainValues, runId: string, parentRunId?: string, tags?: string[], metadata?: Record)`
Called at the start of a Chain run, with the chain name and inputs
and the run ID.
Serialized chain/runnable metadata.
Input values passed to the chain.
Unique run identifier.
Parent run ID for nesting.
LangChain tags (used for LangGraph node detection).
LangChain metadata (agent name, node name, etc.).
### `handleChatModelStart(serialized: Serialized, messages: BaseMessage[][], runId: string, parentRunId?: string, extraParams?: Record, _tags?: string[], _metadata?: Record, runName?: string)`
Called at the start of a Chat Model run, with the prompt(s)
and the run ID.
Serialized chat model metadata.
2D array of chat messages (outer = batches).
Unique run identifier.
Parent run ID for nesting.
Additional invocation parameters.
Optional explicit run name.
### `handleLLMEnd(output: LLMResult, runId: string)`
Called at the end of an LLM/ChatModel run, with the output and the run ID.
The LLMResult containing generations and usage.
Run identifier to look up the span.
### `handleLLMError(error: Error, runId: string)`
Called if an LLM/ChatModel run encounters an error
The error that occurred.
Run identifier to look up the span.
### `handleLLMStart(serialized: Serialized, prompts: string[], runId: string, parentRunId?: string, _extraParams?: Record, _tags?: string[], _metadata?: Record, runName?: string)`
Called at the start of an LLM or Chat Model run, with the prompt(s)
and the run ID.
Serialized LLM metadata.
Array of prompt strings.
Unique run identifier.
Parent run ID for nesting.
Optional explicit run name.
### `handleRetrieverEnd(documents: { metadata?: unknown; pageContent?: string }[], runId: string)`
Called when a retriever returns documents.
Sets the concatenated page content as the tool output.
Array of retrieved documents with `pageContent`.
Run identifier to look up the span.
### `handleRetrieverError(error: Error, runId: string)`
Called when a retriever throws an error.
The error that occurred.
Run identifier to look up the span.
### `handleRetrieverStart(serialized: Serialized, query: string, runId: string, parentRunId?: string)`
Called when a retriever begins fetching documents.
Creates a tool-type span with the retriever name and query.
Serialized retriever metadata.
The search query string.
Unique run identifier.
Parent run ID for nesting.
### `handleToolEnd(output: string, runId: string)`
Called at the end of a Tool run, with the tool output and the run ID.
Tool output (string or ToolMessage-like object).
Run identifier to look up the span.
### `handleToolError(error: Error, runId: string)`
Called if a Tool run encounters an error
The error that occurred.
Run identifier to look up the span.
### `handleToolStart(serialized: Serialized, input: string, runId: string, parentRunId?: string, _tags?: string[], _metadata?: Record, runName?: string)`
Called at the start of a Tool run, with the tool name and input
and the run ID.
Serialized tool metadata.
The input string or structured input for the tool.
Unique run identifier.
Parent run ID for nesting.
Optional explicit run name.
# formatMessages
Source: https://docs.fiddler.ai/sdk-api/langchain-js/format-messages
formatMessages
Array of message objects or strings.
A newline-separated string of formatted messages.
# getConversationId
Source: https://docs.fiddler.ai/sdk-api/langchain-js/get-conversation-id
getConversationId
The active conversation ID, or `undefined` if none is set.
# getLlmContext
Source: https://docs.fiddler.ai/sdk-api/langchain-js/get-llm-context
getLlmContext
A LangChain chat model instance or invocation params
object.
The attached context string, or `undefined` if none is set.
# getProviderFromModel
Source: https://docs.fiddler.ai/sdk-api/langchain-js/get-provider-from-model
getProviderFromModel
Model identifier (e.g., "gpt-4", "claude-3-opus", "gemini-1.5-pro")
Provider name (e.g., "openai", "anthropic", "google")
# getSessionAttributes
Source: https://docs.fiddler.ai/sdk-api/langchain-js/get-session-attributes
getSessionAttributes
A copy of the active session attributes map.
# Introduction
Source: https://docs.fiddler.ai/sdk-api/langchain-js/index
Complete API reference for fiddler-langchain-js
[](https://www.npmjs.com/package/@fiddler-ai/langchain)
## Overview
The `@fiddler-ai/langchain` package provides LangChain JS instrumentation
for Fiddler AI observability. It re-exports the callback handler and
instrumentor from `@fiddler-ai/langgraph` under a LangChain-branded API,
enabling automatic trace capture for LangChain applications with zero
code changes to existing chains.
```bash theme={null}
npm install @fiddler-ai/langchain
```
```typescript theme={null}
import { LangChainInstrumentor, setConversationId } from '@fiddler-ai/langchain';
```
## Components
### Core
The full LangChain JS instrumentation surface, re-exported from `@fiddler-ai/langgraph` under a LangChain-branded API. `LangChainInstrumentor` is the one-time setup and `FiddlerCallbackHandler` emits spans; the `setConversationId`, `setLlmContext`, and `addSessionAttributes` families propagate conversation, session, and retrieval context, while the remaining utilities handle message formatting and serialization.
* [addSessionAttributes](/sdk-api/langchain-js/add-session-attributes)
* [CallbackManagerModule](/sdk-api/langchain-js/callback-manager-module)
* [extractTextContent](/sdk-api/langchain-js/extract-text-content)
* [FiddlerCallbackHandler](/sdk-api/langchain-js/fiddler-callback-handler)
* [formatMessages](/sdk-api/langchain-js/format-messages)
* [getConversationId](/sdk-api/langchain-js/get-conversation-id)
* [getLlmContext](/sdk-api/langchain-js/get-llm-context)
* [getProviderFromModel](/sdk-api/langchain-js/get-provider-from-model)
* [getSessionAttributes](/sdk-api/langchain-js/get-session-attributes)
* [LangChainInstrumentor](/sdk-api/langchain-js/lang-chain-instrumentor)
* [runWithContext](/sdk-api/langchain-js/run-with-context)
* [safeStringify](/sdk-api/langchain-js/safe-stringify)
* [setConversationId](/sdk-api/langchain-js/set-conversation-id)
* [setLlmContext](/sdk-api/langchain-js/set-llm-context)
* [truncate](/sdk-api/langchain-js/truncate)
# LangChainInstrumentor
Source: https://docs.fiddler.ai/sdk-api/langchain-js/lang-chain-instrumentor
Auto-instruments LangChain/LangGraph by monkey-patching
Auto-instruments LangChain/LangGraph by monkey-patching
`CallbackManager._configureSync` (or `.configure`) to inject a
FiddlerCallbackHandler into every invocation.
**Node.js only** — requires `@langchain/core` as a peer dependency and
uses `node:async_hooks` (via context) for context propagation.
Uses a WeakSet to track patched modules so frozen ESM exports
are handled correctly.
## Methods
### `constructor(client: FiddlerClient)`
An initialized FiddlerClient used by the
callback handler to create spans.
### `async instrument(): Promise`
Auto-instrument by dynamically importing `@langchain/core/callbacks/manager`.
### `isInstrumented(): boolean`
Check whether the instrumentor has successfully patched a
`CallbackManager` module.
`true` if auto-instrumentation is active.
### `manuallyInstrument(module: CallbackManagerModule)`
Manually instrument a specific `@langchain/core/callbacks/manager` module.
This is the recommended approach for bundled environments.
The imported `@langchain/core/callbacks/manager` module.
### `uninstrument()`
Remove the patch and restore the original `CallbackManager` behavior.
Safe to call even if the instrumentor was never activated.
# runWithContext
Source: https://docs.fiddler.ai/sdk-api/langchain-js/run-with-context
runWithContext
The function to execute within the new context.
The return value of `fn`.
# safeStringify
Source: https://docs.fiddler.ai/sdk-api/langchain-js/safe-stringify
safeStringify
The value to serialize.
Maximum allowed length (defaults to
MAX\_CONTENT\_LENGTH).
A JSON string (possibly truncated), or empty string.
# setConversationId
Source: https://docs.fiddler.ai/sdk-api/langchain-js/set-conversation-id
setConversationId
Unique conversation or session identifier.
# setLlmContext
Source: https://docs.fiddler.ai/sdk-api/langchain-js/set-llm-context
setLlmContext
A LangChain chat model instance (e.g. `ChatOpenAI`).
The context string to attach.
# truncate
Source: https://docs.fiddler.ai/sdk-api/langchain-js/truncate
truncate
The string to truncate.
Maximum allowed length (defaults to
MAX\_CONTENT\_LENGTH).
The original string if short enough, or a truncated copy.
# add_session_attributes
Source: https://docs.fiddler.ai/sdk-api/langchain/add-session-attributes
Add a session-level attribute that appears on all spans in the current context.
Re-exported from [`fiddler_otel.attributes.add_session_attributes`](/sdk-api/otel/add-session-attributes).
Add a session-level attribute that appears on all spans in the current context.
The attribute is stored in a ContextVar that
`FiddlerSpanProcessor` reads when spans
are started. It is emitted as `fiddler.session.user.{key}` on every span
created in the current thread or async coroutine, and is automatically
propagated from parent spans to child spans.
## Parameters
Logical attribute key. Will appear as `fiddler.session.user.{key}`
in span attributes.
Attribute value. Accepts `str`, `int`, `float`, or `bool`.
Numeric values (`int`/`float`) are stored as numeric attributes and will
appear in the `ValueFloat` column in ClickHouse, enabling range-based
filtering and metrics. String and boolean values appear in the `ValueString`
column and support categorical filtering (booleans are stored as `"true"`
or `"false"`).
## Returns
None Example: default from fiddler\_otel import add\_session\_attributes add\_session\_attributes("user\_id", "user\_12345") add\_session\_attributes("environment", "production") add\_session\_attributes("priority", 7) add\_session\_attributes("score", 0.95)
See the [canonical reference](/sdk-api/otel/add-session-attributes) for the full description and examples.
# add_span_attributes
Source: https://docs.fiddler.ai/sdk-api/langchain/add-span-attributes
Add custom span-level attributes to a specific LangChain component's metadata.
Add custom span-level attributes to a specific LangChain component's metadata.
Attributes are stored under the Fiddler metadata key on the component and
read by [`FiddlerAgentMiddleware`](/sdk-api/langchain/fiddler-agent-middleware)
when creating spans. Unlike `add_session_attributes()`, these are
scoped to individual components and only applied to spans created for
that component (e.g. a particular model, retriever, or tool).
## Parameters
The LangChain component to annotate (modified in place).
Accepts `BaseLanguageModel`,
`BaseRetriever`, or
`BaseTool` instances.
Arbitrary key-value attributes. Keys matching known
`FiddlerSpanAttributes` values are set
directly; all others are prefixed with `fiddler.span.user.`.
## Returns
None Example: default from langchain\_openai import ChatOpenAI from fiddler\_langchain import add\_span\_attributes llm = ChatOpenAI(model="gpt-4o") add\_span\_attributes(llm, model\_tier="premium", use\_case="summarization")
# clear_llm_context
Source: https://docs.fiddler.ai/sdk-api/langchain/clear-llm-context
Remove LLM context from a language model instance.
Remove LLM context from a language model instance.
This is a convenience wrapper that calls `set_llm_context(llm, None)`.
Use it in multi-step agent workflows after a RAG retrieval step to
ensure subsequent non-RAG LLM calls (tool planning, routing, etc.) do
not carry stale context and do not trigger faithfulness evaluation.
The context is stored in the model's metadata and read by
[`FiddlerAgentMiddleware`](/sdk-api/langchain/fiddler-agent-middleware)
when creating LLM spans.
## Parameters
The language model instance or binding to clear context from.
## Raises
**TypeError** – If a `RunnableBinding` is provided but its bound
object is not a `BaseLanguageModel`.
## Returns
None Example: default from langchain\_openai import ChatOpenAI from fiddler\_langchain import set\_llm\_context, clear\_llm\_context llm = ChatOpenAI(model="gpt-4o") retrieved\_context = "Relevant documents joined as a single string..." rag\_prompt = "Answer the question using the context above." planning\_prompt = "Plan the next agent step." # Step 1: RAG retrieval -- attach context for faithfulness evaluation set\_llm\_context(llm, retrieved\_context) response = llm.invoke(rag\_prompt) # faithfulness evaluated # Step 2: Non-RAG planning -- clear context to skip faithfulness evaluation clear\_llm\_context(llm) plan = llm.invoke(planning\_prompt) # no faithfulness evaluation
## SEE ALSO
`set_llm_context()` – Set or clear context on a language model
instance. Passing `None` as the context value is equivalent to
calling `clear_llm_context()`.
# FiddlerAgentMiddleware
Source: https://docs.fiddler.ai/sdk-api/langchain/fiddler-agent-middleware
LangChain V1 middleware that instruments `create_agent` calls with Fiddler tracing.
LangChain V1 middleware that instruments `create_agent` calls with Fiddler tracing.
This middleware produces a clean, flat trace hierarchy for agents built with
`langchain.agents.create_agent`:
* One root **Agent** span (opened in `before_agent`, closed in `after_agent`)
* One **LLM** child span per model invocation (`wrap_model_call` / `awrap_model_call`)
* One **Tool** child span per tool invocation (`wrap_tool_call` / `awrap_tool_call`)
All spans are written to Fiddler's isolated OpenTelemetry context so they do not
interfere with any global tracer that may be active in the application.
Concurrent invocations (async) are fully isolated: Python's `ContextVar` ensures
each `agent.ainvoke()` task tracks its own root span independently.
Usage:
```python theme={null}
from langchain.agents import create_agent
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerAgentMiddleware
client = FiddlerClient(api_key="...", application_id="...", url="...")
agent = create_agent(
model="openai:gpt-4o-mini",
tools=[...],
middleware=[FiddlerAgentMiddleware(client=client, agent_name="my_agent")],
)
agent.invoke({"messages": [...]})
```
## Parameters
The `FiddlerClient` instance.
Label applied to the root Agent span. Defaults to `"agent"`.
## before\_agent()
Opens a root Agent span at the start of each agent invocation.
If the current context already has a Fiddler span (e.g. we are being invoked
as a sub-agent from a tool call), the new agent span is created as a child
of that span so the full multi-agent flow appears in a single trace.
### Returns
`dict[str, Any] | None`
## after\_agent()
Closes the root Agent span at the end of each agent invocation.
### Returns
`dict[str, Any] | None`
## wrap\_model\_call()
Wraps a synchronous model call with an LLM span.
### Returns
`ModelResponse`
## *async* awrap\_model\_call()
Wraps an asynchronous model call with an LLM span.
### Returns
`ModelResponse`
## wrap\_tool\_call()
Wraps a synchronous tool call with a Tool span.
### Returns
`Any`
## *async* awrap\_tool\_call()
Wraps an asynchronous tool call with a Tool span.
### Returns
`Any`
# FiddlerLangChainInstrumentor
Source: https://docs.fiddler.ai/sdk-api/langchain/fiddler-lang-chain-instrumentor
Auto-instrumentor for LangChain V1 agents.
Auto-instrumentor for LangChain V1 agents.
Monkey-patches `langchain.agents.create_agent` so that every agent
created after [`instrument()`](#instrument) is called receives a
[`FiddlerAgentMiddleware`](/sdk-api/langchain/fiddler-agent-middleware) automatically. Callers no longer need to
pass `middleware=[FiddlerAgentMiddleware(...)]` to each `create_agent`
call.
When a call to `create_agent` includes a `name` argument (e.g.
`name='flight_assistant'`), that name is used for the agent in traces.
Otherwise, no agent name is set (empty string), so the agent is not labeled
in the UI. If a call to `create_agent` already includes a
[`FiddlerAgentMiddleware`](/sdk-api/langchain/fiddler-agent-middleware) instance in its `middleware` list, the
instrumentor skips injection for that call so existing manual
configurations are preserved.
Note: Instrumentation persists for the lifetime of the application unless
explicitly removed by calling uninstrument(). Calling instrument() multiple
times is safe — it will not create duplicate middleware.
Usage:
```python theme={null}
import langchain.agents
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(api_key="...", application_id="...", url="...")
instrumentor = FiddlerLangChainInstrumentor(client=client)
instrumentor.instrument()
# Use the module attribute (not a `from ... import` alias) so the call goes
# through the patched version at run-time:
agent = langchain.agents.create_agent(model=..., tools=[...])
agent.invoke({"messages": [...]})
# Remove instrumentation when done:
instrumentor.uninstrument()
```
Note: `instrument()` patches the `langchain.agents` module attribute.
If you prefer `from langchain.agents import create_agent`, call
`instrument()` *before* that import so the local name is bound to the
patched wrapper.
## Parameters
The FiddlerClient instance. **Required**.
## Raises
**RuntimeError** – If the FiddlerClient tracer cannot be initialized.
## instrument()
Enable automatic instrumentation of LangChain V1 agents.
Activates the instrumentor by patching `langchain.agents.create_agent`
to automatically inject [`FiddlerAgentMiddleware`](/sdk-api/langchain/fiddler-agent-middleware) into all agent
instances created afterwards.
This method is idempotent — calling it multiple times has the same effect
as calling it once. After activation, all newly created agents will
automatically include Fiddler's middleware.
### Parameters
Additional instrumentation configuration (currently unused).
### Example
```python theme={null}
import langchain.agents
from fiddler_otel import FiddlerClient
from fiddler_langchain import FiddlerLangChainInstrumentor
client = FiddlerClient(api_key="...", application_id="...", url="...")
instrumentor = FiddlerLangChainInstrumentor(client=client)
instrumentor.instrument()
# Use module attribute so the patched version is called:
agent = langchain.agents.create_agent(model=model, tools=[...])
# Check if instrumentation is active
if instrumentor.is_instrumented_by_opentelemetry:
print("Instrumentation is active")
```
## uninstrument()
Disable automatic instrumentation and restore original behavior.
Deactivates the instrumentor by restoring the original
`langchain.agents.create_agent` function. Agents created after calling
this method will no longer have [`FiddlerAgentMiddleware`](/sdk-api/langchain/fiddler-agent-middleware)
automatically injected.
Note: This does not affect agents that were already created while
instrumentation was active — those will retain their middleware.
### Parameters
Additional uninstrumentation configuration (currently unused).
### Example
```python theme={null}
instrumentor = FiddlerLangChainInstrumentor(client=client)
instrumentor.instrument()
# Create some agents (automatically instrumented)
agent1 = create_agent(model=model, tools=[...])
# Deactivate instrumentation
instrumentor.uninstrument()
# New agents won't be instrumented
agent2 = create_agent(model=model, tools=[...])
# agent1 still has the middleware, agent2 does not
```
## instrumentation\_dependencies()
Returns the package dependencies required for this instrumentor.
### Returns
A collection of package dependency strings.
# Introduction
Source: https://docs.fiddler.ai/sdk-api/langchain/index
Complete API reference for fiddler-langchain
[](https://pypi.org/project/fiddler-langchain/)
## Overview
Complete API reference documentation for the `fiddler-langchain` package, which
instruments [LangChain V1](https://docs.langchain.com/oss/python/langchain/overview)
agents (the `langchain.agents.create_agent` API and middleware pattern) and
exports OpenTelemetry traces to Fiddler.
```bash theme={null}
pip install fiddler-langchain
```
```python theme={null}
from fiddler_langchain import FiddlerLangChainInstrumentor
```
## Components
### Attributes
Span and session attribute helpers. `add_span_attributes` attaches component-scoped attributes to a specific model, retriever, or tool. `add_session_attributes` and `set_conversation_id` are re-exported from `fiddler-otel` and propagate through every span in the current trace context.
* [add\_session\_attributes](/sdk-api/langchain/add-session-attributes)
* [add\_span\_attributes](/sdk-api/langchain/add-span-attributes)
* [set\_conversation\_id](/sdk-api/langchain/set-conversation-id)
### Tracing
Instrumentation entry points and LLM context helpers. `FiddlerLangChainInstrumentor` is the one-time setup that wires the Fiddler exporter into LangChain; `FiddlerAgentMiddleware` is a drop-in agent middleware that emits spans for LLM, tool, and chain steps; `set_llm_context` and `clear_llm_context` attach and remove retrieval context (e.g. retrieved documents) on the active model so it appears on its LLM spans.
* [clear\_llm\_context](/sdk-api/langchain/clear-llm-context)
* [FiddlerAgentMiddleware](/sdk-api/langchain/fiddler-agent-middleware)
* [FiddlerLangChainInstrumentor](/sdk-api/langchain/fiddler-lang-chain-instrumentor)
* [set\_llm\_context](/sdk-api/langchain/set-llm-context)
# set_conversation_id
Source: https://docs.fiddler.ai/sdk-api/langchain/set-conversation-id
Set the conversation ID for the current execution context.
Re-exported from [`fiddler_otel.attributes.set_conversation_id`](/sdk-api/otel/set-conversation-id).
Set the conversation ID for the current execution context.
The conversation ID is propagated to all spans created in the current
thread or async coroutine, allowing the Fiddler dashboard to filter and
display the full ordered sequence of operations for a single conversation.
This value persists until it is called again with a new ID.
## Parameters
Unique identifier for the conversation session.
## Returns
None Example: default import uuid from fiddler\_otel import set\_conversation\_id set\_conversation\_id(str(uuid.uuid4())) # All spans created in this thread or coroutine after this call # carry the same conversation\_id, until set\_conversation\_id is # called again with a new value.
See the [canonical reference](/sdk-api/otel/set-conversation-id) for the full description and examples.
# set_llm_context
Source: https://docs.fiddler.ai/sdk-api/langchain/set-llm-context
Set or clear additional context on a language model for inclusion in LLM spans.
Set or clear additional context on a language model for inclusion in LLM spans.
The context is stored in the model's metadata and read by
[`FiddlerAgentMiddleware`](/sdk-api/langchain/fiddler-agent-middleware) when
creating LLM spans. Supports both plain model instances and
`RunnableBinding`.
Passing `None` removes the context so that subsequent spans no longer
carry it. This is useful in multi-step agent workflows where context set
after a RAG step should not leak into later non-RAG LLM calls.
## Parameters
The language model (or binding) to attach context to.
The context string to attach, or `None` to clear.
## Raises
**TypeError** – If a `RunnableBinding` is provided but its bound
object is not a `BaseLanguageModel`.
## Returns
None Example: default from langchain\_openai import ChatOpenAI from fiddler\_langchain import set\_llm\_context llm = ChatOpenAI(model="gpt-4o") set\_llm\_context(llm, "User prefers concise responses") # Clear context before non-RAG steps set\_llm\_context(llm, None)
# addSessionAttributes
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/add-session-attributes
addSessionAttributes
Key-value pairs to add to the session context.
# CallbackManagerModule
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/callback-manager-module
Shape of the `@langchain/core/callbacks/manager` module export.
Shape of the `@langchain/core/callbacks/manager` module export.
Used by LangGraphInstrumentor.manuallyInstrument when the caller
provides the module reference directly (e.g. in bundled environments
where dynamic `import()` does not work).
## Properties
### `CallbackManager: CallbackManagerLike & Record`
The CallbackManager class with configureSync or configure methods.
# extractTextContent
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/extract-text-content
extractTextContent
The message content in any supported format.
The extracted text as a single string.
# FiddlerCallbackHandler
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/fiddler-callback-handler
LangChain callback handler that creates Fiddler OTel spans for every
LangChain callback handler that creates Fiddler OTel spans for every
chain, LLM, tool, agent, and retriever lifecycle event.
Can be used directly by passing it in the `callbacks` array of a
LangChain/LangGraph invocation, or automatically injected via
LangGraphInstrumentor.
## Properties
### `name: string`
## Methods
### `constructor(client: FiddlerClient)`
An initialized FiddlerClient used to create spans.
### `handleAgentAction(action: AgentAction, runId: string)`
Called when an agent decides to invoke a tool.
Updates the existing agent span with the chosen tool name and input.
The agent action containing tool name and input.
Run identifier to look up the span.
### `handleAgentEnd(action: AgentFinish, runId: string)`
Called when an agent produces its final answer.
Updates the existing agent span with the return values.
The agent finish containing return values.
Run identifier to look up the span.
### `handleChainEnd(outputs: ChainValues, runId: string)`
Called when a chain/graph node completes successfully.
Output values produced by the chain.
Run identifier to look up the span.
### `handleChainError(error: Error, runId: string)`
Called when a chain/graph node throws an error.
The error that occurred.
Run identifier to look up the span.
### `handleChainStart(serialized: Serialized, inputs: ChainValues, runId: string, parentRunId?: string, tags?: string[], metadata?: Record)`
Called when a LangChain chain/graph node begins execution.
Creates a chain or `langgraph_node` span with input and agent
identity attributes.
Serialized chain/runnable metadata.
Input values passed to the chain.
Unique run identifier.
Parent run ID for nesting.
LangChain tags (used for LangGraph node detection).
LangChain metadata (agent name, node name, etc.).
### `handleChatModelStart(serialized: Serialized, messages: BaseMessage[][], runId: string, parentRunId?: string, extraParams?: Record, _tags?: string[], _metadata?: Record, runName?: string)`
Called when a chat model call begins (e.g. ChatOpenAI, ChatAnthropic).
Creates an LLM span with model name, provider, system/user messages,
and optional LLM context.
Serialized chat model metadata.
2D array of chat messages (outer = batches).
Unique run identifier.
Parent run ID for nesting.
Additional invocation parameters.
Optional explicit run name.
### `handleLLMEnd(output: LLMResult, runId: string)`
Called when an LLM call completes.
Sets completion text and token usage attributes on the span.
The LLMResult containing generations and usage.
Run identifier to look up the span.
### `handleLLMError(error: Error, runId: string)`
Called when an LLM call throws an error.
The error that occurred.
Run identifier to look up the span.
### `handleLLMStart(serialized: Serialized, prompts: string[], runId: string, parentRunId?: string, _extraParams?: Record, _tags?: string[], _metadata?: Record, runName?: string)`
Called when a non-chat LLM call begins (e.g. completion models).
Creates an LLM span with model name, provider, and prompt attributes.
Serialized LLM metadata.
Array of prompt strings.
Unique run identifier.
Parent run ID for nesting.
Optional explicit run name.
### `handleRetrieverEnd(documents: { metadata?: unknown; pageContent?: string }[], runId: string)`
Called when a retriever returns documents.
Sets the concatenated page content as the tool output.
Array of retrieved documents with `pageContent`.
Run identifier to look up the span.
### `handleRetrieverError(error: Error, runId: string)`
Called when a retriever throws an error.
The error that occurred.
Run identifier to look up the span.
### `handleRetrieverStart(serialized: Serialized, query: string, runId: string, parentRunId?: string)`
Called when a retriever begins fetching documents.
Creates a tool-type span with the retriever name and query.
Serialized retriever metadata.
The search query string.
Unique run identifier.
Parent run ID for nesting.
### `handleToolEnd(output: string, runId: string)`
Called when a tool invocation completes successfully.
Extracts the output content — handles both plain strings and
LangGraph `ToolMessage` objects that carry a `.content` field.
Tool output (string or ToolMessage-like object).
Run identifier to look up the span.
### `handleToolError(error: Error, runId: string)`
Called when a tool invocation throws an error.
The error that occurred.
Run identifier to look up the span.
### `handleToolStart(serialized: Serialized, input: string, runId: string, parentRunId?: string, _tags?: string[], _metadata?: Record, runName?: string)`
Called when a tool invocation begins.
Creates a tool span with the tool name and input.
Serialized tool metadata.
The input string or structured input for the tool.
Unique run identifier.
Parent run ID for nesting.
Optional explicit run name.
# formatMessages
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/format-messages
formatMessages
Array of message objects or strings.
A newline-separated string of formatted messages.
# getConversationId
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/get-conversation-id
getConversationId
The active conversation ID, or `undefined` if none is set.
# getLlmContext
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/get-llm-context
getLlmContext
A LangChain chat model instance or invocation params
object.
The attached context string, or `undefined` if none is set.
# getProviderFromModel
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/get-provider-from-model
getProviderFromModel
Model identifier (e.g. `"gpt-4"`, `"claude-3-opus"`,
`"gemini-1.5-pro"`).
Provider name (e.g. `"openai"`, `"anthropic"`, `"google"`).
Returns `"unknown"` if no provider can be detected.
# getSessionAttributes
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/get-session-attributes
getSessionAttributes
A copy of the active session attributes map.
# Introduction
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/index
Complete API reference for fiddler-langgraph-js
[](https://www.npmjs.com/package/@fiddler-ai/langgraph)
## Overview
The `@fiddler-ai/langgraph` package provides OpenTelemetry-based
instrumentation for LangGraph JS applications, enabling automatic
trace capture for agentic workflows monitored by Fiddler AI observability.
Mirrors the Python `fiddler-langgraph` SDK API.
```bash theme={null}
npm install @fiddler-ai/langgraph
```
```typescript theme={null}
import { LangGraphInstrumentor, setConversationId } from '@fiddler-ai/langgraph';
```
## Components
### Callback-Handler
The LangChain callback handler that emits Fiddler spans. `FiddlerCallbackHandler` hooks into LangGraph/LangChain run-lifecycle events and translates them into OpenTelemetry spans.
* [FiddlerCallbackHandler](/sdk-api/langgraph-js/fiddler-callback-handler)
### Context
Conversation, session, and retrieval-context helpers. `setConversationId`/`getConversationId` tag spans with a conversation ID, `addSessionAttributes`/`getSessionAttributes` attach session-wide attributes, `setLlmContext`/`getLlmContext` attach retrieval context, and `runWithContext` scopes them to a callback.
* [addSessionAttributes](/sdk-api/langgraph-js/add-session-attributes)
* [getConversationId](/sdk-api/langgraph-js/get-conversation-id)
* [getLlmContext](/sdk-api/langgraph-js/get-llm-context)
* [getSessionAttributes](/sdk-api/langgraph-js/get-session-attributes)
* [runWithContext](/sdk-api/langgraph-js/run-with-context)
* [setConversationId](/sdk-api/langgraph-js/set-conversation-id)
* [setLlmContext](/sdk-api/langgraph-js/set-llm-context)
### Instrumentor
One-time instrumentation setup. `LangGraphInstrumentor` patches the LangChain callback manager (`CallbackManagerModule`) so every LangGraph run automatically emits Fiddler spans.
* [CallbackManagerModule](/sdk-api/langgraph-js/callback-manager-module)
* [LangGraphInstrumentor](/sdk-api/langgraph-js/lang-graph-instrumentor)
### Utils
Internal helpers for span serialization and formatting -- message formatting (`formatMessages`, `extractTextContent`), provider detection (`getProviderFromModel`), and safe truncation/stringification (`truncate`, `safeStringify`).
* [extractTextContent](/sdk-api/langgraph-js/extract-text-content)
* [formatMessages](/sdk-api/langgraph-js/format-messages)
* [getProviderFromModel](/sdk-api/langgraph-js/get-provider-from-model)
* [safeStringify](/sdk-api/langgraph-js/safe-stringify)
* [truncate](/sdk-api/langgraph-js/truncate)
# LangGraphInstrumentor
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/lang-graph-instrumentor
Auto-instruments LangChain/LangGraph by monkey-patching
Auto-instruments LangChain/LangGraph by monkey-patching
`CallbackManager._configureSync` (or `.configure`) to inject a
FiddlerCallbackHandler into every invocation.
**Node.js only** — requires `@langchain/core` as a peer dependency and
uses `node:async_hooks` (via context) for context propagation.
Uses a WeakSet to track patched modules so frozen ESM exports
are handled correctly.
## Methods
### `constructor(client: FiddlerClient)`
An initialized FiddlerClient used by the
callback handler to create spans.
### `async instrument(): Promise`
Auto-instrument by dynamically importing
`@langchain/core/callbacks/manager` and patching its
`CallbackManager`.
### `isInstrumented(): boolean`
Check whether the instrumentor has successfully patched a
`CallbackManager` module.
`true` if auto-instrumentation is active.
### `manuallyInstrument(module: CallbackManagerModule)`
Manually instrument a specific `@langchain/core/callbacks/manager`
module reference.
Use this in bundled environments (Webpack, esbuild, etc.) where
dynamic `import()` of `@langchain/core` may not resolve correctly.
The imported `@langchain/core/callbacks/manager` module.
### `uninstrument()`
Remove the patch and restore the original `CallbackManager` behavior.
Safe to call even if the instrumentor was never activated.
# runWithContext
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/run-with-context
runWithContext
The function to execute within the new context.
The return value of `fn`.
# safeStringify
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/safe-stringify
safeStringify
The value to serialize.
Maximum allowed length (defaults to
MAX\_CONTENT\_LENGTH).
A JSON string (possibly truncated), or empty string.
# setConversationId
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/set-conversation-id
setConversationId
Unique conversation or session identifier.
# setLlmContext
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/set-llm-context
setLlmContext
A LangChain chat model instance (e.g. `ChatOpenAI`).
The context string to attach.
# truncate
Source: https://docs.fiddler.ai/sdk-api/langgraph-js/truncate
truncate
The string to truncate.
Maximum allowed length (defaults to
MAX\_CONTENT\_LENGTH).
The original string if short enough, or a truncated copy.
# add_session_attributes
Source: https://docs.fiddler.ai/sdk-api/langgraph/add-session-attributes
Add a session-level attribute that appears on all spans in the current context.
Add a session-level attribute that appears on all spans in the current context.
The attribute is stored in a ContextVar that
`FiddlerSpanProcessor` reads when spans
are started. It is emitted as `fiddler.session.user.{key}` on every span
created in the current thread or async coroutine, and is automatically
propagated from parent spans to child spans.
## Parameters
Logical attribute key. Will appear as `fiddler.session.user.{key}`
in span attributes.
Attribute value. Accepts `str`, `int`, `float`, or `bool`.
Numeric values (`int`/`float`) are stored as numeric attributes and will
appear in the `ValueFloat` column in ClickHouse, enabling range-based
filtering and metrics. String and boolean values appear in the `ValueString`
column and support categorical filtering (booleans are stored as `"true"`
or `"false"`).
## Returns
None Example: default from fiddler\_otel import add\_session\_attributes add\_session\_attributes("user\_id", "user\_12345") add\_session\_attributes("environment", "production") add\_session\_attributes("priority", 7) add\_session\_attributes("score", 0.95)
# add_span_attributes
Source: https://docs.fiddler.ai/sdk-api/langgraph/add-span-attributes
Add custom span-level attributes to a specific LangChain component's metadata.
Add custom span-level attributes to a specific LangChain component's metadata.
Attributes are stored in the component's metadata and automatically included
in OTel spans when the component executes. Unlike session attributes, these
are scoped to individual components.
## Parameters
The LangChain component to annotate (modified in place).
Arbitrary key-value attributes. Keys matching known
`FiddlerSpanAttributes` values are set directly; all others are
prefixed with `fiddler.span.user.`.
## Returns
None Example: default from langchain\_openai import ChatOpenAI from fiddler\_langgraph import add\_span\_attributes llm = ChatOpenAI(model="gpt-4o") add\_span\_attributes(llm, model\_tier="premium", use\_case="summarization")
# clear_llm_context
Source: https://docs.fiddler.ai/sdk-api/langgraph/clear-llm-context
Remove LLM context from a language model instance.
Remove LLM context from a language model instance.
This is a convenience wrapper that calls `set_llm_context(llm, None)`.
Use it in multi-step agent workflows after a RAG retrieval step to
ensure subsequent non-RAG LLM calls (tool planning, routing, etc.) do
not carry stale context and do not trigger faithfulness evaluation.
The context is stored in the model's metadata and read by
[`LangGraphInstrumentor`](/sdk-api/langgraph/lang-graph-instrumentor) when creating LLM spans.
## Parameters
The language model instance or binding to clear context from.
## Raises
**TypeError** – If a `RunnableBinding` is provided but its bound
object is not a `BaseLanguageModel`.
## Returns
None Example: default from langchain\_openai import ChatOpenAI from fiddler\_langgraph import set\_llm\_context, clear\_llm\_context llm = ChatOpenAI(model="gpt-4o") retrieved\_context = "Relevant documents joined as a single string..." rag\_prompt = "Answer the question using the context above." planning\_prompt = "Plan the next agent step." # Step 1: RAG retrieval -- attach context for faithfulness evaluation set\_llm\_context(llm, retrieved\_context) response = llm.invoke(rag\_prompt) # faithfulness evaluated # Step 2: Non-RAG planning -- clear context to skip faithfulness evaluation clear\_llm\_context(llm) plan = llm.invoke(planning\_prompt) # no faithfulness evaluation
## SEE ALSO
`set_llm_context()` – Set or clear context on a language model
instance. Passing `None` as the context value is equivalent to
calling `clear_llm_context()`.
# FiddlerChain
Source: https://docs.fiddler.ai/sdk-api/langgraph/fiddler-chain
Wrapper for chain/workflow spans with semantic convention helpers.
Re-exported from [`fiddler_otel.span_wrapper.FiddlerChain`](/sdk-api/otel/fiddler-chain).
Wrapper for chain/workflow spans with semantic convention helpers.
Initialize chain span wrapper.
## **enter**()
Enter context and set chain type.
### Returns
`FiddlerChain`
See the [canonical reference](/sdk-api/otel/fiddler-chain) for the full description and examples.
# FiddlerClient
Source: https://docs.fiddler.ai/sdk-api/langgraph/fiddler-client
The main client for instrumenting Generative AI applications with Fiddler observability.
Re-exported from [`fiddler_otel.client.FiddlerClient`](/sdk-api/otel/fiddler-client).
The main client for instrumenting Generative AI applications with Fiddler observability.
This client configures and manages the OpenTelemetry tracer that sends telemetry data
to the Fiddler platform for monitoring, analysis, and debugging of your AI agents
and workflows.
Flush on exit: A shutdown handler is registered via `atexit()` so that pending
spans are flushed and the tracer is shut down when the process exits. For short
scripts or critical workloads, call [`force_flush()`](#force_flush) and [`shutdown()`](#shutdown) explicitly
(e.g. in a `try`/`finally` or signal handler) since `atexit` may not run in all
environments (e.g. SIGKILL, fork).
Asyncio: Tracing works in asyncio (context vars propagate across `await`). When
shutting down from async code, use [`aflush()`](#async-aflush) and [`ashutdown()`](#async-ashutdown) so the event
loop is not blocked; the sync [`force_flush()`](#force_flush) and [`shutdown()`](#shutdown) can block for
up to the flush timeout.
Context manager: Use `with FiddlerClient(...) as client:` to ensure
[`shutdown()`](#shutdown) is called on exit (flush then shutdown; atexit is unregistered).
AWS SageMaker Partner App: When running inside a SageMaker Partner Application,
set the following environment variables to enable SigV4-signed OTLP export — no
code changes are required:
* `AWS_PARTNER_APP_AUTH=true` — opt-in trigger
* `AWS_PARTNER_APP_ARN` — ARN of the SageMaker Partner Application resource
* `AWS_PARTNER_APP_URL` — base URL of the partner app endpoint
AWS credentials are resolved via the standard boto3 credential chain (IAM role
inside a SageMaker runtime, `~/.aws/credentials`, instance profile, environment
variables); no additional env vars are needed when running inside the partner-app
sandbox.
Install the `sagemaker` extra to enable this feature:
```python theme={null}
pip install "fiddler-otel[sagemaker]"
```
In a SageMaker-managed runtime the package is pre-installed.
Initialise the FiddlerClient.
## Parameters
The unique identifier (UUID4) for the application.
Must be a valid UUID4 (e.g. `550e8400-e29b-41d4-a716-446655440000`).
Copy this from the **GenAI Applications** page in the Fiddler UI.
Required in all modes — even when `otlp_enabled=False` — because the
S3 connector uses it to route traces to the correct application.
The API key for authenticating with the Fiddler backend.
Required when `otlp_enabled=True` (the default).
Not required when `otlp_enabled=False` (e.g. S3-routing mode) — omit or pass `''`.
The base URL for your Fiddler instance
(e.g. `https://your-instance.fiddler.ai`).
Required when `otlp_enabled=True` (the default).
Not required when `otlp_enabled=False` — omit or pass `''`.
Must use `http` or `https` scheme.
Controls whether traces are exported directly to the Fiddler
OTLP endpoint. Set to `False` to disable all direct export to Fiddler — useful
when traces must first be written to local files for S3 upload (offline / S3 routing
mode). When `False`, `api_key` and `url` are not required and not validated.
Defaults to `True`.
**Note:** `console_tracer`, `jsonl_capture_enabled`, and
`otlp_json_capture_enabled` are all independent of this flag and can be used in
any combination.
If `True`, span data is **also** printed to the console
(stdout). This is **additive** — traces are still exported to Fiddler via OTLP
(unless `otlp_enabled=False`). Setting this to `True` does **not** suppress or
replace the OTLP export. Useful for local debugging to confirm spans are being
created.
Configuration for span limits, such as the maximum number of
attributes or events. When `None` (default), OpenTelemetry automatically applies
its standard defaults:
* `max_attributes`: 128 (or `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` env var)
* `max_events`: 128 (or `OTEL_SPAN_EVENT_COUNT_LIMIT` env var)
* `max_links`: 128 (or `OTEL_SPAN_LINK_COUNT_LIMIT` env var)
* `max_event_attributes`: 128 (or `OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT` env var)
* `max_link_attributes`: 128 (or `OTEL_LINK_ATTRIBUTE_COUNT_LIMIT` env var)
* `max_span_attribute_length`: None/unlimited
(or `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` env var)
Override these by passing a custom `SpanLimits` object or by setting the
corresponding environment variables.
The sampler for deciding which spans to record. Defaults to the
parent-based always-on OpenTelemetry sampler (100% sampling).
The compression algorithm for exporting traces. Can be
`Compression.Gzip`, `Compression.Deflate`, or `Compression.NoCompression`.
Only used when `otlp_enabled=True`. Defaults to `Compression.Gzip`.
If `True`, span data is **also** saved to a local
JSONL file in a custom Fiddler format. This is **additive** — OTLP export to
Fiddler continues as normal (unless `otlp_enabled=False`). Setting this to
`True` does **not** suppress the OTLP export. Useful for keeping a local copy
of trace data.
**Note:** This JSONL format is **not** compatible with the Fiddler S3 connector.
For S3 ingestion use `otlp_json_capture_enabled=True` instead.
Path to the JSONL file where trace data will be saved.
Only used when `jsonl_capture_enabled=True`.
Override with the `FIDDLER_JSONL_FILE` environment variable.
If `True`, traces are written to local `.json`
files in standard OTLP JSON format (`ExportTraceServiceRequest` envelope). Files
are written to `otlp_json_output_dir`. This format is **directly compatible** with
the Fiddler S3 connector — upload the generated files to S3 and the connector ingests
them without any reformatting. This is **additive** relative to OTLP export; combine
with `otlp_enabled=False` to write to files only (offline / S3 routing mode).
Directory path where OTLP JSON files are written when
`otlp_json_capture_enabled=True`. The directory is created automatically if it
does not exist. Each batch of spans is written to a separate timestamped `.json`
file.
If `True`, large inline base64 content in span
attributes is uploaded to S3 via `POST /v3/files/upload` and replaced with
`fiddler-file://` URIs before the span is exported. This prevents trace loss
at the 10MB Kafka span limit. Requires `otlp_enabled=True` (needs `url` and
`api_key`). Defaults to `False`.
## Raises
* **ValueError** – If `application_id` is not a valid UUID4.
* **ValueError** – If `otlp_enabled=True` and `api_key` is empty or not provided.
* **ValueError** – If `otlp_enabled=True` and `url` is empty, missing a scheme,
or uses a scheme other than `http`/`https`.
**Basic connection to your Fiddler instance**:
```python theme={null}
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID', # UUID4, required in all modes
api_key='YOUR_API_KEY',
url='https://your-instance.fiddler.ai',
)
```
**High-volume applications with custom configuration**:
```python theme={null}
from opentelemetry.sdk.trace import SpanLimits, sampling
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID',
api_key='YOUR_API_KEY',
url='https://your-instance.fiddler.ai',
span_limits=SpanLimits(
max_span_attributes=64, # Reduce from default 128
max_span_attribute_length=2048, # Limit from default None (unlimited)
),
sampler=sampling.TraceIdRatioBased(0.1), # Sample 10% of traces
compression=Compression.Gzip,
)
```
**Local development with console output** (traces still sent to Fiddler):
```python theme={null}
# console_tracer=True prints spans to stdout AND continues to export to Fiddler.
# It does NOT suppress OTLP export.
client = FiddlerClient(
application_id='00000000-0000-0000-0000-000000000000',
api_key='dev-key',
url='http://localhost:4318',
console_tracer=True,
)
```
**Offline / S3 routing mode** (no data sent directly to Fiddler):
```python theme={null}
# otlp_enabled=False → disables direct OTLP export; api_key and url not needed
# otlp_json_capture_enabled=True → writes traces to local .json files in standard
# OTLP JSON format (ExportTraceServiceRequest)
# application_id is still required so the S3 connector routes traces correctly
#
# Upload files from otlp_json_output_dir to your S3 bucket.
# The Fiddler S3 connector reads them directly with no reformatting required.
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID',
otlp_enabled=False,
otlp_json_capture_enabled=True,
otlp_json_output_dir='./fiddler_traces', # default: 'fiddler_traces'
)
```
## **enter**()
Context manager entry.
### Returns
`FiddlerClient`
## **exit**()
Context manager exit — flushes and shuts down the tracer provider.
## force\_flush()
Flush pending spans to the exporter.
### Parameters
Maximum time to wait for flush in milliseconds.
### Returns
True if flush completed within the timeout, False otherwise.
## *async* aflush()
Async version of [`force_flush()`](#force_flush).
Runs the flush in a thread pool so the event loop is not blocked.
### Returns
`bool`
## shutdown()
Shut down the tracer provider after flushing pending spans.
Safe to call multiple times. The atexit handler is unregistered on first call.
## *async* ashutdown()
Async version of [`shutdown()`](#shutdown).
Runs flush and shutdown in a thread pool so the event loop is not blocked.
## get\_tracer\_provider()
Return the OpenTelemetry TracerProvider, initializing it on first call.
### Returns
The configured TracerProvider.
### Raises
**RuntimeError** – If initialization fails.
## update\_resource()
Update the OTel resource with additional attributes.
Must be called **before** [`get_tracer()`](#get_tracer) is invoked.
### Parameters
Key-value pairs to merge into the resource.
### Raises
**ValueError** – If the tracer has already been initialized.
## get\_tracer()
Return an OTel tracer for creating spans.
Initializes the tracer on the first call.
### Returns
The OTel tracer instance.
### Raises
**RuntimeError** – If tracer initialization fails.
## start\_as\_current\_span()
Create a span using a context manager (automatic lifecycle management).
### Parameters
Name for the span.
Span type — `"span"`, `"generation"`, `"chain"`, or `"tool"`.
### Returns
Span wrapper with context manager support.
## start\_span()
Create a span with manual lifecycle control. Caller must call `span.end()`.
### Parameters
Name for the span.
Span type — `"span"`, `"generation"`, `"chain"`, or `"tool"`.
### Returns
Span wrapper requiring an explicit `end()` call.
See the [canonical reference](/sdk-api/otel/fiddler-client) for the full description and examples.
# FiddlerGeneration
Source: https://docs.fiddler.ai/sdk-api/langgraph/fiddler-generation
Wrapper for LLM generation spans with semantic convention helpers.
Re-exported from [`fiddler_otel.span_wrapper.FiddlerGeneration`](/sdk-api/otel/fiddler-generation).
Wrapper for LLM generation spans with semantic convention helpers.
Initialize LLM generation wrapper.
## **enter**()
Enter context and set LLM type.
### Returns
`FiddlerGeneration`
## set\_model()
Set the LLM model name (gen\_ai.request.model).
## set\_system()
Set the LLM system/provider (gen\_ai.system).
## set\_system\_prompt()
Set the system prompt (gen\_ai.llm.input.system).
## set\_user\_prompt()
Set the user prompt (gen\_ai.llm.input.user).
### Parameters
Plain text string, or a list of content parts in OpenAI
multimodal format (e.g. `[{'type': 'text', 'text': '...'},
{'type': 'image_url', 'image_url': {'url': 'data:...'}}]`).
Lists are auto-serialized to JSON.
## set\_completion()
Set the LLM completion/output (gen\_ai.llm.output).
## set\_usage()
Set token usage information (gen\_ai.usage.\*).
### Parameters
Number of input/prompt tokens.
Number of output/completion tokens.
Total tokens; computed from input + output when omitted.
## set\_context()
Set additional context (gen\_ai.llm.context).
## set\_messages()
Set input messages in OpenAI chat format (gen\_ai.input.messages).
Accepts simple format: `[{'role': 'user', 'content': '...'}]`
Auto-converts to OTel format with `parts`.
## set\_output\_messages()
Set output messages in OpenAI chat format (gen\_ai.output.messages).
Accepts simple format: `[{'role': 'assistant', 'content': '...'}]`
Auto-converts to OTel format with `parts`.
## set\_tool\_definitions()
Set available tool definitions for this LLM call (gen\_ai.tool.definitions).
See the [canonical reference](/sdk-api/otel/fiddler-generation) for the full description and examples.
# FiddlerSpan
Source: https://docs.fiddler.ai/sdk-api/langgraph/fiddler-span
Wrapper around OpenTelemetry span with simplified helper methods.
Re-exported from [`fiddler_otel.span_wrapper.FiddlerSpan`](/sdk-api/otel/fiddler-span).
Wrapper around OpenTelemetry span with simplified helper methods.
Initialize wrapper around an OTel span or context manager.
## Parameters
An OTel span or context manager.
Optional `MediaUploader`
for normalizing large base64 content before span export.
## **enter**()
Enter context manager and start span.
### Returns
`FiddlerSpan`
## **exit**()
Exit context manager, record exceptions, and end span.
### Returns
`Literal[False]`
## end()
Explicitly end the span. Must be called when using start\_span().
## set\_input()
Set input data. Auto-serializes dicts/lists to JSON.
## set\_output()
Set output data. Auto-serializes dicts/lists to JSON.
## set\_attribute()
Set a custom attribute on the span.
## update()
Bulk update multiple attributes.
## record\_exception()
Record an exception on the span.
## set\_agent\_name()
Set the agent name (gen\_ai.agent.name).
## set\_agent\_id()
Set the agent ID (gen\_ai.agent.id).
## set\_conversation\_id()
Set the conversation ID (gen\_ai.conversation.id).
See the [canonical reference](/sdk-api/otel/fiddler-span) for the full description and examples.
# FiddlerTool
Source: https://docs.fiddler.ai/sdk-api/langgraph/fiddler-tool
Wrapper for tool/function call spans with semantic convention helpers.
Re-exported from [`fiddler_otel.span_wrapper.FiddlerTool`](/sdk-api/otel/fiddler-tool).
Wrapper for tool/function call spans with semantic convention helpers.
Initialize tool span wrapper.
## **enter**()
Enter context and set tool type.
### Returns
`FiddlerTool`
## set\_tool\_name()
Set the tool/function name (gen\_ai.tool.name).
## set\_tool\_input()
Set the tool input arguments (gen\_ai.tool.input).
## set\_tool\_output()
Set the tool output/result (gen\_ai.tool.output).
## set\_tool\_definitions()
Set available tool definitions (gen\_ai.tool.definitions).
See the [canonical reference](/sdk-api/otel/fiddler-tool) for the full description and examples.
# get_client
Source: https://docs.fiddler.ai/sdk-api/langgraph/get-client
Return the global FiddlerClient singleton (first created in this process).
Re-exported from [`fiddler_otel.client.get_client`](/sdk-api/otel/get-client).
Return the global FiddlerClient singleton (first created in this process).
## Returns
The global client instance.
## Raises
**RuntimeError** – If no FiddlerClient has been initialized.
See the [canonical reference](/sdk-api/otel/get-client) for the full description and examples.
# get_current_span
Source: https://docs.fiddler.ai/sdk-api/langgraph/get-current-span
Get the currently active span as a Fiddler wrapper (inside a traced function).
Re-exported from [`fiddler_otel.decorators.get_current_span`](/sdk-api/otel/get-current-span).
Get the currently active span as a Fiddler wrapper (inside a traced function).
Retrieves the span from the current context and verifies it belongs to Fiddler's tracer
to maintain isolation from other OpenTelemetry tracers.
## Parameters
Wrapper type — `"span"`, `"generation"`, `"chain"`, or `"tool"`.
## Returns
Current span wrapper, or None if no active Fiddler span exists.
See the [canonical reference](/sdk-api/otel/get-current-span) for the full description and examples.
# Introduction
Source: https://docs.fiddler.ai/sdk-api/langgraph/index
Complete API reference for fiddler-langgraph
[](https://pypi.org/project/fiddler-langgraph/)
## Overview
Complete API reference documentation for the `fiddler-langgraph` package, which
instruments [LangGraph](https://langchain-ai.github.io/langgraph/) (and the
underlying LangChain callback system) and exports OpenTelemetry traces to
Fiddler.
```bash theme={null}
pip install fiddler-langgraph
```
```python theme={null}
from fiddler_langgraph import LangGraphInstrumentor
```
## Components
### Instrumentation
Native LangGraph APIs for tracing, attributes, and LLM context. `LangGraphInstrumentor` patches the LangChain callback manager so all LangGraph runs emit Fiddler spans. `add_span_attributes` attaches component-scoped attributes to a specific model, retriever, or tool. `set_llm_context` and `clear_llm_context` attach and remove retrieval context (e.g. retrieved documents) on the active model so it appears on its LLM spans. `add_session_attributes` adds an attribute that appears on every span in the current context, and `set_conversation_id` (re-exported from `fiddler-otel`) tags all spans in the current context with a conversation ID.
* [add\_session\_attributes](/sdk-api/langgraph/add-session-attributes)
* [add\_span\_attributes](/sdk-api/langgraph/add-span-attributes)
* [clear\_llm\_context](/sdk-api/langgraph/clear-llm-context)
* [LangGraphInstrumentor](/sdk-api/langgraph/lang-graph-instrumentor)
* [set\_conversation\_id](/sdk-api/langgraph/set-conversation-id)
* [set\_llm\_context](/sdk-api/langgraph/set-llm-context)
### Re Exported From Otel
`fiddler-langgraph` re-exports the core `fiddler-otel` symbols so you only need to install one package. See the [Fiddler OTel SDK reference](/sdk-api/otel) for full details on each: `FiddlerClient` (the global client; required to construct `LangGraphInstrumentor`), `trace` (decorator that starts a Fiddler span around any function), `get_current_span` (return the active Fiddler span if any), `get_client` (return the global `FiddlerClient` instance), and the manual span wrappers `FiddlerChain`, `FiddlerGeneration`, `FiddlerSpan`, and `FiddlerTool`.
* [FiddlerChain](/sdk-api/langgraph/fiddler-chain)
* [FiddlerClient](/sdk-api/langgraph/fiddler-client)
* [FiddlerGeneration](/sdk-api/langgraph/fiddler-generation)
* [FiddlerSpan](/sdk-api/langgraph/fiddler-span)
* [FiddlerTool](/sdk-api/langgraph/fiddler-tool)
* [get\_client](/sdk-api/langgraph/get-client)
* [get\_current\_span](/sdk-api/langgraph/get-current-span)
* [trace](/sdk-api/langgraph/trace)
# LangGraphInstrumentor
Source: https://docs.fiddler.ai/sdk-api/langgraph/lang-graph-instrumentor
An OpenTelemetry instrumentor for LangGraph applications.
An OpenTelemetry instrumentor for LangGraph applications.
Provides automatic instrumentation for applications built with LangGraph by
monkey-patching LangChain's callback system to inject a custom callback handler
that captures trace data. Once instrumented, all LangGraph operations automatically
generate telemetry data.
Instrumentation persists for the lifetime of the application unless explicitly
removed by calling [`uninstrument()`](#uninstrument). Calling [`instrument()`](#instrument) multiple times
is safe — it will not create duplicate handlers.
Thread Safety: The instrumentation applies globally to the process and affects all
threads. In concurrent environments (multi-threading, async), all contexts share the
same instrumented callback system.
**Basic usage**:
```python theme={null}
from fiddler_langgraph import FiddlerClient, LangGraphInstrumentor
client = FiddlerClient(
application_id="...",
api_key="...",
url="https://your-instance.fiddler.ai",
)
instrumentor = LangGraphInstrumentor(client=client)
instrumentor.instrument()
```
**Removing instrumentation**:
```python theme={null}
# Clean up instrumentation when shutting down
instrumentor.uninstrument()
```
Initialise the LangGraphInstrumentor.
## Parameters
The [`FiddlerClient`](/sdk-api/langgraph/fiddler-client) instance to use for tracing.
## Raises
**ImportError** – If LangGraph is not installed or its version is incompatible.
## instrument()
Instrument LangGraph by monkey-patching `BaseCallbackManager`.
Calling this method multiple times is safe; subsequent calls are no-ops.
### Raises
**ValueError** – If the tracer is not initialized in the FiddlerClient.
## uninstrument()
Remove the instrumentation from LangGraph.
# set_conversation_id
Source: https://docs.fiddler.ai/sdk-api/langgraph/set-conversation-id
Set the conversation ID for the current execution context.
Re-exported from [`fiddler_otel.attributes.set_conversation_id`](/sdk-api/otel/set-conversation-id).
Set the conversation ID for the current execution context.
The conversation ID is propagated to all spans created in the current
thread or async coroutine, allowing the Fiddler dashboard to filter and
display the full ordered sequence of operations for a single conversation.
This value persists until it is called again with a new ID.
## Parameters
Unique identifier for the conversation session.
## Returns
None Example: default import uuid from fiddler\_otel import set\_conversation\_id set\_conversation\_id(str(uuid.uuid4())) # All spans created in this thread or coroutine after this call # carry the same conversation\_id, until set\_conversation\_id is # called again with a new value.
See the [canonical reference](/sdk-api/otel/set-conversation-id) for the full description and examples.
# set_llm_context
Source: https://docs.fiddler.ai/sdk-api/langgraph/set-llm-context
Set or clear additional context information on a language model instance.
Set or clear additional context information on a language model instance.
The context is attached to all spans created for this model and is stored
as `gen_ai.llm.context`. Supports both `BaseLanguageModel` instances
and `RunnableBinding` objects.
Passing `None` removes the context so that subsequent spans no longer
carry it. This is useful in multi-step agent workflows where context set
after a RAG step should not leak into later non-RAG LLM calls.
## Parameters
The language model instance or binding.
The context string to attach, or `None` to clear.
## Raises
**TypeError** – If a `RunnableBinding` is provided but its bound
object is not a `BaseLanguageModel`.
## Returns
None Example: default from langchain\_openai import ChatOpenAI from fiddler\_langgraph import set\_llm\_context llm = ChatOpenAI(model="gpt-4o") set\_llm\_context(llm, "User prefers concise responses in Spanish") # Clear context before non-RAG steps set\_llm\_context(llm, None)
# trace
Source: https://docs.fiddler.ai/sdk-api/langgraph/trace
Decorator for automatic function tracing with input/output capture.
Re-exported from [`fiddler_otel.decorators.trace`](/sdk-api/otel/trace).
Decorator for automatic function tracing with input/output capture.
Uses the global FiddlerClient when `client=` is not passed.
Supports both sync and async functions.
## Parameters
Function to decorate.
Custom span name (defaults to the function name).
Span type — `"span"`, `"generation"`, `"chain"`, or `"tool"`.
Capture function arguments as span input.
Capture the return value as span output.
FiddlerClient instance (defaults to `get_client()`).
LLM model name; sets `gen_ai.request.model`.
User identifier; sets `user.id`.
Version string; sets `service.version`.
LLM provider; sets `gen_ai.system`.
## Returns
The decorated function.
See the [canonical reference](/sdk-api/otel/trace) for the full description and examples.
# FiddlerAgentSpan
Source: https://docs.fiddler.ai/sdk-api/otel-js/fiddler-agent-span
Span wrapper for agent/chain operations with agent identity helpers.
Span wrapper for agent/chain operations with agent identity helpers.
Created via FiddlerClient.startAgent. All setter methods return
`this` for fluent chaining.
## Methods
### `setAgentId(id: string): this`
Set the unique agent identifier.
Agent ID string (e.g. `"travel_agent_v2"`).
`this` for chaining.
### `setAgentName(name: string): this`
Set the human-readable agent name.
Agent display name (e.g. `"travel_agent"`).
`this` for chaining.
# FiddlerClient
Source: https://docs.fiddler.ai/sdk-api/otel-js/fiddler-client
Primary entry point for Fiddler AI observability in JavaScript/TypeScript.
Primary entry point for Fiddler AI observability in JavaScript/TypeScript.
Creates an isolated OpenTelemetry BasicTracerProvider that exports
traces to the Fiddler backend via OTLP HTTP. The provider is **not**
registered globally, so it will not interfere with other OTel
instrumentation in the same process.
**Node.js only** — registers a `process.beforeExit` handler for
automatic shutdown. The `process` guard is safe in non-Node
environments but OTel SDK dependencies require Node.js >= 20.
## Methods
### `constructor(config: FiddlerClientConfig)`
Create a new Fiddler client and initialize the OTel provider.
Client configuration including Fiddler URL, API key,
and application ID.
### `async flush(): Promise`
Force-flush all buffered spans to the Fiddler backend.
Call this before process exit in short-lived scripts to ensure all
spans are exported.
### `async shutdown(): Promise`
Flush remaining spans and shut down the underlying OTel provider.
After shutdown the client cannot create new spans. This method is
idempotent — calling it multiple times is safe.
### `startAgent(name: string, options?: Omit): FiddlerAgentSpan`
Start a new agent/chain span (sets `fiddler.span.type` to `"chain"`).
Human-readable name for the agent span.
Optional parent span for nesting.
A new FiddlerAgentSpan with agent-specific helpers.
### `startGeneration(name: string, options?: Omit): FiddlerGenerationSpan`
Start a new LLM generation span (sets `fiddler.span.type` to `"llm"`).
Human-readable name for the generation span.
Optional parent span for nesting.
A new FiddlerGenerationSpan with LLM-specific helpers.
### `startSpan(name: string, options?: StartSpanOptions): FiddlerSpan`
Start a new span with an optional explicit type.
Human-readable name for the span.
Optional span type and parent span.
A new FiddlerSpan instance.
### `startTool(name: string, options?: Omit): FiddlerToolSpan`
Start a new tool span (sets `fiddler.span.type` to `"tool"`).
Human-readable name for the tool span.
Optional parent span for nesting.
A new FiddlerToolSpan with tool-specific helpers.
# FiddlerClientConfig
Source: https://docs.fiddler.ai/sdk-api/otel-js/fiddler-client-config
Configuration for FiddlerClient.
Configuration for FiddlerClient.
## Properties
### `apiKey: string`
Fiddler API key or personal access token.
### `applicationId: string`
UUID of the Fiddler GenAI application to send traces to.
### `consoleTracer?: boolean`
When `true`, also print spans to the console for local debugging.
### `serviceName?: string`
OTel service name (defaults to `"fiddler-otel"`).
### `serviceVersion?: string`
OTel service version (defaults to `"0.1.0"`).
### `url: string`
Base URL of the Fiddler instance (e.g. `"https://app.fiddler.ai"`).
# FiddlerGenerationSpan
Source: https://docs.fiddler.ai/sdk-api/otel-js/fiddler-generation-span
Span wrapper for LLM generation calls with model, prompt, completion,
Span wrapper for LLM generation calls with model, prompt, completion,
and token-usage helpers.
Created via FiddlerClient.startGeneration. All setter methods
return `this` for fluent chaining.
## Methods
### `setCompletion(text: string): this`
Set the LLM completion/response text.
Alias for FiddlerSpan.setOutput — writes to `gen_ai.llm.output`.
The completion content.
`this` for chaining.
### `setContext(context: string): this`
Set the LLM context (e.g. retrieved RAG context) for this generation.
Context string provided to the LLM.
`this` for chaining.
### `setMessages(messages: Record[]): this`
Set the full input message history sent to the model.
Array of chat messages in OpenAI/Anthropic format.
`this` for chaining.
### `setModel(model: string): this`
Set the model identifier (e.g. `"gpt-4o"`, `"claude-3-opus"`).
Model name or identifier string.
`this` for chaining.
### `setOutputMessages(messages: Record[]): this`
Set the output messages returned by the model.
Array of assistant response messages.
`this` for chaining.
### `setSystem(system: string): this`
Set the LLM provider/system (e.g. `"openai"`, `"anthropic"`).
Provider identifier string.
`this` for chaining.
### `setSystemPrompt(text: string): this`
Set the system prompt text.
The system prompt content.
`this` for chaining.
### `setToolDefinitions(definitions: Record[]): this`
Set the tool/function definitions available to the model.
Array of tool definition objects.
`this` for chaining.
### `setUsage(usage: TokenUsage): this`
Set token usage statistics for this generation.
If `totalTokens` is not provided it is computed as
`inputTokens + outputTokens`.
Token counts for input, output, and total.
`this` for chaining.
### `setUserPrompt(text: string): this`
Set the user prompt text.
Alias for FiddlerSpan.setInput — writes to `gen_ai.llm.input.user`.
The user prompt content.
`this` for chaining.
# FiddlerSpan
Source: https://docs.fiddler.ai/sdk-api/otel-js/fiddler-span
Base wrapper around an OpenTelemetry Span with Fiddler-specific
Base wrapper around an OpenTelemetry Span with Fiddler-specific
attribute helpers.
All setter methods return `this` for fluent chaining.
## Methods
### `constructor(span: Span)`
The underlying OpenTelemetry span instance.
### `end()`
Mark the span as successful and record its end timestamp.
### `recordError(error: string | Error)`
Record an error on this span and set its status to `ERROR`.
An Error instance or error message string.
### `setAttribute(key: string, value: string | number | boolean): this`
Set an arbitrary attribute on the span.
Attribute key (e.g. `"gen_ai.request.model"`).
Attribute value.
`this` for chaining.
### `setConversationId(id: string): this`
Tag this span with a conversation/session identifier.
Unique conversation or session ID.
`this` for chaining.
### `setInput(input: unknown): this`
Set the user-facing input for this span.
Writes to the `gen_ai.llm.input.user` attribute after safely
serializing non-string values.
The input value (string or serializable object).
`this` for chaining.
### `setOutput(output: unknown): this`
Set the output/response for this span.
Writes to the `gen_ai.llm.output` attribute after safely
serializing non-string values.
The output value (string or serializable object).
`this` for chaining.
# FiddlerToolSpan
Source: https://docs.fiddler.ai/sdk-api/otel-js/fiddler-tool-span
Span wrapper for tool/function invocations with tool-specific attribute
Span wrapper for tool/function invocations with tool-specific attribute
helpers.
Created via FiddlerClient.startTool. All setter methods return
`this` for fluent chaining.
## Methods
### `setInput(input: unknown): this`
Set the input passed to the tool.
Overrides the base FiddlerSpan.setInput to write to
`gen_ai.tool.input` instead of `gen_ai.llm.input.user`.
Tool input (string or serializable object).
`this` for chaining.
### `setOutput(output: unknown): this`
Set the output returned by the tool.
Overrides the base FiddlerSpan.setOutput to write to
`gen_ai.tool.output` instead of `gen_ai.llm.output`.
Tool output (string or serializable object).
`this` for chaining.
### `setToolDefinitions(definitions: Record[]): this`
Set the tool/function definitions available in this context.
Array of tool definition objects.
`this` for chaining.
### `setToolName(name: string): this`
Set the canonical tool/function name.
Tool name (e.g. `"web_search"`, `"calculator"`).
`this` for chaining.
# Introduction
Source: https://docs.fiddler.ai/sdk-api/otel-js/index
Complete API reference for fiddler-otel-js
[](https://www.npmjs.com/package/@fiddler-ai/otel)
## Overview
The `@fiddler-ai/otel` package provides OpenTelemetry instrumentation for
Fiddler AI observability. Use it to automatically capture LLM traces,
conversation context, and span attributes from your TypeScript applications.
```bash theme={null}
npm install @fiddler-ai/otel
```
```typescript theme={null}
import { FiddlerClient } from '@fiddler-ai/otel';
```
## Components
### Client
Client initialization. `FiddlerClient` creates an isolated OpenTelemetry tracer provider that exports traces to Fiddler over OTLP HTTP without registering globally, so it does not interfere with other instrumentation in the same process.
* [FiddlerClient](/sdk-api/otel-js/fiddler-client)
### Internal
Configuration and attribute constants. `FiddlerClientConfig` is the client's options object; `SpanAttributes` and `SpanType` enumerate the attribute keys and span-type values Fiddler recognizes.
* [FiddlerClientConfig](/sdk-api/otel-js/fiddler-client-config)
* [SpanAttributes](/sdk-api/otel-js/span-attributes)
* [SpanType](/sdk-api/otel-js/span-type)
### Spans
Manual span helpers. `FiddlerAgentSpan`, `FiddlerGenerationSpan`, and `FiddlerToolSpan` carry semantic conventions for agent steps, LLM calls, and tool invocations; `FiddlerSpan` is the base, with `StartSpanOptions` and `TokenUsage` as supporting types.
* [FiddlerAgentSpan](/sdk-api/otel-js/fiddler-agent-span)
* [FiddlerGenerationSpan](/sdk-api/otel-js/fiddler-generation-span)
* [FiddlerSpan](/sdk-api/otel-js/fiddler-span)
* [FiddlerToolSpan](/sdk-api/otel-js/fiddler-tool-span)
* [StartSpanOptions](/sdk-api/otel-js/start-span-options)
* [TokenUsage](/sdk-api/otel-js/token-usage)
# SpanAttributes
Source: https://docs.fiddler.ai/sdk-api/otel-js/span-attributes
Fiddler span attribute keys.
Fiddler span attribute keys.
Maps logical attribute names to their OpenTelemetry string keys
following the `gen_ai.*` semantic conventions.
## Values
| Constant | Value | Description |
| ------------------------ | ------------------------------ | ---------------------------------------------------------------- |
| `AGENT_ID` | `"gen_ai.agent.id"` | Unique agent identifier. |
| `AGENT_NAME` | `"gen_ai.agent.name"` | Human-readable agent name. |
| `CONVERSATION_ID` | `"gen_ai.conversation.id"` | Conversation or session identifier for multi-turn grouping. |
| `GEN_AI_INPUT_MESSAGES` | `"gen_ai.input.messages"` | Full input message history as JSON. |
| `GEN_AI_OUTPUT_MESSAGES` | `"gen_ai.output.messages"` | Full output messages as JSON. |
| `LLM_CONTEXT` | `"gen_ai.llm.context"` | Retrieved context provided to the LLM (e.g. RAG context). |
| `LLM_INPUT_SYSTEM` | `"gen_ai.llm.input.system"` | System prompt sent to the LLM. |
| `LLM_INPUT_USER` | `"gen_ai.llm.input.user"` | User prompt or input sent to the LLM. |
| `LLM_OUTPUT` | `"gen_ai.llm.output"` | LLM completion/response text. |
| `LLM_REQUEST_MODEL` | `"gen_ai.request.model"` | Requested model identifier (e.g. `"gpt-4o"`). |
| `LLM_SYSTEM` | `"gen_ai.system"` | LLM provider/system (e.g. `"openai"`, `"anthropic"`). |
| `LLM_TOKEN_COUNT_INPUT` | `"gen_ai.usage.input_tokens"` | Number of input/prompt tokens. |
| `LLM_TOKEN_COUNT_OUTPUT` | `"gen_ai.usage.output_tokens"` | Number of output/completion tokens. |
| `LLM_TOKEN_COUNT_TOTAL` | `"gen_ai.usage.total_tokens"` | Total token count. |
| `SESSION_ID` | `"session.id"` | Session identifier. |
| `TOOL_DEFINITIONS` | `"gen_ai.tool.definitions"` | Tool/function definitions as JSON. |
| `TOOL_INPUT` | `"gen_ai.tool.input"` | Tool input payload. |
| `TOOL_NAME` | `"gen_ai.tool.name"` | Tool/function name. |
| `TOOL_OUTPUT` | `"gen_ai.tool.output"` | Tool output payload. |
| `TYPE` | `"fiddler.span.type"` | The Fiddler span type (`"llm"`, `"tool"`, `"chain"`, `"agent"`). |
| `USER_ID` | `"user.id"` | User identifier. |
# SpanType
Source: https://docs.fiddler.ai/sdk-api/otel-js/span-type
Allowed values for the `fiddler.span.type` attribute.
Allowed values for the `fiddler.span.type` attribute.
## Values
| Constant | Value | Description |
| -------- | --------- | -------------------------------- |
| `AGENT` | `"agent"` | An autonomous agent span. |
| `CHAIN` | `"chain"` | A chain or workflow span. |
| `LLM` | `"llm"` | An LLM generation span. |
| `OTHER` | `"other"` | An uncategorized span. |
| `TOOL` | `"tool"` | A tool/function invocation span. |
# StartSpanOptions
Source: https://docs.fiddler.ai/sdk-api/otel-js/start-span-options
Options for creating a new span via FiddlerClient.startSpan.
Options for creating a new span via FiddlerClient.startSpan.
## Properties
### `parent?: FiddlerSpan`
An existing span to use as the logical parent.
### `type?: SpanTypeValue`
The Fiddler span type (e.g. `"llm"`, `"tool"`, `"chain"`).
# TokenUsage
Source: https://docs.fiddler.ai/sdk-api/otel-js/token-usage
Token usage statistics for an LLM call.
Token usage statistics for an LLM call.
## Properties
### `inputTokens?: number`
Number of tokens in the prompt/input.
### `outputTokens?: number`
Number of tokens in the completion/output.
### `totalTokens?: number`
Total token count (computed from input + output if omitted).
# add_session_attributes
Source: https://docs.fiddler.ai/sdk-api/otel/add-session-attributes
Add a session-level attribute that appears on all spans in the current context.
Add a session-level attribute that appears on all spans in the current context.
The attribute is stored in a ContextVar that
[`FiddlerSpanProcessor`](/sdk-api/otel/fiddler-span-processor) reads when spans
are started. It is emitted as `fiddler.session.user.{key}` on every span
created in the current thread or async coroutine, and is automatically
propagated from parent spans to child spans.
## Parameters
Logical attribute key. Will appear as `fiddler.session.user.{key}`
in span attributes.
Attribute value. Accepts `str`, `int`, `float`, or `bool`.
Numeric values (`int`/`float`) are stored as numeric attributes and will
appear in the `ValueFloat` column in ClickHouse, enabling range-based
filtering and metrics. String and boolean values appear in the `ValueString`
column and support categorical filtering (booleans are stored as `"true"`
or `"false"`).
## Returns
None Example: default from fiddler\_otel import add\_session\_attributes add\_session\_attributes("user\_id", "user\_12345") add\_session\_attributes("environment", "production") add\_session\_attributes("priority", 7) add\_session\_attributes("score", 0.95)
# FiddlerChain
Source: https://docs.fiddler.ai/sdk-api/otel/fiddler-chain
Wrapper for chain/workflow spans with semantic convention helpers.
Wrapper for chain/workflow spans with semantic convention helpers.
Initialize chain span wrapper.
## **enter**()
Enter context and set chain type.
### Returns
`FiddlerChain`
# FiddlerClient
Source: https://docs.fiddler.ai/sdk-api/otel/fiddler-client
The main client for instrumenting Generative AI applications with Fiddler observability.
The main client for instrumenting Generative AI applications with Fiddler observability.
This client configures and manages the OpenTelemetry tracer that sends telemetry data
to the Fiddler platform for monitoring, analysis, and debugging of your AI agents
and workflows.
Flush on exit: A shutdown handler is registered via `atexit()` so that pending
spans are flushed and the tracer is shut down when the process exits. For short
scripts or critical workloads, call [`force_flush()`](#force_flush) and [`shutdown()`](#shutdown) explicitly
(e.g. in a `try`/`finally` or signal handler) since `atexit` may not run in all
environments (e.g. SIGKILL, fork).
Asyncio: Tracing works in asyncio (context vars propagate across `await`). When
shutting down from async code, use [`aflush()`](#async-aflush) and [`ashutdown()`](#async-ashutdown) so the event
loop is not blocked; the sync [`force_flush()`](#force_flush) and [`shutdown()`](#shutdown) can block for
up to the flush timeout.
Context manager: Use `with FiddlerClient(...) as client:` to ensure
[`shutdown()`](#shutdown) is called on exit (flush then shutdown; atexit is unregistered).
AWS SageMaker Partner App: When running inside a SageMaker Partner Application,
set the following environment variables to enable SigV4-signed OTLP export — no
code changes are required:
* `AWS_PARTNER_APP_AUTH=true` — opt-in trigger
* `AWS_PARTNER_APP_ARN` — ARN of the SageMaker Partner Application resource
* `AWS_PARTNER_APP_URL` — base URL of the partner app endpoint
AWS credentials are resolved via the standard boto3 credential chain (IAM role
inside a SageMaker runtime, `~/.aws/credentials`, instance profile, environment
variables); no additional env vars are needed when running inside the partner-app
sandbox.
Install the `sagemaker` extra to enable this feature:
```python theme={null}
pip install "fiddler-otel[sagemaker]"
```
In a SageMaker-managed runtime the package is pre-installed.
Initialise the FiddlerClient.
## Parameters
The unique identifier (UUID4) for the application.
Must be a valid UUID4 (e.g. `550e8400-e29b-41d4-a716-446655440000`).
Copy this from the **GenAI Applications** page in the Fiddler UI.
Required in all modes — even when `otlp_enabled=False` — because the
S3 connector uses it to route traces to the correct application.
The API key for authenticating with the Fiddler backend.
Required when `otlp_enabled=True` (the default).
Not required when `otlp_enabled=False` (e.g. S3-routing mode) — omit or pass `''`.
The base URL for your Fiddler instance
(e.g. `https://your-instance.fiddler.ai`).
Required when `otlp_enabled=True` (the default).
Not required when `otlp_enabled=False` — omit or pass `''`.
Must use `http` or `https` scheme.
Controls whether traces are exported directly to the Fiddler
OTLP endpoint. Set to `False` to disable all direct export to Fiddler — useful
when traces must first be written to local files for S3 upload (offline / S3 routing
mode). When `False`, `api_key` and `url` are not required and not validated.
Defaults to `True`.
**Note:** `console_tracer`, `jsonl_capture_enabled`, and
`otlp_json_capture_enabled` are all independent of this flag and can be used in
any combination.
If `True`, span data is **also** printed to the console
(stdout). This is **additive** — traces are still exported to Fiddler via OTLP
(unless `otlp_enabled=False`). Setting this to `True` does **not** suppress or
replace the OTLP export. Useful for local debugging to confirm spans are being
created.
Configuration for span limits, such as the maximum number of
attributes or events. When `None` (default), OpenTelemetry automatically applies
its standard defaults:
* `max_attributes`: 128 (or `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` env var)
* `max_events`: 128 (or `OTEL_SPAN_EVENT_COUNT_LIMIT` env var)
* `max_links`: 128 (or `OTEL_SPAN_LINK_COUNT_LIMIT` env var)
* `max_event_attributes`: 128 (or `OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT` env var)
* `max_link_attributes`: 128 (or `OTEL_LINK_ATTRIBUTE_COUNT_LIMIT` env var)
* `max_span_attribute_length`: None/unlimited
(or `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` env var)
Override these by passing a custom `SpanLimits` object or by setting the
corresponding environment variables.
The sampler for deciding which spans to record. Defaults to the
parent-based always-on OpenTelemetry sampler (100% sampling).
The compression algorithm for exporting traces. Can be
`Compression.Gzip`, `Compression.Deflate`, or `Compression.NoCompression`.
Only used when `otlp_enabled=True`. Defaults to `Compression.Gzip`.
If `True`, span data is **also** saved to a local
JSONL file in a custom Fiddler format. This is **additive** — OTLP export to
Fiddler continues as normal (unless `otlp_enabled=False`). Setting this to
`True` does **not** suppress the OTLP export. Useful for keeping a local copy
of trace data.
**Note:** This JSONL format is **not** compatible with the Fiddler S3 connector.
For S3 ingestion use `otlp_json_capture_enabled=True` instead.
Path to the JSONL file where trace data will be saved.
Only used when `jsonl_capture_enabled=True`.
Override with the `FIDDLER_JSONL_FILE` environment variable.
If `True`, traces are written to local `.json`
files in standard OTLP JSON format (`ExportTraceServiceRequest` envelope). Files
are written to `otlp_json_output_dir`. This format is **directly compatible** with
the Fiddler S3 connector — upload the generated files to S3 and the connector ingests
them without any reformatting. This is **additive** relative to OTLP export; combine
with `otlp_enabled=False` to write to files only (offline / S3 routing mode).
Directory path where OTLP JSON files are written when
`otlp_json_capture_enabled=True`. The directory is created automatically if it
does not exist. Each batch of spans is written to a separate timestamped `.json`
file.
If `True`, large inline base64 content in span
attributes is uploaded to S3 via `POST /v3/files/upload` and replaced with
`fiddler-file://` URIs before the span is exported. This prevents trace loss
at the 10MB Kafka span limit. Requires `otlp_enabled=True` (needs `url` and
`api_key`). Defaults to `False`.
## Raises
* **ValueError** – If `application_id` is not a valid UUID4.
* **ValueError** – If `otlp_enabled=True` and `api_key` is empty or not provided.
* **ValueError** – If `otlp_enabled=True` and `url` is empty, missing a scheme,
or uses a scheme other than `http`/`https`.
**Basic connection to your Fiddler instance**:
```python theme={null}
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID', # UUID4, required in all modes
api_key='YOUR_API_KEY',
url='https://your-instance.fiddler.ai',
)
```
**High-volume applications with custom configuration**:
```python theme={null}
from opentelemetry.sdk.trace import SpanLimits, sampling
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID',
api_key='YOUR_API_KEY',
url='https://your-instance.fiddler.ai',
span_limits=SpanLimits(
max_span_attributes=64, # Reduce from default 128
max_span_attribute_length=2048, # Limit from default None (unlimited)
),
sampler=sampling.TraceIdRatioBased(0.1), # Sample 10% of traces
compression=Compression.Gzip,
)
```
**Local development with console output** (traces still sent to Fiddler):
```python theme={null}
# console_tracer=True prints spans to stdout AND continues to export to Fiddler.
# It does NOT suppress OTLP export.
client = FiddlerClient(
application_id='00000000-0000-0000-0000-000000000000',
api_key='dev-key',
url='http://localhost:4318',
console_tracer=True,
)
```
**Offline / S3 routing mode** (no data sent directly to Fiddler):
```python theme={null}
# otlp_enabled=False → disables direct OTLP export; api_key and url not needed
# otlp_json_capture_enabled=True → writes traces to local .json files in standard
# OTLP JSON format (ExportTraceServiceRequest)
# application_id is still required so the S3 connector routes traces correctly
#
# Upload files from otlp_json_output_dir to your S3 bucket.
# The Fiddler S3 connector reads them directly with no reformatting required.
client = FiddlerClient(
application_id='YOUR_APPLICATION_ID',
otlp_enabled=False,
otlp_json_capture_enabled=True,
otlp_json_output_dir='./fiddler_traces', # default: 'fiddler_traces'
)
```
## **enter**()
Context manager entry.
### Returns
`FiddlerClient`
## **exit**()
Context manager exit — flushes and shuts down the tracer provider.
## force\_flush()
Flush pending spans to the exporter.
### Parameters
Maximum time to wait for flush in milliseconds.
### Returns
True if flush completed within the timeout, False otherwise.
## *async* aflush()
Async version of [`force_flush()`](#force_flush).
Runs the flush in a thread pool so the event loop is not blocked.
### Returns
`bool`
## shutdown()
Shut down the tracer provider after flushing pending spans.
Safe to call multiple times. The atexit handler is unregistered on first call.
## *async* ashutdown()
Async version of [`shutdown()`](#shutdown).
Runs flush and shutdown in a thread pool so the event loop is not blocked.
## get\_tracer\_provider()
Return the OpenTelemetry TracerProvider, initializing it on first call.
### Returns
The configured TracerProvider.
### Raises
**RuntimeError** – If initialization fails.
## update\_resource()
Update the OTel resource with additional attributes.
Must be called **before** [`get_tracer()`](#get_tracer) is invoked.
### Parameters
Key-value pairs to merge into the resource.
### Raises
**ValueError** – If the tracer has already been initialized.
## get\_tracer()
Return an OTel tracer for creating spans.
Initializes the tracer on the first call.
### Returns
The OTel tracer instance.
### Raises
**RuntimeError** – If tracer initialization fails.
## start\_as\_current\_span()
Create a span using a context manager (automatic lifecycle management).
### Parameters
Name for the span.
Span type — `"span"`, `"generation"`, `"chain"`, or `"tool"`.
### Returns
Span wrapper with context manager support.
## start\_span()
Create a span with manual lifecycle control. Caller must call `span.end()`.
### Parameters
Name for the span.
Span type — `"span"`, `"generation"`, `"chain"`, or `"tool"`.
### Returns
Span wrapper requiring an explicit `end()` call.
# FiddlerGeneration
Source: https://docs.fiddler.ai/sdk-api/otel/fiddler-generation
Wrapper for LLM generation spans with semantic convention helpers.
Wrapper for LLM generation spans with semantic convention helpers.
Initialize LLM generation wrapper.
## **enter**()
Enter context and set LLM type.
### Returns
`FiddlerGeneration`
## set\_model()
Set the LLM model name (gen\_ai.request.model).
## set\_system()
Set the LLM system/provider (gen\_ai.system).
## set\_system\_prompt()
Set the system prompt (gen\_ai.llm.input.system).
## set\_user\_prompt()
Set the user prompt (gen\_ai.llm.input.user).
### Parameters
Plain text string, or a list of content parts in OpenAI
multimodal format (e.g. `[{'type': 'text', 'text': '...'},
{'type': 'image_url', 'image_url': {'url': 'data:...'}}]`).
Lists are auto-serialized to JSON.
## set\_completion()
Set the LLM completion/output (gen\_ai.llm.output).
## set\_usage()
Set token usage information (gen\_ai.usage.\*).
### Parameters
Number of input/prompt tokens.
Number of output/completion tokens.
Total tokens; computed from input + output when omitted.
## set\_context()
Set additional context (gen\_ai.llm.context).
## set\_messages()
Set input messages in OpenAI chat format (gen\_ai.input.messages).
Accepts simple format: `[{'role': 'user', 'content': '...'}]`
Auto-converts to OTel format with `parts`.
## set\_output\_messages()
Set output messages in OpenAI chat format (gen\_ai.output.messages).
Accepts simple format: `[{'role': 'assistant', 'content': '...'}]`
Auto-converts to OTel format with `parts`.
## set\_tool\_definitions()
Set available tool definitions for this LLM call (gen\_ai.tool.definitions).
# FiddlerResourceAttributes
Source: https://docs.fiddler.ai/sdk-api/otel/fiddler-resource-attributes
Constants for Fiddler OpenTelemetry resource attributes.
Constants for Fiddler OpenTelemetry resource attributes.
## APPLICATION\_ID *= 'application.id'*
# FiddlerSpan
Source: https://docs.fiddler.ai/sdk-api/otel/fiddler-span
Wrapper around OpenTelemetry span with simplified helper methods.
Wrapper around OpenTelemetry span with simplified helper methods.
Initialize wrapper around an OTel span or context manager.
## Parameters
An OTel span or context manager.
Optional `MediaUploader`
for normalizing large base64 content before span export.
## **enter**()
Enter context manager and start span.
### Returns
`FiddlerSpan`
## **exit**()
Exit context manager, record exceptions, and end span.
### Returns
`Literal[False]`
## end()
Explicitly end the span. Must be called when using start\_span().
## set\_input()
Set input data. Auto-serializes dicts/lists to JSON.
## set\_output()
Set output data. Auto-serializes dicts/lists to JSON.
## set\_attribute()
Set a custom attribute on the span.
## update()
Bulk update multiple attributes.
## record\_exception()
Record an exception on the span.
## set\_agent\_name()
Set the agent name (gen\_ai.agent.name).
## set\_agent\_id()
Set the agent ID (gen\_ai.agent.id).
## set\_conversation\_id()
Set the conversation ID (gen\_ai.conversation.id).
# FiddlerSpanAttributes
Source: https://docs.fiddler.ai/sdk-api/otel/fiddler-span-attributes
Constants for Fiddler OpenTelemetry span attributes.
Constants for Fiddler OpenTelemetry span attributes.
## AGENT\_NAME *= 'gen\_ai.agent.name'*
## AGENT\_ID *= 'gen\_ai.agent.id'*
## CONVERSATION\_ID *= 'gen\_ai.conversation.id'*
## TYPE *= 'fiddler.span.type'*
## SERVICE\_VERSION *= 'service.version'*
## LLM\_INPUT\_SYSTEM *= 'gen\_ai.llm.input.system'*
## LLM\_INPUT\_USER *= 'gen\_ai.llm.input.user'*
## LLM\_OUTPUT *= 'gen\_ai.llm.output'*
## LLM\_CONTEXT *= 'gen\_ai.llm.context'*
## LLM\_REQUEST\_MODEL *= 'gen\_ai.request.model'*
## LLM\_SYSTEM *= 'gen\_ai.system'*
## LLM\_TOKEN\_COUNT\_INPUT *= 'gen\_ai.usage.input\_tokens'*
## LLM\_TOKEN\_COUNT\_OUTPUT *= 'gen\_ai.usage.output\_tokens'*
## LLM\_TOKEN\_COUNT\_TOTAL *= 'gen\_ai.usage.total\_tokens'*
## GEN\_AI\_INPUT\_MESSAGES *= 'gen\_ai.input.messages'*
## GEN\_AI\_OUTPUT\_MESSAGES *= 'gen\_ai.output.messages'*
## TOOL\_INPUT *= 'gen\_ai.tool.input'*
## TOOL\_OUTPUT *= 'gen\_ai.tool.output'*
## TOOL\_NAME *= 'gen\_ai.tool.name'*
## TOOL\_DEFINITIONS *= 'gen\_ai.tool.definitions'*
## SYSTEM\_PROMPT *= 'system\_prompt'*
## GENAI\_SYSTEM\_MESSAGE *= 'gen\_ai.system.message'*
# FiddlerSpanProcessor
Source: https://docs.fiddler.ai/sdk-api/otel/fiddler-span-processor
Span processor that automatically propagates attributes from parent to child spans.
Span processor that automatically propagates attributes from parent to child spans.
## DENORMALIZED\_ATTRIBUTES
## on\_start()
Called when a span starts. Automatically propagates attributes from parent.
### Parameters
The span being started.
The parent context, if any.
## force\_flush()
No buffered state; satisfy `SpanProcessor` interface.
### Returns
`bool`
# FiddlerTool
Source: https://docs.fiddler.ai/sdk-api/otel/fiddler-tool
Wrapper for tool/function call spans with semantic convention helpers.
Wrapper for tool/function call spans with semantic convention helpers.
Initialize tool span wrapper.
## **enter**()
Enter context and set tool type.
### Returns
`FiddlerTool`
## set\_tool\_name()
Set the tool/function name (gen\_ai.tool.name).
## set\_tool\_input()
Set the tool input arguments (gen\_ai.tool.input).
## set\_tool\_output()
Set the tool output/result (gen\_ai.tool.output).
## set\_tool\_definitions()
Set available tool definitions (gen\_ai.tool.definitions).
# get_client
Source: https://docs.fiddler.ai/sdk-api/otel/get-client
Return the global FiddlerClient singleton (first created in this process).
Return the global FiddlerClient singleton (first created in this process).
## Returns
The global client instance.
## Raises
**RuntimeError** – If no FiddlerClient has been initialized.
# get_current_span
Source: https://docs.fiddler.ai/sdk-api/otel/get-current-span
Get the currently active span as a Fiddler wrapper (inside a traced function).
Get the currently active span as a Fiddler wrapper (inside a traced function).
Retrieves the span from the current context and verifies it belongs to Fiddler's tracer
to maintain isolation from other OpenTelemetry tracers.
## Parameters
Wrapper type — `"span"`, `"generation"`, `"chain"`, or `"tool"`.
## Returns
Current span wrapper, or None if no active Fiddler span exists.
# Introduction
Source: https://docs.fiddler.ai/sdk-api/otel/index
Complete API reference for fiddler-otel
[](https://pypi.org/project/fiddler-otel/)
## Overview
Complete API reference documentation for the `fiddler-otel` package, the
framework-agnostic OpenTelemetry-based instrumentation layer for any
Python LLM or agent application. `fiddler-otel` is the canonical source
for Fiddler's span attributes, conversation tracking, and span processing
primitives; the LangChain, LangGraph, and Strands SDKs build on top of it
and re-export several of its symbols.
```bash theme={null}
pip install fiddler-otel
```
```python theme={null}
from fiddler_otel import FiddlerClient, trace
```
## Components
### Attributes
Span and resource attribute helpers. `set_conversation_id` and `add_session_attributes` are propagated through the current trace context; `FiddlerSpanAttributes`, `FiddlerResourceAttributes`, and `SpanType` enumerate the constant keys Fiddler recognizes.
* [add\_session\_attributes](/sdk-api/otel/add-session-attributes)
* [FiddlerResourceAttributes](/sdk-api/otel/fiddler-resource-attributes)
* [FiddlerSpanAttributes](/sdk-api/otel/fiddler-span-attributes)
* [set\_conversation\_id](/sdk-api/otel/set-conversation-id)
* [SpanType](/sdk-api/otel/span-type)
### Client
Client initialization and configuration. `FiddlerClient` wires the OTLP exporter, span processor, and resource attributes into the global tracer provider; `get_client` returns the active singleton.
* [FiddlerClient](/sdk-api/otel/fiddler-client)
* [get\_client](/sdk-api/otel/get-client)
### Decorators
Function-level tracing helpers. `@trace` instruments arbitrary Python functions; `get_current_span` returns the active span for attribute mutation inside a traced call.
* [get\_current\_span](/sdk-api/otel/get-current-span)
* [trace](/sdk-api/otel/trace)
### JSONL Capture
JSONL span exporter for offline analysis. Writes a structured trace record per span to a local file alongside (or instead of) OTLP export, useful for debugging and reproducing customer issues.
* [JSONLSpanExporter](/sdk-api/otel/jsonl-span-exporter)
### Span Processor
Custom OpenTelemetry span processor that enriches outgoing spans with Fiddler resource attributes before export.
* [FiddlerSpanProcessor](/sdk-api/otel/fiddler-span-processor)
### Span Wrapper
Manual span creation primitives for code paths that don't fit the decorator model. `FiddlerSpan` is the base; `FiddlerChain`, `FiddlerGeneration`, and `FiddlerTool` carry semantic conventions for retrieval/agent steps, LLM calls, and tool invocations.
* [FiddlerChain](/sdk-api/otel/fiddler-chain)
* [FiddlerGeneration](/sdk-api/otel/fiddler-generation)
* [FiddlerSpan](/sdk-api/otel/fiddler-span)
* [FiddlerTool](/sdk-api/otel/fiddler-tool)
# JSONLSpanExporter
Source: https://docs.fiddler.ai/sdk-api/otel/jsonl-span-exporter
SpanExporter that captures spans using JSONLSpanCapture.
SpanExporter that captures spans using JSONLSpanCapture.
## Parameters
The JSONLSpanCapture instance to use.
## export()
Export spans by capturing them with JSONLSpanCapture.
### Returns
`SpanExportResult`
# set_conversation_id
Source: https://docs.fiddler.ai/sdk-api/otel/set-conversation-id
Set the conversation ID for the current execution context.
Set the conversation ID for the current execution context.
The conversation ID is propagated to all spans created in the current
thread or async coroutine, allowing the Fiddler dashboard to filter and
display the full ordered sequence of operations for a single conversation.
This value persists until it is called again with a new ID.
## Parameters
Unique identifier for the conversation session.
## Returns
None Example: default import uuid from fiddler\_otel import set\_conversation\_id set\_conversation\_id(str(uuid.uuid4())) # All spans created in this thread or coroutine after this call # carry the same conversation\_id, until set\_conversation\_id is # called again with a new value.
# SpanType
Source: https://docs.fiddler.ai/sdk-api/otel/span-type
Constants for Fiddler OpenTelemetry span types.
Constants for Fiddler OpenTelemetry span types.
## AGENT *= 'agent'*
## CHAIN *= 'chain'*
## TOOL *= 'tool'*
## LLM *= 'llm'*
## OTHER *= 'other'*
# trace
Source: https://docs.fiddler.ai/sdk-api/otel/trace
Decorator for automatic function tracing with input/output capture.
Decorator for automatic function tracing with input/output capture.
Uses the global FiddlerClient when `client=` is not passed.
Supports both sync and async functions.
## Parameters
Function to decorate.
Custom span name (defaults to the function name).
Span type — `"span"`, `"generation"`, `"chain"`, or `"tool"`.
Capture function arguments as span input.
Capture the return value as span output.
FiddlerClient instance (defaults to `get_client()`).
LLM model name; sets `gen_ai.request.model`.
User identifier; sets `user.id`.
Version string; sets `service.version`.
LLM provider; sets `gen_ai.system`.
## Returns
The decorated function.
# AlertCondition
Source: https://docs.fiddler.ai/sdk-api/python-client/alert-condition
Alert trigger conditions for metric comparisons.
Alert trigger conditions for metric comparisons.
Defines the comparison operator used to evaluate whether an alert should trigger
based on the metric value and threshold.
## GREATER
Trigger when metric value > threshold.
Common for drift detection, error rates, latency spikes.
## LESSER
Trigger when metric value \< threshold.
Common for accuracy drops, traffic decreases, availability issues.
### Example
```python theme={null}
# Alert when data drift exceeds 5%
drift_alert = AlertRule(
condition=AlertCondition.GREATER,
threshold=0.05
)
# Alert when model accuracy drops below 90%
accuracy_alert = AlertRule(
condition=AlertCondition.LESSER,
threshold=0.90
)
```
## GREATER *= 'greater'*
## LESSER *= 'lesser'*
# AlertRecord
Source: https://docs.fiddler.ai/sdk-api/python-client/alert-record
Alert record representing a triggered alert instance.
Alert record representing a triggered alert instance.
An AlertRecord captures the details of a specific alert trigger event, including
the metric values, thresholds, and context that caused an AlertRule to fire.
Alert records provide essential data for monitoring analysis and troubleshooting.
## Example
```python theme={null}
# List recent critical alerts
critical_alerts = [
record for record in AlertRecord.list(
alert_rule_id=drift_alert.id,
start_time=datetime.now() - timedelta(days=3)
)
if record.severity == "CRITICAL"
]
# Analyze alert details
for alert in critical_alerts:
print(f"Alert triggered at {alert.created_at}")
print(f"Metric value: {alert.alert_value:.3f}")
print(f"Critical threshold: {alert.critical_threshold:.3f}")
if alert.feature_name:
print(f"Feature: {alert.feature_name}")
print(f"Message: {alert.message}")
print("—")
# Check for alert patterns
hourly_alerts = {}
for alert in AlertRecord.list(alert_rule_id=perf_alert.id):
hour = alert.created_at.hour
hourly_alerts[hour] = hourly_alerts.get(hour, 0) + 1
print("Alerts by hour:", hourly_alerts)
```
Alert records are read-only entities created automatically by the Fiddler
platform when AlertRules trigger. They cannot be created or modified directly
but provide valuable historical data for analysis and debugging.
Initialize an AlertRecord instance.
Creates an alert record object for representing triggered alert instances.
Alert records are typically created automatically by the Fiddler platform
when AlertRules trigger, rather than being instantiated directly by users.
Alert records are read-only entities that capture historical alert
trigger events. They are created automatically by the system and
cannot be modified after creation.
## *classmethod* list()
List alert records triggered by a specific alert rule.
Retrieves historical alert records for analysis and troubleshooting. This method
provides access to all alert trigger events within a specified time range,
enabling pattern analysis and threshold tuning.
### Parameters
The unique identifier of the AlertRule to retrieve records for.
Must be a valid alert rule UUID.
Start time for filtering alert records. If None, defaults to
7 days ago. Used to define the beginning of the query window.
End time for filtering alert records. If None, defaults to
current time. Used to define the end of the query window.
List of field names for result ordering. Prefix with "-" for
descending order (e.g., \["-created\_at"] for newest first).
### Yields
`AlertRecord` – Alert record instances with complete
trigger details and context information.
### Returns
`Iterator[AlertRecord]`
### Example
```python theme={null}
# Get recent alerts for analysis
recent_alerts = list(AlertRecord.list(
alert_rule_id=drift_alert.id,
start_time=datetime.now() - timedelta(days=3),
ordering=["-created_at"] # Newest first
))
# Analyze alert frequency
print(f"Total alerts in last 3 days: {len(recent_alerts)}")
critical_count = sum(1 for a in recent_alerts if a.severity == "CRITICAL")
print(f"Critical alerts: {critical_count}")
# Check alert patterns by feature
feature_alerts = {}
for alert in recent_alerts:
if alert.feature_name:
feature_alerts[alert.feature_name] = feature_alerts.get(alert.feature_name, 0) + 1
print("Alerts by feature:", feature_alerts)
# Analyze threshold violations
for alert in recent_alerts[:5]: # Latest 5 alerts
violation_ratio = alert.alert_value / alert.critical_threshold
print(f"Alert value: {alert.alert_value:.3f} "
f"({violation_ratio:.1%} of threshold)")
```
Results are paginated automatically. The default time range is 7 days
to balance performance with useful historical context. Use ordering
parameters to get the most relevant results first.
# AlertRule
Source: https://docs.fiddler.ai/sdk-api/python-client/alert-rule
Alert rule for automated monitoring and alerting in ML systems.
Alert rule for automated monitoring and alerting in ML systems.
An AlertRule defines conditions that automatically trigger notifications when
ML model metrics exceed specified thresholds. Alert rules are essential for
proactive monitoring of model performance, data drift, and operational issues.
## Example
```python theme={null}
# Create feature drift alert
drift_alert = AlertRule(
name="credit_score_drift",
model_id=model.id,
metric_id="drift_score",
priority=Priority.HIGH,
compare_to=CompareTo.BASELINE,
condition=AlertCondition.GT,
bin_size=BinSize.HOUR,
critical_threshold=0.8,
warning_threshold=0.6,
baseline_id=baseline.id,
columns=["credit_score", "income"]
).create()
# Create performance degradation alert
perf_alert = AlertRule(
name="accuracy_drop",
model_id=model.id,
metric_id="accuracy",
priority=Priority.MEDIUM,
compare_to=CompareTo.TIME_PERIOD,
condition=AlertCondition.LESSER,
bin_size=BinSize.DAY,
critical_threshold=0.85,
compare_bin_delta=7 # Compare to 7 days ago
).create()
# Configure notifications
drift_alert.set_notification_config(
emails=["[ml-team@company.com](mailto:ml-team@company.com)", "[data-team@company.com](mailto:data-team@company.com)"],
pagerduty_services=["ML_ALERTS"],
pagerduty_severity="critical"
)
```
Alert rules continuously monitor metrics and trigger notifications when
thresholds are exceeded. Use appropriate evaluation delays to avoid
false positives from temporary data fluctuations.
Initialize an AlertRule instance.
Creates an alert rule configuration for automated monitoring of ML model metrics.
The alert rule defines conditions that trigger notifications when thresholds are
exceeded, enabling proactive monitoring of model performance and data quality.
## Parameters
Human-readable name for the alert rule. Should be descriptive
and unique within the model context.
UUID of the model this alert rule monitors. Must be a valid
model that exists in the Fiddler platform.
ID of the metric to monitor (e.g., "drift\_score", "accuracy",
"precision", "recall", custom metric IDs).
Alert priority level (HIGH, MEDIUM, LOW). Determines urgency
and routing of notifications.
Comparison method for threshold evaluation:
* BASELINE: Compare against a fixed baseline
* TIME\_PERIOD: Compare against previous time period
* RAW\_VALUE: Compare against absolute threshold
Alert condition (GT, LT, OUTSIDE\_RANGE). Defines when
the alert should trigger relative to the threshold.
Time aggregation window (HOUR, DAY, WEEK). Controls how
data is grouped for metric calculation.
Threshold calculation method (MANUAL or AUTO).
MANUAL uses user-defined thresholds, AUTO calculates
dynamic thresholds based on historical data.
Parameters for automatic threshold calculation.
Used when threshold\_type is AUTO.
Critical alert threshold value. Triggers high-priority
notifications when exceeded.
List of feature columns to monitor. For feature-specific
drift alerts. If None, monitors all features.
UUID of the baseline to compare against. Required when
compare\_to is BASELINE.
UUID of the data segment to monitor. For segment-specific
monitoring (optional).
Number of time bins to compare against. Used with
TIME\_PERIOD comparison (e.g., 7 for week-over-week).
Delay in minutes before evaluating alerts. Helps
avoid false positives from incomplete data.
Custom category for organizing alerts. Useful for grouping
related alerts in dashboards.
## Example
```python theme={null}
# Feature drift alert with baseline comparison
drift_alert = AlertRule(
name="income_drift_detection",
model_id=model.id,
metric_id="drift_score",
priority=Priority.HIGH,
compare_to=CompareTo.BASELINE,
condition=AlertCondition.GT,
bin_size=BinSize.HOUR,
critical_threshold=0.8,
warning_threshold=0.6,
baseline_id=baseline.id,
columns=["income", "credit_score"],
evaluation_delay=15, # 15 minute delay
category="data_quality"
)
# Performance monitoring with time comparison
perf_alert = AlertRule(
name="weekly_accuracy_check",
model_id=model.id,
metric_id="accuracy",
priority=Priority.MEDIUM,
compare_to=CompareTo.TIME_PERIOD,
condition=AlertCondition.LESSER,
bin_size=BinSize.DAY,
critical_threshold=0.85,
compare_bin_delta=7, # Compare to 7 days ago
category="performance"
)
```
After initialization, call create() to persist the alert rule to the
Fiddler platform. Alert rules begin monitoring immediately after creation.
## *classmethod* get()
Retrieve an alert rule by its unique identifier.
Fetches an alert rule from the Fiddler platform using its UUID. This method
returns the complete alert rule configuration including thresholds, notification
settings, and monitoring status.
### Parameters
The unique identifier (UUID) of the alert rule to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The alert rule instance with all
configuration and metadata populated from the server.
### Raises
* **NotFound** – If no alert rule exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Retrieve alert rule by ID
alert_rule = AlertRule.get(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Alert: {alert_rule.name}")
print(f"Metric: {alert_rule.metric_id}")
print(f"Priority: {alert_rule.priority}")
print(f"Critical threshold: {alert_rule.critical_threshold}")
# Check notification configuration
notification_config = alert_rule.get_notification_config()
print(f"Email recipients: {notification_config.emails}")
```
This method makes an API call to fetch the latest alert rule configuration
from the server, including any recent threshold or notification updates.
## *classmethod* list()
Get a list of all alert rules in the organization.
### Parameters
list from the specified model
list rules set on the specified metric id
list rules set on the specified list of columns
order result as per list of fields. \["-field\_name"] for descending
### Returns
paginated list of alert rules for the specified filters
## delete()
Delete an alert rule.
## create()
Create a new alert rule.
### Returns
`AlertRule`
## update()
Update an existing alert rule.
## enable\_notifications()
Enable notifications for an alert rule
## disable\_notifications()
Disable notifications for an alert rule
## set\_notification\_config()
Set notification config for an alert rule
### Parameters
list of emails
list of pagerduty services
severity of pagerduty
list of webhooks UUIDs
### Returns
NotificationConfig object
## get\_notification\_config()
Get notifications config for an alert rule
### Returns
NotificationConfig object
# AlertThresholdAlgo
Source: https://docs.fiddler.ai/sdk-api/python-client/alert-threshold-algo
Threshold determination algorithms for alert rules.
Threshold determination algorithms for alert rules.
Defines how alert thresholds are calculated - either manually specified
or automatically computed based on historical data patterns.
## MANUAL
User-specified static thresholds.
Provides full control but requires domain knowledge to set appropriately.
## STD\_DEV\_AUTO\_THRESHOLD
Automatic thresholds based on standard deviation.
Calculates thresholds as mean ± (multiplier × std\_dev)
from historical data. Adapts to data patterns automatically.
### Example
```python theme={null}
# Manual threshold - user knows the acceptable drift limit
manual_alert = AlertRule(
threshold_type=AlertThresholdAlgo.MANUAL,
critical_threshold=0.1,
warning_threshold=0.05
)
# Auto threshold - let system learn from historical patterns
auto_alert = AlertRule(
threshold_type=AlertThresholdAlgo.STD_DEV_AUTO_THRESHOLD,
auto_threshold_params={
'warning_multiplier': 2.0, # 2 std devs for warning
'critical_multiplier': 3.0 # 3 std devs for critical
}
)
```
## MANUAL *= 'manual'*
## STD\_DEV\_AUTO\_THRESHOLD *= 'standard\_deviation\_auto\_threshold'*
## **str**()
Return the string value of the enum.
### Returns
The enum's string value for serialization and display.
### Example
```python theme={null}
algo = AlertThresholdAlgo.MANUAL
str(algo)
'manual'
```
# ApiError
Source: https://docs.fiddler.ai/sdk-api/python-client/api-error
Raised when the Fiddler API returns an HTTP error response.
Raised when the Fiddler API returns an HTTP error response.
This exception represents errors returned by the Fiddler platform API,
including both client errors (4xx status codes) and server errors
(5xx status codes). It contains detailed information about the error
including the HTTP status code, error message, and any additional
error details provided by the API.
ApiError serves as the base class for more specific API errors like
NotFound and Conflict, but can also be raised directly for other
HTTP error status codes.
## Examples
Handling general API errors:
```python theme={null}
try:
model = client.get_model("nonexistent-model")
except ApiError as e:
print(f"API Error {e.code}: {e.message}")
if e.errors:
print(f"Details: {e.errors}")
```
Handling specific status codes:
```python theme={null}
try:
# API operation
pass
except ApiError as e:
if e.code == 429: # Rate limit
print("Rate limited, retrying later...")
time.sleep(60)
elif e.code >= 500: # Server error
print("Server error, contact support")
else:
print(f"Client error: {e.message}")
```
## reason
## code
# ArtifactStatus
Source: https://docs.fiddler.ai/sdk-api/python-client/artifact-status
Model artifact upload and deployment status.
Model artifact upload and deployment status.
This enum tracks the status of model artifacts in Fiddler, indicating
whether explainability features are available and what type of model
deployment is active.
Artifact Types:
* **No Model**: No artifacts uploaded, monitoring only
* **Surrogate**: Fiddler-generated surrogate model for explainability
* **User Uploaded**: User-provided model artifacts for full explainability
## Examples
Checking artifact status and capabilities:
```python theme={null}
# Check current artifact status
model = fdl.Model.from_name('my_model', project_id=project.id)
if model.artifact_status == fdl.ArtifactStatus.NO_MODEL:
print("Monitoring only - no explainability features")
elif model.artifact_status == fdl.ArtifactStatus.SURROGATE:
print("Surrogate model available - basic explainability")
elif model.artifact_status == fdl.ArtifactStatus.USER_UPLOADED:
print("Full model artifacts - complete explainability")
# Upload model artifacts to enable explainability
if model.artifact_status == fdl.ArtifactStatus.NO_MODEL:
job = model.add_artifact(
model_dir='./model_package/',
deployment_params=fdl.DeploymentParams(
artifact_type=fdl.ArtifactType.PYTHON_PACKAGE
)
)
job.wait()
```
Artifact status affects available explainability features. User-uploaded
artifacts provide the most comprehensive explanation capabilities.
## NO\_MODEL *= 'no\_model'*
No model artifacts have been uploaded.
The model exists in Fiddler for monitoring purposes only. Data drift
detection, performance monitoring, and alerting are available, but
explainability features are not accessible.
Available features:
* Data drift monitoring
* Performance metric tracking
* Alert rule configuration
* Dashboard visualization
* Data publishing and monitoring
Unavailable features:
* Point explainability
* Global feature importance
* Model artifact-based analysis
* Custom explanation methods
This is the default status for newly created models before any
artifacts are uploaded.
## SURROGATE *= 'surrogate'*
Surrogate model generated by Fiddler for explainability.
Fiddler has automatically generated a surrogate model based on your
published data to provide basic explainability features. The surrogate
model approximates your original model's behavior.
Available features:
* Basic point explainability
* Global feature importance
* Approximated explanations
* All monitoring features
Characteristics:
* Automatically generated by Fiddler
* Approximates original model behavior
* Provides reasonable explanation quality
* No additional setup required
Limitations:
* May not perfectly match original model
* Limited to surrogate model capabilities
* Cannot use custom explanation methods
## USER\_UPLOADED *= 'user\_uploaded'*
User-provided model artifacts have been uploaded.
Complete model artifacts have been uploaded, enabling full explainability
features with the actual model. This provides the highest quality
explanations and complete feature access.
Available features:
* Full point explainability with actual model
* Global feature importance from actual model
* Custom explanation methods (if defined)
* Model artifact-based analysis
* All monitoring and surrogate features
Characteristics:
* Uses actual uploaded model
* Highest explanation accuracy
* Supports custom explanation methods
* Complete feature access
Requirements:
* Model artifacts must be properly packaged
* Compatible with Fiddler's deployment environment
* May require specific Python dependencies
# ArtifactType
Source: https://docs.fiddler.ai/sdk-api/python-client/artifact-type
Model artifact types for deployment.
Model artifact types for deployment.
**Values:**
* `SURROGATE` - SURROGATE
Surrogate model artifact generated by Fiddler.
## PYTHON\_PACKAGE *= 'PYTHON\_PACKAGE'*
User-provided Python package artifact.
# AsyncJobFailed
Source: https://docs.fiddler.ai/sdk-api/python-client/async-job-failed
Raised when an asynchronous job fails to execute successfully.
Raised when an asynchronous job fails to execute successfully.
This exception is thrown when long-running operations (such as model training,
data processing, or batch operations) fail to complete successfully. The job
may have been submitted successfully but encountered an error during execution.
Async jobs in Fiddler include operations like model artifact uploads, baseline
computations, batch data ingestion, and other time-intensive operations that
run in the background.
## Examples
Handling async job failures:
```python theme={null}
try:
job = model.upload_artifact(artifact_path)
job.wait() # Wait for completion
except AsyncJobFailed as e:
print(f"Job failed: {e.message}")
# Check job logs or retry operation
```
Monitoring job status to avoid exceptions:
```python theme={null}
job = model.upload_artifact(artifact_path)
while not job.is_complete():
if job.status == JobStatus.FAILED:
print(f"Job failed: {job.error_message}")
break
time.sleep(5)
```
# Baseline
Source: https://docs.fiddler.ai/sdk-api/python-client/baseline
Baseline for drift detection and model performance monitoring.
Baseline for drift detection and model performance monitoring.
A Baseline defines a reference point for comparing production data against expected
patterns. It serves as the foundation for detecting data drift, model performance
degradation, and distributional changes in ML systems.
## Example
```python theme={null}
# Create a static baseline from training data
baseline = Baseline(
name="production_baseline_v1",
model_id=model.id,
environment=EnvType.PRE_PRODUCTION,
dataset_id=training_dataset.id,
type_="STATIC"
).create()
# Create a rolling 30-day baseline
rolling_baseline = Baseline(
name="rolling_30day",
model_id=model.id,
environment=EnvType.PRODUCTION,
type_="ROLLING_WINDOW",
window_bin_size=WindowBinSize.DAY,
offset_delta=30
).create()
# Monitor drift detection
print(f"Baseline '{baseline.name}' has {baseline.row_count} records")
print(f"Created: {baseline.created_at}")
```
Baselines are immutable once created. To modify baseline parameters,
create a new baseline and update your monitoring configurations.
Initialize a Baseline instance.
Creates a baseline configuration for drift detection and monitoring. The baseline
serves as a reference point for comparing production data against expected patterns.
## Parameters
Human-readable name for the baseline. Should be descriptive
and unique within the model context.
UUID of the model this baseline belongs to. Must be a valid
model that exists in the Fiddler platform.
Environment type (PRE\_PRODUCTION or PRODUCTION).
Determines the data environment this baseline monitors.
Baseline type. Supported values:
* "STATIC": Fixed dataset reference (requires dataset\_id)
* "ROLLING\_WINDOW": Sliding time window (requires offset\_delta)
* "PREVIOUS\_PERIOD": Previous time period comparison
UUID of the reference dataset. Required for STATIC baselines,
optional for time-based baselines.
Start timestamp for time-based baselines (Unix timestamp).
Defines the beginning of the reference period.
Time offset in days for rolling/previous period baselines.
For ROLLING\_WINDOW: size of the sliding window.
For PREVIOUS\_PERIOD: how far back to compare.
Aggregation window for time-series analysis.
Controls how data is grouped for comparison.
## Example
```python theme={null}
# Static baseline from training data
baseline = Baseline(
name="training_baseline_v2",
model_id=model.id,
environment=EnvType.PRE_PRODUCTION,
dataset_id=training_dataset.id,
type_="STATIC"
)
# Rolling 7-day window baseline
rolling_baseline = Baseline(
name="weekly_rolling",
model_id=model.id,
environment=EnvType.PRODUCTION,
type_="ROLLING_WINDOW",
offset_delta=7,
window_bin_size=WindowBinSize.DAY
)
# Previous month comparison baseline
monthly_baseline = Baseline(
name="month_over_month",
model_id=model.id,
environment=EnvType.PRODUCTION,
type_="PREVIOUS_PERIOD",
offset_delta=30,
window_bin_size=WindowBinSize.DAY
)
```
After initialization, call create() to persist the baseline to the
Fiddler platform. The baseline configuration cannot be modified
after creation.
## *classmethod* get()
Retrieve a baseline by its unique identifier.
Fetches a baseline from the Fiddler platform using its UUID. This method
returns the complete baseline configuration including metadata and statistics.
### Parameters
The unique identifier (UUID) of the baseline to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The baseline instance with all configuration
and metadata populated from the server.
### Raises
* **NotFound** – If no baseline exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Retrieve baseline by ID
baseline = Baseline.get(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Baseline: {baseline.name}")
print(f"Type: {baseline.type}")
print(f"Environment: {baseline.environment}")
print(f"Records: {baseline.row_count}")
# Check baseline configuration
if baseline.type == "STATIC":
print(f"Reference dataset: {baseline.dataset_id}")
elif baseline.type == "ROLLING_WINDOW":
print(f"Window size: {baseline.offset_delta} days")
print(f"Bin size: {baseline.window_bin_size}")
```
This method makes an API call to fetch the latest baseline information
from the server, including any updated statistics or metadata.
## *classmethod* from\_name()
Get the baseline instance of a model from baseline name
### Parameters
Baseline name
Model identifier
### Returns
Baseline instance
## *classmethod* list()
Get a list of all baselines of a model.
### Returns
`Iterator[Baseline]`
## create()
Create a new baseline.
### Returns
`Baseline`
## delete()
Delete a baseline.
# BaselineCompact
Source: https://docs.fiddler.ai/sdk-api/python-client/baseline-compact
Fetch baseline instance
## id
## name
## fetch()
Fetch baseline instance
### Returns
[`Baseline`](/sdk-api/python-client/baseline)
# BaselineType
Source: https://docs.fiddler.ai/sdk-api/python-client/baseline-type
Baseline computation strategies for data drift detection in Fiddler.
Baseline computation strategies for data drift detection in Fiddler.
Baseline types determine how reference data is defined and used for comparison
with production model behavior. Fiddler supports static and rolling baselines,
each serving different monitoring needs and use cases.
Static Baselines:
* Fixed reference point that doesn't change over time
* Can be created from pre-production data (training/test sets) or production data
* Consistent comparison point for detecting absolute drift
* Ideal for compliance, audit requirements, and stable model environments
* Pre-production static baselines created via model.publish() with PRE\_PRODUCTION environment
* Production static baselines defined using specific time ranges
Rolling Baselines:
* Dynamic sliding window that shifts with time
* Always maintains fixed time distance from current data (e.g., 4 weeks ago)
* Automatically adapts to gradual changes in data patterns
* Excellent for detecting sudden changes or anomalies in time-sensitive data
* Requires window\_bin\_size and offset\_delta parameters
Selection Guidelines:
* Use STATIC for regulatory compliance, model validation, and stable environments
* Use ROLLING for seasonal patterns, evolving data, and operational monitoring
* Static pre-production baselines are recommended for most use cases
* Rolling baselines work best with sufficient historical production data
## Example
```python theme={null}
# Static production baseline using time range
static_baseline = fdl.Baseline(
name="static_baseline",
model_id=model.id,
environment=fdl.EnvType.PRODUCTION,
type_=fdl.BaselineType.STATIC,
start_time=(datetime.now() - timedelta(days=30)).timestamp(),
end_time=(datetime.now() - timedelta(days=7)).timestamp()
).create()
# Rolling production baseline with monthly window
rolling_baseline = fdl.Baseline(
name="rolling_baseline",
model_id=model.id,
environment=fdl.EnvType.PRODUCTION,
type_=fdl.BaselineType.ROLLING,
window_bin_size=fdl.WindowBinSize.MONTH,
offset_delta=1 # 1 month offset
).create()
```
## STATIC
Fixed baseline using historical reference data or specific time ranges
## ROLLING
Dynamic sliding window baseline that shifts with time
## STATIC *= 'STATIC'*
## ROLLING *= 'ROLLING'*
# BinSize
Source: https://docs.fiddler.ai/sdk-api/python-client/bin-size
Time bin sizes for alert rule aggregation.
Time bin sizes for alert rule aggregation.
Defines the time window granularity for aggregating metrics when evaluating alert rules.
Smaller bin sizes provide more frequent monitoring but may be more sensitive to noise.
## Example
```python theme={null}
alert_rule = AlertRule(
name="Hourly Drift Check",
bin_size=BinSize.HOUR,
# ... other parameters
)
```
## HOUR *= 'Hour'*
## DAY *= 'Day'*
## WEEK *= 'Week'*
## MONTH *= 'Month'*
# Column
Source: https://docs.fiddler.ai/sdk-api/python-client/column
Represents a single column in a model schema with its metadata and constraints.
Represents a single column in a model schema with its metadata and constraints.
A Column defines the structure and properties of a data column that will be used
in a Fiddler model. It includes information about the column's data type, value
ranges, categorical values, binning configuration, and other metadata necessary
for proper data validation and monitoring.
This class is used within ModelSchema to define the complete structure of data
that a model expects to receive.
## Examples
Creating a numeric column:
```python theme={null}
column = Column(
name="age",
data_type=DataType.INTEGER,
min=0,
max=120
)
```
Creating a categorical column:
```python theme={null}
column = Column(
name="category",
data_type=DataType.CATEGORY,
categories=["A", "B", "C"]
)
```
Creating a vector column:
```python theme={null}
column = Column(
name="embedding",
data_type=DataType.VECTOR,
n_dimensions=128
)
```
## name
Column name provided by the customer
## data\_type
Data type of the column
## min
Min value of integer/float column
## max
Max value of integer/float column
## categories
List of unique values of a categorical column
## bins
Bins of integer/float column
## replace\_with\_nulls
Replace the list of given values to NULL if found in the events data
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
## n\_dimensions
Number of dimensions of a vector column
# CompareTo
Source: https://docs.fiddler.ai/sdk-api/python-client/compare-to
Comparison baseline types for alert rule thresholds.
Comparison baseline types for alert rule thresholds.
Determines what the current metric value should be compared against when
evaluating alert conditions.
## TIME\_PERIOD
Compare to a historical time period (relative comparison).
Useful for detecting changes over time, seasonal patterns, etc.
When using TIME\_PERIOD, the compare\_bin\_delta parameter specifies
how many time periods back to compare against.
## RAW\_VALUE
Compare to an absolute threshold value (absolute comparison).
Useful for hard limits and business rule enforcement.
When using RAW\_VALUE, compare\_bin\_delta is ignored.
When using TIME\_PERIOD, the allowed compare\_bin\_delta values depend on bin\_size:
### Example
```python theme={null}
# Compare current hourly drift to same hour yesterday (24 hours ago)
alert_rule = AlertRule(
compare_to=CompareTo.TIME_PERIOD,
bin_size=BinSize.HOUR,
compare_bin_delta=24 # 24 hours ago
)
# Compare daily metrics to last week (7 days ago)
alert_rule = AlertRule(
compare_to=CompareTo.TIME_PERIOD,
bin_size=BinSize.DAY,
compare_bin_delta=7 # 7 days ago
)
# Compare current accuracy to absolute minimum (no time comparison)
alert_rule = AlertRule(
compare_to=CompareTo.RAW_VALUE,
threshold=0.85 # Must be above 85%
)
```
## TIME\_PERIOD *= 'time\_period'*
## RAW\_VALUE *= 'raw\_value'*
# Conflict
Source: https://docs.fiddler.ai/sdk-api/python-client/conflict
Raised when a request conflicts with the current state of a resource (HTTP 409).
Raised when a request conflicts with the current state of a resource (HTTP 409).
This exception occurs when attempting to perform an operation that conflicts
with the current state of a resource. This typically happens when trying to
create resources that already exist, or when concurrent modifications cause
state conflicts.
Common scenarios include creating models or projects with names that already
exist, attempting to modify resources that are currently being processed,
or violating business rules that prevent certain operations.
## Examples
Handling resource conflicts:
```python theme={null}
try:
project = client.create_project(name="existing-project")
except Conflict as e:
print(f"Project already exists: {e.message}")
# Use existing project or choose different name
project = client.get_project("existing-project")
```
Handling state conflicts:
```python theme={null}
try:
model.update_status("active")
except Conflict as e:
print(f"Cannot change status: {e.message}")
# Wait for current operation to complete
```
Common Conflict scenarios:
```python theme={null}
- Creating resources with duplicate names
- Modifying resources during processing
- Violating business logic constraints
- Concurrent modification conflicts
```
## code
## reason
# ConnError
Source: https://docs.fiddler.ai/sdk-api/python-client/conn-error
Raised when a connection error occurs during HTTP requests.
Raised when a connection error occurs during HTTP requests.
This exception was used to indicate general connection problems when
trying to communicate with the Fiddler platform, such as network
unreachability, DNS resolution failures, or connection refused errors.
## Examples
Historical usage (now deprecated):
```python theme={null}
try:
# Network API call
pass
except ConnError as e:
print(f"Connection failed: {e.message}")
# Check network connectivity or URL configuration
```
## message
# ConnTimeout
Source: https://docs.fiddler.ai/sdk-api/python-client/conn-timeout
Raised when a connection timeout occurs during HTTP requests.
Raised when a connection timeout occurs during HTTP requests.
This exception was used to indicate that an HTTP request to the Fiddler
platform timed out while waiting for a response. Timeouts can occur due
to network issues, server overload, or requests taking longer than the
configured timeout period.
## Examples
Historical usage (now deprecated):
```python theme={null}
try:
# Long-running API call
pass
except ConnTimeout as e:
print(f"Request timed out: {e.message}")
# Retry with longer timeout or check network
```
## message
# Connection
Source: https://docs.fiddler.ai/sdk-api/python-client/connection
Manages authenticated connections to the Fiddler platform.
Manages authenticated connections to the Fiddler platform.
The Connection class handles all aspects of connecting to and communicating
with the Fiddler platform, including authentication, HTTP client management,
server version compatibility checking, and organization context management.
This class provides the foundation for all API interactions with Fiddler,
managing connection parameters, authentication tokens, and ensuring proper
communication protocols are established.
## Examples
Creating a basic connection:
```python theme={null}
connection = Connection(
url="https://your-fiddler-instance.com",
token="your-api-key"
)
```
Creating a connection with custom timeout and proxy:
```python theme={null}
connection = Connection(
url="https://your-fiddler-instance.com",
token="your-api-key",
timeout=(5.0, 30.0), # (connect_timeout, read_timeout)
proxies={"https": "https://proxy.company.com:8080"}
)
```
Creating a connection without SSL verification:
```python theme={null}
connection = Connection(
url="https://your-fiddler-instance.com",
token="your-api-key",
verify=False, # Not recommended for production
validate=False # Skip version compatibility check
)
```
Initialize a connection to the Fiddler platform.
## Parameters
The base URL to your Fiddler platform instance
API key obtained from the Fiddler UI Credentials tab
Dictionary mapping protocol to proxy URL for HTTP requests
HTTP request timeout settings (float or tuple of connect/read timeouts)
Whether to verify server's TLS certificate (default: True)
Whether to validate server/client version compatibility (default: True)
## Raises
* **ValueError** – If url or token parameters are empty
* **IncompatibleClient** – If server version is incompatible with client version
## *property* client
Get the HTTP request client instance for API communication.
### Returns
Configured HTTP client with authentication headers,
proxy settings, and timeout configurations.
## *property* server\_info
Get server information and metadata from the Fiddler platform.
### Returns
Server information including version, organization details,
and platform configuration.
## *property* server\_version
Get the semantic version of the connected Fiddler server.
### Returns
Semantic version object representing the server version.
## *property* organization\_name
Get the name of the connected organization.
### Returns
Name of the organization associated with this connection.
## *property* organization\_id
Get the UUID of the connected organization.
### Returns
Unique identifier of the organization associated with this connection.
# ConnectionMixin
Source: https://docs.fiddler.ai/sdk-api/python-client/connection-mixin
Mixin class providing connection-related functionality to other classes.
Mixin class providing connection-related functionality to other classes.
ConnectionMixin provides a standardized way for other classes to access
the global Fiddler connection instance and its associated properties.
This mixin enables classes throughout the Fiddler client to access
connection details, HTTP client functionality, and organization context
without directly managing connection state.
This pattern ensures consistent access to connection resources across
all client components while maintaining a clean separation of concerns.
## organization\_name()
Property access to organization name
## organization\_id()
Property access to organization UUID
## get\_organization\_name()
Class method to retrieve organization name
### Returns
`str`
## get\_organization\_id()
Class method to retrieve organization UUID
### Returns
`UUID`
### Examples
Using ConnectionMixin in a custom class:
```python theme={null}
class CustomModel(ConnectionMixin):
def fetch_data(self):
# Access HTTP client through mixin
response = self._client().get('/api/data')
return response.json()
def get_org_info(self):
# Access organization info through mixin
return {
'name': self.organization_name,
'id': str(self.organization_id)
}
```
Using class methods without instantiation:
```python theme={null}
org_name = SomeEntityClass.get_organization_name()
org_id = SomeEntityClass.get_organization_id()
```
## *property* organization\_name
Get the organization name from the connection.
### Returns
Name of the organization associated with the current connection.
## *property* organization\_id
Get the organization UUID from the connection.
### Returns
Unique identifier of the organization associated with the current connection.
## *classmethod* get\_organization\_name()
Get the organization name from the global connection.
### Returns
Name of the organization associated with the current connection.
## *classmethod* get\_organization\_id()
Get the organization UUID from the global connection.
### Returns
Unique identifier of the organization associated with the current connection.
# create_columns_from_df
Source: https://docs.fiddler.ai/sdk-api/python-client/create-columns-from-df
Helper function to create Columns from a pandas DataFrame column dtypes.
Helper function to create Columns from a pandas DataFrame column dtypes.
timedelta, period, interval & object dtypes are converted to string.
Sparse dtypes are not handled.
* **df**: Input pandas DataFrame
## Returns
ModelSchema
# CustomFeature
Source: https://docs.fiddler.ai/sdk-api/python-client/custom-feature
Base class for all custom feature types in Fiddler models.
Base class for all custom feature types in Fiddler models.
CustomFeature provides the foundation for creating specialized feature types
that enhance model monitoring and analysis. Custom features allow you to define
derived metrics, embeddings, and enrichments that extend beyond basic model
inputs and outputs for advanced drift detection and analysis.
This is an abstract base class that should not be instantiated directly.
Instead, use one of its concrete subclasses: Multivariate, VectorFeature,
TextEmbedding, ImageEmbedding, or Enrichment.
## Examples
Creating a multivariate feature from multiple columns:
```python theme={null}
feature = CustomFeature.from_columns(
custom_name="user_behavior_cluster",
cols=["clicks", "views", "time_spent"],
n_clusters=5
)
```
Creating a custom feature from a dictionary:
```python theme={null}
feature_dict = {
"name": "text_sentiment",
"type": "FROM_TEXT_EMBEDDING",
"column": "embedding_col",
"source_column": "review_text"
}
feature = CustomFeature.from_dict(feature_dict)
```
## name
## type
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
## *classmethod* from\_columns()
### Returns
[`Multivariate`](/sdk-api/python-client/multivariate)
## *classmethod* from\_dict()
### Returns
`Any`
## to\_dict()
### Returns
`Dict[str, Any]`
# CustomFeatureType
Source: https://docs.fiddler.ai/sdk-api/python-client/custom-feature-type
Types of custom features for advanced model monitoring.
Types of custom features for advanced model monitoring.
This enum defines different types of custom features that can be created
for advanced monitoring scenarios. Custom features enable monitoring of
complex data types, embeddings, and multi-column relationships.
Feature Categories:
* **Multi-column**: Features derived from multiple input columns
* **Vector-based**: Features from embedding or vector columns
* **Embedding-specific**: Specialized embedding monitoring
* **Enrichment**: Features from data enrichment processes
## Examples
Creating different types of custom features:
```python theme={null}
# Multi-column feature for monitoring column interactions
multivariate_feature = fdl.Multivariate(
name='user_profile',
columns=['age', 'income', 'location'],
monitor_components=True
)
# Vector feature for embedding monitoring
vector_feature = fdl.VectorFeature(
name='product_embedding',
column='product_vector',
n_clusters=10
)
# Text embedding feature with clustering
text_embedding = fdl.TextEmbedding(
name='review_sentiment',
column='review_embedding',
n_clusters=5,
n_tags=10
)
# Image embedding feature
image_embedding = fdl.ImageEmbedding(
name='image_features',
column='image_embedding',
n_clusters=8
)
# Enrichment feature for data validation
enrichment_feature = fdl.Enrichment(
name='email_validation',
enrichment='email_validation',
columns=['email_address'],
config={'strict': True}
## )
```
Custom features enable advanced monitoring capabilities but require
careful configuration to match your specific use case and data structure.
## FROM\_COLUMNS *= 'FROM\_COLUMNS'*
Multi-column derived features (Multivariate).
Used for creating custom features that monitor relationships and
interactions between multiple input columns. Enables detection of
drift patterns across column combinations.
Characteristics:
* Monitors multiple columns as a single feature
* Detects multi-dimensional drift patterns
* Can monitor individual components separately
* Supports complex feature interactions
Use cases:
* Geographic coordinates (latitude, longitude)
* User profiles (age, income, location)
* Product specifications (dimensions, weight, price)
* Time series components (trend, seasonality)
Configuration:
* Specify list of columns to monitor together
* Optional component monitoring
* Clustering for dimensionality reduction
## FROM\_VECTOR *= 'FROM\_VECTOR'*
Single vector column features (VectorFeature).
Used for monitoring embedding vectors or other high-dimensional
numerical arrays as single features. Enables clustering-based
drift detection and embedding analysis.
Characteristics:
* Monitors single vector/embedding column
* Clustering-based drift detection
* Dimensionality reduction visualization
* Vector similarity analysis
Use cases:
* Word embeddings (Word2Vec, GloVe)
* Neural network hidden layer outputs
* Feature vectors from autoencoders
* Learned representations
Configuration:
* Specify vector column name
* Set number of clusters for monitoring
* Optional source column reference
## FROM\_TEXT\_EMBEDDING *= 'FROM\_TEXT\_EMBEDDING'*
Text embedding features (TextEmbedding).
Specialized for monitoring text embeddings with text-specific
analysis capabilities. Includes TF-IDF summarization and
text-aware clustering.
Characteristics:
* Text-specific embedding analysis
* TF-IDF token summarization
* Text-aware clustering
* Semantic drift detection
Use cases:
* BERT, GPT embeddings
* Document embeddings
* Sentence transformers
* Text classification features
Configuration:
* Specify embedding column
* Set number of clusters
* Configure TF-IDF tags per cluster
## FROM\_IMAGE\_EMBEDDING *= 'FROM\_IMAGE\_EMBEDDING'*
Image embedding features (ImageEmbedding).
Specialized for monitoring image embeddings and visual features
extracted from images. Optimized for computer vision model
monitoring.
Characteristics:
* Image-specific embedding analysis
* Visual feature clustering
* Image-aware drift detection
* Computer vision optimizations
Use cases:
* CNN feature extractions
* Image classification embeddings
* Object detection features
* Visual similarity vectors
Configuration:
* Specify embedding column
* Set clustering parameters
* Image-specific preprocessing
## ENRICHMENT *= 'ENRICHMENT'*
Enrichment-derived features (Enrichment).
Used for features created through data enrichment processes
such as validation, transformation, or external data augmentation.
Enables monitoring of enriched data quality and consistency.
Characteristics:
* Derived from enrichment processes
* Data quality monitoring
* Validation result tracking
* Transformation monitoring
Use cases:
* Email validation results
* Address standardization
* Data quality scores
* External API enrichments
Configuration:
* Specify enrichment type
* Configure enrichment parameters
* Set input columns for enrichment
# CustomMetric
Source: https://docs.fiddler.ai/sdk-api/python-client/custom-metric
Custom metric for monitoring business-specific and domain-specific KPIs.
Custom metric for monitoring business-specific and domain-specific KPIs.
CustomMetric enables creation of user-defined metrics that calculate specific
values from model data using SQL-like expressions. Custom metrics extend
Fiddler's built-in monitoring capabilities to support business requirements,
domain-specific quality measures, and complex performance indicators.
## Example
```python theme={null}
# Business conversion rate metric
conversion_rate = CustomMetric(
name="weekly_conversion_rate",
model_id=model.id,
definition="sum(if(prediction_score > 0.7 and converted == 1, 1, 0)) / sum(if(prediction_score > 0.7, 1, 0))",
description="Conversion rate for high-confidence predictions"
).create()
# Data quality metric
missing_rate = CustomMetric(
name="feature_missing_rate",
model_id=model.id,
definition="sum(if(is_null(income), 1, 0)) / count(income)",
description="Percentage of records with missing income values"
).create()
# Fairness metric
fairness_metric = CustomMetric(
name="demographic_parity",
model_id=model.id,
definition="abs((sum(if(gender == 'Male', predicted_churn, 0)) / sum(if(gender == 'Male', 1, 0))) - (sum(if(gender == 'Female', predicted_churn, 0)) / sum(if(gender == 'Female', 1, 0))))",
description="Demographic parity difference between gender groups"
).create()
# Use in alert rule
alert_rule = AlertRule(
name="conversion_rate_alert",
model_id=model.id,
metric_id=conversion_rate.id,
priority=Priority.HIGH,
compare_to=CompareTo.TIME_PERIOD,
condition=AlertCondition.LESSER,
bin_size=BinSize.DAY,
critical_threshold=0.15, # Alert if conversion drops below 15%
compare_bin_delta=7
).create()
```
Custom metrics are calculated during data ingestion and monitoring cycles.
Complex expressions may impact performance, so optimize for efficiency.
Test expressions thoroughly before using in production alert rules.
## create()
Create a new CustomMetric on the Fiddler platform.
Registers this CustomMetric with the Fiddler platform. The expression
must have a name, model\_id, and definition specified before calling create().
### Returns
The same CustomMetric instance with updated server-side attributes
(id, created\_at, etc.).
### Raises
* **ApiError** – If there's an error communicating with the Fiddler API.
* **Conflict** – If a CustomMetric with the same name already exists for this model.
## delete()
Delete this CustomMetric from the Fiddler platform.
Permanently removes the CustomMetric. This action cannot be undone.
Any alert rules or monitors using this CustomMetric must be deleted first.
### Raises
* **NotFound** – If the CustomMetric no longer exists.
* **ApiError** – If there's an error communicating with the Fiddler API.
* **Conflict** – If the CustomMetric is still being used by alert rules or monitors.
## *classmethod* from\_name()
Retrieve a CustomMetric by name and model.
Fetches a CustomMetric from the Fiddler platform using its name
and associated model ID.
### Parameters
The name of the CustomMetric to retrieve.
UUID or string identifier of the associated Model.
### Returns
The CustomMetric instance for the provided parameters.
### Raises
* **NotFound** – If no CustomMetric exists with the specified name and model.
* **ApiError** – If there's an error communicating with the Fiddler API.
## *classmethod* get()
Retrieve a CustomMetric by its unique identifier.
Fetches a CustomMetric from the Fiddler platform using its UUID.
### Parameters
The unique identifier (UUID) of the CustomMetric to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The CustomMetric instance with all its configuration and metadata.
### Raises
* **NotFound** – If no CustomMetric exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
## *classmethod* get\_organization\_id()
Get the organization UUID from the global connection.
### Returns
Unique identifier of the organization associated with the current connection.
## *classmethod* get\_organization\_name()
Get the organization name from the global connection.
### Returns
Name of the organization associated with the current connection.
## *classmethod* list()
List all CustomMetric instances for a model.
Retrieves all CustomMetric instances associated with a specific model.
### Parameters
UUID or string identifier of the Model.
### Yields
CustomMetric instances for each CustomMetric in the model.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Returns
`Iterator[CustomMetric]`
## **init**()
Construct a custom metric instance.
# DataType
Source: https://docs.fiddler.ai/sdk-api/python-client/data-type
Data types supported for model columns in Fiddler.
Data types supported for model columns in Fiddler.
This enum defines the supported data types for model schema columns.
Data types determine how Fiddler processes, validates, and monitors
individual columns in your model's input and output data.
Type Categories:
* **Numeric**: FLOAT, INTEGER - enable statistical analysis
* **Categorical**: BOOLEAN, CATEGORY - enable distribution analysis
* **Textual**: STRING - enable text-based monitoring
* **Temporal**: TIMESTAMP - enable time-based analysis
* **Vector**: VECTOR - enable embedding-based monitoring
## Examples
Defining column data types in model schema:
```python theme={null}
from fiddler import Column, DataType
# Define columns with appropriate data types
columns = [
Column(name='age', data_type=DataType.INTEGER),
Column(name='income', data_type=DataType.FLOAT),
Column(name='is_member', data_type=DataType.BOOLEAN),
Column(name='category', data_type=DataType.CATEGORY),
Column(name='description', data_type=DataType.STRING),
Column(name='created_at', data_type=DataType.TIMESTAMP),
Column(name='embedding', data_type=DataType.VECTOR)
]
# Create model schema
schema = fdl.ModelSchema(columns=columns)
```
Data type validation and monitoring:
```python theme={null}
# Numeric types enable statistical monitoring
if column.data_type.is_numeric():
# Statistical drift detection available
# Range validation enabled
# Distribution analysis supported
pass
# Categorical types enable distribution monitoring
if column.data_type.is_bool_or_cat():
# Category distribution tracking
# New category detection
# Frequency analysis
pass
# Vector types enable embedding monitoring
if column.data_type.is_vector():
# Embedding drift detection
# Clustering analysis
# Dimensionality monitoring
pass
```
Choose data types that accurately represent your data for optimal
monitoring and validation. Incorrect data types may lead to
inappropriate metrics or monitoring failures.
## FLOAT *= 'float'*
Floating-point numerical values.
Used for continuous numerical data with decimal precision. Enables
comprehensive statistical analysis and numerical drift detection.
Characteristics:
* Decimal precision values
* Statistical distribution analysis
* Range and outlier detection
* Correlation analysis support
Monitoring features:
* Mean, median, standard deviation tracking
* Distribution drift detection (KS test, PSI)
* Range violation alerts
* Outlier detection and analysis
Typical use cases:
* Prices, costs, revenues
* Probabilities and confidence scores
* Measurements and sensor readings
* Performance metrics and ratios
* Model prediction scores
Validation: Numeric range checks, NaN detection
## INTEGER *= 'int'*
Integer numerical values.
Used for whole number data without decimal places. Supports numerical
analysis while recognizing discrete nature of integer data.
Characteristics:
* Whole number values only
* Discrete distribution analysis
* Count-based statistics
* Range validation
Monitoring features:
* Count distribution tracking
* Range violation detection
* Discrete value frequency analysis
* Statistical drift detection
Typical use cases:
* Counts and quantities
* Age, years, days
* IDs and identifiers (when numeric)
* Ranking positions
* Categorical codes (when numeric)
Validation: Integer format checks, range validation
## BOOLEAN *= 'bool'*
True/false binary values.
Used for binary flag data with exactly two possible values. Enables
binary distribution analysis and proportion tracking.
Characteristics:
* Exactly two values (True/False, 1/0, Yes/No)
* Binary distribution analysis
* Proportion-based metrics
* Simple categorical handling
Monitoring features:
* True/False ratio tracking
* Binary distribution drift
* Proportion change detection
* Flag frequency analysis
Typical use cases:
* Feature flags and indicators
* Binary classifications
* Yes/No survey responses
* Membership status
* Activation states
Validation: Binary value format checks
## STRING *= 'str'*
Text string values.
Used for textual data of variable length. Supports text-based analysis
and can be combined with text embeddings for advanced monitoring.
Characteristics:
* Variable length text
* Text-based analysis
* String pattern detection
* Encoding-aware processing
Monitoring features:
* Length distribution tracking
* Pattern and format analysis
* Text embedding integration
* String uniqueness analysis
Typical use cases:
* Names and descriptions
* Comments and reviews
* URLs and paths
* Free-form text inputs
* JSON or XML strings
Special considerations:
* Can be converted to embeddings for semantic monitoring
* Supports text enrichment features
* May require text preprocessing
## CATEGORY *= 'category'*
Categorical values with limited distinct options.
Used for data with a finite set of possible values or categories.
Enables categorical distribution analysis and new category detection.
Characteristics:
* Limited set of possible values
* Categorical distribution tracking
* Category frequency analysis
* New category detection
Monitoring features:
* Category distribution drift
* New/missing category alerts
* Frequency change detection
* Category proportion analysis
Typical use cases:
* Product categories
* Geographic regions
* Status codes
* Demographic categories
* Classification labels
Best practices:
* Use for data with \< 1000 unique values
* Consider STRING type for high-cardinality categories
* Define expected categories during schema creation
## TIMESTAMP *= 'timestamp'*
Date and time values.
Used for temporal data including dates, times, and timestamps.
Enables time-based analysis and temporal pattern detection.
Characteristics:
* Date/time information
* Temporal ordering
* Time-based aggregations
* Timezone awareness
Monitoring features:
* Temporal pattern analysis
* Time gap detection
* Seasonal trend monitoring
* Data freshness tracking
Typical use cases:
* Event timestamps
* Creation/modification dates
* Transaction times
* Log timestamps
* Scheduled events
Supported formats:
* Unix timestamps
* ISO 8601 strings
* Pandas datetime objects
* Various date formats (with parsing)
## VECTOR *= 'vector'*
Multi-dimensional numerical vectors (embeddings).
Used for embedding vectors, feature vectors, and other multi-dimensional
numerical data. Enables embedding-based drift detection and clustering analysis.
Characteristics:
* Fixed-dimension numerical arrays
* Embedding-based analysis
* Vector similarity metrics
* Clustering support
Monitoring features:
* Embedding drift detection
* Cluster analysis and visualization
* Vector similarity tracking
* Dimensionality validation
Typical use cases:
* Text embeddings (Word2Vec, BERT, etc.)
* Image embeddings (CNN features)
* User/item embeddings
* Feature vectors from neural networks
* Recommendation system embeddings
Special considerations:
* Requires consistent vector dimensions
* Benefits from custom feature definitions
* Supports clustering and UMAP visualization
## is\_numeric()
Check if the data type is numeric.
### Returns
True if data type is INTEGER or FLOAT
## is\_bool\_or\_cat()
Check if the data type is boolean or categorical.
### Returns
True if data type is BOOLEAN or CATEGORY
## is\_vector()
Check if the data type is vector.
### Returns
True if data type is VECTOR
# Dataset
Source: https://docs.fiddler.ai/sdk-api/python-client/dataset
Represents a dataset containing data published to a Fiddler model.
Represents a dataset containing data published to a Fiddler model.
A Dataset is a collection of data records that have been published to a specific
model in the Fiddler platform. Datasets are automatically created when data is
published using Model.publish() and serve as the foundation for monitoring,
drift detection, and baseline creation.
Key Features:
* **Data Collection**: Organized storage of model input/output data
* **Environment Separation**: Distinct handling of production vs. pre-production data
* **Baseline Source**: Reference data for drift detection and monitoring
* **Analysis Support**: Data download and statistical analysis capabilities
* **Model Integration**: Tight coupling with specific models for context
Dataset Characteristics:
* **Automatic Creation**: Created by Model.publish() operations
* **Model-Scoped**: Each dataset belongs to exactly one model
* **Named Collections**: Unique names within a model for identification
* **Row Tracking**: Automatic counting of data records
* **Environment Typed**: Classified as production or pre-production data
## Example
```python theme={null}
# Retrieve a specific dataset
dataset = Dataset.from_name(
name="training_data_v1",
model_id=model.id
)
print(f"Dataset: {dataset.name}")
print(f"Rows: {dataset.row_count}")
print(f"Model: {dataset.model_id}")
# List all datasets for a model
datasets = list(Dataset.list(model_id=model.id))
print(f"Found {len(datasets)} datasets")
# Find datasets by characteristics
large_datasets = [
ds for ds in Dataset.list(model_id=model.id)
if ds.row_count and ds.row_count > 10000
]
```
Datasets cannot be created directly through the Dataset class. They are
automatically created when data is published to models using Model.publish().
Use the Dataset class for retrieval, listing, and analysis operations.
Initialize a Dataset instance.
Creates a dataset object representing data published to a model. This constructor
is typically used internally when deserializing API responses rather than for
direct dataset creation.
## Parameters
Dataset name, must be unique within the model.
Should be descriptive of the data contents or purpose.
Identifier of the model this dataset belongs to.
Can be provided as UUID object or string representation.
Identifier of the parent project.
Can be provided as UUID object or string representation.
## Example
```python theme={null}
# Internal usage - typically not called directly
dataset = Dataset(
name="training_baseline_v1",
model_id="550e8400-e29b-41d4-a716-446655440000",
project_id="660e8400-e29b-41d4-a716-446655440000"
)
```
Datasets are typically retrieved using Dataset.get(), Dataset.from\_name(),
or Dataset.list() rather than created directly. Direct creation is mainly
used internally by the Fiddler client.
## *classmethod* get()
Retrieve a dataset by its unique identifier.
Fetches a dataset from the Fiddler platform using its UUID. This is the most
direct way to retrieve a dataset when you know its ID.
### Parameters
The unique identifier (UUID) of the dataset to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The dataset instance with all metadata and row count information.
### Raises
* **NotFound** – If no dataset exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get dataset by UUID
dataset = Dataset.get(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved dataset: {dataset.name}")
print(f"Rows: {dataset.row_count}")
print(f"Model: {dataset.model_id}")
# Use dataset for analysis
if dataset.row_count and dataset.row_count > 1000:
print("Large dataset suitable for baseline creation")
```
This method makes an API call to fetch the latest dataset state from the server.
The returned dataset instance reflects the current state in Fiddler.
## *classmethod* from\_name()
Retrieve a dataset by name within a specific model.
Finds and returns a dataset using its name and model context. Dataset names
are unique within a model, making this a reliable lookup method when you
know both the dataset name and model ID.
### Parameters
The name of the dataset to retrieve. Dataset names are unique
within a model and are case-sensitive.
The identifier of the model containing the dataset.
Can be provided as UUID object or string representation.
### Returns
The dataset instance matching the specified name and model.
### Raises
* **NotFound** – If no dataset exists with the specified name in the given model.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get dataset by name for a specific model
dataset = Dataset.from_name(
name="training_baseline",
model_id=model.id
)
print(f"Found dataset: {dataset.name}")
print(f"Rows: {dataset.row_count}")
# Get validation dataset
val_dataset = Dataset.from_name(
name="validation_set_v2",
model_id=model.id
)
# Use for baseline creation
baseline = Baseline.create_from_dataset(
dataset_id=dataset.id,
name="training_baseline"
)
```
Dataset names are case-sensitive and must match exactly. This method
is useful when you know the dataset name from configuration or when
working with named datasets created during model training workflows.
## *classmethod* list()
List all pre-production datasets for a specific model.
Retrieves all datasets that have been published to a model in the pre-production
environment. These datasets are typically used for baselines, training data
analysis, and validation purposes.
### Parameters
The identifier of the model to list datasets for.
Can be provided as UUID object or string representation.
### Yields
`Dataset` – Dataset instances for all pre-production datasets in the model.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Returns
`Iterator[Dataset]`
### Example
```python theme={null}
# List all datasets for a model
for dataset in Dataset.list(model_id=model.id):
print(f"Dataset: {dataset.name}")
print(f" Rows: {dataset.row_count}")
print(f" ID: {dataset.id}")
# Convert to list for analysis
datasets = list(Dataset.list(model_id=model.id))
print(f"Found {len(datasets)} datasets")
# Find datasets by characteristics
large_datasets = [
ds for ds in Dataset.list(model_id=model.id)
if ds.row_count and ds.row_count > 10000
]
print(f"Large datasets: {len(large_datasets)}")
# Get dataset summary statistics
total_rows = sum(
ds.row_count or 0
for ds in Dataset.list(model_id=model.id)
)
print(f"Total rows across all datasets: {total_rows}")
```
This method returns an iterator for memory efficiency and only includes
pre-production datasets. Production data is handled separately through
the monitoring system. Convert to a list with list(Dataset.list(…))
if you need to iterate multiple times.
# DatasetCompact
Source: https://docs.fiddler.ai/sdk-api/python-client/dataset-compact
Lightweight dataset representation for listing and basic operations.
Lightweight dataset representation for listing and basic operations.
A minimal dataset object containing only essential identifiers. Used by
various operations to efficiently reference datasets without fetching
full dataset details and metadata.
This class provides a memory-efficient way to work with dataset references
when you don't need the full dataset functionality but want to access
basic information or fetch the complete dataset when needed.
## Example
```python theme={null}
# From dataset references in other entities
baseline = Baseline.get(id_="baseline-uuid")
dataset_ref = baseline.dataset # Returns DatasetCompact
# Access basic info
print(f"Dataset: {dataset_ref.name}")
print(f"ID: {dataset_ref.id}")
# Fetch full details when needed
full_dataset = dataset_ref.fetch()
print(f"Rows: {full_dataset.row_count}")
print(f"Model: {full_dataset.model_id}")
```
DatasetCompact objects are typically returned by other entities that
reference datasets. Use .fetch() to get the complete Dataset instance
when you need full functionality like row counts or model information.
## id
## name
## fetch()
Fetch the complete Dataset instance.
Retrieves the full Dataset object with all metadata, row counts, and
model associations from the Fiddler platform using this compact dataset's ID.
### Returns
Complete dataset instance with all details and capabilities.
### Example
```python theme={null}
# From dataset reference
compact = baseline.dataset
# Get full dataset details
full_dataset = compact.fetch()
# Now can access full functionality
print(f"Dataset has {full_dataset.row_count} rows")
print(f"Belongs to model: {full_dataset.model_id}")
```
# DatasetDataSource
Source: https://docs.fiddler.ai/sdk-api/python-client/dataset-data-source
Data source for explainability analysis using a sample from a dataset.
Data source for explainability analysis using a sample from a dataset.
DatasetDataSource allows you to perform explainability analysis on a random
sample of data from a specified environment/dataset. This is useful for
understanding general model behavior, analyzing feature importance patterns
across multiple instances, or getting representative explanations.
This data source type is ideal for exploratory analysis, understanding overall
model behavior, or when you want to analyze explanations across a representative
sample rather than specific instances.
## Examples
Creating a dataset data source for production sampling:
```python theme={null}
dataset_source = DatasetDataSource(
env_type="PRODUCTION",
num_samples=100,
env_id="prod_dataset_uuid"
)
```
Creating a dataset data source for validation analysis:
```python theme={null}
validation_source = DatasetDataSource(
env_type="VALIDATION",
num_samples=50
)
```
Creating a dataset data source with default sampling:
```python theme={null}
default_source = DatasetDataSource(
env_type="PRODUCTION"
)
```
## source\_type
## env\_type
## num\_samples
## env\_id
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# DeploymentParams
Source: https://docs.fiddler.ai/sdk-api/python-client/deployment-params
Configuration parameters for deploying a model in the Fiddler platform.
Configuration parameters for deploying a model in the Fiddler platform.
DeploymentParams defines the deployment configuration for a model, including
the artifact type, deployment environment, resource allocation, and container
specifications. These parameters control how the model is packaged, deployed,
and scaled within the Fiddler infrastructure.
This class is used when deploying models to specify the runtime environment,
resource requirements, and deployment strategy that best fits your model's
needs and performance requirements.
## Examples
Creating basic deployment parameters:
```python theme={null}
basic_params = DeploymentParams()
```
Creating deployment with custom resources:
```python theme={null}
custom_params = DeploymentParams(
artifact_type=ArtifactType.PYTHON_PACKAGE,
deployment_type=DeploymentType.BASE_CONTAINER,
replicas=3,
cpu=2,
memory=4096
)
```
Creating deployment with custom container:
```python theme={null}
container_params = DeploymentParams(
artifact_type=ArtifactType.DOCKER_IMAGE,
deployment_type=DeploymentType.CUSTOM_CONTAINER,
image_uri="my-registry.com/my-model:v1.0",
replicas=2,
cpu=4,
memory=8192
)
```
## artifact\_type
## deployment\_type
## image\_uri
## replicas
## cpu
## memory
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# DeploymentType
Source: https://docs.fiddler.ai/sdk-api/python-client/deployment-type
Model deployment types for explainability services.
Model deployment types for explainability services.
**Values:**
* `BASE_CONTAINER` - BASE\_CONTAINER
Base container deployment with standard environment.
## MANUAL *= 'MANUAL'*
Manual deployment configuration.
# DownloadFormat
Source: https://docs.fiddler.ai/sdk-api/python-client/download-format
File formats for downloading and exporting explanation data.
File formats for downloading and exporting explanation data.
This enum defines the supported file formats for downloading explanation
results from Fiddler. Different formats offer different advantages in
terms of performance, compatibility, and data structure preservation.
## PARQUET
Apache Parquet format for efficient columnar storage
## CSV
Comma-separated values format for broad compatibility
### Examples
Downloading explanations in different formats:
```python theme={null}
# Download as Parquet (recommended for large datasets)
parquet_data = model.download_explanations(
format=fdl.DownloadFormat.PARQUET,
chunk_size=1000
)
# Download as CSV (better compatibility)
csv_data = model.download_explanations(
format=fdl.DownloadFormat.CSV,
chunk_size=500
## )
```
Choose format based on your analysis tools and data size requirements.
Parquet is recommended for large datasets due to compression and performance.
## PARQUET *= 'PARQUET'*
Apache Parquet format for efficient columnar data storage.
Parquet is a columnar storage format that provides excellent compression
and query performance. It preserves data types and schema information,
making it ideal for analytical workloads and large datasets.
Advantages:
* Excellent compression ratios
* Fast query performance
* Preserves data types and schema
* Efficient for analytical operations
Best for:
* Large explanation datasets
* Analytical workflows
* Integration with data science tools
* Long-term data storage
## CSV *= 'CSV'*
Comma-separated values format for broad tool compatibility.
CSV is a simple, widely-supported text format that can be opened by
virtually any data analysis tool, spreadsheet application, or programming
language. While less efficient than Parquet, it offers maximum compatibility.
Advantages:
* Universal compatibility
* Human-readable format
* Simple structure
* Supported by all tools
Best for:
* Small to medium datasets
* Sharing with non-technical users
* Quick data inspection
* Integration with legacy systems
# Enrichment
Source: https://docs.fiddler.ai/sdk-api/python-client/enrichment
Represents custom features derived from enrichment operations.
Represents custom features derived from enrichment operations.
Enrichment features apply external processing or analysis to existing columns
to create derived insights. This can include operations like sentiment analysis,
toxicity detection, entity extraction, SQL validation, JSON validation, or any
custom transformation that adds value to the original data.
The feature type is automatically set to CustomFeatureType.ENRICHMENT and enables
domain-specific analysis through configurable enrichment operations.
## Examples
```python theme={null}
# Creating a sentiment analysis enrichment:
sentiment_enrichment = Enrichment(
name="review_sentiment",
columns=["review_text"],
enrichment="sentiment_analysis",
config={
"model": "vader",
"return_scores": True
}
)
# Creating a SQL validation enrichment:
sql_enrichment = Enrichment(
name="sql_validation",
columns=["query_string"],
enrichment="sql_validation",
config={
"dialect": "postgresql",
"strict": True
}
)
# Creating a JSON validation enrichment:
json_enrichment = Enrichment(
name="json_validation",
columns=["json_string"],
enrichment="json_validation",
config={
"strict": True,
"validation_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"prop_1": {"type": "number"}
},
"required": ["prop_1"],
"additionalProperties": False
}
}
)
# Creating a toxicity detection enrichment:
toxicity_enrichment = Enrichment(
name="content_toxicity",
columns=["user_comment"],
enrichment="toxicity_detection",
config={
"threshold": 0.7,
"categories": ["toxic", "severe_toxic", "obscene"]
}
)
```
## type
## columns
## enrichment
## config
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# EnvType
Source: https://docs.fiddler.ai/sdk-api/python-client/env-type
Environment types for data publishing in Fiddler.
Environment types for data publishing in Fiddler.
This enum defines the two primary environment types used when publishing
inference data to Fiddler. The environment type determines how Fiddler
processes, stores, and monitors the data.
## PRODUCTION
Live inference data from production model deployments
## PRE\_PRODUCTION
Static baseline datasets for drift detection reference
### Examples
Publishing pre-production baseline data:
```python theme={null}
# Upload training data as baseline
baseline_job = model.publish(
source='training_data.csv',
environment=fdl.EnvType.PRE_PRODUCTION,
dataset_name='training_baseline'
## )
```
Publishing production inference data:
```python theme={null}
# Stream live inference events
model.publish(
source=inference_events,
environment=fdl.EnvType.PRODUCTION
## )
```
Environment-specific data handling:
```python theme={null}
# Different processing based on environment
if env_type == fdl.EnvType.PRE_PRODUCTION:
# Static dataset - immutable after upload
# Used for baseline calculations
# No time-series monitoring
pass
elif env_type == fdl.EnvType.PRODUCTION:
# Time-series data - continuous monitoring
# Compared against baselines for drift
# Subject to data retention policies
pass
```
Environment types cannot be changed after data publication. Choose the
appropriate environment based on your data's intended use case.
## PRODUCTION *= 'PRODUCTION'*
Production environment for live inference data.
Used for time-series inference data from live model deployments. This data:
* Gets monitored continuously for drift and performance issues
* Is compared against baseline datasets for anomaly detection
* Supports real-time streaming and batch publishing
* Is subject to data retention policies (typically 90 days)
* Enables alert rule evaluation and dashboard visualization
Typical use cases:
* Live model inference results
* Real-time prediction streaming
* Batch inference job outputs
* A/B testing data
* Production model monitoring
Data characteristics:
* Time-series with timestamps
* Continuous data flow
* Variable data volumes
* Monitored for drift patterns
## PRE\_PRODUCTION *= 'PRE\_PRODUCTION'*
Pre-production environment for baseline datasets.
Used for static datasets that serve as reference points for monitoring.
This data:
* Remains immutable after publication
* Serves as baseline for drift detection calculations
* Represents expected model behavior and data distributions
* Is retained indefinitely for comparison purposes
* Does not appear in time-series monitoring charts
Typical use cases:
* Training dataset baselines
* Validation dataset references
* Historical "golden" datasets
* Model performance benchmarks
* Data distribution references
Data characteristics:
* Static, unchanging datasets
* Representative of expected distributions
* Used for statistical comparisons
* No time-series component
# EventIdDataSource
Source: https://docs.fiddler.ai/sdk-api/python-client/event-id-data-source
Data source for explainability analysis using a specific event ID.
Data source for explainability analysis using a specific event ID.
EventIdDataSource allows you to perform explainability analysis on a specific
event that has been previously logged to Fiddler. This is useful when you want
to explain predictions for events that are already stored in your environment,
enabling you to analyze historical predictions and their explanations.
This data source type is ideal for investigating specific incidents, analyzing
historical predictions, or performing post-hoc analysis on logged events.
## Examples
Creating an event data source for production analysis:
```python theme={null}
event_source = EventIdDataSource(
event_id="evt_12345abcdef",
env_id="prod_dataset_uuid",
env_type=EnvType.PRODUCTION
)
```
Creating an event data source for validation analysis:
```python theme={null}
validation_event_source = EventIdDataSource(
event_id="validation_event_789",
env_id="validation_dataset_uuid",
env_type=EnvType.VALIDATION
)
```
## source\_type
## event\_id
## env\_id
## env\_type
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# ExplainMethod
Source: https://docs.fiddler.ai/sdk-api/python-client/explain-method
Explanation methods for model interpretability and feature importance analysis.
Explanation methods for model interpretability and feature importance analysis.
This enum defines the available algorithms for computing feature importance
and generating explanations for model predictions. Different methods provide
different perspectives on how features contribute to model decisions.
Method Categories:
* **SHAP-based**: Unified framework for feature importance (SHAP, FIDDLER\_SHAP)
* **Gradient-based**: Uses model gradients for explanations (IG)
* **Perturbation-based**: Feature permutation and baseline methods
## Examples
Using different explanation methods:
```python theme={null}
# Standard SHAP explanations
shap_explanations = model.explain(
data_source=fdl.RowDataSource(row=sample_data),
explain_method=fdl.ExplainMethod.SHAP
)
# Fiddler's optimized SHAP (recommended)
fast_explanations = model.explain(
data_source=fdl.RowDataSource(row=sample_data),
explain_method=fdl.ExplainMethod.FIDDLER_SHAP
)
# Integrated Gradients for neural networks
ig_explanations = model.explain(
data_source=fdl.RowDataSource(row=sample_data),
explain_method=fdl.ExplainMethod.IG
)
# Permutation importance
perm_explanations = model.explain(
data_source=fdl.RowDataSource(row=sample_data),
explain_method=fdl.ExplainMethod.PERMUTE
## )
```
Method availability depends on model type and artifact configuration.
FIDDLER\_SHAP is recommended for most use cases due to performance optimizations.
## SHAP *= 'SHAP'*
Standard SHAP (SHapley Additive exPlanations) method.
Implements the original SHAP algorithm for computing feature importance
based on game theory. Provides globally consistent and locally accurate
feature attributions that sum to the difference between model output
and expected output.
Characteristics:
* Theoretically grounded in game theory
* Satisfies efficiency, symmetry, dummy, and additivity axioms
* Works with any machine learning model
* Computationally intensive for complex models
Best for:
* Research and academic applications
* When theoretical guarantees are important
* Comparative analysis with other SHAP implementations
## FIDDLER\_SHAP *= 'FIDDLER\_SHAP'*
Fiddler's optimized SHAP implementation for improved performance.
Fiddler's enhanced version of SHAP that provides the same theoretical
guarantees as standard SHAP but with significant performance improvements
and optimizations for production use cases.
Characteristics:
* Same theoretical properties as standard SHAP
* Significant performance optimizations
* Better suited for production environments
* Optimized for Fiddler's infrastructure
Best for:
* Production explainability workflows
* High-volume explanation generation
* Real-time explanation requirements
* Most general-purpose use cases (recommended)
## IG *= 'IG'*
Integrated Gradients method for gradient-based explanations.
Computes feature importance by integrating gradients of the model output
with respect to inputs along a straight path from a baseline to the input.
Particularly effective for neural networks and differentiable models.
Characteristics:
* Uses model gradients for attribution
* Satisfies implementation invariance and sensitivity axioms
* Requires differentiable models
* Effective for neural networks
Best for:
* Neural network models
* Deep learning applications
* When gradient information is available
* Image and text models with embeddings
## PERMUTE *= 'PERMUTE'*
Permutation-based feature importance analysis.
Computes feature importance by measuring the decrease in model performance
when feature values are randomly permuted. Provides model-agnostic
importance scores based on predictive contribution.
Characteristics:
* Model-agnostic approach
* Based on predictive performance impact
* Computationally straightforward
* Provides global feature importance
Best for:
* Model-agnostic analysis
* Understanding overall feature importance
* Comparing feature relevance across models
* When other methods are not applicable
## ZERO\_RESET *= 'ZERO\_RESET'*
Zero baseline reset method for feature ablation analysis.
Computes feature importance by replacing feature values with zero and
measuring the change in model output. Provides insights into how
features contribute relative to a zero baseline.
Characteristics:
* Simple ablation-based approach
* Uses zero as the baseline value
* Fast computation
* May not be suitable for all feature types
Best for:
* Quick feature importance analysis
* Models where zero is a meaningful baseline
* Sparse feature representations
* Initial feature importance exploration
## MEAN\_RESET *= 'MEAN\_RESET'*
Mean baseline reset method for feature ablation analysis.
Computes feature importance by replacing feature values with their
population mean and measuring the change in model output. Uses the
training data mean as a more representative baseline than zero.
Characteristics:
* Ablation-based with mean baseline
* Uses training data statistics
* More representative baseline than zero
* Accounts for feature distributions
Best for:
* Models where mean is a natural baseline
* Features with non-zero typical values
* When training distribution is representative
* Comparative analysis with zero baseline
# File
Source: https://docs.fiddler.ai/sdk-api/python-client/file
Construct a files instance.
Construct a files instance.
## upload()
Upload file. Single or multipart upload depends on file size
### Returns
`File`
## single\_upload()
Single part file upload
### Returns
`File`
## multipart\_upload()
Multi part file upload
### Returns
`File`
# group_by
Source: https://docs.fiddler.ai/sdk-api/python-client/group-by
Group the events by a column. Use this method to form the grouped data for ranking models.
Group the events by a column. Use this method to form the grouped data for ranking models.
## Parameters
The dataframe with flat data
The column to group the data by
Optional path to write the grouped data to. If not specified, data won't be written anywhere
## Returns
Dataframe in grouped format
## Examples
```python theme={null}
COLUMN_NAME = 'col_2'
grouped_df = group_by(df=df, group_by_col=COLUMN_NAME)
```
# HttpError
Source: https://docs.fiddler.ai/sdk-api/python-client/http-error
Base class for all HTTP-related errors.
Base class for all HTTP-related errors.
This is the parent class for HTTP communication errors that can occur
during API requests to the Fiddler platform. While this specific class
is deprecated and no longer thrown directly, it serves as the base for
more specific HTTP error types.
This class is deprecated and not thrown anymore. Use more specific
HTTP error subclasses like ApiError, NotFound, or Conflict instead.
## Examples
Catching any HTTP-related error:
```python theme={null}
try:
# API operation
pass
except HttpError as e:
print(f"HTTP error: {e.message}")
```
# ImageEmbedding
Source: https://docs.fiddler.ai/sdk-api/python-client/image-embedding
Represents custom features derived from image embeddings for visual content analysis.
Represents custom features derived from image embeddings for visual content analysis.
ImageEmbedding extends VectorFeature to handle image-based embeddings, providing
clustering analysis specifically designed for visual content. This feature type
is used to monitor image models and detect visual drift in high-dimensional
embedding spaces.
The feature type is automatically set to CustomFeatureType.FROM\_IMAGE\_EMBEDDING
and applies clustering to image embeddings for visual pattern analysis.
## Examples
Creating an image embedding feature for product photos:
```python theme={null}
image_feature = ImageEmbedding(
name="product_image_clusters",
column="product_embedding",
source_column="product_image_url",
n_clusters=15
)
```
Creating a feature for medical image analysis:
```python theme={null}
medical_feature = ImageEmbedding(
name="xray_pattern_clusters",
column="xray_embedding",
source_column="xray_image_path",
n_clusters=10
)
```
## type
## source\_column
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# IncompatibleClient
Source: https://docs.fiddler.ai/sdk-api/python-client/incompatible-client
Raised when the Python client version is incompatible with the Fiddler platform version.
Raised when the Python client version is incompatible with the Fiddler platform version.
This exception occurs during connection initialization when the client library
version is not compatible with the connected Fiddler platform version. This
ensures that users are aware of version mismatches that could cause unexpected
behavior or missing functionality.
The exception includes both the client version and server version in the error
message to help users understand what versions are involved in the incompatibility.
## Examples
Handling version incompatibility:
```python theme={null}
try:
fdl.init(url="https://old-fiddler.com", token="token")
except IncompatibleClient as e:
print(f"Version mismatch: {e.message}")
# Upgrade client or contact administrator
```
Typical error message format:
```python theme={null}
"Python Client version (3.8.0) is not compatible with your
Fiddler Platform version (3.5.0)."
```
## message
# Introduction
Source: https://docs.fiddler.ai/sdk-api/python-client/index
Complete API reference for fiddler
[](https://pypi.org/project/fiddler-client/)
## Overview
Complete API reference documentation for the fiddler package.
## Components
### Connection
Connection management and initialization
* [Connection](/sdk-api/python-client/connection)
* [ConnectionMixin](/sdk-api/python-client/connection-mixin)
### Constants
Constants module documentation
* [AlertCondition](/sdk-api/python-client/alert-condition)
* [AlertThresholdAlgo](/sdk-api/python-client/alert-threshold-algo)
* [ArtifactStatus](/sdk-api/python-client/artifact-status)
* [ArtifactType](/sdk-api/python-client/artifact-type)
* [BaselineType](/sdk-api/python-client/baseline-type)
* [BinSize](/sdk-api/python-client/bin-size)
* [CompareTo](/sdk-api/python-client/compare-to)
* [CustomFeatureType](/sdk-api/python-client/custom-feature-type)
* [DataType](/sdk-api/python-client/data-type)
* [DeploymentType](/sdk-api/python-client/deployment-type)
* [DownloadFormat](/sdk-api/python-client/download-format)
* [EnvType](/sdk-api/python-client/env-type)
* [ExplainMethod](/sdk-api/python-client/explain-method)
* [JobStatus](/sdk-api/python-client/job-status)
* [ModelInputType](/sdk-api/python-client/model-input-type)
* [ModelTask](/sdk-api/python-client/model-task)
* [Priority](/sdk-api/python-client/priority)
* [WindowBinSize](/sdk-api/python-client/window-bin-size)
### Entities
Core entity classes for interacting with the platform
* [AlertRecord](/sdk-api/python-client/alert-record)
* [AlertRule](/sdk-api/python-client/alert-rule)
* [Baseline](/sdk-api/python-client/baseline)
* [BaselineCompact](/sdk-api/python-client/baseline-compact)
* [CustomMetric](/sdk-api/python-client/custom-metric)
* [Dataset](/sdk-api/python-client/dataset)
* [DatasetCompact](/sdk-api/python-client/dataset-compact)
* [File](/sdk-api/python-client/file)
* [Job](/sdk-api/python-client/job)
* [Model](/sdk-api/python-client/model)
* [ModelCompact](/sdk-api/python-client/model-compact)
* [ModelDeployment](/sdk-api/python-client/model-deployment)
* [Project](/sdk-api/python-client/project)
* [ProjectCompact](/sdk-api/python-client/project-compact)
* [Segment](/sdk-api/python-client/segment)
* [Webhook](/sdk-api/python-client/webhook)
### Exceptions
Exceptions module documentation
* [ApiError](/sdk-api/python-client/api-error)
* [AsyncJobFailed](/sdk-api/python-client/async-job-failed)
* [Conflict](/sdk-api/python-client/conflict)
* [ConnError](/sdk-api/python-client/conn-error)
* [ConnTimeout](/sdk-api/python-client/conn-timeout)
* [HttpError](/sdk-api/python-client/http-error)
* [IncompatibleClient](/sdk-api/python-client/incompatible-client)
* [NotFound](/sdk-api/python-client/not-found)
* [Unsupported](/sdk-api/python-client/unsupported)
### Schemas
Schemas module documentation
* [Column](/sdk-api/python-client/column)
* [CustomFeature](/sdk-api/python-client/custom-feature)
* [DatasetDataSource](/sdk-api/python-client/dataset-data-source)
* [DeploymentParams](/sdk-api/python-client/deployment-params)
* [Enrichment](/sdk-api/python-client/enrichment)
* [EventIdDataSource](/sdk-api/python-client/event-id-data-source)
* [ImageEmbedding](/sdk-api/python-client/image-embedding)
* [ModelSchema](/sdk-api/python-client/model-schema)
* [ModelSpec](/sdk-api/python-client/model-spec)
* [ModelTaskParams](/sdk-api/python-client/model-task-params)
* [Multivariate](/sdk-api/python-client/multivariate)
* [RowDataSource](/sdk-api/python-client/row-data-source)
* [TextEmbedding](/sdk-api/python-client/text-embedding)
* [VectorFeature](/sdk-api/python-client/vector-feature)
* [XaiParams](/sdk-api/python-client/xai-params)
### Utils
Utils module documentation
* [create\_columns\_from\_df](/sdk-api/python-client/create-columns-from-df)
* [group\_by](/sdk-api/python-client/group-by)
# Job
Source: https://docs.fiddler.ai/sdk-api/python-client/job
Represents an asynchronous operation in the Fiddler platform.
Represents an asynchronous operation in the Fiddler platform.
A Job tracks the execution of long-running operations such as data publishing,
model artifact uploads, surrogate model training, and other async tasks. Jobs
provide status monitoring, progress tracking, and error handling for operations
that may take significant time to complete.
Key Features:
* **Status Monitoring**: Real-time tracking of job execution state
* **Progress Tracking**: Percentage completion for running operations
* **Error Reporting**: Detailed error messages and failure reasons
* **Timeout Handling**: Configurable timeouts for job completion
* **Responsive Polling**: Efficient status checking with backoff strategies
Job States:
* **PENDING**: Job queued and waiting to start execution
* **STARTED**: Job actively running with progress updates
* **SUCCESS**: Job completed successfully
* **FAILURE**: Job failed with detailed error information
* **RETRY**: Job being retried after a failure
* **REVOKED**: Job cancelled or terminated
## Example
```python theme={null}
# Get a job by ID
job = Job.get(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Job: {job.name} - Status: {job.status}")
print(f"Progress: {job.progress:.1f}%")
# Wait for job completion
job.wait(timeout=1800) # 30 minute timeout
print("Job completed successfully!")
# Monitor job progress in real-time
for job_update in job.watch(interval=10, timeout=3600):
print(f"{job_update.name}: {job_update.progress:.1f}%")
if job_update.status in [JobStatus.SUCCESS, JobStatus.FAILURE]:
break
```
Jobs are created automatically by async operations and cannot be instantiated
directly. Use Job.get() to retrieve existing jobs and the monitoring methods
to track progress and handle completion.
Initialize a Job instance.
Creates a job object for tracking asynchronous operations. This constructor
is typically used internally when deserializing API responses rather than
for direct job creation.
Jobs are automatically created by async operations (like Model.publish(),
Model.add\_artifact(), etc.) and cannot be created directly. Use Job.get()
to retrieve existing jobs and monitoring methods to track progress.
## *classmethod* get()
Retrieve a job by its unique identifier.
Fetches a job from the Fiddler platform using its UUID. This is the primary
way to retrieve job information for monitoring async operations.
### Parameters
The unique identifier (UUID) of the job to retrieve.
Can be provided as a UUID object or string representation.
Whether to include detailed task execution information.
When True, provides additional debugging and progress details.
### Returns
The job instance with current status, progress, and metadata.
### Raises
* **NotFound** – If no job exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get basic job information
job = Job.get(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Job: {job.name} - Status: {job.status}")
print(f"Progress: {job.progress:.1f}%")
# Get detailed job information for debugging
detailed_job = Job.get(
id_="550e8400-e29b-41d4-a716-446655440000",
verbose=True
)
print(f"Task details: {detailed_job.info}")
# Check for errors
if job.status == JobStatus.FAILURE:
print(f"Error: {job.error_reason}")
print(f"Details: {job.error_message}")
```
This method makes an API call to fetch the latest job state from the server.
Use verbose=True when debugging failed jobs to get additional task details.
## watch()
Monitor job progress with real-time status updates.
Continuously polls the job status at specified intervals and yields updated
job instances. This method provides real-time monitoring of job progress
and automatically handles network errors and retries.
### Parameters
Polling interval in seconds between status checks.
Default is configured by JOB\_POLL\_INTERVAL (typically 5-10 seconds).
Maximum time in seconds to monitor the job before giving up.
Default is configured by JOB\_WAIT\_TIMEOUT (typically 1800 seconds).
### Yields
`Job` – Updated job instances with current status and progress.
### Raises
* **TimeoutError** – If the job doesn't complete within the specified timeout.
* **AsyncJobFailed** – If the job fails during execution (raised by wait() method).
### Returns
`Iterator[Job]`
### Example
```python theme={null}
# Monitor job progress with default settings
job = model.publish(source="large_dataset.csv")
for job_update in job.watch():
print(f"Progress: {job_update.progress:.1f}%")
print(f"Status: {job_update.status}")
if job_update.status in [JobStatus.SUCCESS, JobStatus.FAILURE]:
break
# Custom polling interval and timeout
for job_update in job.watch(interval=30, timeout=7200): # 2 hour timeout
print(f"{job_update.name}: {job_update.progress:.1f}%")
if job_update.status == JobStatus.SUCCESS:
print("Job completed successfully!")
break
elif job_update.status == JobStatus.FAILURE:
print(f"Job failed: {job_update.error_message}")
break
# Progress tracking with custom logic
last_progress = 0
for job_update in job.watch(interval=15):
if job_update.progress > last_progress + 10:
print(f"Progress milestone: {job_update.progress:.1f}%")
last_progress = job_update.progress
```
This method handles network errors gracefully and continues monitoring.
It automatically stops when the job reaches a terminal state (SUCCESS,
FAILURE, or REVOKED). Use shorter intervals for more responsive monitoring
but be mindful of API rate limits.
## wait()
Wait for job completion with automatic progress logging.
Blocks execution until the job completes (successfully or with failure).
Provides automatic progress logging and raises an exception if the job fails.
This is the most convenient method for simple job completion waiting.
### Parameters
Polling interval in seconds between status checks.
Default is configured by JOB\_POLL\_INTERVAL (typically 5-10 seconds).
Maximum time in seconds to wait for job completion.
Default is configured by JOB\_WAIT\_TIMEOUT (typically 1800 seconds).
### Raises
* **TimeoutError** – If the job doesn't complete within the specified timeout.
* **AsyncJobFailed** – If the job fails during execution, includes error details.
### Example
```python theme={null}
# Simple job completion waiting
job = model.publish(source="training_data.csv")
job.wait() # Blocks until completion
print("Data publishing completed!")
# Custom timeout for long-running jobs
job = model.add_artifact(model_dir="./model_package")
job.wait(timeout=3600) # 1 hour timeout
print("Model artifact upload completed!")
# Handle job failures
try:
job = model.publish(source="invalid_data.csv")
job.wait()
except AsyncJobFailed as e:
print(f"Job failed: {e}")
# Handle failure (retry, alert, etc.)
# Fast polling for critical operations
job = model.update_surrogate(dataset_id=dataset.id)
job.wait(interval=5, timeout=600) # 5 second polling, 10 min timeout
```
This method automatically logs progress updates to the logger. For custom
progress handling, use the watch() method instead. The method blocks the
current thread until completion, so consider using watch() for non-blocking
monitoring in async applications.
# JobStatus
Source: https://docs.fiddler.ai/sdk-api/python-client/job-status
Status values for asynchronous job operations in Fiddler.
Status values for asynchronous job operations in Fiddler.
This enum defines the possible states that a Job can be in during its lifecycle.
Jobs represent asynchronous operations such as data publishing, model uploads,
and computation tasks that may take significant time to complete.
Job Lifecycle:
1. **PENDING**: Job created and queued for execution
2. **STARTED**: Job execution has begun
3. **SUCCESS/FAILURE**: Job completed with final status
4. **RETRY**: Job failed but will be retried automatically
5. **REVOKED**: Job was cancelled before completion
## Examples
Monitoring job progress:
```python theme={null}
# Start a data publishing job
job = model.publish(source=data_df, environment=fdl.EnvType.PRODUCTION)
# Wait for completion
job.wait() # Blocks until job completes
# Check final status
if job.status == fdl.JobStatus.SUCCESS:
print("Data published successfully")
else:
print(f"Job failed with status: {job.status}")
```
Polling job status manually:
```python theme={null}
import time
# Check status periodically
while job.status in [fdl.JobStatus.PENDING, fdl.JobStatus.STARTED]:
print(f"Job progress: {job.progress}%")
time.sleep(10)
job.refresh() # Update job status from server
# Handle different completion states
if job.status == fdl.JobStatus.SUCCESS:
print("Operation completed successfully")
elif job.status == fdl.JobStatus.FAILURE:
print(f"Operation failed: {job.error_message}")
elif job.status == fdl.JobStatus.REVOKED:
print("Operation was cancelled")
```
Handling job failures and retries:
```python theme={null}
# Monitor job with retry handling
max_wait_time = 3600 # 1 hour timeout
start_time = time.time()
while (time.time() - start_time) < max_wait_time:
if job.status == fdl.JobStatus.SUCCESS:
break
elif job.status == fdl.JobStatus.FAILURE:
print(f"Job failed permanently: {job.error_message}")
break
elif job.status == fdl.JobStatus.RETRY:
print("Job failed but will be retried automatically")
time.sleep(30)
job.refresh()
```
Job status should be refreshed periodically using job.refresh() to get
the latest status from the server. The job.wait() method provides a
convenient way to block until completion.
## PENDING *= 'PENDING'*
Job is queued and waiting to start execution.
This is the initial status when a job is first created. The job has been
submitted to the system but has not yet begun processing. Jobs may remain
in PENDING status if there are resource constraints or if they are waiting
for dependencies.
Characteristics:
* Job is in the execution queue
* No processing has started yet
* May transition to STARTED when resources become available
* Can be cancelled while in this state
Typical duration: Seconds to minutes depending on system load
## STARTED *= 'STARTED'*
Job execution is currently in progress.
The job has begun processing and is actively running. Progress information
may be available through the job.progress property. Jobs in this state
are consuming system resources and performing the requested operation.
Characteristics:
* Active processing is occurring
* Progress updates may be available
* Cannot be cancelled once started
* Will transition to SUCCESS, FAILURE, or RETRY
Typical duration: Minutes to hours depending on operation complexity
## SUCCESS *= 'SUCCESS'*
Job completed successfully.
The operation has finished successfully and all requested work has been
completed. Results are available and the operation achieved its intended
outcome without errors.
Characteristics:
* Operation completed without errors
* Results are available for use
* Job will not change status again
* Resources have been released
This is a terminal status - the job is complete.
## FAILURE *= 'FAILURE'*
Job failed and will not be retried.
The operation encountered an error and could not be completed successfully.
This is a permanent failure - the job will not be automatically retried.
Error details are typically available in the job.error\_message property.
Characteristics:
* Operation failed due to an error
* No automatic retry will occur
* Error details available in error\_message
* Manual intervention may be required
Common causes:
* Invalid input data or parameters
* Insufficient permissions
* System resource exhaustion
* Data validation failures
This is a terminal status - the job will not continue.
## RETRY *= 'RETRY'*
Job failed but will be automatically retried.
The operation encountered a temporary error but the system will automatically
attempt to retry the job. This typically occurs for transient issues like
network timeouts or temporary resource unavailability.
Characteristics:
* Temporary failure occurred
* System will automatically retry
* May transition back to PENDING or STARTED
* Retry attempts are limited
Common causes:
* Network connectivity issues
* Temporary resource constraints
* Transient system errors
* Database connection timeouts
The job may eventually succeed or transition to FAILURE if retries are exhausted.
## REVOKED *= 'REVOKED'*
Job was cancelled or revoked before completion.
The job was explicitly cancelled by the user or system before it could
complete. This may occur due to user cancellation, system shutdown, or
administrative intervention.
Characteristics:
* Job was cancelled before completion
* No results are available
* Resources have been cleaned up
* Operation did not complete
Common causes:
* User-initiated cancellation
* System maintenance or shutdown
* Administrative intervention
* Timeout or resource limits exceeded
This is a terminal status - the job will not continue.
# Model
Source: https://docs.fiddler.ai/sdk-api/python-client/model
Represents a machine learning model in the Fiddler platform.
Represents a machine learning model in the Fiddler platform.
The Model class is the central entity for ML model monitoring and management.
It encapsulates the model's schema, specification, and metadata, and provides
methods for data publishing, artifact management, and monitoring operations.
Key Concepts:
* **Schema** ([`ModelSchema`](/sdk-api/python-client/model-schema)): Defines the structure and data types of model inputs/outputs
* **Spec** ([`ModelSpec`](/sdk-api/python-client/model-spec)): Defines how columns are used (features, targets, predictions, etc.)
* **Task** ([`ModelTask`](/sdk-api/python-client/model-task)): The ML task type (classification, regression, ranking, etc.)
* **Artifacts** (`ModelArtifact`): Deployable model code and dependencies
* **Surrogates** (`Surrogate`): Simplified models for fast explanations
Lifecycle:
1. Create model with schema/spec (from data or manual definition)
2. Upload model artifacts for serving (optional)
3. Publish baseline/training data for drift detection
4. Publish production data for monitoring
5. Set up alerts and monitoring rules
Common Use Cases:
* **Tabular Models**: Traditional ML models with structured data
* **Text Models**: NLP models with text inputs and embeddings
* **Mixed Models**: Models combining tabular and unstructured data
* **Ranking Models**: Recommendation and search ranking systems
* **LLM Models**: Large language model monitoring
## Parameters
Model name, must be unique within the project version.
Should be descriptive and follow naming conventions.
UUID or string identifier of the parent project.
[`ModelSchema`](/sdk-api/python-client/model-schema) defining column structure and data types.
Can be created manually or generated from data.
[`ModelSpec`](/sdk-api/python-client/model-spec) defining how columns are used (inputs, outputs, targets).
Specifies the model's interface and column roles.
Optional version identifier for model versioning and A/B testing.
Falls back to 'v1' when not specified.
[`ModelInputType`](/sdk-api/python-client/model-input-type) - Type of input data the model processes.
* TABULAR: Structured/tabular data (default)
* TEXT: Natural language text data
* MIXED: Combination of structured and unstructured data
[`ModelTask`](/sdk-api/python-client/model-task) - Machine learning task type.
* BINARY\_CLASSIFICATION: Binary classification (0/1, True/False)
* MULTICLASS\_CLASSIFICATION: Multi-class classification
* REGRESSION: Continuous value prediction
* RANKING: Ranking/recommendation tasks
* LLM: Large language model tasks
* NOT\_SET: Task not specified (default)
[`ModelTaskParams`](/sdk-api/python-client/model-task-params) - Task-specific parameters like classification thresholds,
class weights, or ranking parameters.
Human-readable description of the model's purpose,
training data, or other relevant information.
Column name containing unique identifiers for each
prediction event. Used for event tracking and updates.
Column name containing event timestamps.
Used for time-based analysis and drift detection.
Format string for parsing timestamps in event\_ts\_col.
Examples: '%Y-%m-%d %H:%M:%S', '%Y-%m-%dT%H:%M:%SZ'
[`XaiParams`](/sdk-api/python-client/xai-params) - Configuration for explainability features like
explanation methods and custom feature definitions.
The model exists only locally until .create() is called. Use Model.from\_data()
for automatic schema/spec generation from DataFrames or files.
## Examples
```python theme={null}
# Create model from DataFrame with automatic schema detection
import fiddler as fdl
import pandas as pd
df = pd.DataFrame({
'age': [25, 35, 45],
'income': [50000, 75000, 100000],
'approved': [0, 1, 1] # target
})
model = fdl.Model.from_data(
source=df,
name="credit_approval",
project_id='',
task=fdl.ModelTask.BINARY_CLASSIFICATION,
description="Credit approval model v1.0"
)
model.create()
# Publish production events
events = [
{'age': 30, 'income': 60000, 'prediction': 0.8},
{'age': 40, 'income': 80000, 'prediction': 0.9}
]
event_ids = model.publish(source=events)
# Get model info
print(f"Model: {model.name} (Task: {model.task})")
print(f"Columns: {len(model.schema.columns)}")
print(f"Features: {model.spec.inputs}")
```
## download\_data()
Download production or pre-production event data to a local Parquet or CSV file.
Replaces the removed `get_slice()` and `download_slice()` methods.
### Parameters
Directory where the output file is written. Created if it does
not exist.
Environment to query. One of `EnvType.PRODUCTION` or
`EnvType.PRE_PRODUCTION`.
When `env_type=EnvType.PRE_PRODUCTION`, the UUID of the dataset
to query. Ignored for PRODUCTION.
Start of the query window. PRODUCTION only. Naive datetimes
are interpreted as UTC.
UUID of a saved Segment associated with the model. Mutually
exclusive with `segment_definition`.
Inline FQL (Fiddler Query Language) segment expression.
The segment is applied transiently and not saved. Mutually exclusive
with `segment_id`.
Maximum number of rows to return. When omitted, the server
applies a limit of 1,000 rows. Maximum allowed value is 10,000,000.
Column names to include. When omitted, all model columns are
returned.
Row count per download chunk. You can increase this for faster
downloads when querying fewer than 1000 columns and no vector columns.
When `True`, vector columns are included in the output.
When omitted or `False`, vector columns are filtered out.
Output file format. One of `DownloadFormat.PARQUET` or
`DownloadFormat.CSV`.
### Returns
None. Writes a file named `output.parquet` (default) or `output.csv`
(when `output_format=DownloadFormat.CSV`) into `output_dir`.
### Raises
**ApiError** – If the request fails, the model is a draft model, or no data
matches the query.
### Example
```python theme={null}
from datetime import datetime, timedelta, timezone
import fiddler as fdl
import pandas as pd
model = fdl.Model.from_name(name='my_model', project_id='')
end_time = datetime.now(timezone.utc)
start_time = end_time - timedelta(days=7)
model.download_data(
output_dir='./exports',
env_type=fdl.EnvType.PRODUCTION,
start_time=start_time,
end_time=end_time,
output_format=fdl.DownloadFormat.CSV,
)
df = pd.read_csv('./exports/output.csv')
```
This method is not available for draft models. Caller must have READ
permission on the model's project.
## *classmethod* get()
Retrieve a model by its unique identifier.
Fetches a model from the Fiddler platform using its UUID. This is the most
direct way to retrieve a model when you know its ID.
### Parameters
The unique identifier (UUID) of the model to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The model instance with all its configuration and metadata.
### Raises
* **NotFound** – If no model exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get model by UUID
model = Model.get(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved model: {model.name} (Task: {model.task})")
# Access model properties
print(f"Project ID: {model.project_id}")
print(f"Input columns: {model.spec.inputs}")
print(f"Created: {model.created_at}")
```
This method makes an API call to fetch the latest model state from the server.
The returned model instance reflects the current state in Fiddler.
## *classmethod* from\_name()
Retrieve a model by name within a project.
Finds and returns a model using its name and project context. This is useful
when you know the model name but not its UUID. Supports version-specific
retrieval and latest version lookup.
### Parameters
The name of the model to retrieve. Model names are unique
within a project but may have multiple versions.
UUID or string identifier of the project containing the model.
Specific version name to retrieve. If None, behavior depends
on the 'latest' parameter.
If True and version is None, retrieves the most recently created
version. If False, retrieves the first (oldest) version.
Ignored if version is specified.
### Returns
The model instance matching the specified criteria.
### Raises
* **NotFound** – If no model exists with the specified name/version in the project.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get specific version
model = Model.from_name(
name="fraud_detector",
project_id=project.id,
version="v2.1"
)
# Get latest version
latest_model = Model.from_name(
name="fraud_detector",
project_id=project.id,
latest=True
)
# Get first version (default behavior)
first_model = Model.from_name(
name="fraud_detector",
project_id=project.id
)
print(f"Model versions: {first_model.version} -> {latest_model.version}")
```
When version is None and latest=False, returns the first version created.
This provides consistent behavior for accessing the "original" model version.
## create()
Create the model on the Fiddler platform.
Persists this model instance to the Fiddler platform, making it available
for monitoring, data publishing, and other operations. The model must have
a valid schema, spec, and be associated with an existing project.
### Returns
This model instance, updated with server-assigned fields like
ID, creation timestamp, and other metadata.
### Raises
* **Conflict** – If a model with the same name and version already exists
in the project.
* **ValidationError** – If the model configuration is invalid (e.g., invalid
schema, spec, or task parameters).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Create model from DataFrame
model = Model.from_data(
source=training_df,
name="churn_predictor",
project_id=project.id,
task=ModelTask.BINARY_CLASSIFICATION
)
# Create on platform
created_model = model.create()
print(f"Created model with ID: {created_model.id}")
print(f"Created at: {created_model.created_at}")
# Model is now available for monitoring
assert created_model.id is not None
# Can now publish data, set up alerts, etc.
job = created_model.publish(source=production_data)
```
After successful creation, the model instance is updated in-place with
server-assigned metadata. The same instance can be used for subsequent
operations without needing to fetch it again.
## update()
Update an existing model.
## add\_column()
Add a new column to the model schema.
Updates both the schema and spec to include the new column. This allows
you to extend your model with additional columns after initial creation.
**New in version 3.11.0**
### Parameters
Column object defining the new column's properties (name, data\_type, etc.)
Type of column in spec. One of: 'inputs', 'outputs', 'targets',
'decisions', 'metadata'. Defaults to 'metadata'.
### Raises
* **ValueError** – If column already exists or column\_type is invalid
* **BadRequest** – If column definition is invalid per backend validation
### Example
```python theme={null}
# Add a numeric metadata column
new_col = Column(
name="customer_segment",
data_type=DataType.INTEGER,
min=1,
max=5
)
model.add_column(column=new_col, column_type='metadata')
# Add a categorical feature
category_col = Column(
name="region",
data_type=DataType.CATEGORY,
categories=["US", "EU", "APAC"]
)
model.add_column(column=category_col, column_type='inputs')
```
* Adding a column doesn't populate historical data; new column will be null
for past events
* Column names must be unique within the model
* After adding a column, include it in future event publishing
## *classmethod* list()
List models in a project with optional filtering.
Retrieves all models or model versions within a project. Returns lightweight
ModelCompact objects that can be used to fetch full Model instances when needed.
### Parameters
UUID or string identifier of the project to search within.
Optional model name filter. If provided, returns all versions
of the specified model. If None, returns all models in the project.
### Yields
*ModelCompact* –
Lightweight model objects containing id, name, and version.
Call .fetch() on any ModelCompact to get the full Model instance.
### Returns
`Iterator[ModelCompact]`
### Example
```python theme={null}
# List all models in project
for model_compact in Model.list(project_id=project.id):
print(f"Model: {model_compact.name} v{model_compact.version}")
print(f" ID: {model_compact.id}")
# List all versions of a specific model
for version in Model.list(project_id=project.id, name="fraud_detector"):
print(f"Version: {version.version}")
...
# Get full model details if needed
full_model = version.fetch()
print(f" Task: {full_model.task}")
print(f" Created: {full_model.created_at}")
# Convert to list for counting
models = list(Model.list(project_id=project.id))
print(f"Total models in project: {len(models)}")
```
This method returns an iterator for memory efficiency when dealing with
many models. The ModelCompact objects are lightweight and don't include
full schema/spec information - use .fetch() when you need complete details.
## duplicate()
Duplicate the model instance with the given version name.
This call will not save the model on server. After making changes to the model
instance call .create() to add the model version to Fiddler Platform.
### Parameters
Version name for the new instance
### Returns
Model instance
## *property* datasets
Get all datasets associated with this model.
Returns an iterator over all datasets that have been published to this model,
including both production data and pre-production datasets used for baselines
and drift comparison.
### Yields
[`Dataset`](/sdk-api/python-client/dataset) –
Dataset objects containing metadata and data access methods.
Each dataset represents a collection of events published to the model.
### Example
```python theme={null}
# List all datasets for the model
for dataset in model.datasets:
print(f"Dataset: {dataset.name}")
print(f" Environment: {dataset.environment}")
print(f" Size: {dataset.size} events")
print(f" Created: {dataset.created_at}")
# Find specific dataset
baseline_datasets = [
ds for ds in model.datasets
if ds.environment == EnvType.PRE_PRODUCTION
]
print(f"Found {len(baseline_datasets)} baseline datasets")
```
This includes both production event data and named pre-production datasets.
Use the Dataset objects to download data, analyze distributions, or
set up baseline comparisons for drift detection.
## *property* baselines
Get all baselines configured for this model.
Returns an iterator over all baseline configurations used for drift detection
and performance monitoring. Baselines define reference distributions and
metrics for comparison with production data.
### Yields
[`Baseline`](/sdk-api/python-client/baseline) –
Baseline objects containing configuration and reference data.
Each baseline defines how drift and performance should be measured
against historical or reference datasets.
### Example
```python theme={null}
# List all baselines
for baseline in model.baselines:
print(f"Baseline: {baseline.name}")
print(f" Type: {baseline.type}") # STATIC or ROLLING
print(f" Dataset: {baseline.dataset_name}")
print(f" Created: {baseline.created_at}")
# Find production baseline
prod_baselines = [
bl for bl in model.baselines
if "production" in bl.name.lower()
]
# Use baseline for drift comparison
if prod_baselines:
baseline = prod_baselines[0]
drift_metrics = baseline.compute_drift(recent_data)
```
Baselines are essential for drift detection and alerting. They define
the "normal" behavior against which production data is compared.
Static baselines use fixed reference data, while rolling baselines
update automatically with recent data.
## *property* deployment
Fetch model deployment instance of this model.
### Returns
The deployment configuration for this model.
## *classmethod* from\_data()
Create a Model instance with automatic schema generation from data.
This is the most convenient way to create models when you have training data
or representative samples. The method automatically analyzes the data to
generate appropriate schema (column types) and spec (column roles) definitions.
### Parameters
Data source for schema generation. Can be:
* pandas.DataFrame: Direct data analysis
* Path/str: File path (.csv, .parquet, .json supported)
The data should be representative of your model's inputs/outputs.
Model name, must be unique within the project version.
Use descriptive names like "fraud\_detector\_v1" or "churn\_model".
UUID or string identifier of the parent project.
Optional [`ModelSpec`](/sdk-api/python-client/model-spec) defining column roles. If None, automatic
detection attempts to identify inputs, outputs, and targets
based on column names and data patterns.
Model version identifier. If None, defaults to "v1".
Use semantic versioning like "v1.0", "v2.1", etc.
[`ModelInputType`](/sdk-api/python-client/model-input-type) - Type of input data the model processes.
* TABULAR: Structured/tabular data (default)
* TEXT: Natural language text data
* MIXED: Combination of structured and unstructured data
[`ModelTask`](/sdk-api/python-client/model-task) - Machine learning task type:
* BINARY\_CLASSIFICATION: Binary classification (0/1, True/False)
* MULTICLASS\_CLASSIFICATION: Multi-class classification
* REGRESSION: Continuous value prediction
* RANKING: Ranking/recommendation tasks
* LLM: Large language model tasks
* NOT\_SET: Task not specified (default)
[`ModelTaskParams`](/sdk-api/python-client/model-task-params) - Task-specific parameters:
* binary\_classification\_threshold: Decision threshold (0.5)
* target\_class\_order: Class label ordering
* group\_by: Column for ranking model grouping
* top\_k: Top-k evaluation for ranking
Human-readable description of the model's purpose,
training approach, or other relevant information.
Column name for unique event identifiers. Used for
tracking individual predictions and enabling updates.
Column name for event timestamps. Used for time-based
analysis, drift detection, and temporal monitoring.
Timestamp format string for parsing event\_ts\_col.
Examples: '%Y-%m-%d %H:%M:%S', '%Y-%m-%dT%H:%M:%SZ'
[`XaiParams`](/sdk-api/python-client/xai-params) - Explainability configuration including explanation methods
and custom feature definitions.
Maximum unique values to consider a column categorical.
Columns with more unique values are treated as continuous.
Default is typically 100-1000 depending on data size.
Number of rows to sample for schema generation. Useful
for large datasets to speed up analysis. If None, uses
entire dataset (up to reasonable limits).
### Returns
A new Model instance with automatically generated schema and spec.
The model is not yet created on the platform - call .create() to persist.
### Raises
* **ValueError** – If the data source is invalid or cannot be processed.
* **FileNotFoundError** – If source is a file path that doesn't exist.
* **ValidationError** – If the generated schema/spec is invalid.
### Example
```python theme={null}
import pandas as pd
# Create from DataFrame
df = pd.DataFrame({
'age': [25, 35, 45, 55],
'income': [30000, 50000, 70000, 90000],
'credit_score': [650, 700, 750, 800],
'approved': [0, 1, 1, 1], # target
'prediction': [0.2, 0.8, 0.9, 0.95], # model output
'prediction_score': [0.2, 0.8, 0.9, 0.95] # alternative output
})
model = Model.from_data(
source=df,
name="credit_approval_v1",
project_id=project.id,
task=ModelTask.BINARY_CLASSIFICATION,
description="Credit approval model trained on 2024 data",
event_id_col="application_id", # if present in real data
event_ts_col="timestamp" # if present in real data
)
# Review generated schema
print(f"Columns detected: {len(model.schema.columns)}")
for col in model.schema.columns:
print(f" {col.name}: {col.data_type}")
# Review generated spec
print(f"Inputs: {model.spec.inputs}")
print(f"Outputs: {model.spec.outputs}")
print(f"Targets: {model.spec.targets}")
# Create on platform
model.create()
# Create from file
model = Model.from_data(
source="training_data.csv",
name="file_based_model",
project_id=project.id,
task=ModelTask.REGRESSION,
sample_size=10000 # Sample large files
)
```
The automatic schema generation uses heuristics to detect column types
and roles. Review the generated schema and spec before calling .create()
to ensure they match your model's actual interface. You can modify the
schema and spec after creation if needed.
## delete()
Delete a model and it's associated resources.
### Returns
model deletion job instance
## remove\_column()
Remove a column from the model schema and spec
This method is only to modify model object before creating and
will not save the model on Fiddler Platform. After making
changes to the model instance, call .create() to add the model
to Fiddler Platform.
### Parameters
Column name to be removed
If True, do not raise an error if the column is not found
### Returns
None
### Raises
**KeyError** – If the column name is not found and missing\_ok is False
## publish()
Publish data to the model for monitoring and analysis.
Deprecated: Use [`publish_stream()`](#publish_stream) for streaming events (list of dicts) or [`publish_batch()`](#publish_batch) for batch publishing (file path or DataFrame).
Uploads prediction events, training data, or reference datasets to Fiddler
for monitoring, drift detection, and performance analysis. This is how you
send your model's real-world data to the platform.
### Parameters
Data to publish. Supported formats:
* **File path (str/Path)**: CSV or Parquet files.
> Best for large datasets and batch uploads.
* **DataFrame**: Pandas DataFrame with prediction events.
Good for programmatic uploads and real-time data.
* **List of dicts**: Individual events as dictionaries.
Perfect for streaming/real-time publishing (max 1000 events).
Each dict should match the model's schema.
[`EnvType`](/sdk-api/python-client/env-type) - Data environment type:
* **PRODUCTION**: Live production prediction data.
> Used for real-time monitoring and alerting.
* **PRE\_PRODUCTION**: Training, validation, or baseline data.
Used for drift comparison and model evaluation.
Name for the dataset when using PRE\_PRODUCTION environment.
Creates a named dataset for baseline comparisons.
Not used for PRODUCTION data.
Whether these events update previously published data.
Set to True when republishing corrected predictions or
adding ground truth labels to existing events.
### Returns
Event IDs when source is list of dicts or DataFrame.
Use these IDs to reference specific events later.
* [`Job`](/sdk-api/python-client/job): Async job object when source is a file path.
Use job.wait() to wait for completion or check job.status.
### Raises
* **ValidationError** – If the data doesn't match the model's schema or
contains invalid values.
* **ApiError** – If there's an error uploading the data to Fiddler.
* **ValueError** – If the source format is unsupported or parameters
are incompatible (e.g., dataset\_name with PRODUCTION).
### Example
```python theme={null}
# Publish production events from DataFrame
# (assumes model is a fdl.Model instance; see Model.from_name() or Model.from_data())
import fiddler as fdl
import pandas as pd
prod_df = pd.DataFrame({
'age': [30, 25, 45],
'income': [60000, 45000, 80000],
'prediction': [0.8, 0.3, 0.9],
'timestamp': ['2024-01-01 10:00:00', '2024-01-01 11:00:00', '2024-01-01 12:00:00']
})
# Returns list of event UUIDs
event_ids = model.publish(
source=prod_df,
environment=fdl.EnvType.PRODUCTION
)
print(f"Published {len(event_ids)} events")
# Publish baseline data for drift comparison
job = model.publish(
source="training_data.csv", # File path
environment=fdl.EnvType.PRE_PRODUCTION,
dataset_name="training_baseline_2024"
)
job.wait() # Wait for upload to complete
print(f"Baseline upload status: {job.status}")
# Publish real-time streaming events
events = [
{
'age': 35,
'income': 70000,
'prediction': 0.75,
'event_id': 'pred_001',
'timestamp': '2024-01-01 13:00:00'
},
{
'age': 28,
'income': 55000,
'prediction': 0.45,
'event_id': 'pred_002',
'timestamp': '2024-01-01 13:01:00'
}
]
event_ids = model.publish(source=events)
print(f"Published {len(events)} streaming events")
# Update existing events with ground truth
corrected_events = [
{
'event_id': 'pred_001',
'ground_truth': 1, # Actual outcome
'timestamp': '2024-01-01 13:00:00'
}
]
model.publish(source=corrected_events, update=True)
```
* **Schema Validation**: All published data must match the model's schema.
Column names, types, and value ranges are validated.
* **Event IDs**: Include event\_id\_col if specified in model config for
event tracking and updates.
* **Timestamps**: Include event\_ts\_col for time-based analysis and drift detection.
* **Batch Limits**: List of dicts is limited to 1000 events per call.
Use files or multiple calls for larger datasets.
Production data publishing enables real-time monitoring, alerting, and
drift detection. Pre-production data creates reference datasets for
comparison and model evaluation.
## publish\_stream()
Publish events via streaming for real-time monitoring.
Sends events synchronously to Fiddler. Events are written to an
in-memory queue and asynchronously persisted.
### Parameters
List of event dicts matching the model's schema. Each dict
represents a single prediction event with feature values,
predictions, and optional metadata (event\_id, timestamp).
[`EnvType`](/sdk-api/python-client/env-type) - Data environment type:
* **PRODUCTION**: Live production prediction data.
* **PRE\_PRODUCTION**: Training, validation, or baseline data.
Name for the dataset when using PRE\_PRODUCTION environment.
Not used for PRODUCTION data.
Whether these events update previously published data.
Set to True when adding ground truth labels to existing events.
### Returns
Event IDs for the published events. Use these IDs to
reference specific events later.
### Raises
* **ValidationError** – If the data doesn't match the model's schema.
* **ApiError** – If there's an error publishing the events to Fiddler.
### Example
```python theme={null}
events = [
{
'age': 35,
'income': 70000,
'prediction': 0.75,
'event_id': 'pred_001',
'timestamp': '2024-01-01 13:00:00'
},
{
'age': 28,
'income': 55000,
'prediction': 0.45,
'event_id': 'pred_002',
'timestamp': '2024-01-01 13:01:00'
}
]
event_ids = model.publish_stream(events=events)
print(f"Published {len(event_ids)} streaming events")
# Update existing events with ground truth
ground_truth = [
{'event_id': 'pred_001', 'ground_truth': 1}
]
model.publish_stream(events=ground_truth, update=True)
```
For large datasets, consider using [`publish_batch()`](#publish_batch) with a file
path or DataFrame instead. Streaming is best suited for real-time
or near-real-time event publishing.
## publish\_batch()
Publish data via batch upload for monitoring and analysis.
Uploads data as a file and kicks off an asynchronous server-side job.
Use the returned Job object to track progress. Best suited for large
datasets, historical data uploads, and baseline/reference datasets.
### Parameters
Data to publish. Supported formats:
* **File path (str/Path)**: CSV or Parquet files.
> Best for large datasets already on disk.
* **DataFrame**: Pandas DataFrame with prediction events.
Automatically converted to Parquet (falls back to CSV).
[`EnvType`](/sdk-api/python-client/env-type) - Data environment type:
* **PRODUCTION**: Live production prediction data.
* **PRE\_PRODUCTION**: Training, validation, or baseline data.
Name for the dataset when using PRE\_PRODUCTION environment.
Creates a named dataset for baseline comparisons.
Not used for PRODUCTION data.
Whether these events update previously published data.
Set to True when republishing corrected predictions or
adding ground truth labels to existing events.
### Returns
Async job object. Use `job.wait()`
to block until completion, or check `job.status` to poll.
### Raises
* **ValidationError** – If the data doesn't match the model's schema.
* **ApiError** – If there's an error uploading the data to Fiddler.
* **ValueError** – If the source format is unsupported.
### Example
```python theme={null}
# Batch upload from file
job = model.publish_batch(
source="training_data.csv",
environment=EnvType.PRE_PRODUCTION,
dataset_name="training_baseline_2024"
)
job.wait()
print(f"Upload status: {job.status}")
# Batch upload from DataFrame
import pandas as pd
prod_df = pd.DataFrame({
'age': [30, 25, 45],
'income': [60000, 45000, 80000],
'prediction': [0.8, 0.3, 0.9],
})
job = model.publish_batch(source=prod_df)
job.wait()
```
For real-time or near-real-time event publishing, consider using
[`publish_stream()`](#publish_stream) instead.
## delete\_events()
Delete published production events for this model.
Select events to delete by a time range or by an explicit list of event
IDs (the two are mutually exclusive). Deletion runs asynchronously on the
server; use the returned Job to track progress.
### Parameters
Inclusive start of the time window to delete. Accepts a
timezone-aware `datetime` or an ISO 8601 string with an
explicit offset (e.g. `'2025-11-17T01:00:00Z'`); the
value is normalized to UTC. Naive datetimes are rejected.
Must be given together with `end_time`.
Exclusive end of the time window to delete. Same format
rules as `start_time`. Must be given together with
`start_time`.
List of event IDs to delete, up to the documented
per-call limit. An empty list is treated as "no ID
filter" (the same as `None`). Cannot be combined with a time range.
Whether the server should recompute metrics after
the events are deleted. Applies to time-range deletions only;
combining `update_metrics=True` with `event_ids` is rejected.
### Returns
Async job object. Use `job.wait()`
to block until completion, or check `job.status` to poll.
### Raises
* **ValueError** – If neither a time range nor `event_ids` is given (to
avoid accidentally deleting all events for the model), if
only one of `start_time`/`end_time` is provided, or if
`event_ids` exceeds the documented per-call limit.
* **ApiError** – If there's an error deleting the events on Fiddler.
### Example
```python theme={null}
# Delete a window of production events
job = model.delete_events(
start_time='2025-11-17T01:00:00.000Z',
end_time='2025-11-17T02:00:00.000Z',
update_metrics=True,
)
job.wait()
print(f"Delete status: {job.status}")
# Delete specific events by ID
job = model.delete_events(event_ids=['pred_001', 'pred_002'])
job.wait()
```
Deletion is irreversible. Provide either a time range or `event_ids`
to scope the deletion (not both); a call with no filter is rejected.
## add\_artifact()
Upload and deploy model artifact.
### Parameters
Path to model artifact tar file
Model deployment parameters
### Returns
Async job instance
## update\_artifact()
Update existing model artifact.
### Parameters
Path to model artifact tar file
Model deployment parameters
### Returns
Async job instance
## download\_artifact()
Download existing model artifact.
### Parameters
Path to download model artifact tar file
## add\_surrogate()
Add a new surrogate model
### Parameters
Dataset to be used for generating surrogate model
Model deployment parameters
### Returns
Async job
## update\_surrogate()
Update an existing surrogate model
### Parameters
Dataset to be used for generating surrogate model
Model deployment parameters
### Returns
Async job
# ModelCompact
Source: https://docs.fiddler.ai/sdk-api/python-client/model-compact
Lightweight model representation for listing and basic operations.
Lightweight model representation for listing and basic operations.
A minimal model object containing only essential identifiers. Used by
Model.list() to efficiently return model information without fetching
full schema and configuration details.
## Example
```python theme={null}
# Get from listing
models = list(Model.list(project_id=project.id))
compact_model = models[0]
# Access basic info
print(f"Model: {compact_model.name} v{compact_model.version}")
print(f"ID: {compact_model.id}")
# Fetch full details when needed
full_model = compact_model.fetch()
print(f"Task: {full_model.task}")
print(f"Schema: {len(full_model.schema.columns)} columns")
```
## id
## name
## version
## fetch()
Fetch the complete Model instance.
Retrieves the full Model object with all schema, spec, and configuration
details from the Fiddler platform using this compact model's ID.
### Returns
Complete model instance with all details and capabilities.
### Example
```python theme={null}
# From model listing
compact = next(Model.list(project_id=project.id))
# Get full model details
full_model = compact.fetch()
# Now can access full functionality
full_model.publish(source=data)
print(f"Input columns: {full_model.spec.inputs}")
```
# ModelDeployment
Source: https://docs.fiddler.ai/sdk-api/python-client/model-deployment
Model deployment management for serving infrastructure.
Model deployment management for serving infrastructure.
This class manages containerized model deployments including resource
allocation, scaling, and activation status.
## Examples
Get and update deployment:
```python theme={null}
# Get deployment
deployment = ModelDeployment.of(model_id=model.id)
# Update resources
deployment.replicas = 3
deployment.cpu = 1000
deployment.memory = 2048
job = deployment.update()
job.wait()
```
Initialize ModelDeployment instance.
## Parameters
Model identifier
## update()
Update an existing model deployment.
### Returns
[`Job`](/sdk-api/python-client/job)
## *classmethod* of()
Get model deployment instance of the given model
### Parameters
Model identifier
### Returns
ModelDeployment instance
# ModelInputType
Source: https://docs.fiddler.ai/sdk-api/python-client/model-input-type
Input data types supported by Fiddler models.
Input data types supported by Fiddler models.
This enum defines the different types of input data that models can process.
The input type determines how Fiddler handles data preprocessing, validation,
and monitoring for the model.
## Examples
Defining model input type during onboarding:
```python theme={null}
# Tabular data model (traditional ML)
model = fdl.Model.from_data(
name='credit_model',
source=credit_data,
spec=model_spec,
task=fdl.ModelTask.BINARY_CLASSIFICATION,
input_type=fdl.ModelInputType.TABULAR
)
# Text-based model (NLP)
model = fdl.Model.from_data(
name='sentiment_model',
source=text_data,
spec=model_spec,
task=fdl.ModelTask.MULTICLASS_CLASSIFICATION,
input_type=fdl.ModelInputType.TEXT
)
# Mixed data model (multimodal)
model = fdl.Model.from_data(
name='multimodal_model',
source=mixed_data,
spec=model_spec,
task=fdl.ModelTask.REGRESSION,
input_type=fdl.ModelInputType.MIXED
## )
```
Input type affects data validation, preprocessing, and available
monitoring features. Choose the type that best matches your model's
primary input data format.
## TABULAR *= 'structured'*
Structured tabular data with rows and columns.
Used for traditional machine learning models that operate on structured
datasets with well-defined features. This includes most supervised learning
models trained on CSV data, database tables, or pandas DataFrames.
Characteristics:
* Fixed schema with defined columns
* Numeric and categorical features
* Traditional ML algorithms (trees, linear models, etc.)
* Standard drift detection on individual features
Typical use cases:
* Credit scoring models
* Fraud detection systems
* Customer churn prediction
* Sales forecasting models
* Risk assessment models
Supported data types: All DataType enum values
## TEXT *= 'text'*
Natural language text data.
Used for models that primarily process text inputs such as NLP models,
language models, and text classification systems. Enables text-specific
monitoring and embedding-based drift detection.
Characteristics:
* Text strings as primary input
* Embedding-based feature monitoring
* Text-specific preprocessing
* Language model optimizations
Typical use cases:
* Sentiment analysis models
* Document classification
* Named entity recognition
* Text summarization models
* Chatbots and conversational AI
Special considerations:
* May require text embedding custom features
* Supports text-specific enrichments
* Drift detection on embedding vectors
## MIXED *= 'mixed'*
Combination of structured and unstructured data.
Used for multimodal models that process both structured tabular data and
unstructured data like text, images, or embeddings. Enables comprehensive
monitoring across different data types.
Characteristics:
* Multiple data modalities
* Complex feature interactions
* Flexible schema definitions
* Advanced custom feature support
Typical use cases:
* Recommendation systems (user features + content)
* Fraud detection (transaction data + text descriptions)
* Medical diagnosis (structured data + images/text)
* E-commerce search (product features + text queries)
* Content moderation (metadata + text/images)
Special considerations:
* Requires careful schema design
* May need multiple custom feature types
* Complex drift monitoring setup
# ModelSchema
Source: https://docs.fiddler.ai/sdk-api/python-client/model-schema
Defines the complete schema structure for a model's input data.
Defines the complete schema structure for a model's input data.
ModelSchema contains the specification of all columns that a model expects
to receive, including their data types, constraints, and metadata. This schema
is used by Fiddler for data validation, monitoring, and analysis purposes.
The schema acts as a contract between your model and Fiddler, ensuring that
incoming data conforms to expected formats and enabling proper drift detection,
data quality monitoring, and other features.
## Examples
Creating a model schema:
```python theme={null}
schema = ModelSchema(
columns=[
Column(name="age", data_type=DataType.INTEGER, min=0, max=120),
Column(name="income", data_type=DataType.FLOAT, min=0),
Column(name="category", data_type=DataType.CATEGORY,
categories=["A", "B", "C"])
]
)
```
Accessing columns by name:
```python theme={null}
age_column = schema["age"]
print(age_column.data_type)
```
Adding a new column:
```python theme={null}
new_column = Column(name="score", data_type=DataType.FLOAT)
schema["score"] = new_column
```
Removing a column:
```python theme={null}
del schema["age"]
```
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
## schema\_version
Schema version
## columns
List of columns
## **getitem**()
Get column by name
### Returns
[`Column`](/sdk-api/python-client/column)
## **setitem**()
Set column by name
## **delitem**()
Delete column by name
# ModelSpec
Source: https://docs.fiddler.ai/sdk-api/python-client/model-spec
Defines how model columns are categorized and used along with model task configuration.
Defines how model columns are categorized and used along with model task configuration.
ModelSpec provides a comprehensive specification of how different columns in your
model's data should be interpreted and used. It categorizes columns into inputs,
outputs, targets, decisions, and metadata, and allows for custom feature definitions
that enhance model monitoring and analysis capabilities.
This specification is crucial for Fiddler to understand your model's structure,
enabling proper monitoring, drift detection, bias analysis, and explainability
features. It acts as the contract between your model and Fiddler's monitoring
infrastructure.
* **custom\_features** (*List* *\[Multivariate* *|* *VectorFeature* *|* *TextEmbedding* *|* *ImageEmbedding* *|* *Enrichment* *]*)
## Examples
Creating a basic model spec for classification:
```python theme={null}
spec = ModelSpec(
inputs=["age", "income", "credit_score"],
outputs=["prediction", "probability"],
targets=["approved"],
metadata=["customer_id", "timestamp"]
)
```
Creating a spec with custom features:
```python theme={null}
from fiddler.schemas.custom_features import Multivariate, TextEmbedding
spec = ModelSpec(
inputs=["user_clicks", "session_time", "review_text_embedding"],
outputs=["recommendation_score"],
targets=["user_rating"],
metadata=["user_id", "session_id"],
custom_features=[
Multivariate(
name="user_behavior",
columns=["user_clicks", "session_time"],
n_clusters=5
),
TextEmbedding(
name="review_clusters",
column="review_text_embedding",
source_column="review_text",
n_clusters=8
)
]
)
```
Creating a spec for ranking models:
```python theme={null}
ranking_spec = ModelSpec(
inputs=["query_features", "doc_features", "relevance_score"],
outputs=["ranking_score"],
targets=["click_through"],
decisions=["final_ranking"],
metadata=["query_id", "doc_id"]
)
```
## schema\_version
Schema version
## inputs
Feature columns
## outputs
Prediction columns
## targets
Label columns
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
## decisions
Decisions columns
## metadata
Metadata columns
## custom\_features
Custom feature definitions
## remove\_column()
Remove a column name from spec if it exists.
# ModelTask
Source: https://docs.fiddler.ai/sdk-api/python-client/model-task
Machine learning task types supported by Fiddler.
Machine learning task types supported by Fiddler.
This enum defines the different types of ML tasks that Fiddler can monitor.
The task type determines which metrics are calculated, how performance is
measured, and what monitoring capabilities are available.
Task-Specific Features:
* **Classification**: Accuracy, precision, recall, F1, AUC, confusion matrix
* **Regression**: MAE, MSE, RMSE, R², residual analysis
* **Ranking**: NDCG, MAP, precision\@k, ranking-specific metrics
* **LLM**: Token-based metrics, response quality, safety metrics
## Examples
Configuring models for different tasks:
```python theme={null}
# Binary classification (fraud detection)
fraud_model = fdl.Model.from_data(
name='fraud_detector',
source=fraud_data,
spec=model_spec,
task=fdl.ModelTask.BINARY_CLASSIFICATION,
task_params=fdl.ModelTaskParams(
binary_classification_threshold=0.5
)
)
# Multiclass classification (sentiment analysis)
sentiment_model = fdl.Model.from_data(
name='sentiment_analyzer',
source=sentiment_data,
spec=model_spec,
task=fdl.ModelTask.MULTICLASS_CLASSIFICATION,
task_params=fdl.ModelTaskParams(
target_class_order=['negative', 'neutral', 'positive']
)
)
# Regression (price prediction)
price_model = fdl.Model.from_data(
name='price_predictor',
source=price_data,
spec=model_spec,
task=fdl.ModelTask.REGRESSION
)
# Ranking (recommendation system)
ranking_model = fdl.Model.from_data(
name='recommender',
source=ranking_data,
spec=model_spec,
task=fdl.ModelTask.RANKING,
task_params=fdl.ModelTaskParams(
group_by='user_id',
top_k=10
)
)
# LLM (language model)
llm_model = fdl.Model.from_data(
name='chatbot',
source=conversation_data,
spec=model_spec,
task=fdl.ModelTask.LLM
## )
```
Task type cannot be changed after model creation. Choose carefully
based on your model's primary objective and output format.
## BINARY\_CLASSIFICATION *= 'binary\_classification'*
Two-class classification tasks.
Used for models that predict one of two possible outcomes or classes.
Enables binary classification metrics and threshold-based analysis.
Available metrics:
* Accuracy, Precision, Recall, F1-score
* AUC-ROC, AUC-PR curves
* Confusion matrix analysis
* Threshold optimization tools
Typical use cases:
* Fraud detection (fraud/legitimate)
* Email spam filtering (spam/ham)
* Medical diagnosis (positive/negative)
* Credit approval (approve/deny)
* Churn prediction (churn/retain)
Required outputs: Single probability score or binary prediction
Task parameters: binary\_classification\_threshold
## MULTICLASS\_CLASSIFICATION *= 'multiclass\_classification'*
Multi-class classification tasks.
Used for models that predict one of multiple possible classes or categories.
Supports comprehensive multiclass performance analysis and class-specific metrics.
Available metrics:
* Per-class precision, recall, F1-score
* Macro and micro-averaged metrics
* Confusion matrix with multiple classes
* Class distribution analysis
Typical use cases:
* Document categorization (multiple topics)
* Image classification (multiple objects)
* Sentiment analysis (positive/neutral/negative)
* Product categorization
* Intent classification in chatbots
Required outputs: Class probabilities or single class prediction
Task parameters: target\_class\_order, class\_weights
## REGRESSION *= 'regression'*
Continuous value prediction tasks.
Used for models that predict numerical values on a continuous scale.
Enables regression-specific metrics and residual analysis.
Available metrics:
* Mean Absolute Error (MAE)
* Mean Squared Error (MSE)
* Root Mean Squared Error (RMSE)
* R-squared (coefficient of determination)
* Residual distribution analysis
Typical use cases:
* Price prediction
* Sales forecasting
* Risk scoring (continuous scores)
* Demand forecasting
* Performance rating prediction
Required outputs: Single continuous numerical value
Task parameters: None (uses standard regression metrics)
## RANKING *= 'ranking'*
Ranking and recommendation tasks.
Used for models that rank items or provide ordered recommendations.
Supports ranking-specific metrics and list-wise evaluation.
Available metrics:
* Normalized Discounted Cumulative Gain (NDCG)
* Mean Average Precision (MAP)
* Precision\@K, Recall\@K
* Mean Reciprocal Rank (MRR)
* Hit Rate analysis
Typical use cases:
* Search result ranking
* Product recommendations
* Content recommendation systems
* Information retrieval
* Personalized ranking
Required outputs: Ranked list of items with scores
Task parameters: group\_by (session/user ID), top\_k
Special data format: Grouped by query/session identifier
## LLM *= 'llm'*
Large language model and generative AI tasks.
Used for language models, chatbots, and generative AI applications.
Enables LLM-specific monitoring including safety, quality, and performance metrics.
Available metrics:
* Response quality metrics
* Safety and toxicity detection
* Hallucination detection
* Token-based analysis
* Latency and throughput metrics
Typical use cases:
* Chatbots and conversational AI
* Text generation models
* Question-answering systems
* Code generation models
* Content creation assistants
Special features:
* Guardrails integration
* Safety monitoring
* Prompt and response analysis
* Token usage tracking
## NOT\_SET *= 'not\_set'*
Placeholder for undefined or unspecified tasks.
Used as a default value when the model task has not been explicitly
defined. Should be replaced with an appropriate task type during
model configuration.
This value should not be used for production models as it limits
available monitoring capabilities and metrics.
## is\_classification()
Check if the task is a classification type.
### Returns
True if task is binary or multiclass classification
## is\_regression()
Check if the task is regression.
### Returns
True if task is regression
# ModelTaskParams
Source: https://docs.fiddler.ai/sdk-api/python-client/model-task-params
Configuration parameters for different model task types and evaluation metrics.
Configuration parameters for different model task types and evaluation metrics.
ModelTaskParams defines task-specific parameters that control how models are
evaluated and monitored within Fiddler. Different model types (classification,
regression, ranking) require different parameters to properly compute metrics
and perform analysis.
These parameters are essential for accurate metric computation, proper baseline
establishment, and meaningful performance monitoring across different model
types and use cases.
## Examples
Configuration for binary classification:
```python theme={null}
binary_params = ModelTaskParams(
binary_classification_threshold=0.5,
target_class_order=["negative", "positive"]
)
```
Configuration for multi-class classification with class weights:
```python theme={null}
multiclass_params = ModelTaskParams(
target_class_order=["class_a", "class_b", "class_c"],
class_weights=[0.3, 0.5, 0.2],
weighted_ref_histograms=True
)
```
Configuration for ranking models:
```python theme={null}
ranking_params = ModelTaskParams(
group_by="query_id",
top_k=10,
target_class_order=["not_relevant", "relevant", "highly_relevant"]
)
```
Configuration for imbalanced datasets:
```python theme={null}
imbalanced_params = ModelTaskParams(
binary_classification_threshold=0.3,
class_weights=[0.1, 0.9],
weighted_ref_histograms=True
)
```
## binary\_classification\_threshold
Threshold for labels
## target\_class\_order
Order of target classes
## group\_by
Query/session id column for ranking models
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
## top\_k
Top k results to consider when computing ranking metrics
## class\_weights
Weight of each classes
## weighted\_ref\_histograms
Whether baseline histograms must be weighted or not while drift metrics
# Multivariate
Source: https://docs.fiddler.ai/sdk-api/python-client/multivariate
Represents custom features derived from multiple columns using clustering analysis.
Represents custom features derived from multiple columns using clustering analysis.
Multivariate features combine multiple numeric columns into a single derived feature
using k-means clustering algorithms. This enables monitoring of multivariate drift
and detecting unusual combinations that might not be apparent when monitoring
columns individually.
The feature type is automatically set to CustomFeatureType.FROM\_COLUMNS and uses
clustering to group similar combinations of column values for drift detection.
## Examples
Creating a user behavior multivariate feature:
```python theme={null}
behavior_feature = Multivariate(
name="user_engagement_cluster",
columns=["page_views", "session_duration", "clicks"],
n_clusters=8,
monitor_components=True
)
```
Creating a system performance multivariate feature:
```python theme={null}
perf_feature = Multivariate(
name="system_health",
columns=["cpu_usage", "memory_usage", "response_time"],
n_clusters=5,
monitor_components=False
)
```
## type
## n\_clusters
## centroids
## columns
## monitor\_components
## *classmethod* validate\_columns()
### Returns
`List[str]`
## *classmethod* validate\_n\_clusters()
### Returns
`int`
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# NotFound
Source: https://docs.fiddler.ai/sdk-api/python-client/not-found
Raised when a requested resource is not found (HTTP 404).
Raised when a requested resource is not found (HTTP 404).
This exception is thrown when attempting to access or manipulate a resource
that does not exist in the Fiddler platform. This can include models,
projects, datasets, alerts, or any other entity that cannot be located
using the provided identifier.
Common scenarios include using incorrect IDs, attempting to access deleted
resources, or referencing resources that haven't been created yet.
## Examples
Handling missing resources:
```python theme={null}
try:
model = client.get_model("wrong-model-id")
except NotFound as e:
print(f"Model not found: {e.message}")
# Create the model or check the correct ID
```
Checking if a resource exists:
```python theme={null}
try:
project = client.get_project("my-project")
print("Project exists")
except NotFound:
print("Project doesn't exist, creating it...")
project = client.create_project(name="my-project")
```
Common NotFound scenarios:
```python theme={null}
- Accessing models, projects, or datasets with wrong IDs
- Referencing deleted or expired resources
- Typos in resource names or identifiers
```
## code
## reason
# Priority
Source: https://docs.fiddler.ai/sdk-api/python-client/priority
Alert priority levels for notification routing and escalation.
Alert priority levels for notification routing and escalation.
Determines the urgency and routing behavior of alert notifications.
Higher priority alerts may trigger immediate notifications, phone calls,
or escalation to on-call personnel.
## HIGH
Critical issues requiring immediate attention.
Examples: Model completely down, severe data quality issues.
## MEDIUM
Important issues that should be addressed promptly.
Examples: Performance degradation, moderate drift.
## LOW
Minor issues for awareness and trend monitoring.
Examples: Small drift increases, non-critical feature changes.
### Example
```python theme={null}
# High priority for production model failures
critical_alert = AlertRule(
name="Model Down",
priority=Priority.HIGH
)
# Medium priority for performance monitoring
perf_alert = AlertRule(
name="Accuracy Drop",
priority=Priority.MEDIUM
)
```
## HIGH *= 'HIGH'*
## MEDIUM *= 'MEDIUM'*
## LOW *= 'LOW'*
# Project
Source: https://docs.fiddler.ai/sdk-api/python-client/project
Represents a project container for organizing ML models and resources.
Represents a project container for organizing ML models and resources.
A Project is the top-level organizational unit in Fiddler that groups related
machine learning models, datasets, and monitoring configurations. Projects provide
logical separation, access control, and resource management for ML monitoring workflows.
Key Features:
* **Model Organization**: Container for related ML models and their versions
* **Resource Isolation**: Separate namespaces prevent naming conflicts
* **Access Management**: Project-level permissions and access control
* **Monitoring Coordination**: Centralized monitoring and alerting configuration
* **Lifecycle Management**: Coordinated creation, updates, and deletion of resources
Project Lifecycle:
1. **Creation**: Create project with unique name within organization
2. **Model Addition**: Add models using Model.create() or Model.from\_data()
3. **Configuration**: Set up monitoring, alerts, and baseline comparisons
4. **Operations**: Publish data, monitor performance, manage alerts
5. **Maintenance**: Update configurations, add new model versions
6. **Cleanup**: Delete project when no longer needed (removes all contained resources)
## Example
```python theme={null}
# Create a new project for fraud detection models
project = Project(name="fraud-detection-2024")
project = project.create()
print(f"Created project: {project.name} (ID: {project.id})")
# Add a model to the project
model = Model.from_data(
source=training_df,
name="xgboost_v1",
project_id=project.id,
task=ModelTask.BINARY_CLASSIFICATION
)
model.create()
# List all models in the project
models = list(project.models)
print(f"Project contains {len(models)} models")
# Access project models
for model_compact in project.models:
full_model = model_compact.fetch()
print(f"Model: {full_model.name} (Task: {full_model.task})")
```
Projects are permanent containers - once created, the name cannot be changed.
Deleting a project removes all contained models, datasets, and configurations.
Consider the organizational structure carefully before creating projects.
Initialize a Project instance.
Creates a new Project object with the specified name. The project is not
created on the Fiddler platform until .create() is called.
## Parameters
Project name, must be unique within the organization.
Should follow slug-like naming conventions:
* Use lowercase letters, numbers, hyphens, and underscores
* Start with a letter or number
* Be descriptive of the project's purpose
* Examples: "fraud-detection", "churn\_prediction\_2024", "nlp-models"
## Example
```python theme={null}
# Create project instance for fraud detection
project = Project(name="fraud-detection-prod")
# Create project instance for experimentation
experiment_project = Project(name="ml-experiments-q1-2024")
# Create on platform
created_project = project.create()
print(f"Project created with ID: {created_project.id}")
```
The project exists only locally until .create() is called. Project names
cannot be changed after creation, so choose descriptive, permanent names.
Consider your organization's naming conventions and project structure.
## *classmethod* get()
Retrieve a project by its unique identifier.
Fetches a project from the Fiddler platform using its UUID. This is the most
direct way to retrieve a project when you know its ID.
### Parameters
The unique identifier (UUID) of the project to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The project instance with all metadata and configuration.
### Raises
* **NotFound** – If no project exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get project by UUID
project = Project.get(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved project: {project.name}")
print(f"Created: {project.created_at}")
# Access project models
model_count = len(list(project.models))
print(f"Project contains {model_count} models")
```
This method makes an API call to fetch the latest project state from the server.
The returned project instance reflects the current state in Fiddler.
## *classmethod* from\_name()
Retrieve a project by name.
Finds and returns a project using its name within the organization. This is useful
when you know the project name but not its UUID. Project names are unique within
an organization, making this a reliable lookup method.
### Parameters
The name of the project to retrieve. Project names are unique
within an organization and are case-sensitive.
### Returns
The project instance matching the specified name.
### Raises
* **NotFound** – If no project exists with the specified name in the organization.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Get project by name
project = Project.from_name(name="fraud-detection")
print(f"Found project: {project.name} (ID: {project.id})")
print(f"Created: {project.created_at}")
# Use project to list models
for model in project.models:
print(f"Model: {model.name} v{model.version}")
# Get project for specific environment
prod_project = Project.from_name(name="fraud-detection-prod")
staging_project = Project.from_name(name="fraud-detection-staging")
```
Project names are case-sensitive and must match exactly. Use this method
when you have a known project name from configuration or user input.
## *classmethod* list()
List all projects in the organization.
Retrieves all projects that the current user has access to within the organization.
Returns an iterator for memory efficiency when dealing with many projects.
### Yields
`Project` – Project instances for all accessible projects.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Returns
`Iterator[Project]`
### Example
```python theme={null}
# List all projects
for project in Project.list():
print(f"Project: {project.name}")
print(f" ID: {project.id}")
print(f" Created: {project.created_at}")
print(f" Models: {len(list(project.models))}")
# Convert to list for counting and filtering
projects = list(Project.list())
print(f"Total accessible projects: {len(projects)}")
# Find projects by name pattern
prod_projects = [
p for p in Project.list()
if "prod" in p.name.lower()
]
print(f"Production projects: {len(prod_projects)}")
# Get project summaries
for project in Project.list():
model_count = len(list(project.models))
print(f"{project.name}: {model_count} models")
```
This method returns an iterator for memory efficiency. Convert to a list
with list(Project.list()) if you need to iterate multiple times or get
the total count. The iterator fetches projects lazily from the API.
## create()
Create the project on the Fiddler platform.
Persists this project instance to the Fiddler platform, making it available
for adding models, configuring monitoring, and other operations. The project
must have a unique name within the organization.
### Returns
This project instance, updated with server-assigned fields like
ID, creation timestamp, and other metadata.
### Raises
* **Conflict** – If a project with the same name already exists in the organization.
* **ValidationError** – If the project configuration is invalid (e.g., invalid name format).
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Create a new project
project = Project(name="customer-churn-analysis")
created_project = project.create()
print(f"Created project with ID: {created_project.id}")
print(f"Created at: {created_project.created_at}")
# Project is now available for adding models
assert created_project.id is not None
# Add a model to the newly created project
model = Model.from_data(
source=training_data,
name="churn_model_v1",
project_id=created_project.id
)
model.create()
```
After successful creation, the project instance is updated in-place with
server-assigned metadata. The same instance can be used for subsequent
operations without needing to fetch it again.
## *classmethod* get\_or\_create()
Get an existing project by name or create a new one if it doesn't exist.
This is a convenience method that attempts to retrieve a project by name,
and if not found, creates a new project with that name. Useful for idempotent
project setup in automation scripts and deployment pipelines.
### Parameters
The name of the project to retrieve or create. Must follow
project naming conventions (slug-like format).
### Returns
Either the existing project with the specified name,
or a newly created project if none existed.
### Raises
* **ValidationError** – If the project name format is invalid.
* **ApiError** – If there's an error communicating with the Fiddler API.
### Example
```python theme={null}
# Safe project setup - get existing or create new
project = Project.get_or_create(name="fraud-detection-prod")
print(f"Using project: {project.name} (ID: {project.id})")
# Idempotent setup in deployment scripts
project = Project.get_or_create(name="ml-pipeline-staging")
# Add models safely - project guaranteed to exist
model = Model.from_data(
source=data,
name="model_v1",
project_id=project.id
)
model.create()
# Use in configuration management
environments = ["dev", "staging", "prod"]
projects = {}
for env in environments:
projects[env] = Project.get_or_create(name=f"fraud-detection-{env}")
```
This method is idempotent - calling it multiple times with the same name
will return the same project. It logs when creating a new project for
visibility in automation scenarios.
## delete()
Delete the project and all its contained resources.
Permanently removes the project from the Fiddler platform, including all
associated models, datasets, baselines, alerts, and monitoring configurations.
This operation cannot be undone.
### Raises
* **NotFound** – If the project doesn't exist or has already been deleted.
* **ApiError** – If there's an error communicating with the Fiddler API.
* **PermissionError** – If the user doesn't have permission to delete the project.
This operation is irreversible and will delete ALL resources associated
with the project, including:
* All models and their versions
* All datasets and published data
* All baselines and monitoring configurations
* All alert rules and notification settings
* All access permissions and project settings
### Example
```python theme={null}
# Delete a test or temporary project
test_project = Project.from_name(name="temp-experiment")
test_project.delete()
print(f"Deleted project: {test_project.name}")
# Safe deletion with confirmation
project = Project.from_name(name="old-project")
model_count = len(list(project.models))
if model_count == 0:
project.delete()
print("Empty project deleted")
else:
print(f"Project has {model_count} models - deletion cancelled")
# Cleanup projects by pattern
for project in Project.list():
if project.name.startswith("temp-"):
print(f"Deleting temporary project: {project.name}")
project.delete()
```
Requires Fiddler platform version 25.2.0 or higher. Consider exporting
important data or configurations before deletion. There is no recovery
mechanism once a project is deleted.
## *property* models
Fetch all the models of this project.
### Yields
[`ModelCompact`](/sdk-api/python-client/model-compact) – Lightweight model objects for this project.
# ProjectCompact
Source: https://docs.fiddler.ai/sdk-api/python-client/project-compact
Lightweight project representation for listing and basic operations.
Lightweight project representation for listing and basic operations.
A minimal project object containing only essential identifiers. Used by
various listing operations to efficiently return project information without
fetching full project details and associated resources.
This class provides a memory-efficient way to work with project references
when you don't need the full project functionality but want to access
basic information or fetch the complete project when needed.
## Example
```python theme={null}
# From project references in other entities
model = Model.get(id_="model-uuid")
project_compact = model.project # Returns ProjectCompact
# Access basic info
print(f"Project: {project_compact.name}")
print(f"ID: {project_compact.id}")
# Fetch full details when needed
full_project = project_compact.fetch()
print(f"Created: {full_project.created_at}")
print(f"Models: {len(list(full_project.models))}")
```
ProjectCompact objects are typically returned by other entities that
reference projects. Use .fetch() to get the complete Project instance
when you need full functionality like listing models or project operations.
## id
## name
## fetch()
Fetch project instance.
### Returns
Complete project instance with all details.
# RowDataSource
Source: https://docs.fiddler.ai/sdk-api/python-client/row-data-source
Data source for explainability analysis using a single row of data.
Data source for explainability analysis using a single row of data.
RowDataSource allows you to perform explainability analysis on a specific
data row by providing the row data directly. This is useful when you want
to explain a particular prediction or analyze feature importance for a
specific instance without referencing stored data.
This data source type is ideal for real-time explanations, ad-hoc analysis,
or when you have specific data points that you want to analyze independently
of your stored datasets.
## Examples
Creating a row data source for a loan application:
```python theme={null}
row_source = RowDataSource(
row={
"age": 35,
"income": 75000,
"credit_score": 720,
"employment_years": 8,
"loan_amount": 250000
}
)
```
Creating a row data source for image classification:
```python theme={null}
image_row_source = RowDataSource(
row={
"image_features": [0.1, 0.5, 0.3, ...],
"metadata": "product_image_001.jpg",
"category": "electronics"
}
)
```
## source\_type
## row
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# Segment
Source: https://docs.fiddler.ai/sdk-api/python-client/segment
Data segment for targeted monitoring and cohort analysis.
Data segment for targeted monitoring and cohort analysis.
Segment defines subsets of model data based on specific criteria using SQL-like
expressions. Segments enable cohort analysis, A/B testing evaluation, targeted
monitoring of specific populations, and fairness analysis across different groups.
## Example
```python theme={null}
# High-value customer segment
high_value_segment = Segment(
name="high_value_customers",
model_id=model.id,
definition="customer_lifetime_value > 10000 and account_age_days > 365",
description="Customers with high LTV and established accounts"
).create()
# Geographic segment
west_coast_segment = Segment(
name="west_coast_users",
model_id=model.id,
definition="state == 'CA' or state == 'OR' or state == 'WA'",
description="Users from West Coast states"
).create()
# Risk-based segment
high_risk_segment = Segment(
name="high_risk_applications",
model_id=model.id,
definition="credit_score < 600 or debt_to_income > 0.4",
description="Loan applications with elevated risk factors"
).create()
# Age-based demographic segment
young_adults_segment = Segment(
name="young_adults",
model_id=model.id,
definition="age >= 18 and age <= 35",
description="Young adult demographic (18-35 years)"
).create()
# Use segment in alert rule for targeted monitoring
segment_alert = AlertRule(
name="high_value_drift_alert",
model_id=model.id,
metric_id="drift_score",
priority=Priority.HIGH,
compare_to=CompareTo.BASELINE,
condition=AlertCondition.GT,
bin_size=BinSize.HOUR,
critical_threshold=0.7,
baseline_id=baseline.id,
segment_id=high_value_segment.id
).create()
```
Segments are evaluated during data processing and can be used with any
monitoring metric. Complex segment definitions may impact performance,
so optimize for efficiency. Segments are particularly useful for fairness
monitoring and business-critical cohort analysis.
## create()
Create a new Segment on the Fiddler platform.
Registers this Segment with the Fiddler platform. The expression
must have a name, model\_id, and definition specified before calling create().
### Returns
The same Segment instance with updated server-side attributes
(id, created\_at, etc.).
### Raises
* **ApiError** – If there's an error communicating with the Fiddler API.
* **Conflict** – If a Segment with the same name already exists for this model.
## delete()
Delete this Segment from the Fiddler platform.
Permanently removes the Segment. This action cannot be undone.
Any alert rules or monitors using this Segment must be deleted first.
### Raises
* **NotFound** – If the Segment no longer exists.
* **ApiError** – If there's an error communicating with the Fiddler API.
* **Conflict** – If the Segment is still being used by alert rules or monitors.
## *classmethod* from\_name()
Retrieve a Segment by name and model.
Fetches a Segment from the Fiddler platform using its name
and associated model ID.
### Parameters
The name of the Segment to retrieve.
UUID or string identifier of the associated Model.
### Returns
The Segment instance for the provided parameters.
### Raises
* **NotFound** – If no Segment exists with the specified name and model.
* **ApiError** – If there's an error communicating with the Fiddler API.
## *classmethod* get()
Retrieve a Segment by its unique identifier.
Fetches a Segment from the Fiddler platform using its UUID.
### Parameters
The unique identifier (UUID) of the Segment to retrieve.
Can be provided as a UUID object or string representation.
### Returns
The Segment instance with all its configuration and metadata.
### Raises
* **NotFound** – If no Segment exists with the specified ID.
* **ApiError** – If there's an error communicating with the Fiddler API.
## *classmethod* get\_organization\_id()
Get the organization UUID from the global connection.
### Returns
Unique identifier of the organization associated with the current connection.
## *classmethod* get\_organization\_name()
Get the organization name from the global connection.
### Returns
Name of the organization associated with the current connection.
## *classmethod* list()
List all Segment instances for a model.
Retrieves all Segment instances associated with a specific model.
### Parameters
UUID or string identifier of the Model.
### Yields
Segment instances for each Segment in the model.
### Raises
**ApiError** – If there's an error communicating with the Fiddler API.
### Returns
`Iterator[Segment]`
## **init**()
Construct a segment instance.
# TextEmbedding
Source: https://docs.fiddler.ai/sdk-api/python-client/text-embedding
Represents custom features derived from text embeddings with TF-IDF analysis.
Represents custom features derived from text embeddings with TF-IDF analysis.
TextEmbedding extends VectorFeature to handle text-based embeddings with additional
text-specific analysis capabilities. It combines vector clustering with TF-IDF
analysis to provide both semantic clustering and keyword extraction for text data.
The feature type is automatically set to CustomFeatureType.FROM\_TEXT\_EMBEDDING
and uses clustering combined with TF-IDF summarization for drift computation.
## Examples
```python theme={null}
# Creating a text embedding feature for review analysis:
text_feature = TextEmbedding(
name="review_sentiment_clusters",
column="review_embedding",
source_column="review_text",
n_clusters=8,
n_tags=20
)
# Creating a feature for document classification:
doc_feature = TextEmbedding(
name="document_topic_clusters",
column="doc_embedding",
source_column="document_content",
n_clusters=12,
n_tags=15
)
```
## type
## source\_column
## n\_tags
## tf\_idf
## *classmethod* validate\_n\_tags()
### Returns
`int`
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# Unsupported
Source: https://docs.fiddler.ai/sdk-api/python-client/unsupported
Raised when an unsupported operation is attempted.
Raised when an unsupported operation is attempted.
This exception occurs when users try to perform operations that are not
supported by the current Fiddler platform configuration, client version,
or specific model/dataset setup. This can include feature limitations,
deprecated functionality, or operations not available for certain model types.
Common scenarios include attempting to use features not available in the
current platform edition, trying deprecated API methods, or performing
operations incompatible with the model's configuration.
## Examples
Handling unsupported operations:
```python theme={null}
try:
model.enable_advanced_feature()
except Unsupported as e:
print(f"Feature not available: {e.message}")
# Use alternative approach or upgrade platform
```
Common unsupported scenarios:
```python theme={null}
- Using enterprise features on basic plans
- Attempting operations on incompatible model types
- Calling deprecated API methods
```
## message
# VectorFeature
Source: https://docs.fiddler.ai/sdk-api/python-client/vector-feature
Represents custom features derived from a single vector column using clustering analysis.
Represents custom features derived from a single vector column using clustering analysis.
VectorFeature processes high-dimensional vector data (like embeddings or feature
vectors) by applying k-means clustering to create discrete clusters that can be
monitored for distribution changes over time. This is particularly useful for
monitoring embedding drift in high-dimensional spaces.
The feature type is automatically set to CustomFeatureType.FROM\_VECTOR and creates
meaningful groupings from vector data for drift detection and anomaly identification.
## source\_column
Optional original column if this feature is derived from an embedding
### Examples
Creating a feature from a general embedding column:
```python theme={null}
vector_feature = VectorFeature(
name="embedding_clusters",
column="user_embedding",
n_clusters=10
)
```
Creating a feature from model hidden states:
```python theme={null}
hidden_feature = VectorFeature(
name="hidden_state_clusters",
column="model_hidden_layer",
n_clusters=15,
source_column="input_features"
)
```
## type
## n\_clusters
## centroids
## column
## *classmethod* validate\_n\_clusters()
### Returns
`int`
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
# Webhook
Source: https://docs.fiddler.ai/sdk-api/python-client/webhook
Webhook for integrating external notification systems with Fiddler alerts.
Webhook for integrating external notification systems with Fiddler alerts.
A Webhook represents an integration endpoint for delivering alert notifications
to external systems like Slack, Microsoft Teams, or custom HTTP endpoints.
Webhooks enable real-time alert delivery to team communication tools.
## name
Human-readable name for the webhook. Should be descriptive
and indicate the target system or team.
## url
The webhook endpoint URL provided by the external system.
This is the destination where alert notifications will be sent.
## provider
The webhook provider type (`WebhookProvider`).
Determines message formatting and delivery method.
### Example
```python theme={null}
# Create Slack webhook for critical alerts
slack_webhook = Webhook(
name="critical-alerts-slack",
url="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
provider=WebhookProvider.SLACK
).create()
# Create Microsoft Teams webhook for team notifications
teams_webhook = Webhook(
name="ml-team-notifications",
url="https://outlook.office.com/webhook/xxxxx/IncomingWebhook/xxxxx",
provider=WebhookProvider.MS_TEAMS
).create()
# List all webhooks
webhooks = list(Webhook.list())
for webhook in webhooks:
print(f"Webhook: {webhook.name} ({webhook.provider})")
# Update webhook URL
webhook = Webhook.from_name("critical-alerts-slack")
webhook.url = "https://hooks.slack.com/services/NEW/WEBHOOK/URL"
webhook.update()
```
Webhook URLs are sensitive credentials that should be kept secure.
The provider type determines how messages are formatted and delivered.
Test webhooks thoroughly before using them in production alert rules.
Initialize a Webhook instance.
Creates a webhook configuration for integrating external notification systems
with Fiddler alert notifications. The webhook defines where and how alert
messages should be delivered to external platforms.
## Parameters
Human-readable name for the webhook. Should be descriptive and
indicate the target system, team, or purpose (e.g., "ml-team-slack",
"critical-alerts-teams").
The webhook endpoint URL provided by the external system. This is
the destination where Fiddler will send HTTP POST requests with
alert notification payloads.
The webhook provider type (SLACK, MS\_TEAMS, or GENERIC).
Determines message formatting, payload structure, and delivery
method for optimal integration with the target platform.
## Example
```python theme={null}
# Slack webhook for ML team alerts
slack_webhook = Webhook(
name="ml-team-slack-alerts",
url="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
provider=WebhookProvider.SLACK
)
# Microsoft Teams webhook for data quality alerts
teams_webhook = Webhook(
name="data-quality-teams",
url="https://outlook.office.com/webhook/xxxxx/IncomingWebhook/xxxxx",
provider=WebhookProvider.MS_TEAMS
)
# Generic webhook for custom integrations
custom_webhook = Webhook(
name="custom-monitoring-system",
url="https://monitoring.company.com/webhooks/fiddler",
provider=WebhookProvider.GENERIC
)
```
After initialization, call create() to register the webhook with the
Fiddler platform. Webhook URLs are sensitive and should be kept secure.
## *classmethod* get()
Get the webhook instance using webhook id
* **Params uuid:**
UUID belongs to the Webhook
### Returns
Webhook object
## *classmethod* from\_name()
Get the webhook instance using webhook name
### Returns
`Webhook`
## *classmethod* list()
Get a list of all webhooks in the organization
### Returns
`Iterator[Webhook]`
## create()
Create a new webhook
* **Params name:**
name of webhook
* **Params url:**
webhook url
* **Params provider:**
Either 'SLACK' or 'MS\_TEAMS'
### Returns
Created Webhook object.
## update()
Update an existing webhook.
## delete()
Delete an existing webhook.
# WindowBinSize
Source: https://docs.fiddler.ai/sdk-api/python-client/window-bin-size
Time granularities for rolling baseline window aggregation.
Time granularities for rolling baseline window aggregation.
Window bin sizes define the time intervals used for rolling baseline calculations.
They determine how far back in time the rolling baseline looks and at what
granularity the data is aggregated. This parameter is only used with rolling
baselines and works in conjunction with offset\_delta.
Rolling Baseline Mechanics:
* Window bin size sets the granularity of the sliding window
* offset\_delta determines how many bins to look back
* Together they define the rolling window: offset\_delta × window\_bin\_size
* Example: WEEK + offset\_delta=4 creates a 4-week rolling window
Granularity Trade-offs:
* **Finer granularity (HOUR)**: More responsive to recent changes, higher sensitivity
* **Coarser granularity (MONTH)**: More stable patterns, reduced noise
* **Medium granularity (DAY/WEEK)**: Balanced responsiveness and stability
Selection Guidelines:
* HOUR: High-frequency models with rapid data changes
* DAY: Standard operational monitoring for most models
* WEEK: Weekly business cycles, batch processing patterns
* MONTH: Long-term trends, seasonal patterns, strategic monitoring
## Example
```python theme={null}
# Daily rolling baseline looking back 30 days
daily_rolling = fdl.Baseline(
name="daily_rolling_baseline",
model_id=model.id,
environment=fdl.EnvType.PRODUCTION,
type_=fdl.BaselineType.ROLLING,
window_bin_size=fdl.WindowBinSize.DAY,
offset_delta=30
).create()
# Weekly rolling baseline looking back 8 weeks
weekly_rolling = fdl.Baseline(
name="weekly_rolling_baseline",
model_id=model.id,
environment=fdl.EnvType.PRODUCTION,
type_=fdl.BaselineType.ROLLING,
window_bin_size=fdl.WindowBinSize.WEEK,
offset_delta=8
).create()
```
Data Volume Considerations:
```python theme={null}
- Ensure sufficient data volume within each bin for statistical reliability
- Finer granularities require higher prediction frequencies
- Consider your model's prediction patterns when selecting bin size
- Balance between responsiveness and statistical stability
```
## HOUR
Hourly time bins for high-frequency rolling baselines
## DAY
Daily time bins for standard rolling baseline monitoring
## WEEK
Weekly time bins for trend analysis and batch patterns
## MONTH
Monthly time bins for long-term seasonal pattern detection
## HOUR *= 'Hour'*
## DAY *= 'Day'*
## WEEK *= 'Week'*
## MONTH *= 'Month'*
# XaiParams
Source: https://docs.fiddler.ai/sdk-api/python-client/xai-params
Configuration parameters for explainability (XAI) analysis in Fiddler models.
Configuration parameters for explainability (XAI) analysis in Fiddler models.
XaiParams defines the configuration for explainability analysis, including
custom explanation methods and default explanation strategies. These parameters
control how feature importance, SHAP values, and other explainability metrics
are computed for your model.
This configuration is essential for models that require custom explanation
logic or when you want to override the default explanation methods provided
by Fiddler's built-in explainability features.
## Examples
Creating XAI parameters with custom methods:
```python theme={null}
xai_params = XaiParams(
custom_explain_methods=["custom_shap", "custom_lime", "domain_specific"],
default_explain_method="custom_shap"
)
```
Creating XAI parameters with only default method:
```python theme={null}
simple_xai_params = XaiParams(
default_explain_method="integrated_gradients"
)
```
Creating XAI parameters for multiple explanation strategies:
```python theme={null}
multi_xai_params = XaiParams(
custom_explain_methods=[
"business_rule_explainer",
"feature_interaction_explainer",
"counterfactual_explainer"
],
default_explain_method="business_rule_explainer"
)
```
Creating empty XAI parameters (use Fiddler defaults):
```python theme={null}
default_xai_params = XaiParams()
```
## custom\_explain\_methods
User-defined explain\_custom method of the model object defined in package.py
## model\_config
Configuration for the model, should be a dictionary conforming to \[ConfigDict]\[pydantic.config.ConfigDict].
## default\_explain\_method
Default explanation method
# Creates Alert Rules
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/create-alert
POST /v3/alert-rules
Creates Alert Rules
# createNotificationForAlertRule
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/create-notification-for-alert-rule
POST /v3/alert-rules/{alert_id}/notification
Create Notification for an Alert Rule
# Deletes an Alert Rule
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/delete-alert-rule
DELETE /v3/alert-rules/{alert_id}
Deletes an Alert Rule
# List latest alert record for each time bucket during specified time_bucket_start and time_bucket_end for the given alert rule
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/get-alert-record-history
GET /v3/alert-rules/{alert_id}/record-history
List latest alert record for each time bucket during specified time_bucket_start and time_bucket_end for the given alert rule
# Returns Alert Rule for the given id
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/get-alert-rule
GET /v3/alert-rules/{alert_id}
Returns Alert Rule for the given id
# List of all alert records during specified time_bucket_start and time_bucket_end for the given alert rule
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/get-alert-rule-records
GET /v3/alert-rules/{alert_id}/records
List of all alert records during specified time_bucket_start and time_bucket_end for the given alert rule
# Lists all alert rules configured for a model.
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/get-alert-rules
GET /v3/alert-rules
Lists all alert rules configured for a model.
# List of all alert rule summary in the given time_bucket_start and time_bucket_end
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/get-alert-rules-summary
GET /v3/alert-rules/summary
List of all alert rule summary in the given time_bucket_start and time_bucket_end
# Get thresholds that were used to evaluate the given time bin using the given alert rule.
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/get-alert-thresholds-for-time-bin
GET /v3/alert-rules/{alert_id}/get-thresholds
Get thresholds that were used to evaluate the given time bin using the given alert rule.
# Returns notification for the given Alert Rule id
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/get-notification-for-alert-rule
GET /v3/alert-rules/{alert_id}/notification
Returns notification for the given Alert Rule id
# Alert Rules REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/index
Discover Fiddler’s Alert Rules REST API guide. Learn to list alerts by time, create alert rules, send notifications for alerts, and more.
`POST /v3/alert-rules/{alert_id}/notification`
`POST /v3/alert-rules`
`DELETE /v3/alert-rules/{alert_id}`
`GET /v3/alert-rules/{alert_id}/get-thresholds`
`GET /v3/alert-rules/{alert_id}/record-history`
`GET /v3/alert-rules/{alert_id}/records`
`GET /v3/alert-rules/summary`
`GET /v3/alert-rules`
`GET /v3/alert-rules/{alert_id}`
`GET /v3/alert-rules/{alert_id}/notification`
`POST /v3/alert-rules/{alert_id}/test-notification`
`PATCH /v3/alert-rules/{alert_id}/notification`
`PATCH /v3/alert-rules/{alert_id}`
# Send a test notification for the given Alert Rule
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/test-notification-for-alert-rule
POST /v3/alert-rules/{alert_id}/test-notification
Send a test notification for the given Alert Rule to verify notification configuration
# Update by id
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/update-alert-rule
PATCH /v3/alert-rules/{alert_id}
Update Alert Rule by id
# Update by id
Source: https://docs.fiddler.ai/sdk-api/rest-api/alert-rules/update-notification-for-alert-rule
PATCH /v3/alert-rules/{alert_id}/notification
Update Notification by Alert Rule id
# Create New Application
Source: https://docs.fiddler.ai/sdk-api/rest-api/applications/create-application
POST /v3/applications
Create New Application
# Delete Application
Source: https://docs.fiddler.ai/sdk-api/rest-api/applications/delete-application
DELETE /v3/applications/{application_id}
Delete application and all associated resources asynchronously
# Get Application
Source: https://docs.fiddler.ai/sdk-api/rest-api/applications/get-application
GET /v3/applications/{application_id}
Get application by ID
# Get Application Metrics Metadata
Source: https://docs.fiddler.ai/sdk-api/rest-api/applications/get-application-metrics
GET /v3/applications/{application_id}/metrics
Retrieves available metrics and aggregations for a GenAI application
# List applications
Source: https://docs.fiddler.ai/sdk-api/rest-api/applications/get-applications
GET /v3/applications
Get list of applications in a project for provided query and filters
# Applications
Source: https://docs.fiddler.ai/sdk-api/rest-api/applications/index
REST API endpoints for managing applications in Fiddler Platform.
`POST /v3/applications`
`DELETE /v3/applications/{application_id}`
`GET /v3/applications/{application_id}`
`GET /v3/applications/{application_id}/metrics`
`GET /v3/applications`
`GET /v3/applications/{application_id}/catalog`
`GET /v3/applications/{application_id}/catalog/values`
# List catalog entries
Source: https://docs.fiddler.ai/sdk-api/rest-api/applications/list-catalog
GET /v3/applications/{application_id}/catalog
Paginated, searchable list of entity names for an application. Returns entity names with semantic concept mappings, data types, first/last seen timestamps, and observation counts.
Use query parameters to filter by entity type, data type, or semantic concept name. The `search` parameter performs a case-insensitive substring match on entity names.
# List distinct values for a catalog entity
Source: https://docs.fiddler.ai/sdk-api/rest-api/applications/list-catalog-values
GET /v3/applications/{application_id}/catalog/values
Paginated, searchable list of distinct values for a specific entity within an application. Useful for populating filter dropdowns in the Trace Explorer UI.
Both `entity_type` and `name` are required — values are always scoped to a specific entity.
# List attributes
Source: https://docs.fiddler.ai/sdk-api/rest-api/attributes/get-attributes
GET /v3/attributes
Get list of attributes
# Attributes
Source: https://docs.fiddler.ai/sdk-api/rest-api/attributes/index
REST API endpoints for managing attributes in Fiddler Platform.
`GET /v3/attributes`
# add baseline to a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/baseline/add-baseline
POST /v3/baselines
Adds a baseline to a model
# Delete baseline from a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/baseline/delete-baseline
DELETE /v3/baselines/{baseline_id}
Delete baseline from a model
# Get baseline details
Source: https://docs.fiddler.ai/sdk-api/rest-api/baseline/get-baseline
GET /v3/baselines/{baseline_id}
Get baseline details
# List of baselines based on user permissions and filters
Source: https://docs.fiddler.ai/sdk-api/rest-api/baseline/get-baselines
GET /v3/baselines
List of baselines based on user permissions and filters
# List of baselines of a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/baseline/get-model-baselines
GET /v3/models/{model_id}/baselines
List of baselines of a model
# Baselines REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/baseline/index
Discover Fiddler’s Baseline feature. Learn how to add, retrieve, delete, and list baselines for a model based on user permissions and filters.
`POST /v3/baselines`
`DELETE /v3/baselines/{baseline_id}`
`GET /v3/baselines/{baseline_id}`
`GET /v3/baselines`
`GET /v3/models/{model_id}/baselines`
# Create new custom metric
Source: https://docs.fiddler.ai/sdk-api/rest-api/custom-metrics/create-custom-metric
POST /v3/custom-metrics
Create new custom metric
# Delete custom metric by uuid
Source: https://docs.fiddler.ai/sdk-api/rest-api/custom-metrics/delete-custom-metric
DELETE /v3/custom-metrics/{uuid}
Delete custom metric by uuid
# Detail info of a custom metric
Source: https://docs.fiddler.ai/sdk-api/rest-api/custom-metrics/get-custom-metric
GET /v3/custom-metrics/{uuid}
Detail info of a custom metric
# List all custom metrics
Source: https://docs.fiddler.ai/sdk-api/rest-api/custom-metrics/get-custom-metrics
GET /v3/custom-metrics
List all custom metrics
# List all custom metrics for a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/custom-metrics/get-model-custom-metrics
GET /v3/models/{model_id}/custom-metrics
List all custom metrics for a model
# Custom Metrics REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/custom-metrics/index
Discover Fiddler’s Custom Metric feature. Learn how to add, retrieve, and delete custom metrics from you Fiddler models.
`POST /v3/custom-metrics`
`DELETE /v3/custom-metrics/{uuid}`
`GET /v3/custom-metrics/{uuid}`
`GET /v3/custom-metrics`
`GET /v3/models/{model_id}/custom-metrics`
# Get environment of a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/environment/get-environment
GET /v3/environments/{env_id}
Retrieve details of a specific environment associated with a model.
# List Environments
Source: https://docs.fiddler.ai/sdk-api/rest-api/environment/get-environments
GET /v3/environments
Retrieve a list of environments from authorized projects.
# List of environments of a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/environment/get-model-environments
GET /v3/models/{model_id}/environments
Retrieve a list of environments associated with a specific model.
# Environments REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/environment/index
Learn about Fiddler’s environments API . Learn how to list pre-production and production environment datasets.
`GET /v3/environments/{env_id}`
`GET /v3/environments`
`GET /v3/models/{model_id}/environments`
# Add new Evals dataset items
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/add-evals-dataset-items
POST /v3/evals/datasets/{dataset_id}/items
# Bulk add spans to dataset
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/bulk-add-spans-to-dataset
POST /v3/evals/datasets/{dataset_id}/items/from-spans
Accept a list of explicit span references and a field mapping, fetch span attributes from ClickHouse, apply the mapping server-side, and write one dataset item per span. All-or-nothing — if any span cannot be resolved, no items are written and a 422 is returned with the list of unresolvable spans.
`start_time` and `end_time` are required to bound the ClickHouse lookup to the relevant monthly partitions. Without them, the query would scan every partition for the application. The FE should pass the datagrid's current time range; MCP agents should pass the time range of the spans being exported.
Mapping values are span attribute keys as returned by `POST /v3/spans/fields` (e.g. `"system.gen_ai.usage.input_tokens"`, `"user.cost_usd"`). Each key maps to the corresponding value from the span's `span_attributes` dict.
# Create an Evals dataset
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/create-evals-dataset
POST /v3/evals/datasets
# Delete a dataset
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/delete-evals-dataset
DELETE /v3/evals/datasets/{dataset_id}
# Get a dataset by id
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/get-evals-dataset-by-id
GET /v3/evals/datasets/{dataset_id}
# List dataset items
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/get-evals-dataset-items
GET /v3/evals/datasets/{dataset_id}/items
Get list of items for a dataset
# Get dataset schema
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/get-evals-dataset-schema
GET /v3/evals/datasets/{dataset_id}/schema
Discover what field keys exist in each JSON bucket (inputs, expected_outputs, metadata, extras) across this dataset's items, along with a coverage count and inferred data type for each key. Useful for pre-populating a field mapper UI or for an MCP agent to understand what data is available before constructing a mapping. Returns empty objects for all buckets if the dataset has no items.
# Evals
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/index
REST API endpoints for managing evals in Fiddler Platform.
`POST /v3/evals/datasets/{dataset_id}/items`
`POST /v3/evals/datasets/{dataset_id}/items/from-spans`
`POST /v3/evals/datasets`
`DELETE /v3/evals/datasets/{dataset_id}`
`GET /v3/evals/datasets/{dataset_id}`
`GET /v3/evals/datasets/{dataset_id}/schema`
`GET /v3/evals/datasets/{dataset_id}/items`
`GET /v3/evals/datasets`
`PATCH /v3/evals/datasets/{dataset_id}`
# List Evals datasets
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/list-evals-datasets
GET /v3/evals/datasets
Retrieve datasets with pagination, search, ordering, and filters.
# Update a dataset
Source: https://docs.fiddler.ai/sdk-api/rest-api/evals/update-evals-dataset
PATCH /v3/evals/datasets/{dataset_id}
# Evaluation
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluation/index
REST API endpoints for managing evaluation in Fiddler Platform.
`POST /v3/evals/score`
# Score inputs using an evaluator
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluation/score-inputs
POST /v3/evals/score
Score inputs using an evaluator
# Create New Evaluator Rule
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluator-rules/create-evaluator-rule
POST /v3/evaluator-rules
Create a new Evaluator Rule
# Delete Rule Evaluator for an application
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluator-rules/delete-rule-evaluator
DELETE /v3/evaluator-rules/{evaluator_rule_id}
Delete Rule Evaluator
# List evaluator backfills
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluator-rules/get-evaluator-backfills
GET /v3/genai-enrichment-backfills
Get list of evaluator backfill jobs for provided query and filters
# List Rule evaluators
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluator-rules/get-rule-evaluators
GET /v3/evaluator-rules
Get list of Rule evaluators setup for provided query and filters
# Evaluator Rules REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluator-rules/index
Discover Fiddler’s Evaluator Rules REST API guide. Learn to list and create evaluator rules, update or delete a rule, map input keys, and list evaluator backfills.
`POST /v3/evaluator-rules`
`DELETE /v3/evaluator-rules/{evaluator_rule_id}`
`POST /v3/evaluator-rules/map-input-keys`
`GET /v3/genai-enrichment-backfills`
`GET /v3/evaluator-rules`
`PATCH /v3/evaluator-rules/{evaluator_rule_id}`
# Get attribute names and input fields
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluator-rules/map-input-keys-rule-evaluator
POST /v3/evaluator-rules/map-input-keys
Get the possible attribute names detected in the application and the input fields with data type to fill out for the evaluator name
# Update Rule Evaluator
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluator-rules/update-rule-evaluator
PATCH /v3/evaluator-rules/{evaluator_rule_id}
Update the Rule evaluator
# Archive Evaluator
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluators/archive-evaluator
POST /v3/evaluators/{evaluator_id}/archive
Archive an evaluator. Sets the evaluator's status to ARCHIVED. Archived evaluators are excluded from active queries and cannot be used in new evaluator rules.
# Create New Evaluator
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluators/create-evaluator
POST /v3/evaluators
Create a new Evaluator
# Get Evaluator by ID
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluators/get-evaluator
GET /v3/evaluators/{evaluator_id}
Get details of a specific evaluator by its ID
# List evaluator config schemas
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluators/get-evaluator-config-schemas
GET /v3/evaluators/get-config-options
Get list of evaluator config schema for each enrichment type with data type
# List evaluators
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluators/get-evaluators
GET /v3/evaluators
Get list of evaluators for provided query and filters
# Evaluators REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluators/index
Discover Fiddler’s Evaluators REST API guide. Learn to list and create evaluators, get an evaluator by ID, update or archive evaluators, and fetch evaluator configuration options.
`POST /v3/evaluators/{evaluator_id}/archive`
`POST /v3/evaluators`
`GET /v3/evaluators/{evaluator_id}`
`GET /v3/evaluators/get-config-options`
`GET /v3/evaluators`
`PATCH /v3/evaluators/{evaluator_id}`
# Update Evaluator
Source: https://docs.fiddler.ai/sdk-api/rest-api/evaluators/update-evaluator
PATCH /v3/evaluators/{evaluator_id}
Update an evaluator's name and/or configuration. Creates a new version of the evaluator and archives the previous version. At least one field (name or enrichment_config) must be provided.
# Delete events in Fiddler Platform
Source: https://docs.fiddler.ai/sdk-api/rest-api/events/delete-events
DELETE /v3/events
Delete events in Fiddler Platform
# Events
Source: https://docs.fiddler.ai/sdk-api/rest-api/events/index
REST API endpoints for managing events in Fiddler Platform.
`DELETE /v3/events`
`POST /v3/events`
`PATCH /v3/events`
# Publish events to Fiddler Platform
Source: https://docs.fiddler.ai/sdk-api/rest-api/events/publish-events
POST /v3/events
Publish events to Fiddler Platform
# Publish update events to Fiddler Platform
Source: https://docs.fiddler.ai/sdk-api/rest-api/events/publish-events-update
PATCH /v3/events
Publish update events to Fiddler Platform
# Create New Experiment
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/create-experiment
POST /v3/evals/experiments
Create a new experiment
# Delete Experiment
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/delete-experiment
DELETE /v3/evals/experiments/{experiment_id}
Delete an experiment
# getEvalScores
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/get-eval-scores
GET /v3/evals/experiments/{experiment_id}/scores
Get list of evaluation scores for an experiment
# Get Experiment
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/get-experiment
GET /v3/evals/experiments/{experiment_id}
Get experiment by ID
# Get Experiment Metrics
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/get-experiment-metrics
GET /v3/evals/experiments/{experiment_id}/metrics
Get per-evaluator aggregate metrics for an experiment. Auto-detects each evaluator's chart type based on score cardinality: numeric_range (>10 distinct numeric values, histogram), numeric_categorical (<=10 distinct numeric values, distribution bars), or categorical (label-only scores, distribution with ratios).
# Get Experiment Row Metrics
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/get-experiment-metrics-rows
GET /v3/evals/experiments/{experiment_id}/metrics/rows
Get top and bottom performing rows via percentile-based outlier detection. Numeric evaluators use P10/P90 thresholds; categorical evaluators flag labels that differ from the mode. Rows are ranked by how many evaluators flagged them as outliers.
# List experiment results
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/get-experiment-results
GET /v3/evals/experiments/{experiment_id}/results
Get a paginated list of results for an experiment. Each result contains the experiment item outputs, dataset inputs and expected outputs, and all associated evaluation scores (including the evaluator name).
**Filtering** (`filter` parameter — JSON-encoded QueryCondition DSL): - `experiment_item_id` — filter by one or more item IDs
(operators: `equal`, `not_equal`, `in`, `not_in`)
**Ordering** (`ordering` parameter): - `timestamp` / `-timestamp` — sort by item timestamp (default: `-timestamp`) - `created_at` / `-created_at` — sort by backend creation time - `scores.score_value` / `-scores.score_value` — sort by first score value - `scores.{score_name}` / `-scores.{score_name}` — sort by named score value
(e.g. `scores.accuracy`, `-scores.toxicity`)
# List experiments
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/get-experiments
GET /v3/evals/experiments
Get list of experiments for provided query and filters
# Experiments
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/index
REST API endpoints for managing experiments in Fiddler Platform.
`POST /v3/evals/experiments`
`DELETE /v3/evals/experiments/{experiment_id}`
`GET /v3/evals/experiments/{experiment_id}`
`GET /v3/evals/experiments/{experiment_id}/metrics`
`GET /v3/evals/experiments/{experiment_id}/metrics/rows`
`GET /v3/evals/experiments/{experiment_id}/scores`
`GET /v3/evals/experiments/{experiment_id}/results`
`GET /v3/evals/experiments`
`PATCH /v3/evals/experiments/{experiment_id}`
`POST /v3/evals/experiments/{experiment_id}/results`
`POST /v3/evals/experiments/{experiment_id}/items`
# Update Experiment
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/update-experiment
PATCH /v3/evals/experiments/{experiment_id}
Update an experiment
# Upload new experiment items
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/upload-experiment-items
POST /v3/evals/experiments/{experiment_id}/items
Upload new experiment items
# Upload experiment results
Source: https://docs.fiddler.ai/sdk-api/rest-api/experiments/upload-experiment-results
POST /v3/evals/experiments/{experiment_id}/results
Upload new experiment results, each with an item and its associated scores.
# Complete multi-part upload
Source: https://docs.fiddler.ai/sdk-api/rest-api/file-upload/complete-multi-part-upload
POST /v3/files/multipart-complete
Completes the multi-part upload process for a large file in Fiddler.
# Upload file in a single part
Source: https://docs.fiddler.ai/sdk-api/rest-api/file-upload/file-upload
POST /v3/files/upload
Uploads a file in a single part to Fiddler.
# File Upload REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/file-upload/index
Explore API integration with Fiddler’s platform. Learn about file uploads, from single-part to multi-part options, and optimize your workflow effortlessly.
`POST /v3/files/multipart-complete`
`POST /v3/files/multipart-init`
`POST /v3/files/multipart-upload`
`POST /v3/files/upload`
# Initiate multi-part upload
Source: https://docs.fiddler.ai/sdk-api/rest-api/file-upload/initiate-multi-part-upload
POST /v3/files/multipart-init
Initiates a multi-part upload process for a large file to Fiddler.
# Upload file a part of the file
Source: https://docs.fiddler.ai/sdk-api/rest-api/file-upload/upload-part
POST /v3/files/multipart-upload
Uploads a part of a large file to Fiddler as part of a multi-part upload process.
# FQL Expressions
Source: https://docs.fiddler.ai/sdk-api/rest-api/fql-expressions/index
REST API endpoints for managing fql expressions in Fiddler Platform.
`GET /v3/fql-expressions`
# List available FQL functions
Source: https://docs.fiddler.ai/sdk-api/rest-api/fql-expressions/list-fql-expressions
GET /v3/fql-expressions
Returns the list of FQL functions available for GenAI custom metrics, including function metadata, parameter definitions, and return types. The response is used by the frontend for autocomplete suggestions, signature hints, and documentation in the FQL editor.
Currently returns GenAI functions only (from the GenAI function registry plus the `attribute` construct). Extensible to ML context in the future via a query parameter.
# Create GenAI Alert Rule
Source: https://docs.fiddler.ai/sdk-api/rest-api/genai-alert-rules/create-gen-ai-alert-rule
POST /v3/genai-alert-rules
Creates a new GenAI Alert Rule
# Deletes a GenAI Alert Rule
Source: https://docs.fiddler.ai/sdk-api/rest-api/genai-alert-rules/delete-gen-ai-alert-rule
DELETE /v3/genai-alert-rules/{alert_rule_id}
Deletes a GenAI Alert Rule
# Get GenAI Alert Rule
Source: https://docs.fiddler.ai/sdk-api/rest-api/genai-alert-rules/get-gen-ai-alert-rule
GET /v3/genai-alert-rules/{alert_rule_id}
Retrieves a specific GenAI Alert Rule by its ID
# GenAI Alert Rules
Source: https://docs.fiddler.ai/sdk-api/rest-api/genai-alert-rules/index
REST API endpoints for managing genai alert rules in Fiddler Platform.
`POST /v3/genai-alert-rules`
`DELETE /v3/genai-alert-rules/{alert_rule_id}`
`GET /v3/genai-alert-rules/{alert_rule_id}`
`GET /v3/genai-alert-rules`
`POST /v3/genai-alert-rules/{alert_rule_id}/test-notification`
# List GenAI Alert Rules
Source: https://docs.fiddler.ai/sdk-api/rest-api/genai-alert-rules/list-gen-ai-alert-rules
GET /v3/genai-alert-rules
Lists all GenAI Alert Rules with pagination
# Send Test Notification
Source: https://docs.fiddler.ai/sdk-api/rest-api/genai-alert-rules/test-gen-ai-alert-rule-notification
POST /v3/genai-alert-rules/{alert_rule_id}/test-notification
Sends a test notification for a GenAI Alert Rule using the configured notification settings. The notification uses the actual alert email template with placeholder breach values so recipients can preview what a real alert notification will look like.
# REST API
Source: https://docs.fiddler.ai/sdk-api/rest-api/index
Reference for the Fiddler REST API. Every operation in the public OpenAPI specification is documented here, grouped by resource.
The Fiddler REST API is organized around REST: it has predictable, resource-oriented URLs, accepts and returns JSON-encoded payloads, and uses standard HTTP response codes, authentication, and verbs. Use it to ingest events, manage models, configure alerts, run explainability queries, and more.
All endpoints require Bearer token authentication. See [Authentication](/sdk-api/python-client/connection) for details on obtaining a token.
`/alert-rules` endpoints
`/applications` endpoints
`/attributes` endpoints
`/baseline` endpoints
`/custom-metrics` endpoints
`/environment` endpoints
`/evals` endpoints
`/evaluation` endpoints
`/evaluator-rules` endpoints
`/evaluators` endpoints
`/events` endpoints
`/experiments` endpoints
`/file-upload` endpoints
`/fql-expressions` endpoints
`/genai-alert-rules` endpoints
`/jobs` endpoints
`/llm-gateway` endpoints
`/model` endpoints
`/projects` endpoints
`/queries` endpoints
`/scores` endpoints
`/segments` endpoints
`/server-info` endpoints
`/sessions` endpoints
`/spans` endpoints
`/traces` endpoints
`/user-access-keys` endpoints
`/users` endpoints
## API Response types
Every Fiddler API response is wrapped in an envelope whose `kind` field identifies the response type: `NORMAL`, `PAGINATED`, or `ERROR`.
### Normal response
Returned by operations that respond with a single resource or result. The payload is carried in `data`.
```json theme={null}
{
"api_version": "3.0",
"kind": "NORMAL",
"data": {}
}
```
### Paginated response
Returned by list operations. The `data` object carries the current page in `items`, alongside pagination metadata.
```json theme={null}
{
"api_version": "3.0",
"kind": "PAGINATED",
"data": {
"page_size": 10,
"item_count": 10,
"total": 100,
"page_count": 10,
"page_index": 1,
"offset": 0,
"items": []
}
}
```
### Error response
Returned when a request fails. The `error` object carries an HTTP-aligned `code`, a human-readable `message`, and an `errors` array with per-issue detail.
```json theme={null}
{
"api_version": "3.0",
"kind": "ERROR",
"error": {
"code": 404,
"message": "The requested resource was not found.",
"errors": [
{
"reason": "ResourceNotFound",
"message": "Model 'abc123' was not found in this project.",
"help": ""
}
]
}
}
```
## Rate Limiting
Rate limits apply to the endpoint categories listed below. When a limit is exceeded, the API returns **`429 Too Many Requests`**. These are **default limits and are deployment-overridable** — your Fiddler administrator can tune them per environment, and rate limiting can be enabled or disabled per deployment.
### Default limits
| Endpoint category | Limit |
| ---------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
| Metadata **read** — `GET` (list/read) on projects, applications, evaluators, evaluator rules, and related metadata resources | 30 req/s, 1,000 req/min |
| Metadata **write** — `POST`/`PATCH`/`DELETE` on the same resources | 10 req/s, 300 req/min |
| Spans query — `POST /v3/spans/query` | 10 req/s, 120 req/min |
| ML event deletion — `DELETE /v3/events` | 30 req/day |
| GenAI bulk delete — `DELETE /v3/traces`, `DELETE /v3/sessions` | 5 req/s, 100 req/day |
### Response headers
Responses from rate-limited endpoints include the following headers:
| Header | Description |
| ----------------------- | ------------------------------------------------------------------------------- |
| `X-RateLimit-Limit` | The active rate-limit policy for the endpoint (for example, `30 per 1 second`). |
| `X-RateLimit-Remaining` | Requests remaining in the current window. |
| `X-RateLimit-Reset` | Unix timestamp (in seconds) at which the current window resets. |
| `Retry-After` | Seconds to wait before retrying. Sent only on a `429` response. |
### Per-token limits
Limits are enforced **per access token**. Each endpoint keeps its own independent counter, so exhausting the limit on one endpoint does not consume the budget of any other.
### Handling 429 responses
When a request returns `429 Too Many Requests`, wait at least the number of seconds indicated by the `Retry-After` header, then retry using **exponential backoff with jitter**. Each operation that enforces a limit also documents its `429 Too Many Requests` response on its API reference page (see the resource groups above).
# Get async job details for a job id
Source: https://docs.fiddler.ai/sdk-api/rest-api/jobs/get-job
GET /v3/jobs/{job_id}
Get async job details for a job id
# Get details of all background/async jobs
Source: https://docs.fiddler.ai/sdk-api/rest-api/jobs/get-jobs
GET /v3/jobs
Get details of all background/async jobs
# Jobs REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/jobs/index
Learn how to create requests and responses to retrieve async job details by job ID and view all background/async jobs with the Fiddler Platform.
`GET /v3/jobs/{job_id}`
`GET /v3/jobs`
# Create new LLM provider
Source: https://docs.fiddler.ai/sdk-api/rest-api/llm-gateway/create-llm-provider
POST /v3/llm-gateway/providers
Create a new LLM provider with credentials and models
# Delete LLM provider
Source: https://docs.fiddler.ai/sdk-api/rest-api/llm-gateway/delete-llm-provider
DELETE /v3/llm-gateway/providers/{provider}
Delete an LLM provider by name
# Get available provider options
Source: https://docs.fiddler.ai/sdk-api/rest-api/llm-gateway/get-llm-provider-options
GET /v3/llm-gateway/providers/options
Get available models for each supported LLM provider
# List LLM providers
Source: https://docs.fiddler.ai/sdk-api/rest-api/llm-gateway/get-llm-providers
GET /v3/llm-gateway/providers
Get list of LLM providers with pagination support
# LLM Gateway
Source: https://docs.fiddler.ai/sdk-api/rest-api/llm-gateway/index
REST API endpoints for managing llm gateway in Fiddler Platform.
`POST /v3/llm-gateway/providers`
`DELETE /v3/llm-gateway/providers/{provider}`
`GET /v3/llm-gateway/providers/options`
`GET /v3/llm-gateway/providers`
`POST /v3/llm-gateway/test-connection`
`PUT /v3/llm-gateway/providers/{provider}`
# Test LLM connection
Source: https://docs.fiddler.ai/sdk-api/rest-api/llm-gateway/test-llm-connection
POST /v3/llm-gateway/test-connection
Test that a registered model + credential combination works by sending
a minimal LLM completion call. Returns HTTP `200` with success/failure
for LLM-level results. Returns `4xx` for invalid inputs (unknown model,
unknown credential, malformed request) before any LLM call is made.
# Update LLM provider
Source: https://docs.fiddler.ai/sdk-api/rest-api/llm-gateway/update-llm-provider
PUT /v3/llm-gateway/providers/{provider}
Update an existing LLM provider's models and credentials. This is a replace operation - pass the complete desired state. For credentials: include existing ones (name/uuid) to keep them, add new ones (name/credential_config) to create them, omit any to remove them. For models: pass the complete list of desired models. Any fields not included will be removed.
# Add a new model under a project
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/add-model
POST /v3/models
Add a new model under a project
# Delete a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/delete-model
DELETE /v3/models/{model_id}
Delete a model
# Upload artifacts associated with the model
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/deploy-model-artifact
POST /v3/models/{model_id}/deploy-artifact
Upload artifacts associated with the model
# Update artifacts associated with the model
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/deploy-model-artifact-update
PUT /v3/models/{model_id}/deploy-artifact
Update artifacts associated with the model
# Deploy a surrogate model
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/deploy-surrogate
POST /v3/models/{model_id}/deploy-surrogate
Deploy a surrogate model
# Update a surrogate model
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/deploy-surrogate-update
PUT /v3/models/{model_id}/deploy-surrogate
Update a surrogate model
# Get details of a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/get-model
GET /v3/models/{model_id}
Details of a model for given model id
# Details of all columns
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/get-model-all-columns
GET /v3/models/{model_id}/columns
Details of all columns for a model
# getModelColumnV3
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/get-model-column
GET /v3/models/{model_id}/columns/{column_id}
Details of a specific column for a model
# List models
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/get-models
GET /v3/models
Get list of model for provided query and filters
# Model REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/index
Dive into our guide to the Model REST API. Learn to list models, add models to a project, get details, update fields, generate models from samples, and more.
`POST /v3/models`
`DELETE /v3/models/{model_id}`
`POST /v3/models/{model_id}/deploy-surrogate`
`GET /v3/models/{model_id}/columns`
`POST /v3/model-factory`
`GET /v3/models/{model_id}`
`GET /v3/models/{model_id}/columns/{column_id}`
`GET /v3/models`
`PUT /v3/models/{model_id}/deploy-surrogate`
`PUT /v3/models/{model_id}/deploy-artifact`
`PATCH /v3/models/{model_id}`
`POST /v3/models/{model_id}/deploy-artifact`
# Generate model from the given data sample
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/model-factory
POST /v3/model-factory
Generate model from the given data sample
# Update the fields of a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/model/update-model
PATCH /v3/models/{model_id}
Update the fields of a model.
**Schema and spec consistency.** This endpoint does not enforce
consistency between the model `schema` (the list of columns) and
the model `spec` (which assigns each column a role such as
`inputs`, `outputs`, `targets`, `decisions`, `metadata`, or
`custom_features`). If you add a column to `schema` without also
adding it to the appropriate `spec` array, the column will be
ingested into the raw events table but will not appear anywhere
else in the Fiddler UI.
**PATCH replaces nested objects wholesale.** When you PATCH the
`spec` (or `schema`) field, the request body replaces the entire
nested object — any role array you omit (`inputs`, `outputs`,
`targets`, `decisions`, `metadata`, `custom_features`) is reset
to empty on the server. Always read the current model first and
merge your additions into the existing arrays before sending the
PATCH.
To add columns safely, follow the multi-step sequence shown in the
code samples on the right:
1. `GET /v3/models/{model_id}` — retrieve the current `schema` and
`spec`.
2. `PATCH /v3/models/{model_id}` with an updated `schema`
containing the existing columns plus the new columns.
3. `PATCH /v3/models/{model_id}` with an updated `spec` that
preserves every existing role array and adds the new columns
to the appropriate one.
The Fiddler Python SDK and the Fiddler UI enforce this consistency
automatically; this note applies to direct REST API integrations.
# Create new project
Source: https://docs.fiddler.ai/sdk-api/rest-api/projects/create-project
POST /v3/projects
Create new project
# Delete project by id
Source: https://docs.fiddler.ai/sdk-api/rest-api/projects/delete-project
DELETE /v3/projects/{project_id}
Delete project by id
# Detail info of a project for the specified project id
Source: https://docs.fiddler.ai/sdk-api/rest-api/projects/get-project
GET /v3/projects/{project_id}
Detail info of a project
# List of projects
Source: https://docs.fiddler.ai/sdk-api/rest-api/projects/get-projects
GET /v3/projects
List of projects
# Projects REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/projects/index
Learn about REST API projects on the Fiddler platform. Explore how to list projects, create a new project, and get detailed info for a specified project ID.
`POST /v3/projects`
`DELETE /v3/projects/{project_id}`
`GET /v3/projects/{project_id}`
`GET /v3/projects`
# API to fetch metrics used to plot monitoring charts
Source: https://docs.fiddler.ai/sdk-api/rest-api/queries/get-queries
POST /v3/queries
API to fetch metrics used to plot monitoring charts
# Queries
Source: https://docs.fiddler.ai/sdk-api/rest-api/queries/index
REST API endpoints for managing queries in Fiddler Platform.
`POST /v3/queries`
# Bulk create scores
Source: https://docs.fiddler.ai/sdk-api/rest-api/scores/bulk-create-scores
POST /v3/scores/bulk
Create up to 100 scores in a single request. Per-item validation errors are collected and returned; valid items are committed together in a single batch insert. Infrastructure errors bubble as 500.
# Create a score
Source: https://docs.fiddler.ai/sdk-api/rest-api/scores/create-score
POST /v3/scores
Create a new score or annotation. Evaluator scores are read-only through this API — only human, LLM, and code sources are accepted. Returns 201 on success.
# Delete a score
Source: https://docs.fiddler.ai/sdk-api/rest-api/scores/delete-score
DELETE /v3/scores/{score_id}
Delete a score by ID. Returns 204 No Content on success. Idempotent — returns 204 even if the score does not exist. Returns 403 if source is evaluator.
# Get a single score
Source: https://docs.fiddler.ai/sdk-api/rest-api/scores/get-score
GET /v3/scores/{score_id}
Retrieve a single score by its ID.
# Scores
Source: https://docs.fiddler.ai/sdk-api/rest-api/scores/index
REST API endpoints for managing scores in Fiddler Platform.
`POST /v3/scores/bulk`
`POST /v3/scores`
`DELETE /v3/scores/{score_id}`
`GET /v3/scores/{score_id}`
`GET /v3/scores`
`PATCH /v3/scores/{score_id}`
# List scores
Source: https://docs.fiddler.ai/sdk-api/rest-api/scores/list-scores
GET /v3/scores
List scores with optional filters. Supports filtering by target (span, trace, session), score name, source, level, type, annotator, config name, and time range. All filter parameters accept comma-separated values for multi-value matching.
Ordering: use the `ordering` query param to sort results. Prefix a field name with `-` for descending. Default is `-created_at`. Valid fields: created_at, name, score_type, source, score_level, annotator_id, config_name, timestamp. The `timestamp` field sorts by the underlying target's timestamp (span/trace/session ingest time), which is not exposed as a separate field in the response.
# Update a score
Source: https://docs.fiddler.ai/sdk-api/rest-api/scores/update-score
PATCH /v3/scores/{score_id}
Partially update a score. Only value fields (value, label, text, reasoning, metadata) can be updated. Identity and structural fields (name, score_type, source, targets) cannot be changed — create a new score instead. Returns 403 if source is evaluator.
# Create new segment
Source: https://docs.fiddler.ai/sdk-api/rest-api/segments/create-segment
POST /v3/segments
Create new segment
# Delete segment by uuid
Source: https://docs.fiddler.ai/sdk-api/rest-api/segments/delete-segment
DELETE /v3/segments/{uuid}
Delete segment by uuid
# List all segments for a model
Source: https://docs.fiddler.ai/sdk-api/rest-api/segments/get-model-segments
GET /v3/models/{model_id}/segments
List all segments for a model
# Detail info of a segment
Source: https://docs.fiddler.ai/sdk-api/rest-api/segments/get-segment
GET /v3/segments/{uuid}
Detail info of a segment
# List all segments
Source: https://docs.fiddler.ai/sdk-api/rest-api/segments/get-segments
GET /v3/segments
List all segments
# Segments REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/segments/index
Discover Fiddler’s Segments REST API guide. Learn to list, create, view details, delete segments by UUID, and list all segments for a model.
`POST /v3/segments`
`DELETE /v3/segments/{uuid}`
`GET /v3/segments/{uuid}`
`GET /v3/segments`
`GET /v3/models/{model_id}/segments`
# Get server info
Source: https://docs.fiddler.ai/sdk-api/rest-api/server-info/get-server-info
GET /v3/server-info
Get detailed information about the Fiddler server.
# Server Info REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/server-info/index
Learn API guidelines for retrieving server info with Fiddler’s platform. Discover how to create requests, receive responses, and maximize our solutions.
`GET /v3/server-info`
# Delete Sessions
Source: https://docs.fiddler.ai/sdk-api/rest-api/sessions/delete-sessions
DELETE /v3/sessions
Delete one or more sessions and all associated data (spans, attributes, evaluator scores, and span content) from the application. This operation is idempotent — returns 204 even if the sessions do not exist.
# Sessions
Source: https://docs.fiddler.ai/sdk-api/rest-api/sessions/index
REST API endpoints for managing sessions in Fiddler Platform.
`DELETE /v3/sessions`
# Get span field coverage
Source: https://docs.fiddler.ai/sdk-api/rest-api/spans/get-span-fields
POST /v3/spans/fields
Given a filter over spans, return the union of span attribute keys and evaluator outputs that appear in the matching spans, along with a coverage count for each field.
Useful for schema discovery and data completeness analysis — e.g. before adding spans to an evaluation dataset, an agent or UI can call this endpoint to understand which attributes are present and how consistently they appear across the selected spans, without fetching full span payloads.
The query is bounded internally to the first 10,000 matching spans. When this cap is reached, `sampled` is set to `true` and counts are approximate.
# Spans
Source: https://docs.fiddler.ai/sdk-api/rest-api/spans/index
REST API endpoints for managing spans in Fiddler Platform.
`POST /v3/spans/fields`
`POST /v3/spans/query`
# Query spans
Source: https://docs.fiddler.ai/sdk-api/rest-api/spans/query-spans
POST /v3/spans/query
Search, filter, and paginate through spans with advanced criteria
# Delete Traces
Source: https://docs.fiddler.ai/sdk-api/rest-api/traces/delete-traces
DELETE /v3/traces
Delete one or more traces and all associated data (spans, attributes, evaluator scores, and span content) from the application. This operation is idempotent — returns 204 even if the traces do not exist.
# Traces
Source: https://docs.fiddler.ai/sdk-api/rest-api/traces/index
REST API endpoints for managing traces in Fiddler Platform.
`DELETE /v3/traces`
# Create a new API key.
Source: https://docs.fiddler.ai/sdk-api/rest-api/user-access-keys/create-user-access-key
POST /v3/user-access-keys
Create a new API key for the authenticated user. Requires a unique name per user per organization. The full composite API key (key_id.secret) is returned once in the response and cannot be retrieved again. Enforces MAX_ACCESS_KEYS_PER_USER limit (default 5, new table only).
# Delete an API key.
Source: https://docs.fiddler.ai/sdk-api/rest-api/user-access-keys/delete-user-access-key
DELETE /v3/user-access-keys/{key_id}
Delete an API key. The API key is permanently removed and can no longer be used for authentication.
# Get a single API key.
Source: https://docs.fiddler.ai/sdk-api/rest-api/user-access-keys/get-user-access-key
GET /v3/user-access-keys/{key_id}
Get metadata for a single API key owned by the authenticated user. Returns `404` if the API key is not found or belongs to another user.
# User Access Keys REST API Guide
Source: https://docs.fiddler.ai/sdk-api/rest-api/user-access-keys/index
Manage Fiddler API keys with the User Access Keys REST API guide. Learn to list, create, retrieve, update, and delete the access keys that authenticate your requests.
`POST /v3/user-access-keys`
`DELETE /v3/user-access-keys/{key_id}`
`GET /v3/user-access-keys/{key_id}`
`GET /v3/user-access-keys`
`PATCH /v3/user-access-keys/{key_id}`
# List the caller's API keys.
Source: https://docs.fiddler.ai/sdk-api/rest-api/user-access-keys/list-user-access-keys
GET /v3/user-access-keys
List the authenticated user's API keys. Returns API key metadata only — secrets and hashes are never returned. Results are paginated.
# Update an API key.
Source: https://docs.fiddler.ai/sdk-api/rest-api/user-access-keys/update-user-access-key
PATCH /v3/user-access-keys/{key_id}
Update an API key's mutable fields. Supports updating the API key name and expiration. The new expires_at must be a future timestamp (greater than current time) and cannot exceed 100 years from now.
# This API is used to get identity and access related information of a user. It also provides details into the last successful login of the user.
Source: https://docs.fiddler.ai/sdk-api/rest-api/users/get-users
GET /v3/users
List users
# Users
Source: https://docs.fiddler.ai/sdk-api/rest-api/users/index
REST API endpoints for managing users in Fiddler Platform.
`GET /v3/users`
# clear_llm_context
Source: https://docs.fiddler.ai/sdk-api/strands/clear-llm-context
Remove any previously set LLM context from a Model instance.
Remove any previously set LLM context from a Model instance.
After calling this function, subsequent spans created for the model will
not carry `gen_ai.llm.context` unless it is set again via
`set_llm_context()`.
This is equivalent to `set_llm_context(model, None)`.
Works automatically in both synchronous and asynchronous contexts.
## Parameters
The Model instance to clear context from
## Example
```python theme={null}
from strands.models.openai import OpenAIModel
from fiddler_strandsagents import set_llm_context, clear_llm_context
model = OpenAIModel(api_key="...", model_id="gpt-4")
set_llm_context(model, "RAG retrieved documents...")
# ... RAG LLM call ...
clear_llm_context(model)
# ... subsequent non-RAG LLM calls have no context ...
```
# FiddlerInstrumentationHook
Source: https://docs.fiddler.ai/sdk-api/strands/fiddler-instrumentation-hook
Centralized instrumentation hook for Strands agents with Fiddler integration.
Centralized instrumentation hook for Strands agents with Fiddler integration.
This hook provider automatically captures telemetry data from Strands agents
and enriches OpenTelemetry spans with Fiddler-specific attributes. It handles
three key events: before invocation, after tool execution, and after model calls.
The hook is automatically injected when using StrandsAgentInstrumentor, so
users typically don't need to manually add it to their agents.
## Example
```python theme={null}
from strands import Agent
from fiddler_strandsagents import FiddlerInstrumentationHook
# Manual usage (not needed with StrandsAgentInstrumentor)
agent = Agent(
model=model,
system_prompt="...",
hooks=[FiddlerInstrumentationHook()]
)
```
## register\_hooks()
Register hook callbacks with the Strands agent registry.
This method is called by the Strands framework to register callbacks
for various agent lifecycle events.
### Parameters
The HookRegistry to register callbacks with
## tool\_end()
Handle the completion of a tool invocation.
Enriches the current OpenTelemetry span with:
* Custom attributes set via set\_span\_attributes() (prefixed 'fiddler.span.user.')
* gen\_ai.tool.input — JSON-serialised input arguments passed to the tool
* gen\_ai.tool.output — Text extracted from the tool result content
### Parameters
AfterToolCallEvent (or legacy AfterToolInvocationEvent) containing
tool execution details
## model\_end()
Handle the completion of a model invocation.
Enriches the current OpenTelemetry span with:
* gen\_ai.llm.context — retrieved context set via set\_llm\_context()
* Custom attributes set via set\_span\_attributes() (prefixed 'fiddler.span.user.')
* gen\_ai.llm.output — text extracted from the model's response message
### Parameters
AfterModelCallEvent (or legacy AfterModelInvocationEvent) containing
model execution details
## before\_invocation()
Handle the start of an agent invocation.
Enriches the current OpenTelemetry span with:
* Conversation ID set via set\_conversation\_id()
* Session attributes set via set\_session\_attributes()
* gen\_ai.llm.input.user — text of the last user message in the invocation
Session attributes and conversation ID are also written to
`event.agent.trace_attributes` so that Strands (>=1.19.0) propagates
them natively to all child spans (model invoke, tool call, event-loop
cycle, etc.) without needing manual parent-to-child denormalisation in
a SpanProcessor.
### Parameters
BeforeInvocationEvent containing agent invocation details
# FiddlerSpanProcessor
Source: https://docs.fiddler.ai/sdk-api/strands/fiddler-span-processor
Backwards-compatible alias for StrandsSpanProcessor.
# FiddlerSpanProcessor
`FiddlerSpanProcessor` is a backwards-compatible alias for [`StrandsSpanProcessor`](/sdk-api/strands/strands-span-processor). They reference the same class -- `StrandsSpanProcessor` is the canonical name and `FiddlerSpanProcessor` remains exported so existing code continues to work.
```python theme={null}
from fiddler_strandsagents import FiddlerSpanProcessor, StrandsSpanProcessor
assert FiddlerSpanProcessor is StrandsSpanProcessor # True
```
For the full reference, see [StrandsSpanProcessor](/sdk-api/strands/strands-span-processor).
> **Recommendation:** new code should import `StrandsSpanProcessor` directly.
# get_conversation_id
Source: https://docs.fiddler.ai/sdk-api/strands/get-conversation-id
Get the conversation ID for the current agent invocation.
Get the conversation ID for the current agent invocation.
Retrieves the conversation ID that was previously set using set\_conversation\_id().
Works in both synchronous and asynchronous contexts automatically.
## Parameters
The Strands Agent instance to retrieve the conversation ID from
## Returns
The conversation ID string, or empty string if none has been set
# get_llm_context
Source: https://docs.fiddler.ai/sdk-api/strands/get-llm-context
Get the LLM context for the current model.
Get the LLM context for the current model.
Retrieves the context string that was previously set using set\_llm\_context().
Works automatically in both synchronous and asynchronous contexts.
## Parameters
The Model instance to retrieve context from
## Returns
The LLM context string, or empty string if none has been set
## Example
```python theme={null}
from fiddler_strandsagents import set_llm_context, get_llm_context
set_llm_context(model, "Important background information")
context = get_llm_context(model)
print(context) # "Important background information"
```
# get_session_attributes
Source: https://docs.fiddler.ai/sdk-api/strands/get-session-attributes
Get the session attributes for the current agent invocation.
Get the session attributes for the current agent invocation.
Retrieves session attributes that were previously set using set\_session\_attributes().
Works in both synchronous and asynchronous contexts automatically.
## Parameters
The Strands Agent instance to retrieve session attributes from
## Returns
Dictionary of session attribute key-value pairs, or empty dict if none exist
# get_span_attributes
Source: https://docs.fiddler.ai/sdk-api/strands/get-span-attributes
Get span attributes from a Model or AgentTool object.
Get span attributes from a Model or AgentTool object.
Retrieves custom attributes that were previously set using set\_span\_attributes().
Returns an empty dictionary if no attributes have been set.
## Parameters
The Model or AgentTool instance to retrieve attributes from
## Returns
Dictionary of attribute key-value pairs, or empty dict if none exist
# Introduction
Source: https://docs.fiddler.ai/sdk-api/strands/index
Complete API reference for fiddler-strandsagents
[](https://pypi.org/project/fiddler-strands/)
## Overview
Complete API reference documentation for the `fiddler-strands` package, which
instruments the [Strands Agents SDK](https://strandsagents.com/) and exports
OpenTelemetry traces to Fiddler.
The package is published to PyPI as **`fiddler-strands`** and imported in
Python as **`fiddler_strandsagents`**:
```bash theme={null}
pip install fiddler-strands
```
```python theme={null}
from fiddler_strandsagents import StrandsAgentInstrumentor
```
## Components
### Conversation And Session
Conversation and session attribute helpers. `set_conversation_id` / `get_conversation_id` set and read the conversation ID for the current execution context. `set_session_attributes` / `get_session_attributes` replace and read the session attribute dict for the current context.
* [get\_conversation\_id](/sdk-api/strands/get-conversation-id)
* [get\_session\_attributes](/sdk-api/strands/get-session-attributes)
* [set\_conversation\_id](/sdk-api/strands/set-conversation-id)
* [set\_session\_attributes](/sdk-api/strands/set-session-attributes)
### Instrumentation
`StrandsAgentInstrumentor` installs Fiddler tracing into the Strands Agents runtime. `FiddlerInstrumentationHook` is the Strands hook used internally by the instrumentor.
* [FiddlerInstrumentationHook](/sdk-api/strands/fiddler-instrumentation-hook)
* [StrandsAgentInstrumentor](/sdk-api/strands/strands-agent-instrumentor)
### Llm Context
`set_llm_context` attaches context (e.g. retrieved documents) to a Strands `Model` so it appears on its LLM spans. `clear_llm_context` removes previously set LLM context. `get_llm_context` reads the current LLM context for a model.
* [clear\_llm\_context](/sdk-api/strands/clear-llm-context)
* [get\_llm\_context](/sdk-api/strands/get-llm-context)
* [set\_llm\_context](/sdk-api/strands/set-llm-context)
### Span Attributes
`set_span_attributes` and `get_span_attributes` set and read per-span custom attributes.
* [get\_span\_attributes](/sdk-api/strands/get-span-attributes)
* [set\_span\_attributes](/sdk-api/strands/set-span-attributes)
### Span Processor
`StrandsSpanProcessor` propagates parent attributes and patches Strands' missing `gen_ai.system.message` event. `FiddlerSpanProcessor` is a backwards-compatible alias for `StrandsSpanProcessor` (same class object).
* [FiddlerSpanProcessor](/sdk-api/strands/fiddler-span-processor)
* [StrandsSpanProcessor](/sdk-api/strands/strands-span-processor)
# set_conversation_id
Source: https://docs.fiddler.ai/sdk-api/strands/set-conversation-id
Set the conversation ID for the current agent invocation.
Set the conversation ID for the current agent invocation.
The conversation ID is used to group related agent invocations together,
enabling conversation-level tracing and monitoring in Fiddler's platform.
This ID will persist until it is explicitly changed by calling this function
again with a new value.
## Parameters
The Strands Agent instance to associate with the conversation
Unique identifier for the conversation (e.g., session ID, user ID)
## Example
```python theme={null}
from strands import Agent
from fiddler_strandsagents import set_conversation_id
agent = Agent(model=model, system_prompt="...")
set_conversation_id(agent, "session_12345")
```
# set_llm_context
Source: https://docs.fiddler.ai/sdk-api/strands/set-llm-context
Set or clear additional context information for LLM interactions.
Set or clear additional context information for LLM interactions.
The LLM context allows you to provide additional background information
or available options that can be tracked alongside model invocations.
This context will be added to telemetry spans as 'gen\_ai.llm.context'
and can be used for debugging or analysis in Fiddler's platform.
The context persists until explicitly changed by calling this function
again with a new value, or cleared by passing `None`. Clearing is
useful in multi-step agent workflows where context set after a RAG step
should not leak into later non-RAG LLM calls.
Works automatically in both synchronous and asynchronous contexts.
## Parameters
The Model instance to attach context to
Context string providing additional information about available
options, constraints, or background for the LLM interaction.
Pass `None` to clear any previously set context.
## Example
```python theme={null}
from strands.models.openai import OpenAIModel
from fiddler_strandsagents import set_llm_context
model = OpenAIModel(api_key="...", model_id="gpt-4")
set_llm_context(
model,
'Available hotels: Hilton, Marriott, Hyatt...'
)
# Now when the model is invoked, the context will be
# included in the telemetry span
agent = Agent(model=model, system_prompt="You are a travel assistant")
response = agent("Which hotel should I book?")
# Clear context before non-RAG steps
set_llm_context(model, None)
```
# set_session_attributes
Source: https://docs.fiddler.ai/sdk-api/strands/set-session-attributes
Add Fiddler-specific session attributes to an agent's metadata.
Add Fiddler-specific session attributes to an agent's metadata.
Session attributes provide context about the agent's execution environment,
such as user roles, cost centers, or any custom metadata that should be
tracked across all invocations within a session.
## Parameters
The Strands Agent instance to add session attributes to
Key-value pairs of session attributes (str, int, float, or bool values)
## Example
```python theme={null}
from strands import Agent
from fiddler_strandsagents import set_session_attributes
agent = Agent(model=model, system_prompt="...")
set_session_attributes(agent, role="customer_support", region="us-west")
```
# set_span_attributes
Source: https://docs.fiddler.ai/sdk-api/strands/set-span-attributes
Set custom attributes on a Model or AgentTool that can be accessed by logging hooks.
Set custom attributes on a Model or AgentTool that can be accessed by logging hooks.
This function stores key-value pairs as attributes on the object, making
them accessible to hooks during model invocation events. Attributes are
automatically scoped to async or sync contexts.
## Parameters
The object to set the attribute on (Model or AgentTool instance)
Key-value pairs of attributes to set (str, int, float, or bool values)
## Example
```python theme={null}
from strands.models.openai import OpenAIModel
from fiddler_strandsagents import set_span_attributes
model = OpenAIModel(api_key="...", model_id="gpt-4")
set_span_attributes(model, model_id="gpt-4", temperature=0.7)
```
# StrandsAgentInstrumentor
Source: https://docs.fiddler.ai/sdk-api/strands/strands-agent-instrumentor
OpenTelemetry instrumentor for Strands AI agents.
OpenTelemetry instrumentor for Strands AI agents.
This instrumentor automatically injects FiddlerInstrumentationHook into all
Strands Agent instances, enabling automatic observability without manual hook
configuration. It also registers a StrandsSpanProcessor for attribute denormalization.
The instrumentor follows the OpenTelemetry instrumentation pattern and can be
enabled/disabled dynamically.
## Example
```python theme={null}
from strands.telemetry import StrandsTelemetry
from fiddler_strandsagents import StrandsAgentInstrumentor
telemetry = StrandsTelemetry()
telemetry.setup_otlp_exporter()
# Enable instrumentation
instrumentor = StrandsAgentInstrumentor(telemetry)
instrumentor.instrument()
# Create agents - hooks are automatically injected
agent = Agent(model=model, system_prompt="...")
```
Initialize the Strands Agent instrumentor.
## Parameters
StrandsTelemetry instance for configuring trace exporters
## instrumentation\_dependencies()
Return the list of packages required for instrumentation.
### Returns
Collection of package names with version constraints
## instrument()
Enable automatic instrumentation of Strands agents.
Activates the instrumentor by registering the StrandsSpanProcessor with
the tracer provider and patching Agent.**init** to automatically inject
FiddlerInstrumentationHook into all agent instances.
This method is idempotent - calling it multiple times has the same effect
as calling it once. After activation, all newly created agents will
automatically include Fiddler's instrumentation hook.
### Parameters
**\*\*kwargs** – Additional instrumentation configuration (currently unused)
### Example
```python theme={null}
from strands import Agent
from strands.telemetry import StrandsTelemetry
from fiddler_strandsagents import StrandsAgentInstrumentor
# Set up telemetry
telemetry = StrandsTelemetry()
telemetry.setup_otlp_exporter()
# Activate instrumentation
instrumentor = StrandsAgentInstrumentor(telemetry)
instrumentor.instrument()
# Agents created after this point are automatically instrumented
agent = Agent(model=model, system_prompt="...")
# Check if instrumentation is active
if instrumentor.is_instrumented_by_opentelemetry:
print("Instrumentation is active")
```
## uninstrument()
Disable automatic instrumentation and restore original behavior.
Deactivates the instrumentor by restoring the original Agent.**init**
method. Agents created after calling this method will no longer have
FiddlerInstrumentationHook automatically injected.
Note: This does not affect agents that were already created while
instrumentation was active - those will retain their hooks.
### Parameters
**\*\*kwargs** – Additional uninstrumentation configuration (currently unused)
### Example
```python theme={null}
from fiddler_strandsagents import StrandsAgentInstrumentor
instrumentor = StrandsAgentInstrumentor(telemetry)
instrumentor.instrument()
# Create some agents (automatically instrumented)
agent1 = Agent(model=model, system_prompt="...")
# Deactivate instrumentation
instrumentor.uninstrument()
# New agents won't be instrumented
agent2 = Agent(model=model, system_prompt="...")
# agent1 still has the hook, agent2 does not
```
## *property* is\_instrumented\_by\_opentelemetry
Check whether instrumentation is currently active.
### Returns
True if the instrumentor has been activated via instrument() and
not yet deactivated via uninstrument(), False otherwise
### Example
```python theme={null}
instrumentor = StrandsAgentInstrumentor(telemetry)
print(instrumentor.is_instrumented_by_opentelemetry) # False
instrumentor.instrument()
print(instrumentor.is_instrumented_by_opentelemetry) # True
instrumentor.uninstrument()
print(instrumentor.is_instrumented_by_opentelemetry) # False
```
# StrandsSpanProcessor
Source: https://docs.fiddler.ai/sdk-api/strands/strands-span-processor
Propagates parent attributes via `CoreFiddlerSpanProcessor` plus Strands-specific behavior.
Propagates parent attributes via `CoreFiddlerSpanProcessor` plus Strands-specific behavior.
On every span start this processor:
1. Delegates to `CoreFiddlerSpanProcessor` for standard attribute denormalization.
2. For `chat` spans only: copies the parent's `system_prompt` into a
`gen_ai.system.message` event as a workaround for Strands not always emitting
that event on the chat span upstream (strands-agents/sdk-python#822).
## on\_start()
Called when a span starts. Automatically propagates attributes from parent.
### Parameters
The span being started.
The parent context, if any.