LogoLogo
👨‍💻 API Reference📣 Release Notes📺 Request a Demo
  • Introduction to Fiddler
    • Monitor, Analyze, and Protect your ML Models and Gen AI Applications
  • Fiddler Doc Chatbot
  • First Steps
    • Getting Started With Fiddler Guardrails
    • Getting Started with LLM Monitoring
    • Getting Started with ML Model Observability
  • Tutorials & Quick Starts
    • LLM and GenAI
      • LLM Evaluation - Compare Outputs
      • LLM Monitoring - Simple
    • Fiddler Free Guardrails
      • Guardrails - Quick Start Guide
      • Guardrails - Faithfulness
      • Guardrails - Safety
      • Guardrails FAQ
    • ML Observability
      • ML Monitoring - Simple
      • ML Monitoring - NLP Inputs
      • ML Monitoring - Class Imbalance
      • ML Monitoring - Model Versions
      • ML Monitoring - Ranking
      • ML Monitoring - Regression
      • ML Monitoring - Feature Impact
      • ML Monitoring - CV Inputs
  • Glossary
    • Product Concepts
      • Baseline
      • Custom Metric
      • Data Drift
      • Embedding Visualization
      • Fiddler Guardrails
      • Fiddler Trust Service
      • LLM and GenAI Observability
      • Metric
      • Model Drift
      • Model Performance
      • ML Observability
      • Trust Score
  • Product Guide
    • LLM Application Monitoring & Protection
      • LLM-Based Metrics
      • Embedding Visualizations for LLM Monitoring and Analysis
      • Selecting Enrichments
      • Enrichments (Private Preview)
      • Guardrails for Proactive Application Protection
    • Optimize Your ML Models and LLMs with Fiddler's Comprehensive Monitoring
      • Alerts
      • Package-Based Alerts (Private Preview)
      • Class Imbalanced Data
      • Enhance ML and LLM Insights with Custom Metrics
      • Data Drift: Monitor Model Performance Changes with Fiddler's Insights
      • Ensuring Data Integrity in ML Models And LLMs
      • Embedding Visualization With UMAP
      • Fiddler Query Language
      • Model Versions
      • How to Effectively Use the Monitoring Chart UI
      • Performance Tracking
      • Model Segments: Analyze Cohorts for Performance Insights and Bias Detection
      • Statistics
      • Monitoring ML Model and LLM Traffic
      • Vector Monitoring
    • Enhance Model Insights with Fiddler's Slice and Explain
      • Events Table in RCA
      • Feature Analytics Creation
      • Metric Card Creation
      • Performance Charts Creation
      • Performance Charts Visualization
    • Master AI Monitoring: Create, Customize, and Compare Dashboards
      • Creating Dashboards
      • Dashboard Interactions
      • Dashboard Utilities
    • Adding and Editing Models in the UI
      • Model Editor UI
      • Model Schema Editing Guide
    • Fairness
    • Explainability
      • Model: Artifacts, Package, Surrogate
      • Global Explainability: Visualize Feature Impact and Importance in Fiddler
      • Point Explainability
      • Flexible Model Deployment
        • On Prem Manual Flexible Model Deployment XAI
  • Technical Reference
    • Python Client API Reference
    • Python Client Guides
      • Installation and Setup
      • Model Onboarding
        • Create a Project and Onboard a Model for Observation
        • Model Task Types
        • Customizing your Model Schema
        • Specifying Custom Missing Value Representations
      • Publishing Inference Data
        • Creating a Baseline Dataset
        • Publishing Batches Of Events
        • Publishing Ranking Events
        • Streaming Live Events
        • Updating Already Published Events
        • Deleting Events From Fiddler
      • Creating and Managing Alerts
      • Explainability Examples
        • Adding a Surrogate Model
        • Uploading Model Artifacts
        • Updating Model Artifacts
        • ML Framework Examples
          • Scikit Learn
          • Tensorflow HDF5
          • Tensorflow Savedmodel
          • Xgboost
        • Model Task Examples
          • Binary Classification
          • Multiclass Classification
          • Regression
          • Uploading A Ranking Model Artifact
    • Integrations
      • Data Pipeline Integrations
        • Airflow Integration
        • BigQuery Integration
        • Integration With S3
        • Kafka Integration
        • Sagemaker Integration
        • Snowflake Integration
      • ML Platform Integrations
        • Integrate Fiddler with Databricks for Model Monitoring and Explainability
        • Datadog Integration
        • ML Flow Integration
      • Alerting Integrations
        • PagerDuty Integration
    • Comprehensive REST API Reference
      • Projects REST API Guide
      • Model REST API Guide
      • File Upload REST API Guide
      • Custom Metrics REST API Guide
      • Segments REST API Guide
      • Baselines REST API Guide
      • Jobs REST API Guide
      • Alert Rules REST API Guide
      • Environments REST API Guide
      • Explainability REST API Guide
      • Server Info REST API Guide
      • Events REST API Guide
      • Fiddler Trust Service REST API Guide
    • Fiddler Free Guardrails Documentation
  • Configuration Guide
    • Authentication & Authorization
      • Adding Users
      • Overview of Role-Based Access Control
      • Email Authentication
      • Okta OIDC SSO Integration
      • Azure AD OIDC SSO Integration
      • Ping Identity SAML SSO Integration
      • Mapping LDAP Groups & Users to Fiddler Teams
    • Application Settings
    • Supported Browsers
  • History
    • Release Notes
    • Python Client History
    • Compatibility Matrix
    • Product Maturity Definitions
Powered by GitBook

© 2024 Fiddler Labs, Inc.

On this page
  • Creating Alert Rules
  • Understanding Alert Thresholds
  • Example Implementations
  • Example 1: Data Integrity Alert with Static Threshold
  • Example 2: Data Integrity Alert with Historical Comparison
  • Example 3: Performance Alert with Time-Based Comparison
  • Retrieving Alert Rules
  • Removing Alert Rules
  • Accessing Triggered Alerts
  • Configuring Notifications

Was this helpful?

  1. Technical Reference
  2. Python Client Guides

Creating and Managing Alerts

PreviousDeleting Events From FiddlerNextExplainability Examples

Last updated 1 month ago

Was this helpful?

The Fiddler API client provides programmatic control for alert management alongside the Fiddler UI, enabling these key workflows:

  • Create alert rules

  • Remove alert rules

  • Retrieve all configured alert rules

  • Access triggered alert history

📘 Note: For UI-based alert configuration, refer to the .

Creating Alert Rules

The Fiddler client can be used to create a variety of alert rules types including Data Drift, Performance, Data Integrity, Service Metrics, and Custom Metrics.

The Fiddler client supports multiple alert rule types including:

  • Data Drift

  • Performance

  • Data Integrity

  • Service Metrics

  • Custom Metrics

For a complete list of supported alert metrics (specified via the metric_id parameter), see our .

Understanding Alert Thresholds

When configuring thresholds:

  • Absolute thresholds (CompareTo.RAW_VALUE): For percentage-based metrics like null_violation_percentage, express values as percentages (e.g., 10 for 10%).

  • Relative thresholds (CompareTo.TIME_PERIOD): Express values as decimal fractions regardless of metric type (e.g., 0.1 for 10%)

Example Implementations

Example 1: Data Integrity Alert with Static Threshold

This example creates an alert that monitors for missing values in the age column. It triggers notifications when null values exceed 5% (warning) or 10% (critical) of daily values.

MODEL_ID = '299c7b40-b87c-4dad-bb94-251dbcd3cbdf'

alert_rule = AlertRule(
    name='Bank Churn Missing Values Static Percent',
    model_id=MODEL_ID,
    metric_id='null_violation_percentage',
    priority=Priority.HIGH,
    compare_to=CompareTo.RAW_VALUE,
    condition=AlertCondition.GREATER,
    bin_size=BinSize.DAY,
    critical_threshold=10,
    warning_threshold=5,
    columns=['age'],
).create()

notifications = alert_rule.set_notification_config(
    emails=['your.email@example.com', 'alerts.group@example.com'],
)

Example 2: Data Integrity Alert with Historical Comparison

This example creates an alert that compares today's missing values against yesterday's values. It triggers when null values increase by 5% (warning) or 10% (critical) compared to the previous day.

MODEL_ID = '299c7b40-b87c-4dad-bb94-251dbcd3cbdf'

alert_rule = AlertRule(
    name='Bank Churn Missing Values Rolling Historical Percent',
    model_id=MODEL_ID,
    metric_id='null_violation_percentage',
    priority=Priority.HIGH,
    compare_to=CompareTo.TIME_PERIOD,
    compare_bin_delta=1,
    condition=AlertCondition.GREATER,
    bin_size=BinSize.DAY,
    critical_threshold=0.1,
    warning_threshold=0.05,
    columns=['age'],
).create()

notifications = alert_rule.set_notification_config(
    emails=['your.email@example.com', 'alerts.group@example.com'],
)

Example 3: Performance Alert with Time-Based Comparison

This example creates a performance alert that monitors for precision changes. It compares today's precision with yesterday's values and triggers when precision decreases by 5% (warning) or 10% (critical).

MODEL_ID = '4531bfd9-2ca2-4a7b-bb5a-136c8da09ca2'

alert_rule = AlertRule(
    name='Bank Churn Precision Relative',
    model_id=MODEL_ID,
    metric_id='precision',
    priority=Priority.HIGH,
    compare_to=CompareTo.TIME_PERIOD,
    compare_bin_delta=1,
    condition=AlertCondition.GREATER,
    bin_size=BinSize.DAY,
    critical_threshold=0.1,
    warning_threshold=0.05,
).create()

notifications = alert_rule.set_notification_config(
    emails=['your.email@example.com', 'alerts.group@example.com'],
)

🚧 Please note, the possible values for compare_bin_delta vs bin_size are:

Bin Size
Allowed Compare bin delta

BinSize.Hour

[1, 24, 24 * 7, 24 * 30, 24 * 90]

BinSize.Day

[1, 7, 30, 90]

BinSize.Week

[1]

BinSize.Month

[1]

Retrieving Alert Rules

MODEL_ID = '4531bfd9-2ca2-4a7b-bb5a-136c8da09ca2'

alert_rules = AlertRule.list(
    model_id=MODEL_ID,  # Optional parameter
    metric_id='jsd',  # Optional parameter
    columns=['age'],  # Optional parameter
    ordering=[
        'critical_threshold'
    ],  # Add **-** prefix for descending sort ['-critical_threshold']
)

Tip: All filter parameters are optional. Omit them to retrieve all alert rules.

Removing Alert Rules

# Delete from a list of alerts
MODEL_ID = '4531bfd9-2ca2-4a7b-bb5a-136c8da09ca2'
alert_rules = AlertRule.list(model_id=MODEL_ID)

for alert_rule in alert_rules:
    if alert_rule.name == 'Bank Churn Precision Relative':
        alert_rule.delete()
        break

# Delete using the alert rule's unique identifier
ALERT_RULE_ID = '6da9c3c0-a9fa-4ab6-8b64-8d07b0736e77'
rule = AlertRule.get(id_=ALERT_RULE_ID)
rule.delete()

Tip: Find the alert rule ID in the Alert Rule tab of your Fiddler Alerts page.

Accessing Triggered Alerts

from datetime import datetime

triggered_alerts = AlertRecord.list(
    alert_rule_id=ALERT_RULE_ID,     # Required: ID of the alert rule
    start_time=datetime(2024, 9, 1), # Optional: Start of time range
    end_time=datetime(2024, 9, 24),  # Optional: End of time range
    ordering=['alert_time_bucket'],  # Optional: Sort order
                                     # Use '-alert_time_bucket' for descending
)

Configuring Notifications

After creating an alert rule, configure how notifications are sent when the rule triggers. Fiddler supports these notification channels:

  • Email

  • PagerDuty

  • Slack webhooks

  • Custom webhooks

You can specify multiple notification types, and each type can have multiple recipients.

ALERT_RULE_ID = '72e8835b-cde2-4dd2-a435-a35d4b51196b'
rule = AlertRule.get(id_=ALERT_RULE_ID)

rule.set_notification_config(
    emails=['your.email@example.com', 'alerts.group@example.com'],
    webhooks=['8b403d99-530a-4c5a-a519-89688d65ddc1'],  # Webhook UUID
    pagerduty_services=[
        'pagerduty_service_1',
        'pagerduty_service_2',
    ],  # PagerDuty service names
    pagerduty_severity='critical',  # Only applies to PagerDuty
)

Important: PagerDuty, Slack webhooks, and custom webhooks must be pre-configured by your Fiddler administrator.

The method retrieves alert rules that match your specified criteria. This method returns a Python iterator of matching rules.

To delete an alert rule, call the method on an object. You can retrieve the rule object using either:

The method with filters

The method with the rule's unique identifier

Use the method to retrieve alerts triggered by a specific rule:

alert setup guide
Alert Rule list with Copy alert rule ID highlighted in an alert rule's context menu

Need help? Contact us at .

💡
help@fiddler.ai
metrics reference
AlertRule.list()
delete()
AlertRule
list()
get()
AlertRecord.list()