LogoLogo
👨‍💻 API Reference📣 Release Notes📺 Request a Demo
  • Introduction to Fiddler
    • Monitor, Analyze, and Protect your ML Models and Gen AI Applications
  • Fiddler Doc Chatbot
  • First Steps
    • Getting Started With Fiddler Guardrails
    • Getting Started with LLM Monitoring
    • Getting Started with ML Model Observability
  • Tutorials & Quick Starts
    • LLM and GenAI
      • LLM Evaluation - Compare Outputs
      • LLM Monitoring - Simple
    • Fiddler Free Guardrails
      • Guardrails - Quick Start Guide
      • Guardrails - Faithfulness
      • Guardrails - Safety
      • Guardrails FAQ
    • ML Observability
      • ML Monitoring - Simple
      • ML Monitoring - NLP Inputs
      • ML Monitoring - Class Imbalance
      • ML Monitoring - Model Versions
      • ML Monitoring - Ranking
      • ML Monitoring - Regression
      • ML Monitoring - Feature Impact
      • ML Monitoring - CV Inputs
  • Glossary
    • Product Concepts
      • Baseline
      • Custom Metric
      • Data Drift
      • Embedding Visualization
      • Fiddler Guardrails
      • Fiddler Trust Service
      • LLM and GenAI Observability
      • Metric
      • Model Drift
      • Model Performance
      • ML Observability
      • Trust Score
  • Product Guide
    • LLM Application Monitoring & Protection
      • LLM-Based Metrics
      • Embedding Visualizations for LLM Monitoring and Analysis
      • Selecting Enrichments
      • Enrichments (Private Preview)
      • Guardrails for Proactive Application Protection
    • Optimize Your ML Models and LLMs with Fiddler's Comprehensive Monitoring
      • Alerts
      • Package-Based Alerts (Private Preview)
      • Class Imbalanced Data
      • Enhance ML and LLM Insights with Custom Metrics
      • Data Drift: Monitor Model Performance Changes with Fiddler's Insights
      • Ensuring Data Integrity in ML Models And LLMs
      • Embedding Visualization With UMAP
      • Fiddler Query Language
      • Model Versions
      • How to Effectively Use the Monitoring Chart UI
      • Performance Tracking
      • Model Segments: Analyze Cohorts for Performance Insights and Bias Detection
      • Statistics
      • Monitoring ML Model and LLM Traffic
      • Vector Monitoring
    • Enhance Model Insights with Fiddler's Slice and Explain
      • Events Table in RCA
      • Feature Analytics Creation
      • Metric Card Creation
      • Performance Charts Creation
      • Performance Charts Visualization
    • Master AI Monitoring: Create, Customize, and Compare Dashboards
      • Creating Dashboards
      • Dashboard Interactions
      • Dashboard Utilities
    • Adding and Editing Models in the UI
      • Model Editor UI
      • Model Schema Editing Guide
    • Fairness
    • Explainability
      • Model: Artifacts, Package, Surrogate
      • Global Explainability: Visualize Feature Impact and Importance in Fiddler
      • Point Explainability
      • Flexible Model Deployment
        • On Prem Manual Flexible Model Deployment XAI
  • Technical Reference
    • Python Client API Reference
    • Python Client Guides
      • Installation and Setup
      • Model Onboarding
        • Create a Project and Onboard a Model for Observation
        • Model Task Types
        • Customizing your Model Schema
        • Specifying Custom Missing Value Representations
      • Publishing Inference Data
        • Creating a Baseline Dataset
        • Publishing Batches Of Events
        • Publishing Ranking Events
        • Streaming Live Events
        • Updating Already Published Events
        • Deleting Events From Fiddler
      • Creating and Managing Alerts
      • Explainability Examples
        • Adding a Surrogate Model
        • Uploading Model Artifacts
        • Updating Model Artifacts
        • ML Framework Examples
          • Scikit Learn
          • Tensorflow HDF5
          • Tensorflow Savedmodel
          • Xgboost
        • Model Task Examples
          • Binary Classification
          • Multiclass Classification
          • Regression
          • Uploading A Ranking Model Artifact
    • Integrations
      • Data Pipeline Integrations
        • Airflow Integration
        • BigQuery Integration
        • Integration With S3
        • Kafka Integration
        • Sagemaker Integration
        • Snowflake Integration
      • ML Platform Integrations
        • Integrate Fiddler with Databricks for Model Monitoring and Explainability
        • Datadog Integration
        • ML Flow Integration
      • Alerting Integrations
        • PagerDuty Integration
    • Comprehensive REST API Reference
      • Projects REST API Guide
      • Model REST API Guide
      • File Upload REST API Guide
      • Custom Metrics REST API Guide
      • Segments REST API Guide
      • Baselines REST API Guide
      • Jobs REST API Guide
      • Alert Rules REST API Guide
      • Environments REST API Guide
      • Explainability REST API Guide
      • Server Info REST API Guide
      • Events REST API Guide
      • Fiddler Trust Service REST API Guide
    • Fiddler Free Guardrails Documentation
  • Configuration Guide
    • Authentication & Authorization
      • Adding Users
      • Overview of Role-Based Access Control
      • Email Authentication
      • Okta OIDC SSO Integration
      • Azure AD OIDC SSO Integration
      • Ping Identity SAML SSO Integration
      • Mapping LDAP Groups & Users to Fiddler Teams
    • Application Settings
    • Supported Browsers
  • History
    • Release Notes
    • Python Client History
    • Compatibility Matrix
    • Product Maturity Definitions
Powered by GitBook

© 2024 Fiddler Labs, Inc.

On this page
  • Overview
  • Supported Metric Types
  • Alert Configurations
  • Why do we need alerts?
  • Sample Alert Email
  • Integrations

Was this helpful?

  1. Product Guide
  2. Optimize Your ML Models and LLMs with Fiddler's Comprehensive Monitoring

Alerts

PreviousOptimize Your ML Models and LLMs with Fiddler's Comprehensive MonitoringNextPackage-Based Alerts (Private Preview)

Last updated 2 months ago

Was this helpful?

Overview

Fiddler enables users to set up alert rules to track a model's health and performance over time. Fiddler alerts also enable users to dig into triggered alerts and perform root cause analysis to discover what is causing a model to degrade. Users can set up alerts using the and the Fiddler UI as demonstrated below.

Supported Metric Types

You can get alerts for the following metrics:

    • The volume of traffic received by the model over time informs us of the overall system's health.

    • Metrics which can be used to monitor basic aggregations over columns.

    • Model performance can be poor if models trained on a specific dataset encounter different data in production.

    • Three types of violations can occur at model inference: missing feature values, type mismatches (e.g. sending a float input for a categorical feature type) or range mismatches (e.g. sending an unknown US State for a State categorical feature).

    • The model performance tells us how well a model performs on its task. A poorly performing model can have significant business implications.

    • Create your own custom metric formulas and create Alerts on those metrics giving you ultimate flexibility in alerting behavior.

Alert Configurations

Comparison Types

There are two options for alert threshold comparison:

  • Absolute — Compare the metric to an absolute value

    • Example: if traffic for a given hour is less than a threshold of 1,000, trigger alert.

  • Relative — Compare the metric to a previous period

    • Example: if traffic is down 10% or more than it was at the same time one week ago, trigger alert.

Alert Rule Priority & Severity

  • Priority: Whether you're setting up an alert rule to keep tabs on a model in a test environment or for production scenarios, Fiddler has you covered. Easily set the Alert Rule Priority to indicate the importance of any given Alert Rule. Users can select from Low, Medium, and High priorities.

  • Severity: Up to two threshold values can be specified for additional flexibility. A Critical severity threshold value is always required when setting up an Alert Rule, and a Warning threshold value is optional.

Why do we need alerts?

  • It’s not possible to manually track all metrics 24/7.

  • Sensible alerts are your first line of defense, and they are meant to warn about issues in production.

Setting up Alert Rules

To create a new alert in the Fiddler UI, click Add Alert on the Alerts tab.

  1. Fill in the Alert Rule form with basic details like alert name, project, and model.

  2. Choose an Alert Type (Traffic, Data Drift, Data Integrity, Performance, Statistic, or Custom Metric) and set up specific metrics, bin size, and columns.

  3. Define comparison methods, thresholds, and notification preferences. Click Add Alert Rule to finish.

Alert Notification options

You can select the following types of notifications for your alert.

Delete an Alert Rule

Delete an existing alert by clicking on the overflow button (⋮) on the right-hand side of any Alert Rule record and clicking Delete. To make any other changes to an Alert Rule, you will need to delete the alert and create a new one with the desired specifications.

Triggered Alert Revisions

Say goodbye to stale alerts! Triggered Alert Revisions mark a leap forward in alert intelligence, giving you the confidence to act decisively and optimize your operations.

Alerts now adapt to changing data. If new information emerges that alters an alert's severity or value, the alert automatically updates you with highlights in the user interface and revised notifications. This approach empowers you to:

  • Make informed decisions based on real-time data: No more relying on outdated or inaccurate alerts.

  • Focus on critical issues: Updated alerts prioritize the most relevant information.

Inspect Alert experience

Triggered Alert revision experience

Sample Alert Email

Here's a sample of an email that's sent if an alert is triggered:

Integrations

The Integrations tab is a read-only view of all the integrations your Admin has enabled for use. As of today, users can configure their Alert Rules to notify them via email or Pager Duty services.

Admins can add new integrations by clicking on the setting cog icon in the main navigation bar and selecting the integration tab of interest.

Pause alert notification

This feature allows users to temporarily pause and resume notifications for specific alerts without affecting their evaluation and triggering mechanisms. It enhances user experience by providing efficient notification management.\

How to Use

Using the Fiddler User Interface (UI)

  • Locate the Alert Tool: Navigate to the alert rule table and identify the desired alert.

  • Toggle Notifications:

    • Click the notification bell icon.

    • The icon updates to indicate the new state (paused or resumed).

  • Confirm Action:

    • A loading indicator and a toast notification confirm the action.

Using the Fiddler Client API

For programmatic control, use the Fiddler client API's alert-rules method with the enable_notification argument.

Note

  • No Impact on Evaluation: Pausing notifications does not affect the evaluation of alert conditions. The alert tool will continue assessing conditions and triggering alerts as usual.

In order to create and configure alerts using the Fiddler API client see .

Details: Refer to the for a complete explanation of API functionalities.

Fiddler API Client
Traffic
Statistics
Data Drift
Data Integrity
Performance
Custom Metrics
Alert Configuration with Fiddler Client
Fiddler documentation
Inspect Alert experience
Triggered Alert revision experience

Questions? to a product expert or a demo.

Need help? Contact us at .

❓
💡
Talk
request
help@fiddler.ai