LogoLogo
👨‍💻 API Reference📣 Release Notes📺 Request a Demo
  • Introduction to Fiddler
    • Monitor, Analyze, and Protect your ML Models and Gen AI Applications
  • Fiddler Doc Chatbot
  • First Steps
    • Getting Started With Fiddler Guardrails
    • Getting Started with LLM Monitoring
    • Getting Started with ML Model Observability
  • Tutorials & Quick Starts
    • LLM and GenAI
      • LLM Evaluation - Compare Outputs
      • LLM Monitoring - Simple
    • Fiddler Free Guardrails
      • Guardrails - Quick Start Guide
      • Guardrails - Faithfulness
      • Guardrails - Safety
      • Guardrails FAQ
    • ML Observability
      • ML Monitoring - Simple
      • ML Monitoring - NLP Inputs
      • ML Monitoring - Class Imbalance
      • ML Monitoring - Model Versions
      • ML Monitoring - Ranking
      • ML Monitoring - Regression
      • ML Monitoring - Feature Impact
      • ML Monitoring - CV Inputs
  • Glossary
    • Product Concepts
      • Baseline
      • Custom Metric
      • Data Drift
      • Embedding Visualization
      • Fiddler Guardrails
      • Fiddler Trust Service
      • LLM and GenAI Observability
      • Metric
      • Model Drift
      • Model Performance
      • ML Observability
      • Trust Score
  • Product Guide
    • LLM Application Monitoring & Protection
      • LLM-Based Metrics
      • Embedding Visualizations for LLM Monitoring and Analysis
      • Selecting Enrichments
      • Enrichments (Private Preview)
      • Guardrails for Proactive Application Protection
    • Optimize Your ML Models and LLMs with Fiddler's Comprehensive Monitoring
      • Alerts
      • Package-Based Alerts (Private Preview)
      • Class Imbalanced Data
      • Enhance ML and LLM Insights with Custom Metrics
      • Data Drift: Monitor Model Performance Changes with Fiddler's Insights
      • Ensuring Data Integrity in ML Models And LLMs
      • Embedding Visualization With UMAP
      • Fiddler Query Language
      • Model Versions
      • How to Effectively Use the Monitoring Chart UI
      • Performance Tracking
      • Model Segments: Analyze Cohorts for Performance Insights and Bias Detection
      • Statistics
      • Monitoring ML Model and LLM Traffic
      • Vector Monitoring
    • Enhance Model Insights with Fiddler's Slice and Explain
      • Events Table in RCA
      • Feature Analytics Creation
      • Metric Card Creation
      • Performance Charts Creation
      • Performance Charts Visualization
    • Master AI Monitoring: Create, Customize, and Compare Dashboards
      • Creating Dashboards
      • Dashboard Interactions
      • Dashboard Utilities
    • Adding and Editing Models in the UI
      • Model Editor UI
      • Model Schema Editing Guide
    • Fairness
    • Explainability
      • Model: Artifacts, Package, Surrogate
      • Global Explainability: Visualize Feature Impact and Importance in Fiddler
      • Point Explainability
      • Flexible Model Deployment
        • On Prem Manual Flexible Model Deployment XAI
  • Technical Reference
    • Python Client API Reference
    • Python Client Guides
      • Installation and Setup
      • Model Onboarding
        • Create a Project and Onboard a Model for Observation
        • Model Task Types
        • Customizing your Model Schema
        • Specifying Custom Missing Value Representations
      • Publishing Inference Data
        • Creating a Baseline Dataset
        • Publishing Batches Of Events
        • Publishing Ranking Events
        • Streaming Live Events
        • Updating Already Published Events
        • Deleting Events From Fiddler
      • Creating and Managing Alerts
      • Explainability Examples
        • Adding a Surrogate Model
        • Uploading Model Artifacts
        • Updating Model Artifacts
        • ML Framework Examples
          • Scikit Learn
          • Tensorflow HDF5
          • Tensorflow Savedmodel
          • Xgboost
        • Model Task Examples
          • Binary Classification
          • Multiclass Classification
          • Regression
          • Uploading A Ranking Model Artifact
    • Integrations
      • Data Pipeline Integrations
        • Airflow Integration
        • BigQuery Integration
        • Integration With S3
        • Kafka Integration
        • Sagemaker Integration
        • Snowflake Integration
      • ML Platform Integrations
        • Integrate Fiddler with Databricks for Model Monitoring and Explainability
        • Datadog Integration
        • ML Flow Integration
      • Alerting Integrations
        • PagerDuty Integration
    • Comprehensive REST API Reference
      • Projects REST API Guide
      • Model REST API Guide
      • File Upload REST API Guide
      • Custom Metrics REST API Guide
      • Segments REST API Guide
      • Baselines REST API Guide
      • Jobs REST API Guide
      • Alert Rules REST API Guide
      • Environments REST API Guide
      • Explainability REST API Guide
      • Server Info REST API Guide
      • Events REST API Guide
      • Fiddler Trust Service REST API Guide
    • Fiddler Free Guardrails Documentation
  • Configuration Guide
    • Authentication & Authorization
      • Adding Users
      • Overview of Role-Based Access Control
      • Email Authentication
      • Okta Integration
      • SSO with Azure AD
      • Ping Identity SAML SSO Integration
      • Mapping LDAP Groups & Users to Fiddler Teams
    • Application Settings
    • Supported Browsers
  • History
    • Release Notes
    • Python Client History
    • Compatibility Matrix
    • Product Maturity Definitions
Powered by GitBook

© 2024 Fiddler Labs, Inc.

On this page
  • Overview
  • Google Sheet Components
  • Generating the Recipe YAML File
  • Downloading and Integrating the YAML File
  • Configuring and Running the Notebook
  • Final Notes

Was this helpful?

  1. Product Guide
  2. Optimize Your ML Models and LLMs with Fiddler's Comprehensive Monitoring

Package-Based Alerts (Private Preview)

Private Preview Notice This feature is in early access, and we are actively enhancing it. We recommend testing on a single model first before enabling it more broadly. Please reach out to the Fiddler team for guidance.

Private Preview Limitations

  • This feature is not enabled by default and requires manual configuration.

  • Changes to existing alerts may require manual intervention.

  • Auto-thresholding logic is still being refined based on real-world feedback.

The Package-Based Alerts feature streamlines the process of configuring and generating alert rules, significantly reducing manual effort. By leveraging a Google Sheet to input key data and generate a recipe YAML file, this feature enables efficient and accurate alert rule creation. This ensures more relevant alerts, enhancing monitoring capabilities and allowing for proactive issue resolution.

Overview

The Packaged Based Alerts feature is designed to facilitate the configuration and generation of alert rules with minimal manual intervention. The process involves:

  1. Populating a multi-sheet Google Sheet with necessary data.

  2. Generating a recipe YAML file by clicking a dedicated button.

  3. Downloading and referencing the YAML file in an alert rules notebook.

  4. Updating configuration settings before running the notebook to generate the alerts.

Google Sheet Components

The feature uses several interconnected sheets within a single Google Sheet document:

1. Main Sheet

  • Purpose: Central dashboard for creating the recipe YAML file.

  • Functionality: Hosts the “Generate Alert Recipe” button that triggers the script.

2. Recipe Parameters

  • Purpose: Input key details such as:

    • Project Name

    • Model Name

    • Feature Priority Lists: Define:

      • High-priority features (for high-priority alerts)

      • Medium-priority features (for medium-priority alerts)

3. Mapping Sheets

  • Examples: Metric ID Map, Metric Type Map, Threshold Type Map, etc.

  • Purpose: Provide dropdown values and mappings (e.g., monitor types, threshold algorithms like standard deviation mapped to a specific threshold type).

  • Note: These maps should not be modified unless you need to add new monitor types or update corresponding metrics.

Generating the Recipe YAML File

Follow these steps to generate the YAML file that will be used to create alert rules:

  1. Enter Data in Recipe Parameters:

    • Provide the project name, model name, and list of high-priority features.

    • The remaining features will automatically be treated as medium priority.

  2. Click the “Generate Alert Recipe” Button:

    • Located on the main sheet.

    • Clicking this button triggers an integrated script.

    • The script collects data from all sheets (including mapping data) and generates alert_recipes.yaml.

  3. Script Execution & Permissions:

    • When you click the button, you may be prompted to log in and grant permission for the script to access your Google Drive.

    • Once permissions are granted, the script runs and creates the YAML file in your Google Drive.

Downloading and Integrating the YAML File

  1. Download the YAML File:

    • After generation, use the provided link or navigate to your Google Drive to download the YAML file.

    • Save the file in an easy-to-find location, as you'll need to refer to its path in the notebook.

  2. Reference the File in the Notebook:

    • Open the alert rules notebook.

    • Update the file path in the notebook configuration to point to the downloaded alert_recipes.yaml.

Configuring and Running the Notebook

Before executing the notebook to generate alert rules, perform the following configuration:

  1. Update the Configuration File:

    • Use the provided config.py example as a starting point.

    • Create your configuration file (e.g., copy config.py.example to config.py).

    • Fill in the necessary details:

      • Token

      • Path to the alert_recipes.yaml file

    • Note: Notification channel settings are optional for this process.

  2. Execute the Notebook:

    • With the configuration in place, run the notebook.

    • The notebook will parse the alert_recipe.yaml file (which includes your project name, model name, and alert configurations) and generate the corresponding alert rules.

Final Notes

  • Permission Flow: Ensure you complete any Google Drive authentication prompts during script execution.

  • Mapping Integrity: Do not modify the mapping sheets unless you are intentionally adding or updating monitor types.

  • Troubleshooting:

    • If alert rules are not generated as expected, verify:

      • Data accuracy in the Google Sheet.

      • The correct path and token entries in your configuration file.

      • You have provided the necessary permissions for the script.

PreviousAlertsNextClass Imbalanced Data

Last updated 1 month ago

Was this helpful?


Questions? to a product expert or a demo.

Need help? Contact us at .

❓
💡
Talk
request
help@fiddler.ai