Package based Alerts (Private preview)

Private Preview Notice You are getting early access to the feature, and we are actively enhancing it. We recommend testing on a single model first before enabling it more broadly. Please reach out to the Fiddler team for guidance.

Private Preview Limitations

• This feature is not enabled by default and requires manual configuration.

• Changes to existing alerts may require manual intervention.

• Auto-thresholding logic is still being refined based on real-world feedback.

This guide provides step-by-step instructions for generating alert rules using the Packaged Based Alerts feature. The feature leverages a Google Sheet to create a recipe YAML file, which is then used by a notebook to generate alert rules.

---

1. Overview

The Packaged Based Alerts feature is designed to help you configure and generate alert rules with minimal manual intervention. The process involves:

1. Populating a multi-sheet Google Sheet with necessary data.

2. Generating a recipe YAML file by clicking a dedicated button.

3. Downloading and referencing the YAML file in an alert rules notebook.

4. Updating configuration settings before running the notebook to generate the alerts.

---

2. Google Sheet Components

The feature uses several interconnected sheets within a single Google Sheet document:

a. Main Sheet

• Purpose: Central dashboard for creating the recipe YAML file.

• Functionality: Hosts the “Generate Alert Recipe” button that triggers the script.

b. Recipe Parameters

• Purpose: Input key details such as:

  • Project Name

  • Model Name

  • Feature Priority Lists: Define:

    • High-priority features (for high-priority alerts)

    • Medium-priority features (for medium-priority alerts)

c. Mapping Sheets

• Examples: Metric ID Map, Metric Type Map, Threshold Type Map, etc.

• Purpose: Provide dropdown values and mappings (e.g., monitor types, threshold algorithms like standard deviation mapped to a specific threshold type).

• Note: These maps should not be modified unless you need to add new monitor types or update corresponding metrics.

---

3. Generating the Recipe YAML File

Follow these steps to generate the YAML file that will be used to create alert rules:

1. Enter Data in Recipe Parameters:

   • Provide the project name, model name, and list of high-priority features.

   • The remaining features will automatically be treated as medium priority.

2. Click the “Generate Alert Recipe” Button:

   • Located on the main sheet.

   • Clicking this button triggers an integrated script.

   • The script collects data from all sheets (including mapping data) and generates alert_recipes.yaml.

3. Script Execution & Permissions:

   • When you click the button, you may be prompted to log in and grant permission for the script to access your Google Drive.

   • Once permissions are granted, the script runs and creates the YAML file in your Google Drive.

---

4. Downloading and Integrating the YAML File

1. Download the YAML File:

   • After generation, use the provided link or navigate to your Google Drive to download the YAML file.

   • Save the file in an easy-to-find location, as you'll need to refer to its path in the notebook.

2. Reference the File in the Notebook:

   • Open the alert rules notebook.

   • Update the file path in the notebook configuration to point to the downloaded alert_recipes.yaml.

---

5. Configuring and Running the Notebook

Before executing the notebook to generate alert rules, perform the following configuration:

1. Update the Configuration File:

   • Use the provided config.py example as a starting point.

   • Create your configuration file (e.g., copy config.py.example to config.py).

   • Fill in the necessary details:

     • Token

     • Path to the alert_recipes.yaml file

   • Note: Notification channel settings are optional for this process.

2. Execute the Notebook:

   • With the configuration in place, run the notebook.

   • The notebook will parse the alert_recipe.yaml file (which includes your project name, model name, and alert configurations) and generate the corresponding alert rules.

---

6. Final Notes

• Permission Flow: Ensure you complete any Google Drive authentication prompts during script execution.

• Mapping Integrity: Do not modify the mapping sheets unless you are intentionally adding or updating monitor types.

• Troubleshooting:

  • If alert rules are not generated as expected, verify:

    • Data accuracy in the Google Sheet.

    • The correct path and token entries in your configuration file.

    • You have provided the necessary permissions for the script.

For further assistance, please reach out to the Fiddler support team via the Slack channel.