Alerts
Overview
Fiddler enables users to set up alert rules to track a model's health and performance over time. Fiddler alerts also enable users to dig into triggered alerts and perform root cause analysis to discover what is causing a model to degrade. Users can set up alerts using the Fiddler API Client and the Fiddler UI as demonstrated below.
Supported Metric Types
You can get alerts for the following metrics:
The volume of traffic received by the model over time informs us of the overall system's health.
Metrics which can be used to monitor basic aggregations over columns.
Model performance can be poor if models trained on a specific dataset encounter different data in production.
Three types of violations can occur at model inference: missing feature values, type mismatches (e.g. sending a float input for a categorical feature type) or range mismatches (e.g. sending an unknown US State for a State categorical feature).
The model performance tells us how well a model performs on its task. A poorly performing model can have significant business implications.
Create your own custom metric formulas and create Alerts on those metrics giving you ultimate flexibility in alerting behavior.
Alert Configurations
Comparison Types
There are two options for alert threshold comparison:
Absolute — Compare the metric to an absolute value
Example: if traffic for a given hour is less than a threshold of 1,000, trigger alert.
Relative — Compare the metric to a previous period
Example: if traffic is down 10% or more than it was at the same time one week ago, trigger alert.
Alert Rule Priority & Severity
Priority: Whether you're setting up an alert rule to keep tabs on a model in a test environment or for production scenarios, Fiddler has you covered. Easily set the Alert Rule Priority to indicate the importance of any given Alert Rule. Users can select from Low, Medium, and High priorities.
Severity: Up to two threshold values can be specified for additional flexibility. A Critical severity threshold value is always required when setting up an Alert Rule, and a Warning threshold value is optional.
Why do we need alerts?
It’s not possible to manually track all metrics 24/7.
Sensible alerts are your first line of defense, and they are meant to warn about issues in production.
Setting up Alert Rules
To create a new alert in the Fiddler UI, click Add Alert on the Alerts tab.
Fill in the Alert Rule form with basic details like alert name, project, and model.
Choose an Alert Type (Traffic, Data Drift, Data Integrity, Performance, Statistic, or Custom Metric) and set up specific metrics, bin size, and columns.
Define comparison methods, thresholds, and notification preferences. Click Add Alert Rule to finish.
In order to create and configure alerts using the Fiddler API client see Alert Configuration with Fiddler Client.
Alert Notification options
You can select the following types of notifications for your alert.
Delete an Alert Rule
Delete an existing alert by clicking on the overflow button (⋮) on the right-hand side of any Alert Rule record and clicking Delete. To make any other changes to an Alert Rule, you will need to delete the alert and create a new one with the desired specifications.
Triggered Alert Revisions
Say goodbye to stale alerts! Triggered Alert Revisions mark a leap forward in alert intelligence, giving you the confidence to act decisively and optimize your operations.
Alerts now adapt to changing data. If new information emerges that alters an alert's severity or value, the alert automatically updates you with highlights in the user interface and revised notifications. This approach empowers you to:
Make informed decisions based on real-time data: No more relying on outdated or inaccurate alerts.
Focus on critical issues: Updated alerts prioritize the most relevant information.
Inspect Alert experience
Triggered Alert revision experience
Sample Alert Email
Here's a sample of an email that's sent if an alert is triggered:
Integrations
The Integrations tab is a read-only view of all the integrations your Admin has enabled for use. As of today, users can configure their Alert Rules to notify them via email or Pager Duty services.
Admins can add new integrations by clicking on the setting cog icon in the main navigation bar and selecting the integration tab of interest.
Pause alert notification
This feature allows users to temporarily pause and resume notifications for specific alerts without affecting their evaluation and triggering mechanisms. It enhances user experience by providing efficient notification management.\
How to Use
Using the Fiddler User Interface (UI)
Locate the Alert Tool: Navigate to the alert rule table and identify the desired alert.
Toggle Notifications:
Click the notification bell icon.
The icon updates to indicate the new state (paused or resumed).
Confirm Action:
A loading indicator and a toast notification confirm the action.
Using the Fiddler Client API
For programmatic control, use the Fiddler client API's alert-rules method with the enable_notification argument.
Details: Refer to the Fiddler documentation for a complete explanation of API functionalities.
Note
No Impact on Evaluation: Pausing notifications does not affect the evaluation of alert conditions. The alert tool will continue assessing conditions and triggering alerts as usual.
Last updated
Was this helpful?
