Alerts
Overview
Fiddler enables users to set up alert rules to track a model's health and performance over time. Fiddler alerts also enable users to dig into triggered alerts and perform root cause analysis to discover what is causing a model to degrade. Users can set up alerts using both the Fiddler UI and the Fiddler API Client.
Supported Metric Types
You can get alerts for the following metrics:
The volume of traffic received by the model over time informs us of the overall system's health.
Metrics which can be used to monitor basic aggregations over columns.
Data Drift β Predictions and all features
Model performance can be poor if models trained on a specific dataset encounter different data in production.
Data Integrity β All features
Three types of violations can occur at model inference: missing feature values, type mismatches (e.g. sending a float input for a categorical feature type) or range mismatches (e.g. sending an unknown US State for a State categorical feature).
The model performance tells us how well a model performs on its task. A poorly performing model can have significant business implications.
Alert Configurations
Comparison Types
You have two options for deciding when to be alerted:
Absolute β Compare the metric to an absolute value
e.g. if traffic for a given hour is less than 1000, then alert.
Relative β Compare the metric to a previous period
e.g. if traffic is down 10% or more than it was at the same time one week ago, then alert.
Alert Rule Priority & Severity
Whether you're setting up an alert rule to keep tabs on a model in a test environment, or data for production scenarios, Fiddler has you covered. Easily set the Alert Rule Priority to indicate the importance of any given Alert Rule. Users can select from Low, Medium, and High priorities.
Users can now specify up to two threshold values, Critical and Warning severities for additional flexibility. Critical severity is always required when setting up an Alert Rule, but a Warning can be optionally set as well.
Why do we need alerts?
Itβs not possible to manually track all metrics 24/7.
Sensible alerts are your first line of defense, and they are meant to warn about issues in production.
What should I do when I receive an alert?
Click on the link in the email to go to the tab where the alert originated (e.g. Data Drift).
Under the Monitoring tab, more information can be obtained from the drill below the main chart.
You can also examine the data in the Analyze tab. You can use SQL to slice and dice the data and use custom visualization tools and operators to make sense of the modelβs behaviour within the time range under consideration.
Sample Alert Email
Here's a sample of an email that's sent if an alert is triggered:
Integrations
Fiddler supports the following alert notification integrations:
Email
Slack
PagerDuty
Webhook