Fiddler allows you to set up alerts for a model or group of models in a project. View your alerts by clicking Alerts in the Monitor page.
What kinds of alerts are supported??¶
You can get alerts for the following metrics: * Drift score - predictions and all features * Data integrity - all features * Service metrics
Support for alerts about outliers is coming soon
You have two options for deciding when to be alerted:
- Compare the metric to an absolute value. For example, if traffic for a given hour is less than 1000, then alert.
- Compare the metric to a previous time period. For example, if traffic is over 10% lower than the corresponding time period a week ago, then alert.
You can set a threshold for either comparison, between 10%-90%.
Why do we need alerts?¶
- It’s not possible to manually track all metrics 24/7
- Sensible alerts are your first line of defense, and they are meant to warn about issues in production.
What should I do when I receive an alert?¶
- Click on the link in the email to go to the tab where the alert originated (e.g. Data Drift).
- In Fiddler’s Monitor UI, more information can usually be obtained from the drill down below the main chart.
- You can also examine the data in the Analyze tab. You can use SQL to slice and dice the data, and use custom visualization tools and operators to make sense of the model’s behavior on these segments.
Setting up Alerts¶
Create a new alert by clicking the
*Add Alert* button in the top right inside the Monitor pane. Configure the alert, and click
*Save* when you are done.
|Alert||Service Metrics||Data Drift|
|Type||Service metrics||Data drift|
|Metric||Traffic, Latency, Error||Prediction or Feature|
|Name||N/A||Prediction or Feature names|
|Window Size||1 hour||1 hour|
|Compare To||Previous window or Abs value||Previous window or Abs Value|
|Condition||Increased or Decreased||Drift greater or lesser|
|Threshold||10-90 in Percentage||10-90 in Percentage|
|User email||User email|
Once an alert is set, it can be viewed under ‘Alert Configuration’ page.
Delete an existing alert by clicking on the three dots under the Actions column. You can only edit the email for an alert; to make any other changes, you need to delete the alert and create a new one.
Here is a sample of the email you would receive when an alert is triggered:
- See our article on The Rise of MLOps Monitoring