Supported Metric for different Alert Types in Alert Rules
Following is the list of metrics, with corresponding alert type and model task, for which an alert rule can be created.
Enum Values | Supported for Alert Types (ModelTask restriction if any) | Description |
---|---|---|
fdl.Metric.PSI | fdl.AlertType.DATA_DRIFT | Population Stability Index |
fdl.Metric.JSD | fdl.AlertType.DATA_DRIFT | Jensen–Shannon divergence |
fdl.Metric.MISSING_VALUE | fdl.AlertType.DATA_INTEGRITY | Missing Value |
fdl.Metric.TYPE_VIOLATION | fdl.AlertType.DATA_INTEGRITY | Type Violation |
fdl.Metric.RANGE_VIOLATION | fdl.AlertType.DATA_INTEGRITY | Range violation |
fdl.Metric.TRAFFIC | fdl.AlertType.SERVICE_METRICS | Traffic Count |
fdl.Metric.ACCURACY | fdl.AlertType.PERFORMANCE (fdl.ModelTask.BINARY_CLASSIFICATION, fdl.ModelTask.MULTICLASS_CLASSIFICATION) | Accuracy |
fdl.Metric.RECALL | fdl.AlertType.PERFORMANCE (fdl.ModelTask.BINARY_CLASSIFICATION) | Recall |
fdl.Metric.FPR | fdl.AlertType.PERFORMANCE (fdl.ModelTask.BINARY_CLASSIFICATION) | False Positive Rate |
fdl.Metric.PRECISION | fdl.AlertType.PERFORMANCE (fdl.ModelTask.BINARY_CLASSIFICATION) | Precision |
fdl.Metric.TPR | fdl.AlertType.PERFORMANCE (fdl.ModelTask.BINARY_CLASSIFICATION) | True Positive Rate |
fdl.Metric.AUC | fdl.AlertType.PERFORMANCE (fdl.ModelTask.BINARY_CLASSIFICATION) | Area under the ROC Curve |
fdl.Metric.F1_SCORE | fdl.AlertType.PERFORMANCE (fdl.ModelTask.BINARY_CLASSIFICATION) | F1 score |
fdl.Metric.ECE | fdl.AlertType.PERFORMANCE (fdl.ModelTask.BINARY_CLASSIFICATION) | Expected Calibration Error |
fdl.Metric.R2 | fdl.AlertType.PERFORMANCE (fdl.ModelTask.REGRESSION) | R Squared |
fdl.Metric.MSE | fdl.AlertType.PERFORMANCE (fdl.ModelTask.REGRESSION) | Mean squared error |
fdl.Metric.MAPE | fdl.AlertType.PERFORMANCE (fdl.ModelTask.REGRESSION) | Mean Absolute Percentage Error |
fdl.Metric.WMAPE | fdl.AlertType.PERFORMANCE (fdl.ModelTask.REGRESSION) | Weighted Mean Absolute Percentage Error |
fdl.Metric.MAE | fdl.AlertType.PERFORMANCE (fdl.ModelTask.REGRESSION) | Mean Absolute Error |
fdl.Metric.LOG_LOSS | fdl.AlertType.PERFORMANCE (fdl.ModelTask.MULTICLASS_CLASSIFICATION) | Log Loss |
fdl.Metric.MAP | fdl.AlertType.PERFORMANCE (fdl.ModelTask.RANKING) | Mean Average Precision |
fdl.Metric.MEAN_NDCG | fdl.AlertType.PERFORMANCE (fdl.ModelTask.RANKING) | Normalized Discounted Cumulative Gain |
import fiddler as fdl
client.add_alert_rule(
name = "perf-gt-5prec-1hr-1d-ago",
project_name = 'project-a',
model_name = 'binary_classification_model-a',
alert_type = fdl.AlertType.PERFORMANCE,
metric = fdl.Metric.PRECISION, <----
bin_size = fdl.BinSize.ONE_HOUR,
compare_to = fdl.CompareTo.TIME_PERIOD,
compare_period = fdl.ComparePeriod.ONE_DAY,
warning_threshold = 0.05,
critical_threshold = 0.1,
condition = fdl.AlertCondition.GREATER,
priority = fdl.Priority.HIGH,
notifications_config = notifications_config
)
[AlertRule(alert_rule_uuid='9b8711fa-735e-4a72-977c-c4c8b16543ae',
organization_name='some_org_name',
project_id='project-a',
model_id='binary_classification_model-a',
name='perf-gt-5prec-1hr-1d-ago',
alert_type=AlertType.PERFORMANCE,
metric=Metric.PRECISION, <---
priority=Priority.HIGH,
compare_to='CompareTo.TIME_PERIOD,
compare_period=ComparePeriod.ONE_DAY,
compare_threshold=None,
raw_threshold=None,
warning_threshold=0.05,
critical_threshold=0.1,
condition=AlertCondition.GREATER,
bin_size=BinSize.ONE_HOUR)]