Request demo

fdl.Metric

Supported Metric for different Alert Types in Alert Rules

Following is the list of metrics, with corresponding alert type and model task, for which an alert rule can be created.

Enum ValuesSupported for Alert Types
(ModelTask restriction if any)		Description
fdl.Metric.PSIfdl.AlertType.DATA_DRIFTPopulation Stability Index
fdl.Metric.JSDfdl.AlertType.DATA_DRIFTJensen–Shannon divergence
fdl.Metric.MISSING_VALUEfdl.AlertType.DATA_INTEGRITYMissing Value
fdl.Metric.TYPE_VIOLATIONfdl.AlertType.DATA_INTEGRITYType Violation
fdl.Metric.RANGE_VIOLATIONfdl.AlertType.DATA_INTEGRITYRange violation
fdl.Metric.TRAFFICfdl.AlertType.SERVICE_METRICSTraffic Count
fdl.Metric.ACCURACYfdl.AlertType.PERFORMANCE
(fdl.ModelTask.BINARY_CLASSIFICATION,
fdl.ModelTask.MULTICLASS_CLASSIFICATION)		Accuracy
fdl.Metric.RECALLfdl.AlertType.PERFORMANCE
(fdl.ModelTask.BINARY_CLASSIFICATION)		Recall
fdl.Metric.FPRfdl.AlertType.PERFORMANCE
(fdl.ModelTask.BINARY_CLASSIFICATION)		False Positive Rate
fdl.Metric.PRECISIONfdl.AlertType.PERFORMANCE
(fdl.ModelTask.BINARY_CLASSIFICATION)		Precision
fdl.Metric.TPRfdl.AlertType.PERFORMANCE
(fdl.ModelTask.BINARY_CLASSIFICATION)		True Positive Rate
fdl.Metric.AUCfdl.AlertType.PERFORMANCE
(fdl.ModelTask.BINARY_CLASSIFICATION)		Area under the ROC Curve
fdl.Metric.F1_SCOREfdl.AlertType.PERFORMANCE
(fdl.ModelTask.BINARY_CLASSIFICATION)		F1 score
fdl.Metric.ECEfdl.AlertType.PERFORMANCE
(fdl.ModelTask.BINARY_CLASSIFICATION)		Expected Calibration Error
fdl.Metric.R2fdl.AlertType.PERFORMANCE
(fdl.ModelTask.REGRESSION)		R Squared
fdl.Metric.MSEfdl.AlertType.PERFORMANCE
(fdl.ModelTask.REGRESSION)		Mean squared error
fdl.Metric.MAPEfdl.AlertType.PERFORMANCE
(fdl.ModelTask.REGRESSION)		Mean Absolute Percentage Error
fdl.Metric.WMAPEfdl.AlertType.PERFORMANCE
(fdl.ModelTask.REGRESSION)		Weighted Mean Absolute Percentage Error
fdl.Metric.MAEfdl.AlertType.PERFORMANCE
(fdl.ModelTask.REGRESSION)		Mean Absolute Error
fdl.Metric.LOG_LOSSfdl.AlertType.PERFORMANCE
(fdl.ModelTask.MULTICLASS_CLASSIFICATION)		Log Loss
fdl.Metric.MAPfdl.AlertType.PERFORMANCE
(fdl.ModelTask.RANKING)		Mean Average Precision
fdl.Metric.MEAN_NDCGfdl.AlertType.PERFORMANCE
(fdl.ModelTask.RANKING)		Normalized Discounted Cumulative Gain
import fiddler as fdl

client.add_alert_rule(
    name = "perf-gt-5prec-1hr-1d-ago",
    project_name = 'project-a',
    model_name = 'binary_classification_model-a',
    alert_type = fdl.AlertType.PERFORMANCE,
    metric = fdl.Metric.PRECISION, <----
    bin_size = fdl.BinSize.ONE_HOUR, 
    compare_to = fdl.CompareTo.TIME_PERIOD,
    compare_period = fdl.ComparePeriod.ONE_DAY,
    warning_threshold = 0.05,
    critical_threshold = 0.1,
    condition = fdl.AlertCondition.GREATER,
    priority = fdl.Priority.HIGH,
    notifications_config = notifications_config
)

[AlertRule(alert_rule_uuid='9b8711fa-735e-4a72-977c-c4c8b16543ae',
           organization_name='some_org_name',
           project_id='project-a',
           model_id='binary_classification_model-a',
           name='perf-gt-5prec-1hr-1d-ago',
           alert_type=AlertType.PERFORMANCE,
           metric=Metric.PRECISION, <---
           priority=Priority.HIGH,
           compare_to='CompareTo.TIME_PERIOD,
           compare_period=ComparePeriod.ONE_DAY,
           compare_threshold=None,
           raw_threshold=None,
           warning_threshold=0.05,
           critical_threshold=0.1,
           condition=AlertCondition.GREATER,
           bin_size=BinSize.ONE_HOUR)]