CustomMetric

CustomMetric

CustomMetric

Custom metric for monitoring business-specific and domain-specific KPIs.

CustomMetric enables creation of user-defined metrics that calculate specific values from model data using SQL-like expressions. Custom metrics extend Fiddler’s built-in monitoring capabilities to support business requirements, domain-specific quality measures, and complex performance indicators.

Inherits all attributes from :class:`~fiddler.entities.CustomExpression`.

Example

# Business conversion rate metric
conversion_rate = CustomMetric(
    name=”weekly_conversion_rate”,
    model_id=model.id,
    definition=”SUM(CASE WHEN prediction_score > 0.7 AND converted = 1 THEN 1 ELSE 0 END) / SUM(CASE WHEN prediction_score > 0.7 THEN 1 ELSE 0 END)”,
    description=”Conversion rate for high-confidence predictions”
).create()

# Data quality metric
missing_rate = CustomMetric(
    name=”feature_missing_rate”,
    model_id=model.id,
    definition=”SUM(CASE WHEN income IS NULL THEN 1 ELSE 0 END) / COUNT(\*)”,
    description=”Percentage of records with missing income values”
).create()

# Fairness metric
fairness_metric = CustomMetric(
    name=”demographic_parity”,
    model_id=model.id,
    definition=”ABS(AVG(CASE WHEN gender = ‘M’ THEN prediction_score ELSE NULL END) - AVG(CASE WHEN gender = ‘F’ THEN prediction_score ELSE NULL END))”,
    description=”Demographic parity difference between gender groups”
).create()

# Use in alert rule
alert_rule = AlertRule(
    name=”conversion_rate_alert”,
    model_id=model.id,
    metric_id=conversion_rate.id,
    priority=Priority.HIGH,
    compare_to=CompareTo.TIME_PERIOD,
    condition=AlertCondition.LT,
    bin_size=BinSize.DAY,
    critical_threshold=0.15,  # Alert if conversion drops below 15%
    compare_bin_delta=7
).create()

Custom metrics are calculated during data ingestion and monitoring cycles. Complex expressions may impact performance, so optimize for efficiency. Test expressions thoroughly before using in production alert rules.

__init__(name, model_id, definition, description=None)

Construct a custom metric instance.

Return type: None