Monitoring & Alerting Overview - Fiddler Documentation

Route Fiddler AI observability alerts to your existing incident management and communication tools. Integrate with observability platforms, on-call systems, and team collaboration tools to ensure AI issues are detected, triaged, and resolved quickly.

Why Alert Integration Matters

AI models fail in unique ways—drift, data quality issues, performance degradation, safety violations. Fiddler’s alert integrations ensure your team responds immediately:

Unified Incident Management - AI alerts flow into the same systems as infrastructure alerts
Faster Response Times - On-call engineers notified via existing escalation policies
Context-Rich Alerts - Model context, affected predictions, and root cause analysis included
Reduced Alert Fatigue - Intelligent grouping and deduplication across tools
Automated Remediation - Trigger workflows to rollback models or scale resources

Integration Categories

📊 Observability Platforms

Send Fiddler metrics and alerts to enterprise observability platforms for unified monitoring. Supported Platforms:

Datadog - Application performance monitoring and infrastructure observability ✓ GA

Common Use Cases:

Correlate AI model issues with infrastructure metrics
Build unified dashboards combining Fiddler + infrastructure data
Use Datadog’s anomaly detection on Fiddler metrics
Alert on compound conditions (model drift + high latency)

🚨 Incident Management

Connect alerts to on-call systems for immediate engineer notification. Supported Platforms:

PagerDuty - Incident management and on-call scheduling ✓ GA

Common Use Cases:

Page on-call ML engineers for critical model failures
Escalate unresolved AI incidents automatically
Track MTTR (Mean Time To Resolution) for model issues
Integrate with incident runbooks and response workflows

💬 Team Collaboration

Send alerts to team communication tools for visibility and collaboration. Supported Platforms:

Slack - Team messaging and collaboration ✓ GA (Coming Soon)
Microsoft Teams - Enterprise communication platform ✓ GA (Coming Soon)

Common Use Cases:

Notify ML team channel when drift is detected
Alert data science team on data quality issues
Share model performance reports automatically
Collaborative incident triage in team channels

Observability Platform Integrations

Datadog

Integrate Fiddler with Datadog for unified application and AI monitoring. Why Datadog + Fiddler:

Unified Dashboards - Combine infrastructure, application, and AI model metrics
Correlated Alerts - Alert on compound conditions (e.g., “high model drift + high API latency”)
Service Map Integration - See model health in Datadog service dependency graphs
Anomaly Detection - Leverage Datadog’s ML-based alerting on Fiddler metrics

Key Features:

Metric Export - Send Fiddler drift, performance, and data quality metrics to Datadog
Event Streaming - Stream model events (predictions, drift detections) as Datadog events
Alert Forwarding - Route Fiddler alerts to Datadog for unified incident management
Tag Propagation - Maintain consistent tagging across platforms (model, environment, team)

Status: ✓ GA - Production-ready Get Started with Datadog → Quick Start:

from fiddler import FiddlerClient

client = FiddlerClient(api_key="fid_...")

# Configure Datadog integration
client.add_datadog_integration(
    api_key="datadog_api_key",
    app_key="datadog_app_key",
    site="datadoghq.com",  # or datadoghq.eu

    # Metric export configuration
    export_metrics=True,
    metric_prefix="fiddler.model.",
    tags=["env:production", "team:ml-platform"],

    # Event export configuration
    export_events=True,
    event_priority="normal"  # or "low"
)

Example Datadog Dashboard:

# Datadog dashboard combining infrastructure + AI metrics
widgets:
  - title: "Model Latency vs API Latency"
    type: timeseries
    queries:
      - metric: fiddler.model.latency.p95
        scope: model:fraud_detector
      - metric: trace.flask.request.duration.p95
        scope: service:fraud-api

  - title: "Model Drift Detection"
    type: query_value
    queries:
      - metric: fiddler.model.drift.score
        aggregation: max
    conditional_formats:
      - comparator: ">"
        value: 0.1
        palette: "red"

Incident Management Integrations

PagerDuty

Route critical AI alerts to on-call engineers via PagerDuty. Why PagerDuty + Fiddler:

On-Call Escalation - Page the right ML engineer based on escalation policies
Incident Deduplication - Prevent alert storms from related model issues
Incident Timeline - Track when AI issues were detected, acknowledged, resolved
Postmortem Integration - Include model context in incident reports

Key Features:

Severity Mapping - Map Fiddler alert criticality to PagerDuty severity levels
Service Integration - Associate alerts with PagerDuty services (e.g., “Fraud Detection Service”)
Custom Payloads - Include model metadata, drift scores, affected predictions
Bidirectional Updates - Acknowledge/resolve incidents in PagerDuty or Fiddler

Status: ✓ GA - Production-ready Get Started with PagerDuty → Quick Start:

from fiddler import FiddlerClient

client = FiddlerClient(api_key="fid_...")

# Configure PagerDuty integration
client.add_pagerduty_integration(
    integration_key="pagerduty_integration_key",
    service_name="ML Models - Production",

    # Alert routing rules
    severity_mapping={
        "critical": "critical",  # Fiddler → PagerDuty severity
        "high": "error",
        "medium": "warning",
        "low": "info"
    }
)

# Create alert with PagerDuty notification
client.create_alert(
    name="Critical Model Drift",
    project="fraud-detection",
    model="fraud_detector_v3",
    metric="drift_score",
    threshold=0.15,
    severity="critical",
    notification_channels=["pagerduty"]
)

Example PagerDuty Incident:

{
  "incident_key": "fiddler_drift_fraud_detector_v3",
  "type": "trigger",
  "description": "High drift detected on fraud_detector_v3",
  "details": {
    "model": "fraud_detector_v3",
    "project": "fraud-detection",
    "drift_score": 0.23,
    "affected_features": ["transaction_amount", "merchant_category"],
    "time_window": "2024-11-10 14:00 - 14:30 UTC",
    "fiddler_url": "https://app.fiddler.ai/projects/fraud-detection/models/fraud_detector_v3/drift"
  },
  "client": "Fiddler AI Observability",
  "client_url": "https://app.fiddler.ai"
}

Team Collaboration Integrations

Slack (Coming Soon)

Planned Features:

Channel Notifications - Post alerts to team Slack channels
Interactive Messages - Acknowledge, snooze, or resolve alerts from Slack
Scheduled Reports - Daily/weekly model performance summaries
Threaded Discussions - Collaborate on incident resolution in threads

Example Configuration:

# Future Slack integration API
client.add_slack_integration(
    webhook_url="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXX",
    channel="#ml-alerts",
    username="Fiddler",
    icon_emoji=":robot_face:"
)

Microsoft Teams (Coming Soon)

Planned Features:

Adaptive Cards - Rich, interactive alert notifications
Team Channels - Route alerts to relevant team channels
Bot Commands - Query model status from Teams chat
Integration with Workflows - Trigger Teams workflows on alerts

Alert Routing Patterns

Pattern 1: Severity-Based Routing

Route alerts to different channels based on severity:

# Critical alerts → PagerDuty (page on-call)
client.create_alert(
    name="Model Offline",
    severity="critical",
    notification_channels=["pagerduty"]
)

# High alerts → Slack (notify team)
client.create_alert(
    name="High Drift Detected",
    severity="high",
    notification_channels=["slack"]
)

# Medium alerts → Datadog (track as events)
client.create_alert(
    name="Minor Data Quality Issue",
    severity="medium",
    notification_channels=["datadog"]
)

Pattern 2: Team-Based Routing

Different teams get different alerts:

# ML Engineering team
client.create_alert(
    name="Model Performance Degradation",
    project="fraud-detection",
    notification_channels=["slack"],
    slack_channel="#ml-engineering",
    tags=["team:ml-engineering"]
)

# Data Engineering team
client.create_alert(
    name="Data Pipeline Failure",
    project="fraud-detection",
    notification_channels=["slack"],
    slack_channel="#data-engineering",
    tags=["team:data-engineering"]
)

# On-Call team (escalation)
client.create_alert(
    name="Critical Model Failure",
    project="fraud-detection",
    notification_channels=["pagerduty"],
    pagerduty_service="ml-models-production",
    tags=["team:on-call"]
)

Pattern 3: Composite Alerting

Alert on compound conditions across multiple platforms:

# Example: Alert if BOTH model drift is high AND API latency is high
# (Indicates model update needed + production impact)

# Configure in Datadog (using Fiddler + Datadog metrics)
datadog_composite_alert = {
    "name": "Model Drift + High Latency",
    "query": """
        avg(last_5m):fiddler.model.drift.score{model:fraud_detector} > 0.1
        AND
        avg(last_5m):trace.flask.request.duration.p95{service:fraud-api} > 500
    """,
    "message": "@pagerduty-ml-oncall Model drift detected with production latency impact",
    "tags": ["model:fraud_detector", "alert:composite"]
}

Metric Export Patterns

Export Fiddler Metrics to Datadog

from fiddler import FiddlerClient

client = FiddlerClient(api_key="fid_...")

# Configure metric export
client.configure_metric_export(
    destination="datadog",
    metrics=[
        "drift_score",
        "data_quality_score",
        "prediction_latency_p95",
        "prediction_count",
        "error_rate"
    ],
    export_interval=60,  # seconds
    tags=["source:fiddler", "env:production"]
)

Exported Metrics:

fiddler.model.drift.score - Overall drift score (0-1)
fiddler.model.drift.feature.<feature_name> - Per-feature drift
fiddler.model.performance.<metric> - Model performance metrics
fiddler.model.data_quality.score - Data quality score
fiddler.model.predictions.count - Prediction volume
fiddler.model.predictions.latency - Prediction latency percentiles

Query Fiddler Metrics in Datadog

# Datadog Metrics Query Language (MQL)
queries = {
    "high_drift_models": """
        avg:fiddler.model.drift.score{*} by {model} > 0.1
    """,
    "low_performance_models": """
        avg:fiddler.model.performance.accuracy{*} by {model} < 0.85
    """,
    "high_volume_models": """
        sum:fiddler.model.predictions.count{*} by {model}.as_count()
    """
}

Alert Lifecycle Management

Alert States

Fiddler alerts transition through these states:

Triggered → Acknowledged → Investigating → Resolved → Closed
                ↓                                      ↑
            Escalated --------------------------------┘

Synchronization with External Tools:

PagerDuty: Bidirectional state sync (acknowledge, resolve)
Datadog: Event-based updates
Slack: Interactive message updates

Alert Deduplication

Prevent alert storms with intelligent deduplication:

# Configure deduplication rules
client.configure_alert_deduplication(
    project="fraud-detection",
    model="fraud_detector_v3",

    # Group related alerts
    grouping_window=300,  # seconds
    grouping_keys=["model", "metric"],

    # Suppress similar alerts
    suppression_window=3600,  # seconds
    max_alerts_per_window=3
)

Custom Webhook Integrations

For platforms not natively supported, use generic webhooks:

# Send alerts to any webhook endpoint
client.add_webhook_integration(
    name="custom-alerting-system",
    url="https://your-system.com/webhook/fiddler",
    method="POST",
    headers={
        "Authorization": "Bearer your-token",
        "Content-Type": "application/json"
    },
    payload_template={
        "alert_name": "{{alert.name}}",
        "model": "{{model.name}}",
        "severity": "{{alert.severity}}",
        "timestamp": "{{alert.timestamp}}",
        "details": "{{alert.details}}"
    }
)

# Use webhook in alerts
client.create_alert(
    name="Custom Alert",
    notification_channels=["webhook:custom-alerting-system"]
)

Webhook Payload Example:

{
  "alert_name": "High Drift Detected",
  "model": "fraud_detector_v3",
  "project": "fraud-detection",
  "severity": "high",
  "timestamp": "2024-11-10T14:32:01Z",
  "metric": "drift_score",
  "current_value": 0.23,
  "threshold": 0.15,
  "fiddler_url": "https://app.fiddler.ai/projects/fraud-detection/models/fraud_detector_v3",
  "affected_features": ["transaction_amount", "merchant_category"],
  "recommended_actions": [
    "Investigate feature distribution changes",
    "Consider model retraining",
    "Review data pipeline for issues"
  ]
}

Monitoring Integration Health

Track Integration Status

# Check integration health
integrations = client.list_integrations()
for integration in integrations:
    status = client.get_integration_health(integration.name)
    print(f"{integration.name}: {status.status}")
    print(f"  Last successful send: {status.last_success}")
    print(f"  Failed alerts (24h): {status.failed_count}")

Alerts on Integration Failures

# Alert if alert delivery fails
client.create_meta_alert(
    name="Alert Delivery Failure",
    trigger="integration_failure",
    integration="pagerduty",
    threshold=3,  # failures
    time_window=3600,  # seconds
    notification_channels=["email"]  # use different channel!
)

Best Practices

Alert Fatigue Prevention

1. Use Appropriate Severity Levels:

# Reserve "critical" for page-worthy issues
severity_guidelines = {
    "critical": "Model offline, safety violations, production outages",
    "high": "Significant drift, performance degradation",
    "medium": "Minor data quality issues, gradual drift",
    "low": "Informational, trend observations"
}

2. Implement Alert Throttling:

client.create_alert(
    name="Drift Detection",
    threshold=0.1,
    evaluation_window=300,  # 5 minutes
    cooldown_period=3600,   # Don't re-alert for 1 hour
    max_alerts_per_day=5    # Cap daily alerts
)

3. Use Alert Grouping:

# Group related alerts into single notification
client.configure_alert_grouping(
    group_name="Feature Drift Alerts",
    alerts=["drift_feature_1", "drift_feature_2", "drift_feature_3"],
    send_as_digest=True,
    digest_interval=1800  # 30 minutes
)

Incident Response Runbooks

Include runbook links in alert payloads:

client.create_alert(
    name="High Model Drift",
    metadata={
        "runbook_url": "https://wiki.company.com/ml/runbooks/drift-response",
        "on_call_team": "ml-platform",
        "escalation_policy": "ml-models-production",
        "sla": "30 minutes to acknowledge, 2 hours to investigate"
    }
)

Security & Compliance

Secure Credential Management

Never hardcode credentials:

import os

# Use environment variables
client.add_pagerduty_integration(
    integration_key=os.environ['PAGERDUTY_INTEGRATION_KEY']
)

# Or use secret management systems
from secret_manager import get_secret
client.add_datadog_integration(
    api_key=get_secret('datadog-api-key'),
    app_key=get_secret('datadog-app-key')
)

Alert Data Privacy

PII Redaction in Alerts:

client.configure_alert_privacy(
    redact_pii=True,
    pii_fields=["email", "ssn", "phone_number"],
    redaction_string="[REDACTED]"
)

Audit Logging

Track alert delivery:

# Query alert delivery audit log
audit_log = client.get_alert_audit_log(
    start_time="2024-11-01",
    end_time="2024-11-10",
    include_payload=True
)

for entry in audit_log:
    print(f"Alert: {entry.alert_name}")
    print(f"Delivered to: {entry.channel}")
    print(f"Status: {entry.status}")
    print(f"Timestamp: {entry.timestamp}")

Troubleshooting

Common Issues

Alerts Not Delivered:

Verify integration credentials are valid and not expired
Check network connectivity from Fiddler to external platform
Ensure webhook endpoints are reachable (not blocked by firewall)
Validate alert thresholds are actually being triggered

Duplicate Alerts:

Enable alert deduplication with appropriate time windows
Check if multiple notification channels are configured
Verify integration isn’t configured twice

Missing Alert Context:

Ensure include_context=True in alert configuration
Check payload template includes necessary fields
Verify external platform supports rich payloads (some SMS gateways don’t)

Integration Selector

Choose the right integration for your use case:

Your Need	Recommended Integration	Why
On-call engineer paging	PagerDuty	Escalation policies, incident management
Infrastructure correlation	Datadog	Unified metrics, correlated dashboards
Team notifications	Slack (Coming Soon)	Channel-based, collaborative triage
Custom internal tools	Generic Webhooks	Flexible, integrate with any HTTP endpoint
Multi-tool strategy	Datadog + PagerDuty	Metrics + incidents in one workflow

Cloud Platforms - Deploy Fiddler on AWS, Azure, GCP
Data Platforms - Ingest data from Snowflake, Kafka
ML Platforms - Integrate with Databricks, MLflow
Agentic AI - Monitor LangGraph and Strands Agents

​Why Alert Integration Matters

​Integration Categories

​📊 Observability Platforms

​🚨 Incident Management

​💬 Team Collaboration

​Observability Platform Integrations

​Datadog

​Incident Management Integrations

​PagerDuty

​Team Collaboration Integrations

​Slack (Coming Soon)

​Microsoft Teams (Coming Soon)

​Alert Routing Patterns

​Pattern 1: Severity-Based Routing

​Pattern 2: Team-Based Routing

​Pattern 3: Composite Alerting

​Metric Export Patterns

​Export Fiddler Metrics to Datadog

​Query Fiddler Metrics in Datadog

​Alert Lifecycle Management

​Alert States

​Alert Deduplication

​Custom Webhook Integrations

​Monitoring Integration Health

​Track Integration Status

​Alerts on Integration Failures

​Best Practices

​Alert Fatigue Prevention

​Incident Response Runbooks

​Security & Compliance

​Secure Credential Management

​Alert Data Privacy

​Audit Logging

​Troubleshooting

​Common Issues

​Integration Selector

​Related Integrations

Why Alert Integration Matters

Integration Categories

📊 Observability Platforms

🚨 Incident Management

💬 Team Collaboration

Observability Platform Integrations

Datadog

Incident Management Integrations

PagerDuty

Team Collaboration Integrations

Slack (Coming Soon)

Microsoft Teams (Coming Soon)

Alert Routing Patterns

Pattern 1: Severity-Based Routing

Pattern 2: Team-Based Routing

Pattern 3: Composite Alerting

Metric Export Patterns

Export Fiddler Metrics to Datadog

Query Fiddler Metrics in Datadog

Alert Lifecycle Management

Alert States

Alert Deduplication

Custom Webhook Integrations

Monitoring Integration Health

Track Integration Status

Alerts on Integration Failures

Best Practices

Alert Fatigue Prevention

Incident Response Runbooks

Security & Compliance

Secure Credential Management

Alert Data Privacy

Audit Logging

Troubleshooting

Common Issues

Integration Selector

Related Integrations