Data Platforms Overview - Fiddler Documentation

Integrate Fiddler with your existing data infrastructure to seamlessly ingest training data, production events, and ground truth labels. From cloud data warehouses to real-time streaming platforms, Fiddler connects to the data sources you already use.

Why Data Integration Matters

AI observability requires continuous data flow from your ML pipelines and applications. Fiddler’s data platform integrations enable:

Automated Data Ingestion - Pull training datasets and production events without manual uploads
Real-Time Monitoring - Stream prediction events for immediate drift and performance detection
Unified Data Pipeline - Single integration point for all your ML data sources
Ground Truth Enrichment - Automatically join production predictions with delayed labels
Historical Analysis - Query data warehouse for model performance over time

Integration Categories

🏢 Data Warehouses

Connect Fiddler to your cloud data warehouse for batch data ingestion and historical analysis. Supported Platforms:

Snowflake - Cloud data warehouse with zero-copy data sharing ✓ GA
Google BigQuery - Serverless data warehouse with SQL analytics ✓ GA

Common Use Cases:

Import training datasets from data warehouse tables
Query historical model predictions for performance analysis
Join production events with delayed ground truth labels
Export Fiddler metrics back to warehouse for BI tools

📊 Data Streaming

Stream real-time prediction events and feedback directly to Fiddler for immediate observability. Supported Platforms:

Apache Kafka - Distributed event streaming platform ✓ GA
Amazon S3 - Object storage with event notifications ✓ GA

Common Use Cases:

Stream model predictions in real-time from production services
Monitor agentic AI interactions as they occur
Trigger alerts on data quality issues within seconds
Capture ground truth feedback from user interactions

🔄 Orchestration & Pipelines

Integrate Fiddler into your ML workflow orchestration for automated monitoring at every pipeline stage. Supported Platforms:

Apache Airflow - Workflow orchestration platform ✓ GA
AWS SageMaker Pipelines - Managed ML pipeline service ✓ GA

Common Use Cases:

Automatically upload datasets when training pipelines complete
Trigger model evaluation as part of CI/CD workflows
Schedule periodic drift checks and performance reports
Orchestrate ground truth label collection and enrichment

Data Warehouse Integrations

Snowflake

Why Snowflake + Fiddler:

Zero-Copy Data Sharing - No data duplication, direct queries to Snowflake
Secure Data Access - OAuth 2.0 and key-pair authentication
Scalable Analytics - Leverage Snowflake’s compute for large datasets
Cost-Effective - Pay only for queries executed, no data transfer fees

Quick Start:

-- Create Fiddler integration user
CREATE USER fiddler_user PASSWORD='...';
CREATE ROLE fiddler_role;
GRANT ROLE fiddler_role TO USER fiddler_user;

-- Grant permissions to your ML data
GRANT USAGE ON DATABASE ml_data TO ROLE fiddler_role;
GRANT USAGE ON SCHEMA ml_data.predictions TO ROLE fiddler_role;
GRANT SELECT ON ALL TABLES IN SCHEMA ml_data.predictions TO ROLE fiddler_role;

Full Snowflake Setup Guide →

Google BigQuery

Why BigQuery + Fiddler:

Serverless Architecture - No infrastructure management
SQL-Based Queries - Familiar interface for data teams
Federated Queries - Join Fiddler data with other GCP sources
Machine Learning - BigQuery ML model monitoring integration

Quick Start:

from fiddler import FiddlerClient

# Connect Fiddler to BigQuery
client = FiddlerClient(api_key="fid_...")
client.add_bigquery_connection(
    project_id="your-gcp-project",
    dataset_id="ml_predictions",
    credentials_path="service-account-key.json"
)

# Import dataset from BigQuery table
client.upload_dataset_from_bigquery(
    project="your-project",
    dataset="ml_predictions",
    table="credit_risk_predictions"
)

Full BigQuery Setup Guide →

Streaming Integrations

Apache Kafka

Why Kafka + Fiddler:

Real-Time Monitoring - Sub-second latency from prediction to observability
High Throughput - Handle millions of events per second
Event Replay - Replay historical events for testing and validation
Exactly-Once Semantics - Guaranteed delivery for critical predictions

Architecture Pattern:

Prediction Service
  ↓ (publish)
Kafka Topic: model-predictions
  ↓ (consume)
Fiddler Kafka Consumer
  ↓ (ingest)
Fiddler Platform

Quick Start:

from fiddler import FiddlerClient

# Configure Kafka consumer for Fiddler
client = FiddlerClient(api_key="fid_...")
client.add_kafka_source(
    bootstrap_servers="kafka-broker:9092",
    topic="model-predictions",
    group_id="fiddler-consumer",

    # Message format
    value_deserializer="json",
    schema={
        "prediction_id": "string",
        "model_id": "string",
        "prediction": "float",
        "features": "object"
    }
)

Full Kafka Setup Guide →

Amazon S3

Why S3 + Fiddler:

Batch Processing - Ingest large datasets efficiently
Event Notifications - Automatic processing when new files arrive
Data Lake Integration - Monitor models trained on S3 data lakes
Cost-Effective Storage - Archive historical predictions in S3

Integration Patterns: Pattern 1: Scheduled Batch Upload

from fiddler import FiddlerClient

client = FiddlerClient(api_key="fid_...")

# Upload dataset from S3
client.upload_dataset_from_s3(
    project="fraud-detection",
    dataset="production-predictions",
    s3_uri="s3://my-bucket/predictions/2024-11-10.parquet",
    format="parquet"
)

Pattern 2: Event-Driven Upload

# AWS Lambda triggered by S3 event
import json
from fiddler import FiddlerClient

def lambda_handler(event, context):
    # Extract S3 bucket and key from event
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    # Upload to Fiddler
    client = FiddlerClient(api_key=os.environ['FIDDLER_API_KEY'])
    client.upload_dataset_from_s3(
        project="my-project",
        dataset="streaming-data",
        s3_uri=f"s3://{bucket}/{key}"
    )

Full S3 Setup Guide →

Orchestration & Pipeline Integrations

Apache Airflow

Why Airflow + Fiddler:

Automated Workflows - Trigger Fiddler operations as DAG tasks
Dependency Management - Ensure data quality before model training
Scheduling - Periodic drift checks and model evaluations
Observability - Monitor ML pipelines and models in one platform

Example DAG:

from airflow import DAG
from airflow.operators.python import PythonOperator
from fiddler_airflow import FiddlerDatasetUploadOperator, FiddlerModelEvaluationOperator
from datetime import datetime

dag = DAG(
    'ml_pipeline_with_monitoring',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily'
)

# Train model
train_task = PythonOperator(
    task_id='train_model',
    python_callable=train_model_func,
    dag=dag
)

# Upload training data to Fiddler
upload_task = FiddlerDatasetUploadOperator(
    task_id='upload_to_fiddler',
    project='credit-risk',
    dataset='training_data',
    data_source='s3://my-bucket/training-data.csv',
    dag=dag
)

# Evaluate model in Fiddler
eval_task = FiddlerModelEvaluationOperator(
    task_id='evaluate_model',
    project='credit-risk',
    model='risk_model_v2',
    baseline_dataset='training_data',
    dag=dag
)

train_task >> upload_task >> eval_task

Full Airflow Setup Guide →

AWS SageMaker Pipelines

Why SageMaker Pipelines + Fiddler:

Native AWS Integration - Seamless with SageMaker Partner AI App
End-to-End ML Workflows - From data prep to model monitoring
Model Registry Integration - Automatic monitoring setup for registered models
Cost Optimization - Leverage existing SageMaker infrastructure

Example Pipeline:

from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.steps import ProcessingStep, TrainingStep
from sagemaker.workflow.functions import Join
from fiddler_sagemaker import FiddlerMonitoringStep

# Define SageMaker pipeline steps
processing_step = ProcessingStep(...)
training_step = TrainingStep(...)

# Add Fiddler monitoring step
fiddler_step = FiddlerMonitoringStep(
    name="SetupFiddlerMonitoring",
    model_name=training_step.properties.ModelName,
    baseline_dataset_s3_uri=processing_step.properties.ProcessingOutputConfig.Outputs["baseline"].S3Output.S3Uri,
    fiddler_project="sagemaker-models",
    depends_on=[training_step]
)

# Create pipeline
pipeline = Pipeline(
    name="MLPipelineWithFiddler",
    steps=[processing_step, training_step, fiddler_step]
)

Full SageMaker Pipelines Guide →

Integration Selector

Not sure which data integration to use? Here’s a quick decision guide:

Your Data Source	Recommended Integration	Why
Snowflake data warehouse	Snowflake connector	Zero-copy sharing, direct queries
BigQuery tables	BigQuery connector	Serverless, SQL-based, GCP-native
Real-time prediction streams	Kafka integration	Sub-second latency, high throughput
S3 data lake	S3 integration	Batch processing, event-driven uploads
Airflow ML pipelines	Airflow operators	Automated workflows, task dependencies
SageMaker workflows	SageMaker Pipelines	Native AWS integration, model registry

Getting Started

Prerequisites

Before setting up data integrations, ensure you have:

Fiddler Account - Cloud or on-premises instance
API Key - Generate from Fiddler UI Settings
Data Source Access - Credentials with read permissions
Network Connectivity - Firewall rules allowing Fiddler → Data Source

General Setup Pattern

All data integrations follow this pattern: 1. Configure Connection

from fiddler import FiddlerClient

client = FiddlerClient(
    api_key="fid_...",
    url="https://app.fiddler.ai"
)

# Add data source
client.add_data_source(
    name="my-data-warehouse",
    type="snowflake",  # or bigquery, kafka, s3
    connection_params={...}
)

2. Test Connection

# Verify connectivity
status = client.test_data_source("my-data-warehouse")
print(f"Connection status: {status}")

3. Ingest Data

# Upload dataset from data source
client.upload_dataset_from_source(
    project="my-project",
    dataset="training-data",
    source="my-data-warehouse",
    query="SELECT * FROM predictions WHERE date > '2024-01-01'"
)

Advanced Patterns

Pattern 1: Multi-Source Data Enrichment

Combine data from multiple sources for comprehensive monitoring:

# Production predictions from Kafka
client.add_kafka_source(
    topic="production-predictions",

    # ... additional connection settings
)

# Ground truth labels from Snowflake
client.add_snowflake_source(
    database="ml_data",
    schema="labels",

    # ... additional connection settings
)

# Join predictions with labels
client.configure_ground_truth_enrichment(
    project="fraud-detection",
    prediction_source="production-predictions",
    label_source="ml_data.labels.fraud_labels",
    join_key="transaction_id",
    time_window_hours=24
)

Pattern 2: Data Quality Validation

Validate data quality before ingestion:

from fiddler import DataValidator

validator = DataValidator()
validator.add_rule("age", min_value=18, max_value=100)
validator.add_rule("income", not_null=True)
validator.add_rule("credit_score", pattern=r'^\d{3}$')

# Validate before upload
is_valid = client.upload_dataset_from_source(
    ...,
    validator=validator,
    on_validation_failure="abort"  # or "warn" or "ignore"
)

Pattern 3: Incremental Updates

Efficiently update datasets with only new data:

# Initial full load
client.upload_dataset_from_source(
    project="my-project",
    dataset="user-events",
    source="my-bigquery",
    query="SELECT * FROM events"
)

# Daily incremental updates (scheduled via Airflow)
client.update_dataset_from_source(
    project="my-project",
    dataset="user-events",
    source="my-bigquery",
    query="SELECT * FROM events WHERE date = CURRENT_DATE()",
    mode="append"  # or "upsert"
)

Data Format Requirements

Baseline/Training Data

Must include:

Features - All model input features
Predictions - Model outputs (for validation)
Metadata (optional) - Additional context fields

Example Schema:

{
  "transaction_id": "string",
  "amount": "float",
  "merchant_category": "string",
  "prediction": "float",
  "timestamp": "datetime"
}

Production Event Data

Must include:

Event ID - Unique identifier
Timestamp - Event time
Features - Model inputs
Predictions - Model outputs
Model Version (optional) - For multi-model monitoring

Example Kafka Message:

{
  "event_id": "evt_12345",
  "timestamp": "2024-11-10T14:32:01Z",
  "model": "fraud_detector_v3",
  "features": {
    "amount": 127.50,
    "merchant": "AMZN"
  },
  "prediction": 0.23
}

Security & Compliance

Authentication Methods

Snowflake:

Username/Password
Key Pair Authentication (recommended for production)
OAuth 2.0

BigQuery:

Service Account JSON key
Application Default Credentials
Workload Identity (GKE)

Kafka:

SASL/PLAIN
SASL/SCRAM
mTLS

S3:

IAM Role (recommended for AWS deployments)
Access Key / Secret Key
Cross-account access via IAM role assumption

Data Privacy

Encryption in Transit - TLS 1.3 for all data transfers
Encryption at Rest - Data encrypted in Fiddler storage
PII Redaction - Automatically detect and redact sensitive fields
Data Retention - Configurable retention policies per dataset

Network Security

Firewall Rules:

Source: Fiddler Platform IP ranges
Destination: Your Data Source
Ports:
  - Snowflake: 443 (HTTPS)
  - BigQuery: 443 (HTTPS)
  - Kafka: 9092/9093 (SASL/SSL)
  - S3: 443 (HTTPS)

Private Connectivity:

AWS PrivateLink - For SageMaker Partner AI App
VPC Peering - Direct connection to data sources
VPN Tunnels - Secure connectivity for on-premises sources

Monitoring Data Pipeline Health

Connection Health Checks

# Check all data source connections
sources = client.list_data_sources()
for source in sources:
    status = client.test_data_source(source.name)
    print(f"{source.name}: {status.status} (latency: {status.latency_ms}ms)")

Data Ingestion Metrics

Monitor data pipeline performance:

Ingestion Latency - Time from source to Fiddler
Throughput - Events per second processed
Error Rate - Failed ingestion attempts
Data Freshness - Time since last successful update

Alerts on Pipeline Failures

# Configure alert for data pipeline issues
client.create_alert(
    name="Data Pipeline Failure",
    trigger="data_ingestion_failed",
    source="kafka-production-predictions",
    notification_channels=["email", "pagerduty"]
)

Troubleshooting

Common Issues

Connection Timeouts:

Check network connectivity and firewall rules
Verify credentials are current and have proper permissions
Ensure data source is reachable from Fiddler’s network

Schema Mismatches:

Validate data types match Fiddler’s expected schema
Check for null values in required fields
Ensure timestamp fields use supported formats (ISO 8601)

High Latency:

For Kafka: Check consumer lag and partition count
For Data Warehouses: Optimize queries, add indexes
For S3: Use Parquet or ORC instead of CSV

Data Quality Issues:

Enable data validation rules before ingestion
Set up alerts for out-of-range values
Configure automatic PII redaction

Cloud Platforms - Deploy Fiddler on AWS, Azure, GCP
ML Platforms - Integrate with Databricks, MLflow
Agentic AI - Monitor LangGraph and Strands Agents
Monitoring & Alerting - Send alerts to incident management tools

​Why Data Integration Matters

​Integration Categories

​🏢 Data Warehouses

​📊 Data Streaming

​🔄 Orchestration & Pipelines

​Data Warehouse Integrations

​Snowflake

​Google BigQuery

​Streaming Integrations

​Apache Kafka

​Amazon S3

​Orchestration & Pipeline Integrations

​Apache Airflow

​AWS SageMaker Pipelines

​Integration Selector

​Getting Started

​Prerequisites

​General Setup Pattern

​Advanced Patterns

​Pattern 1: Multi-Source Data Enrichment

​Pattern 2: Data Quality Validation

​Pattern 3: Incremental Updates

​Data Format Requirements

​Baseline/Training Data

​Production Event Data

​Security & Compliance

​Authentication Methods

​Data Privacy

​Network Security

​Monitoring Data Pipeline Health

​Connection Health Checks

​Data Ingestion Metrics

​Alerts on Pipeline Failures

​Troubleshooting

​Common Issues

​Related Integrations

Why Data Integration Matters

Integration Categories

🏢 Data Warehouses

📊 Data Streaming

🔄 Orchestration & Pipelines

Data Warehouse Integrations

Snowflake

Google BigQuery

Streaming Integrations

Apache Kafka

Amazon S3

Orchestration & Pipeline Integrations

Apache Airflow

AWS SageMaker Pipelines

Integration Selector

Getting Started

Prerequisites

General Setup Pattern

Advanced Patterns

Pattern 1: Multi-Source Data Enrichment

Pattern 2: Data Quality Validation

Pattern 3: Incremental Updates

Data Format Requirements

Baseline/Training Data

Production Event Data

Security & Compliance

Authentication Methods

Data Privacy

Network Security

Monitoring Data Pipeline Health

Connection Health Checks

Data Ingestion Metrics

Alerts on Pipeline Failures

Troubleshooting

Common Issues

Related Integrations