Skip to main content
When enabled, Fiddler’s event deduplication feature ensures that the same inference event is stored and counted only once within the deduplication windows, even when events are published multiple times due to retries, overlapping batch uploads, or client-side errors. Event deduplication is not enabled by default. To enable it for your deployment, contact Fiddler support.

​
Prerequisites

Event deduplication requires that a unique identifier column is defined on your model:
  • Set event_id_col to the name of the column containing unique event identifiers when onboarding your model using Model.from_data() or Model().
  • Events that do not have a value in this column (null or missing) are always accepted and stored without deduplication.
model = fdl.Model.from_data(
    name='your_model_name',
    project_id=fdl.Project.from_name('your_project_name').id,
    source=sample_df,
    spec=model_spec,
    task=model_task,
    event_id_col='your_event_id_column',  # required for deduplication
    event_ts_col='your_timestamp_column',
)

​
How Deduplication Works

Fiddler applies multiple independent layers of deduplication as events move through the ingestion pipeline. Each layer targets a different source of duplication.
LayerWhat it preventsWindow
Request retry protectionSame HTTP request processed twice due to client retry30 minutes
Within-batch deduplicationDuplicate event IDs in the same publish callEntire batch
Against recently stored eventsEvent IDs that already exist in recent ingestion history1 hour (default)
Across concurrent publish callsSame event ID arriving in parallel chunks5 minutes (default)
Storage-level idempotencySame internal batch resubmitted due to infrastructure retry7 days
Request retry protection and storage-level idempotency are always active. The remaining layers are part of event deduplication.

​
Request Retry Protection

When the Fiddler client SDK retries a publish request due to a network timeout or transient error, Fiddler detects the retry and returns the original result without reprocessing the events. This prevents duplicate job submissions and duplicate event ingestion caused by client-side retry logic.
  • Window: 30 minutes. A retried request received within 30 minutes of the original is recognized and deduplicated.
  • Always active: This layer requires no configuration and is always enabled.

​
Within-Batch Deduplication

If the same event ID appears more than once within a single publish call (for example, duplicate rows in a DataFrame or CSV), Fiddler keeps the first occurrence and drops subsequent duplicates before the events are stored.
  • Window: The entire publish batch.

​
Against Recently Stored Events

Before storing incoming events, Fiddler checks whether their event IDs already exist in recently ingested data. If a match is found, the incoming event is dropped.
  • Window: 1 hour (default). Event IDs are checked against events ingested within the past hour.

​
Across Concurrent Publish Calls

Large batch uploads are internally split into chunks and processed in parallel. If the same event ID appears across multiple chunks — for example, when overlapping files are uploaded concurrently — Fiddler uses a short-lived shared cache to detect and drop the duplicates.
  • Window: 5 minutes (default). Event IDs seen within the past 5 minutes are held in cache and used to deduplicate overlapping chunks.

​
Storage-Level Idempotency

If an internal processing failure causes the same batch of events to be retried and resubmitted to storage, Fiddler’s storage layer silently ignores the duplicate insert using a per-chunk idempotency token.
  • Always active: This layer requires no configuration.
  • It protects against infrastructure-level retries and does not deduplicate individual events by event ID.
Note: If all events in a publish call are identified as duplicates, the call succeeds with no new events stored — it is not treated as an error.

​
Without Deduplication

When event deduplication is not enabled, duplicate events published via retries or overlapping batches are accepted and stored. This may affect the accuracy of aggregated metrics such as event counts, drift scores, and model performance indicators shown in monitoring dashboards, charts, and alerts. To enable deduplication for your deployment, contact Fiddler support.

​
Limitations

  • Deduplication applies to insert operations only. Update operations (used to add ground truth labels or metadata) are never deduplicated.
  • Events with a null or missing event ID are never deduplicated. They are always accepted and stored, and receive a system-generated identifier.
  • Enabling deduplication does not retroactively correct metrics affected by previously published duplicates.