S3 Trace Ingestion

Overview

The Fiddler S3 Connector allows you to ingest OpenTelemetry (OTLP) trace data from Amazon S3 into Fiddler without requiring any live SDK integration. Your application writes OTLP trace files to an S3 bucket (or a compatible object store), and Fiddler’s ingestion service automatically discovers, parses, and forwards those traces into the observability platform. This is the recommended approach for:

Air-gapped or high-security deployments where direct outbound connections from the application to Fiddler are not permitted
Batch ingestion pipelines that transform logs or events into OTLP format and stage them in S3
Custom log transformers that convert raw LangGraph or other framework logs into the Fiddler OTLP format

When to use the SDK insteadIf your application can make direct outbound HTTPS requests, the Fiddler LangGraph SDK or Fiddler OTel SDK provide zero-config auto-instrumentation with no file staging required. Use S3 ingestion only when direct SDK integration is not possible.

Architecture

Prerequisites

A Fiddler account with a GenAI Application created — you will need its Application UUID
A valid Fiddler API key (from Settings → Credentials) — this is used to authenticate the worker with the OTEL Collector
An Amazon S3 bucket (or S3-compatible store) that Fiddler’s worker can read from
IAM permissions on the bucket — see IAM Setup below

OTLP File Format

Files placed in S3 must be valid OTLP data in one of the supported formats below. All formats use the standard ExportTraceServiceRequest schema.

Supported file extensions

Format	Extensions	Description
OTLP JSON	`.json`, `.json.gz`, `.json.zst`	One `ExportTraceServiceRequest` JSON envelope per file.
OTLP JSONL	`.jsonl`, `.jsonl.gz`, `.jsonl.zst`, `.ndjson`, `.ndjson.gz`, `.ndjson.zst`	One `ExportTraceServiceRequest` JSON envelope per line, each with a single `resourceSpans` entry.
OTLP Protobuf	`.binpb`, `.binpb.gz`, `.binpb.zst`	Binary serialized `ExportTraceServiceRequest` protobuf.

Compression is transparent — files with .gz (gzip) or .zst (zstd) suffixes are decompressed automatically before parsing. Configure file_extensions on the ingestion source to match the format your pipeline produces.

Required JSON structure

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "application.id",
            "value": { "stringValue": "<your-fiddler-application-uuid>" }
          },
          {
            "key": "service.name",
            "value": { "stringValue": "your-service-name" }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": { "name": "your-tracer-name", "version": "1.0.0" },
          "spans": [
            {
              "traceId": "<trace-id>",
              "spanId": "<span-id>",
              "parentSpanId": "<parent-span-id-or-empty-string-for-root>",
              "name": "agent_invocation",
              "kind": 1,
              "startTimeUnixNano": "1744200000000000000",
              "endTimeUnixNano": "1744200005500000000",
              "status": { "code": 1 },
              "attributes": [
                {
                  "key": "fiddler.span.type",
                  "value": { "stringValue": "llm" }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Critical fields

Field	Description	Notes
`resource.attributes["application.id"]`	Fiddler Application UUID	Must match the application UUID in your Fiddler instance. This routes spans to the correct application in the UI.
`startTimeUnixNano` / `endTimeUnixNano`	Span timestamps as nanoseconds since Unix epoch	Must reflect the actual time of each event. Wrong timestamps cause spans to appear in the wrong time range in the UI.
`traceId`	16-byte trace identifier	Accepted as base64 (`S/kvNXezTaajzpKdDg5HNg==`) or hex (`4bf92f3577b34da6a3ce929d0e0e4736`). Fiddler auto-normalizes hex to base64.
`spanId`	8-byte span identifier	Same encoding rules as `traceId`.
`parentSpanId`	Parent span ID	Set to `""` (empty string) for root spans.
`fiddler.span.type`	Span type for Fiddler’s UI	Recommended values: `llm`, `tool`. See Supported span types below.

application.id must be in the fileThe application.id resource attribute inside the OTLP file is the source of truth for routing spans to the correct Fiddler application. The application_ids field on the ingestion source (see below) is used for access control only — it does not override the application.id in the span data. If these do not match, spans will be ingested but will not appear under your application in the UI.

Supported span types and attributes

`fiddler.span.type`	Key attributes
`llm`	`gen_ai.system`, `gen_ai.request.model`, `gen_ai.llm.input.user`, `gen_ai.llm.input.system`, `gen_ai.llm.output`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`
`tool`	`gen_ai.tool.name`, `gen_ai.tool.input`, `gen_ai.tool.output`
`agent`	`gen_ai.agent.name`, `gen_ai.agent.id`
`chain` (legacy — LangChain)	`gen_ai.agent.name`, `gen_ai.agent.id`, `gen_ai.conversation.id`

For the full attribute reference including all supported keys and their types, see Span and Resource Attributes. Custom span attributes can be added using the fiddler.span.user.* namespace:

{ "key": "fiddler.span.user.risk_rating", "value": { "stringValue": "Moderate-High" } }

Platform enablement requiredThe S3 connector must be enabled for your Fiddler environment before use. Contact your Fiddler account team or platform admin to request enablement. Once confirmed, proceed with the steps below to set up your ingestion source.

Setting Up the Ingestion Source

Create an ingestion source via the Fiddler REST API to tell the connector where to look in S3.

API endpoint

POST /v3/ingestion-sources

Request body

{
  "name": "my-agent-traces",
  "description": "Production LangGraph agent traces from ECS Fargate",
  "provider": "s3",
  "region": "us-west-2",
  "bucket": "my-company-traces",
  "prefix": "fiddler/prod/",
  "scan_interval_seconds": 60,
  "file_extensions": [".json"],
  "credential_type": "iam_role",
  "role_arn": "arn:aws:iam::123456789012:role/my-ingestion-role",
  "application_ids": ["<your-fiddler-application-uuid>"]
}

Request fields

Field	Required	Description
`name`	✅	Unique name for this ingestion source (1–255 characters)
`bucket`	✅	S3 bucket name (without `s3://` prefix)
`application_ids`	✅	List containing exactly one Fiddler application UUID (v1 enforces 1:1)
`provider`	❌	Object store provider. Default: `"s3"`. One of: `s3`, `gcs`, `azure`
`region`	❌	AWS region of the bucket (e.g. `"us-west-2"`). Optional
`prefix`	❌	S3 key prefix to scan (e.g. `"traces/prod/"`). Default: `""` (scan entire bucket)
`file_extensions`	❌	List of file extensions to process. Default: `[".json"]`. Supported values: `.json`, `.json.gz`, `.json.zst`, `.jsonl`, `.jsonl.gz`, `.jsonl.zst`, `.ndjson`, `.ndjson.gz`, `.ndjson.zst`, `.binpb`, `.binpb.gz`, `.binpb.zst`
`credential_type`	❌	`"iam_role"` (default) to use an IAM role, or `"access_key"` for static access key/secret
`access_key_id`	❌	Required when `credential_type` is `"access_key"`. Write-only, never returned in responses
`secret_access_key`	❌	Required when `credential_type` is `"access_key"`. Write-only, never returned in responses
`role_arn`	❌	IAM role ARN to assume via STS (cross-account). Omit to use the worker’s node IAM role
`external_id`	❌	STS external ID for cross-account role assumption. Write-only, never returned in responses
`endpoint_url`	❌	Custom endpoint URL for S3-compatible stores (e.g. MinIO). Default: `null`
`scan_interval_seconds`	❌	How often the manager scans for new files. Default: `60`. Minimum: `10`
`description`	❌	Optional description (max 1000 characters)

Example using Python

import requests

FIDDLER_URL = "https://your-instance.fiddler.ai"
API_KEY = "your-api-key"
APPLICATION_ID = "your-application-uuid"

response = requests.post(
    f"{FIDDLER_URL}/v3/ingestion-sources",
    headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    },
    json={
        "name": "my-agent-traces",
        "description": "Agent traces from S3",
        "provider": "s3",
        "region": "us-west-2",
        "bucket": "my-traces-bucket",
        "prefix": "traces/",
        "scan_interval_seconds": 60,
        "file_extensions": [".json"],
        "credential_type": "iam_role",
        "application_ids": [APPLICATION_ID],
    },
)
print(response.json())

IAM Setup

The Fiddler worker needs read access to your S3 bucket. The recommended approach is an IAM role.

Minimum required IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FiddlerS3ReadAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-traces-bucket",
        "arn:aws:s3:::my-traces-bucket/traces/*"
      ]
    }
  ]
}

Encrypted buckets (SSE-KMS)

If your bucket is encrypted with a customer-managed KMS key (CMK), the worker needs decrypt permission on that key in addition to the S3 actions above. Without it, s3:GetObject succeeds at the S3 authorization step but the request fails at the decryption step, and the file is marked failed with an error like:

AccessDenied ... not authorized to perform: kms:Decrypt on resource: arn:aws:kms:...:key/...
because no identity-based policy allows the kms:Decrypt action

Add the following statement to the role’s IAM policy, alongside the FiddlerS3ReadAccess statement:

{
  "Sid": "FiddlerKmsDecrypt",
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:us-west-2:123456789012:key/<your-kms-key-id>"
}

kms:Decrypt is the only permission strictly required for the read path; kms:DescribeKey is optional but commonly included for diagnostics.

KMS access is gated by two policies evaluated together — the role’s IAM policy and the KMS key policy. If the key policy keeps the default arn:aws:iam::123456789012:root / kms:* delegation, granting kms:Decrypt in the IAM policy above is sufficient. If that delegation was removed or restricted, you must also add the worker role as an allowed principal in the KMS key policy with kms:Decrypt and kms:DescribeKey.

Buckets encrypted with SSE-S3 (AES-256) or the AWS-managed aws/s3 key do not require any extra KMS permissions.

Options

Option	How to configure	When to use
EKS Node IAM Role	Grant the Fiddler worker node’s IAM role access to your bucket. Omit `role_arn` in the API request.	Same AWS account, simple setup
Cross-account IAM Role	Create a role in your account with a trust policy allowing Fiddler’s worker role to assume it. Set `role_arn` in the API request.	Cross-account or tighter security boundary

Monitoring File Processing Status

Aggregate stats

Get a summary count of files by status and total spans ingested:

GET /v3/ingestion-sources/{source_id}/stats

curl -H "Authorization: Bearer <api-key>" \
  "https://your-instance.fiddler.ai/v3/ingestion-sources/<source-id>/stats"

Example response:

{
  "data": {
    "source_id": "<source-uuid>",
    "total_files": 10,
    "pending": 2,
    "processing": 1,
    "completed": 6,
    "failed": 1,
    "total_spans_ingested": 342,
    "last_file_completed_at": "2026-04-10T09:38:22Z",
    "last_file_failed_at": null
  }
}

Per-file list

List individual files with their status:

GET /v3/ingestion-sources/{source_id}/files

curl -H "Authorization: Bearer <api-key>" \
  "https://your-instance.fiddler.ai/v3/ingestion-sources/<source-id>/files"

File statuses

Status	Meaning
`pending`	Discovered by the manager, queued for processing. Also the status a file returns to after a transient error when retries remain (`retries` counter increments each time).
`processing`	Worker is currently downloading and parsing the file
`completed`	Successfully ingested. `spans_count` shows how many spans were sent
`failed`	Permanently failed. Either a non-retryable error (e.g. malformed OTLP data, auth error) or all retries exhausted. Check `error_message` for details. Use the retry API to re-queue.

Example response

{
  "data": {
    "items": [
      {
        "id": 1,
        "s3_key": "traces/sample_trace.json",
        "file_size_bytes": 10097,
        "status": "completed",
        "error_message": null,
        "spans_count": 4,
        "retries": 0,
        "max_retries": 3,
        "processed_at": "2026-04-10T09:38:22Z",
        "created_at": "2026-04-10T09:38:18Z"
      }
    ]
  }
}

Retry a failed file

Retry a single failed file:

POST /v3/ingestion-sources/{source_id}/files/{file_id}/retry

Bulk retry all failed files for a source at once:

POST /v3/ingestion-sources/{source_id}/retry-failed

Testing the Connection

Before uploading production traces, verify that the ingestion source can reach your bucket:

POST /v3/ingestion-sources/{source_id}/test

curl -X POST \
  -H "Authorization: Bearer <api-key>" \
  "https://your-instance.fiddler.ai/v3/ingestion-sources/<source-id>/test"

A successful response returns { "status": "success", "files_found": N } where files_found is the number of files discovered under the configured prefix. A failure returns { "status": "error", "message": "<error detail>" } (e.g. AccessDenied, NoSuchBucket).

Generating OTLP Files with the Fiddler SDK

If your application can write files locally (but cannot send traces directly to Fiddler), you can use the Fiddler LangGraph SDK’s built-in OTLP file capture and upload the files to S3 separately.

from fiddler_langgraph import FiddlerClient

client = FiddlerClient(
    application_id="your-application-uuid",
    otlp_enabled=False,           # Do not send traces directly to Fiddler
    otlp_json_capture_enabled=True,
    otlp_json_output_dir="./traces",  # Directory to write OTLP JSON files
)

Each LangGraph invocation writes one OTLP JSON file to ./traces/. Upload these files to your S3 bucket using the AWS CLI or any S3 client:

aws s3 sync ./traces/ s3://my-traces-bucket/traces/

For environments where local file writes are also not possible (e.g. ECS Fargate with read-only filesystems), generate the OTLP JSON in memory and stream it directly to S3 using boto3.client('s3').put_object() without writing to disk.

Troubleshooting

Spans not appearing in the UI

Symptom	Likely cause	Fix
File `status: completed` but no spans visible	Time range filter in UI doesn’t cover the span timestamps	Change the UI date range to match `startTimeUnixNano` in your files
File `status: completed` but spans under wrong application	`application.id` in the file doesn’t match the Fiddler application UUID	Update your transformer to embed the correct `application.id` as a resource attribute
File `status: failed` with `invalid TraceID length`	`traceId` / `spanId` encoded incorrectly	Use 32-char hex or standard base64. Fiddler auto-normalizes hex; avoid other encodings
File `status: failed` with `AccessDenied`	Worker IAM role lacks `s3:GetObject` on your bucket	Update the IAM policy — see IAM Setup
File `status: failed` with `kms:Decrypt` `AccessDenied`	Bucket uses SSE-KMS and the worker role lacks `kms:Decrypt` on the CMK	Add `kms:Decrypt` + `kms:DescribeKey` to the role’s IAM policy, and ensure the KMS key policy allows the role — see Encrypted buckets (SSE-KMS)
File `status: failed` with `CredentialError`	`OTEL_COLLECTOR_AUTH_TOKEN` not configured on the worker	Contact your Fiddler admin to configure the collector auth token
Files never appear (stuck in `pending`)	Manager cannot list the bucket	Check `s3:ListBucket` permission and that `prefix` is correct

Verifying file format locally

Use the following Python snippets to validate your OTLP files before uploading: JSON (.json):

import json
from google.protobuf.json_format import ParseDict
from opentelemetry.proto.trace.v1.trace_pb2 import ResourceSpans

with open("my_trace.json") as f:
    data = json.load(f)

for rs_dict in data["resourceSpans"]:
    try:
        rs = ParseDict(rs_dict, ResourceSpans())
        print(f"OK: {sum(len(ss.spans) for ss in rs.scope_spans)} spans")
    except Exception as e:
        print(f"ERROR: {e}")

JSONL (.jsonl):

import json
from google.protobuf.json_format import ParseDict
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest

with open("my_trace.jsonl") as f:
    for i, line in enumerate(f, 1):
        line = line.strip()
        if not line:
            continue
        try:
            data = json.loads(line)
            req = ParseDict(data, ExportTraceServiceRequest())
            spans = sum(len(ss.spans) for rs in req.resource_spans for ss in rs.scope_spans)
            print(f"Line {i} OK: {spans} spans")
        except Exception as e:
            print(f"Line {i} ERROR: {e}")

Protobuf (.binpb):

from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest

with open("my_trace.binpb", "rb") as f:
    data = f.read()

try:
    req = ExportTraceServiceRequest()
    req.ParseFromString(data)
    spans = sum(len(ss.spans) for rs in req.resource_spans for ss in rs.scope_spans)
    print(f"OK: {spans} spans across {len(req.resource_spans)} resourceSpans")
except Exception as e:
    print(f"ERROR: {e}")

File Naming and Organization

The S3 connector processes every file under the configured prefix that matches the configured file_extensions. Supported extensions include .json, .jsonl, .binpb, and their compressed variants (.gz, .zst). Once a file is processed (whether completed or failed), it is not reprocessed unless you call the retry API. Recommended S3 key structure:

traces/
  2026/04/10/
    agent-run-abc123.json
    agent-run-def456.json
  2026/04/11/
    agent-run-ghi789.json

This date-partitioned layout makes it easy to manage retention policies and audit ingestion history.

Exporting OTel Traces to Fiddler — client-side protobuf export for custom pipelines
Fiddler LangGraph SDK — direct SDK integration for LangGraph applications
OpenTelemetry Integration — live OTLP export for custom agent frameworks
Fiddler OTel SDK — decorator-based tracing for custom Python agents

Agentic AI & LLM Frameworks

Data Platforms & Pipelines

ML Platforms & Tools

Cloud Platforms & Deployment

Monitoring & Alerting

S3 Trace Ingestion

S3 Trace Ingestion

Overview

Architecture

Prerequisites

OTLP File Format

Supported file extensions

Required JSON structure

Critical fields

Supported span types and attributes

Setting Up the Ingestion Source

API endpoint

Request body

Request fields

Example using Python

IAM Setup

Minimum required IAM policy

Encrypted buckets (SSE-KMS)

Options

Monitoring File Processing Status

Aggregate stats

Per-file list

File statuses

Example response

Retry a failed file

Testing the Connection

Generating OTLP Files with the Fiddler SDK

Troubleshooting

Spans not appearing in the UI

Verifying file format locally

File Naming and Organization

​S3 Trace Ingestion

​Overview

​Architecture

​Prerequisites

​OTLP File Format

​Supported file extensions

​Required JSON structure

​Critical fields

​Supported span types and attributes

​Setting Up the Ingestion Source

​API endpoint

​Request body

​Request fields

​Example using Python

​IAM Setup

​Minimum required IAM policy

​Encrypted buckets (SSE-KMS)

​Options

​Monitoring File Processing Status

​Aggregate stats

​Per-file list

​File statuses

​Example response

​Retry a failed file

​Testing the Connection

​Generating OTLP Files with the Fiddler SDK

​Troubleshooting

​Spans not appearing in the UI

​Verifying file format locally

​File Naming and Organization

​Related Documentation

S3 Trace Ingestion

Overview

Architecture

Prerequisites

OTLP File Format

Supported file extensions

Required JSON structure

Critical fields

Supported span types and attributes

Setting Up the Ingestion Source

API endpoint

Request body

Request fields

Example using Python

IAM Setup

Minimum required IAM policy

Encrypted buckets (SSE-KMS)

Options

Monitoring File Processing Status

Aggregate stats

Per-file list

File statuses

Example response

Retry a failed file

Testing the Connection

Generating OTLP Files with the Fiddler SDK

Troubleshooting

Spans not appearing in the UI

Verifying file format locally

File Naming and Organization

Related Documentation