Multi-Application Trace Routing

Multi-application trace routing lets a single OTLP pipeline — one endpoint, one API key — carry spans for many Fiddler applications at once. You tag each span (or its resource) with an application.id; Fiddler authorizes every span against the applications your key is allowed to write to, stores the authorized ones under the right application, and reports the rest back to you.

Overview

In the single-application flow (Exporting OTel Traces to Fiddler), each request targets one application — you set a fiddler-application-id header and every span in that request lands in that one application. Multi-application routing removes that one-application-per-request limit. A single stream can mix spans for any number of applications, and Fiddler does the routing for you:

One endpoint, one API key. A shared gateway or collector fronting many applications sends everything to the same /v1/traces endpoint with a single API key — no per-application pipeline, and no separate key wired in per application.
No routing infrastructure to build or run. You do not stand up and maintain your own collector or configure routing connectors to split traffic by application. You tag each span with its application.id — one attribute, set once in your emitter’s config — and Fiddler handles the routing server-side.
Authorization-aware routing. Fiddler verifies each span’s application.id against the applications your API key is permitted to write to, and rejects the rest — so a span can never reach an application your API key cannot write to. This bounds routing to your API key’s scope; it does not isolate a shared gateway’s callers from each other — see Trust Model and Caller-Supplied Application IDs.
Everything in one call. Authorized and unauthorized spans can coexist in the same batch; the authorized ones are ingested and the unauthorized ones are reported back in the response.

When to use thisReach for multi-application routing when a single component — an AI gateway, a shared OpenTelemetry Collector, or a common egress service — emits telemetry for several Fiddler applications and you want to send it all through one authenticated channel instead of one pipeline per application.For a single application, the single-application header flow is all you need and continues to work unchanged.

How It Works

Architecture of multi-application trace routing: two producers carry many applications' spans over one endpoint — a gateway (AgentGateway or a request-path proxy) that copies application.id onto each span from a caller's request header (per span), and a batch or replay pipeline (for example, replaying spans stored in S3) that assembles each ResourceSpans with one application's application.id on its resource (per resource) — both send to one OTLP endpoint with one API key. The Fiddler OpenTelemetry collector, which is separate from the gateway and authorizes rather than produces, then checks each span's application.id (resource-preferred, span-fallback, fail-closed) against the applications the key may write to, storing authorized spans under their application and returning the rest in the partialSuccess response.

Tag each span with its application

Set application.id on each span (or on its resource, when a whole ResourceSpans belongs to one application). One batch can contain spans for many different applications.

Send with one API key

POST the batch to /v1/traces with a single Authorization: Bearer <YOUR_API_KEY> header. No fiddler-application-id header is used in this flow — routing is driven by each span’s application.id.

Fiddler authorizes each span

Fiddler resolves the full set of applications your API key may write to and matches each span’s application.id against that set — no single application is declared up front.

Authorized spans are stored, the rest are reported

Authorized spans are ingested under their application. Unauthorized spans are not stored; instead they are counted and named in the response so you can act on them.

The call returns HTTP 200 with an OTLP partialSuccess block whenever any span is rejected — see The partialSuccess Response. Authorized spans in the same batch are still ingested.

Why one API key can serve many applications. In the single-application flow you declare one application up front (via the fiddler-application-id header) and Fiddler answers a single yes/no. Multi-application routing instead authorizes against your API key’s full set of permitted applications and matches each span’s application.id against it. Nothing is declared up front, so one API key serves every application it is allowed to write to.

Prerequisites

One or more Fiddler applications created — you will need each Application UUID.
A valid Fiddler API key (from Settings > Credentials) whose user has write access to each application you intend to send to. Access is governed by your role assignments — see Access Control.
Multi-application routing enabled on your Fiddler deployment.

Multi-application routing is an opt-in capability. If sending a multi-application batch returns everything as rejected, confirm it is enabled on your deployment and that your API key has write access to the target applications — contact your Fiddler Customer Success Manager if you are unsure.A newly created application takes a few minutes to propagate. Fiddler caches each API key’s authorized-application list (the collector’s default is 5 minutes), so an application you just created can be rejected until that cache refreshes — wait a few minutes and resend.

How the Application ID Reaches Each Span

This is the question every shared-pipeline setup has to answer, so it is worth stating plainly. The component that emits telemetry — a gateway, a collector, an egress service — is usually not the component that knows which Fiddler application a request belongs to. The caller knows that. So the application ID has to travel from the caller, through the emitter, and onto each span as the application.id attribute. Where and how you set it depends on what is emitting the spans:

One application per process (an instrumented service) — set application.id on the resource, via an environment variable or the SDK. See One Application per Process.
Many applications from one process (a gateway, or a service that switches application per request) — set application.id per span. See Many Applications from One Process.
A batch or replay pipeline — set application.id as you assemble each span; see the protobuf example below.

Resource or Span by Topology

Fiddler resolves each span’s application resource-preferred, span-fallback: a resource-level application.id decides for every span under it, a span-level application.id is used only when the resource sets none, and a span with neither fails closed (rejected as unauthorized). Which level you set depends on your topology — and for the shared-pipeline case this page targets, resource-level is the wrong choice:

One application per process (an instrumented service) — an OpenTelemetry TracerProvider carries exactly one Resource, so a resource-level application.id tags every span the process emits with the same application. That is correct when a process serves one application — the single-application flow.
Many applications from one process (a gateway, collector, or shared exporter — what this page is about) — set application.id per span, and do not set it on the resource.

On a shared emitter, a resource-level application.id silently overrides every per-request value. The process has one resource and resource beats span, so a resource-level value pins everything it sends to that one application and ignores the per-span value — with no error. For multi-application routing, set application.id per span and leave the resource unset. (A batch or replay pipeline is the exception: it groups each application’s spans under their own ResourceSpans, so a resource-level value there is correct — see the protobuf example below.)

Setting application.id

One Application per Process

When a process serves a single application, set application.id on the resource. The cheapest form is environment variables — no code — and works with any OpenTelemetry SDK:

OTEL_EXPORTER_OTLP_ENDPOINT="https://your-instance.fiddler.ai"
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer <YOUR_API_KEY>"
OTEL_RESOURCE_ATTRIBUTES="application.id=<APPLICATION_UUID>,service.name=my-agent-service"

Or in code, with the OpenTelemetry SDK’s Resource helper (rather than hand-building protobuf):

from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "application.id": "<APPLICATION_UUID>",
    "service.name": "my-agent-service",
})
# pass `resource` when you build your TracerProvider

This is still multi-application routing — just spread across processes: point many such single-application services at the same Fiddler endpoint and API key, and Fiddler routes each service’s spans to its own application. No per-service pipeline, and no routing config to build.

Many Applications from One Process

When one process emits spans for several applications — a gateway, or a service that switches application per request — set application.id per span and leave it off the resource (see the trap above). With the OpenTelemetry SDK, set it on each span as you create it, and do not put application.id on the Resource:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# Configure the exporter once — no application.id on the Resource.
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://your-instance.fiddler.ai/v1/traces",
    headers={"Authorization": "Bearer <YOUR_API_KEY>"},
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

# Per request: tag the span with the caller's application ID — this routes it.
with tracer.start_as_current_span("chat") as span:
    span.set_attribute("application.id", "<APPLICATION_UUID>")
    span.set_attribute("fiddler.span.type", "llm")
    # ... your LLM call and other span attributes

# BatchSpanProcessor exports asynchronously; in a short-lived script, flush
# before exit or the spans never leave the process.
provider.shutdown()

With AgentGateway, the caller sends the application ID as a request header and a CEL expression copies it onto each span — the same shape AgentGateway already uses for gen_ai.conversation.id:

frontendPolicies:
  tracing:
    # ... host / protocol / auth as in the AgentGateway integration guide
    attributes:
      # Per-request application ID, copied from the caller's header onto each span.
      application.id: |
        request.headers["x-fiddler-application-id"] != "" ? request.headers["x-fiddler-application-id"] : ""
      gen_ai.conversation.id: |
        request.headers["x-fiddler-conversation-id"] != "" ? request.headers["x-fiddler-conversation-id"] : ""

Two changes from the single-application AgentGateway integration config are required:

Move application.id out of resources and into attributes. The single-application config sets resources.application.id to a fixed value; leaving it there pins every span to one application (resource beats span) and the header is ignored.
Drop the fiddler-application-id request header that the single-application config adds via requestHeaderModifier. On a multi-application deployment that header is ignored, so it is dead config — removing it is a cleanup, not a correctness fix.

The caller then sends its application ID on every request, e.g. X-Fiddler-Application-Id: <APPLICATION_UUID> (exactly as it already sends X-Fiddler-Conversation-Id).

Assembling batches as protobuf (pipelines, replay, ETL)

When you hold spans as data rather than running a live tracer — replaying spans from object storage, running an ETL job, or backfilling history — assemble the OTLP payload directly. Group each application’s spans under their own ResourceSpans with that application’s application.id on the resource (here a resource-level value is correct, because each ResourceSpans belongs to exactly one application), and send the whole batch with one API key.

import gzip
import httpx
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
from opentelemetry.proto.common.v1.common_pb2 import AnyValue, InstrumentationScope, KeyValue
from opentelemetry.proto.resource.v1.resource_pb2 import Resource
from opentelemetry.proto.trace.v1.trace_pb2 import ResourceSpans, ScopeSpans, Span, Status

FIDDLER_URL   = "https://your-instance.fiddler.ai"
FIDDLER_API_KEY = "your-api-key"
APP_A = "11111111-1111-1111-1111-111111111111"
APP_B = "22222222-2222-2222-2222-222222222222"


def llm_span(name: str) -> Span:
    return Span(
        trace_id=bytes.fromhex("4bf92f3577b34da6a3ce929d0e0e4736"),
        span_id=bytes.fromhex("00f067aa0ba902b7"),
        name=name,
        kind=Span.SPAN_KIND_INTERNAL,
        start_time_unix_nano=1_700_000_000_000_000_000,
        end_time_unix_nano=1_700_000_001_000_000_000,
        status=Status(code=Status.STATUS_CODE_OK),
        attributes=[
            KeyValue(key="fiddler.span.type",    value=AnyValue(string_value="llm")),
            KeyValue(key="gen_ai.request.model", value=AnyValue(string_value="gpt-4o")),
        ],
    )


def app_resource_spans(app_uuid: str, span_name: str) -> ResourceSpans:
    return ResourceSpans(
        resource=Resource(attributes=[
            KeyValue(key="application.id", value=AnyValue(string_value=app_uuid)),
        ]),
        scope_spans=[ScopeSpans(
            scope=InstrumentationScope(name="my-tracer", version="1.0.0"),
            spans=[llm_span(span_name)],
        )],
    )


# One batch, two applications
payload = ExportTraceServiceRequest(resource_spans=[
    app_resource_spans(APP_A, "chat-app-a"),
    app_resource_spans(APP_B, "chat-app-b"),
]).SerializeToString()

response = httpx.post(
    f"{FIDDLER_URL}/v1/traces",
    content=gzip.compress(payload),
    headers={
        "Authorization": f"Bearer {FIDDLER_API_KEY}",
        "Content-Type": "application/x-protobuf",
        "Content-Encoding": "gzip",
    },
    timeout=30.0,
)
response.raise_for_status()  # 200 even when some spans are rejected — inspect partial_success

For the full span/attribute schema (span types, LLM attributes, typing) and a value-to-AnyValue helper, see Exporting OTel Traces to Fiddler and Span and Resource Attributes.

Producer Support

Any producer that can set application.id per span can drive multi-application routing. Today that is:

Producer	Per-request multi-application routing	How
OpenTelemetry SDK	✅ Supported	`span.set_attribute("application.id", ...)` per span
AgentGateway	✅ Supported	CEL maps a request header to the `application.id` span attribute
LiteLLM	⚠️ Single application per proxy	Set one `application.id` for the proxy via `OTEL_RESOURCE_ATTRIBUTES`

LiteLLM does not currently support per-request application routing. A caller’s per-request application.id (whether sent as a header or in request metadata) is captured only inside namespaced metadata attributes such as metadata.requester_custom_headers, which Fiddler resolves after the routing decision has already been made. Routing keys off the bare application.id at ingest, and LiteLLM never sets that per request — so a per-request application ID cannot influence which application a span lands in. (This is the same reason per-request gen_ai.conversation.id works but application.id does not: a conversation ID is a display field resolved late in the pipeline, while an application ID is a routing field needed at ingest — same transport, different timing.) Use LiteLLM for single-application export: set one application.id for the proxy via OTEL_RESOURCE_ATTRIBUTES. Per-request routing through LiteLLM would require a Fiddler-supplied callback that sets application.id at emit time — contact your Fiddler Customer Success Manager if you need it.

Migrating from the Single-Application Header

Once multi-application routing is enabled, the fiddler-application-id header is ignored. Routing is driven entirely by each span’s application.id attribute. A client that identified its application only through that header — with no application.id on the resource or span — will have every span fail closed (rejected as unauthorized).Fiddler’s documented integrations already set application.id alongside the header: both the LiteLLM and AgentGateway guides set the application.id attribute as well, so they keep working. Only a client that was trimmed down to header-only is affected. Before enabling multi-application routing, confirm every producer sets application.id on the resource or span.

Trust Model and Caller-Supplied Application IDs

Multi-application routing authorizes the API key, not the caller. Fiddler checks each span’s application.id against the applications your API key is permitted to write to, and rejects the rest. This protects Fiddler’s boundary between tenants: a caller cannot route a span to an application your API key has no access to. It does not, on its own, isolate a shared gateway’s callers from one another. The application.id on a span is caller-supplied input, and an API key that can write to many applications can write to any of them — so any caller of that gateway can tag a span as any application the gateway’s API key can reach. If a shared gateway must keep its own callers in separate applications, enforce that at the gateway — for example, derive the application ID from an authenticated caller identity rather than trusting a raw client header. Per-span authorization protects Fiddler’s tenancy boundary, not the gateway’s internal one.

The partialSuccess Response

When every span is authorized, the response is a plain 200 with an empty body. When one or more spans are rejected, Fiddler still returns 200 and populates the standard OTLP partialSuccess block:

Field	Meaning
`rejectedSpans`	How many spans were not stored because the API key was not authorized for their `application.id`
`errorMessage`	A human-readable summary naming the unauthorized `application.id` values

{
  "partialSuccess": {
    "rejectedSpans": 1,
    "errorMessage": "1 span(s) rejected: unauthorized application IDs [22222222-2222-2222-2222-222222222222]"
  }
}

partialSuccess is not an error — the HTTP status is still 200 and the authorized spans in the batch are ingested. Do not treat a populated partialSuccess as a failed request; instead, read rejectedSpans and errorMessage to see which applications were rejected and why.

The most common cause of a rejection is an API key without write access to the named application. Grant the API key’s user the appropriate role on that application (see below) and resend.

Access Control

Authorization is per span: Fiddler compares each span’s application.id against the set of applications the API key’s user is allowed to write to. A span is stored only if its application is in that set; otherwise it is rejected and reported in partialSuccess. To let an API key send to an application, ensure its user has a role that grants write access to that application’s project. See Role-Based Access Control for the available roles, and API Keys for creating and managing API keys.

Frequently Asked Questions

How does the application ID get onto a span through a gateway?

The caller sends it as a request header and the gateway copies it onto each span as the application.id attribute. For AgentGateway this is a one-line CEL expression — see Many Applications from One Process. The gateway itself does not know which application a request belongs to; the caller does.

Do I still need the fiddler-application-id header?

No. In the multi-application flow, routing is driven by each span’s application.id and the header is ignored. The single-application fiddler-application-id header still works for one-application-per-request exports on deployments without multi-application routing.

Resource-level or span-level application.id — which wins?

Resource-level takes precedence. If a ResourceSpans sets application.id on its resource, that value applies to every span under it; the span-level attribute is a fallback used only when the resource does not set one. From a shared emitter, set it per span so one process can route to many applications — see the trap above.

What happens to the authorized spans if some spans are rejected?

They are still ingested. Rejection is per span — authorized and unauthorized spans can share a batch, and only the unauthorized ones are dropped and reported in partialSuccess.

What if a span has no application.id at all?

It is treated as unauthorized (fail-closed): it is not stored and is counted in rejectedSpans.

What if application.id is malformed, or names an application that doesn't exist?

Both are rejected the same way as any unauthorized application — counted in rejectedSpans and named in errorMessage. Fiddler matches each span’s application.id against the set of applications your API key can write to; a well-formed UUID that no application matches, or a value that isn’t a valid UUID at all, is simply not in that set (Fiddler does not validate UUID format at ingest). Only a missing or empty application.id behaves differently — it fails closed (see the previous question).

Why is every span coming back rejected?

Common causes: the application was just created and hasn’t propagated yet — Fiddler caches each API key’s authorized-application list (the collector’s default is 5 minutes), so a brand-new application is rejected until that cache refreshes; wait a few minutes and resend. Otherwise: the API key does not have write access to the target applications; multi-application routing is not enabled on your deployment; or a fixed resource-level application.id (or header-only setup) is overriding the per-span value. Verify the API key’s roles, confirm the capability is enabled with your Fiddler Customer Success Manager, and check that application.id is set per span.

AgentGateway Integration

Configure AgentGateway’s tracing, including the CEL header-to-attribute mapping.

Exporting OTel Traces to Fiddler

The single-application export flow, span schema, and attribute mapping.

OpenTelemetry Integration

Live agent instrumentation via the OTel SDK.

Span and Resource Attributes

Attribute typing, custom attributes, and how they flow into metrics.

Role-Based Access Control

Roles that govern which applications an API key can write to.

API Keys

Create and manage the API keys used for trace ingestion.

Agentic AI & LLM Frameworks

Data Platforms & Pipelines

ML Platforms & Tools

Cloud Platforms & Deployment

Monitoring & Alerting

Overview

How It Works

Prerequisites

How the Application ID Reaches Each Span

Resource or Span by Topology

Setting application.id

One Application per Process

Many Applications from One Process

Producer Support

Migrating from the Single-Application Header

Trust Model and Caller-Supplied Application IDs

The partialSuccess Response

Access Control

Frequently Asked Questions

AgentGateway Integration

Exporting OTel Traces to Fiddler

OpenTelemetry Integration

Span and Resource Attributes

Role-Based Access Control

API Keys

​Overview

​How It Works

​Prerequisites

​How the Application ID Reaches Each Span

​Resource or Span by Topology

​Setting application.id

​One Application per Process

​Many Applications from One Process

​Producer Support

​Migrating from the Single-Application Header

​Trust Model and Caller-Supplied Application IDs

​The partialSuccess Response

​Access Control

​Frequently Asked Questions

​Related Documentation

AgentGateway Integration

Exporting OTel Traces to Fiddler

OpenTelemetry Integration

Span and Resource Attributes

Role-Based Access Control

API Keys

Overview

How It Works

Prerequisites

How the Application ID Reaches Each Span

Resource or Span by Topology

Setting application.id

One Application per Process

Many Applications from One Process

Producer Support

Migrating from the Single-Application Header

Trust Model and Caller-Supplied Application IDs

The partialSuccess Response

Access Control

Frequently Asked Questions

Related Documentation