> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fiddler.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Application Trace Routing

> Send OpenTelemetry traces for many Fiddler applications through one endpoint and one API key. Fiddler authorizes and routes each span by its application.id, so a single shared pipeline can serve every application.

Multi-application trace routing lets a **single** OTLP pipeline — one endpoint, one API key — carry spans for **many** Fiddler applications at once. You tag each span (or its resource) with an `application.id`; Fiddler authorizes every span against the applications your key is allowed to write to, stores the authorized ones under the right application, and reports the rest back to you.

***

## Overview

In the single-application flow ([Exporting OTel Traces to Fiddler](/integrations/agentic-ai/otel-trace-export)), each request targets one application — you set a `fiddler-application-id` header and every span in that request lands in that one application.

Multi-application routing removes that one-application-per-request limit. A single stream can mix spans for any number of applications, and Fiddler does the routing for you:

* **One endpoint, one API key.** A shared gateway or collector fronting many applications sends everything to the same `/v1/traces` endpoint with a single API key — no per-application pipeline, and no separate key wired in per application.
* **No routing infrastructure to build or run.** You do not stand up and maintain your own collector or configure routing connectors to split traffic by application. You tag each span with its `application.id` — one attribute, set once in your emitter's config — and Fiddler handles the routing server-side.
* **Authorization-aware routing.** Fiddler verifies each span's `application.id` against the applications your API key is permitted to write to, and rejects the rest — so a span can never reach an application your API key cannot write to. This bounds routing to your API key's scope; it does not isolate a shared gateway's callers from each other — see [Trust Model and Caller-Supplied Application IDs](#trust-model-and-caller-supplied-application-ids).
* **Everything in one call.** Authorized and unauthorized spans can coexist in the same batch; the authorized ones are ingested and the unauthorized ones are reported back in the response.

<Info>
  **When to use this**

  Reach for multi-application routing when a single component — an AI gateway, a shared OpenTelemetry Collector, or a common egress service — emits telemetry for several Fiddler applications and you want to send it all through one authenticated channel instead of one pipeline per application.

  For a single application, the [single-application header flow](/integrations/agentic-ai/otel-trace-export) is all you need and continues to work unchanged.
</Info>

***

## How It Works

<img src="https://mintcdn.com/fiddlerai/dA2-JpO7n2Bn2uc-/images/multi-application-trace-routing-architecture.svg?fit=max&auto=format&n=dA2-JpO7n2Bn2uc-&q=85&s=cfcbda4b94c255ecd6dcc085010bc615" alt="Architecture of multi-application trace routing: two producers carry many applications' spans over one endpoint — a gateway (AgentGateway or a request-path proxy) that copies application.id onto each span from a caller's request header (per span), and a batch or replay pipeline (for example, replaying spans stored in S3) that assembles each ResourceSpans with one application's application.id on its resource (per resource) — both send to one OTLP endpoint with one API key. The Fiddler OpenTelemetry collector, which is separate from the gateway and authorizes rather than produces, then checks each span's application.id (resource-preferred, span-fallback, fail-closed) against the applications the key may write to, storing authorized spans under their application and returning the rest in the partialSuccess response." width="880" height="700" data-path="images/multi-application-trace-routing-architecture.svg" />

<Steps>
  <Step title="Tag each span with its application">
    Set `application.id` on each span (or on its resource, when a whole `ResourceSpans` belongs to one application). One batch can contain spans for many different applications.
  </Step>

  <Step title="Send with one API key">
    POST the batch to `/v1/traces` with a single `Authorization: Bearer <YOUR_API_KEY>` header. No `fiddler-application-id` header is used in this flow — routing is driven by each span's `application.id`.
  </Step>

  <Step title="Fiddler authorizes each span">
    Fiddler resolves the full set of applications your API key may write to and matches each span's `application.id` against that set — no single application is declared up front.
  </Step>

  <Step title="Authorized spans are stored, the rest are reported">
    Authorized spans are ingested under their application. Unauthorized spans are not stored; instead they are counted and named in the response so you can act on them.
  </Step>
</Steps>

The call returns **HTTP 200** with an OTLP `partialSuccess` block whenever any span is rejected — see [The partialSuccess Response](#the-partialsuccess-response). Authorized spans in the same batch are still ingested.

<Note>
  **Why one API key can serve many applications.** In the single-application flow you declare one application up front (via the `fiddler-application-id` header) and Fiddler answers a single yes/no. Multi-application routing instead authorizes against your API key's **full set** of permitted applications and matches each span's `application.id` against it. Nothing is declared up front, so one API key serves every application it is allowed to write to.
</Note>

***

## Prerequisites

* One or more Fiddler applications created — you will need each **Application UUID**.
* A valid **Fiddler API key** (from **Settings** > **Credentials**) whose user has **write access** to each application you intend to send to. Access is governed by your role assignments — see [Access Control](#access-control).
* Multi-application routing **enabled on your Fiddler deployment**.

<Note>
  Multi-application routing is an opt-in capability. If sending a multi-application batch returns everything as rejected, confirm it is enabled on your deployment and that your API key has write access to the target applications — contact your Fiddler Customer Success Manager if you are unsure.

  **A newly created application takes a few minutes to propagate.** Fiddler caches each API key's authorized-application list (the collector's default is 5 minutes), so an application you just created can be rejected until that cache refreshes — wait a few minutes and resend.
</Note>

***

## How the Application ID Reaches Each Span

This is the question every shared-pipeline setup has to answer, so it is worth stating plainly.

The component that emits telemetry — a gateway, a collector, an egress service — is usually **not** the component that knows which Fiddler application a request belongs to. The **caller** knows that. So the application ID has to travel from the caller, through the emitter, and onto each span as the `application.id` attribute. Where and how you set it depends on what is emitting the spans:

* **One application per process** (an instrumented service) — set `application.id` on the resource, via an environment variable or the SDK. See [One Application per Process](#one-application-per-process).
* **Many applications from one process** (a gateway, or a service that switches application per request) — set `application.id` per span. See [Many Applications from One Process](#many-applications-from-one-process).
* **A batch or replay pipeline** — set `application.id` as you assemble each span; see the protobuf example below.

### Resource or Span by Topology

Fiddler resolves each span's application **resource-preferred, span-fallback**: a resource-level `application.id` decides for every span under it, a span-level `application.id` is used only when the resource sets none, and a span with neither fails closed (rejected as unauthorized). Which level you set depends on your topology — and for the shared-pipeline case this page targets, resource-level is the wrong choice:

* **One application per process** (an instrumented service) — an OpenTelemetry `TracerProvider` carries exactly one `Resource`, so a resource-level `application.id` tags every span the process emits with the same application. That is correct when a process serves one application — the [single-application flow](/integrations/agentic-ai/otel-trace-export).
* **Many applications from one process** (a gateway, collector, or shared exporter — what this page is about) — set `application.id` **per span**, and do **not** set it on the resource.

<Warning>
  **On a shared emitter, a resource-level `application.id` silently overrides every per-request value.** The process has one resource and resource beats span, so a resource-level value pins *everything* it sends to that one application and ignores the per-span value — with no error. For multi-application routing, set `application.id` per span and leave the resource unset. (A batch or replay pipeline is the exception: it groups each application's spans under their own `ResourceSpans`, so a resource-level value there is correct — see the protobuf example below.)
</Warning>

***

## Setting application.id

### One Application per Process

When a process serves a single application, set `application.id` on the **resource**. The cheapest form is environment variables — no code — and works with any OpenTelemetry SDK:

```bash theme={null}
OTEL_EXPORTER_OTLP_ENDPOINT="https://your-instance.fiddler.ai"
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer <YOUR_API_KEY>"
OTEL_RESOURCE_ATTRIBUTES="application.id=<APPLICATION_UUID>,service.name=my-agent-service"
```

Or in code, with the OpenTelemetry SDK's `Resource` helper (rather than hand-building protobuf):

```python theme={null}
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "application.id": "<APPLICATION_UUID>",
    "service.name": "my-agent-service",
})
# pass `resource` when you build your TracerProvider
```

This is still multi-application routing — just spread across processes: point many such single-application services at the **same** Fiddler endpoint and API key, and Fiddler routes each service's spans to its own application. No per-service pipeline, and no routing config to build.

### Many Applications from One Process

When one process emits spans for several applications — a gateway, or a service that switches application per request — set `application.id` **per span** and leave it off the resource (see [the trap above](#resource-or-span-by-topology)).

**With the OpenTelemetry SDK**, set it on each span as you create it, and do not put `application.id` on the `Resource`:

```python theme={null}
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# Configure the exporter once — no application.id on the Resource.
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://your-instance.fiddler.ai/v1/traces",
    headers={"Authorization": "Bearer <YOUR_API_KEY>"},
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

# Per request: tag the span with the caller's application ID — this routes it.
with tracer.start_as_current_span("chat") as span:
    span.set_attribute("application.id", "<APPLICATION_UUID>")
    span.set_attribute("fiddler.span.type", "llm")
    # ... your LLM call and other span attributes

# BatchSpanProcessor exports asynchronously; in a short-lived script, flush
# before exit or the spans never leave the process.
provider.shutdown()
```

**With AgentGateway**, the caller sends the application ID as a request header and a CEL expression copies it onto each span — the same shape AgentGateway already uses for `gen_ai.conversation.id`:

```yaml theme={null}
frontendPolicies:
  tracing:
    # ... host / protocol / auth as in the AgentGateway integration guide
    attributes:
      # Per-request application ID, copied from the caller's header onto each span.
      application.id: |
        request.headers["x-fiddler-application-id"] != "" ? request.headers["x-fiddler-application-id"] : ""
      gen_ai.conversation.id: |
        request.headers["x-fiddler-conversation-id"] != "" ? request.headers["x-fiddler-conversation-id"] : ""
```

Two changes from the single-application [AgentGateway integration](/integrations/agentic-ai/agentgateway-integration) config are required:

* **Move `application.id` out of `resources` and into `attributes`.** The single-application config sets `resources.application.id` to a fixed value; leaving it there pins every span to one application (resource beats span) and the header is ignored.
* **Drop the `fiddler-application-id` request header** that the single-application config adds via `requestHeaderModifier`. On a multi-application deployment that header is ignored, so it is dead config — removing it is a cleanup, not a correctness fix.

The caller then sends its application ID on every request, e.g. `X-Fiddler-Application-Id: <APPLICATION_UUID>` (exactly as it already sends `X-Fiddler-Conversation-Id`).

<Accordion title="Assembling batches as protobuf (pipelines, replay, ETL)">
  When you hold spans as data rather than running a live tracer — replaying spans from object storage, running an ETL job, or backfilling history — assemble the OTLP payload directly. Group each application's spans under their own `ResourceSpans` with that application's `application.id` on the resource (here a resource-level value is correct, because each `ResourceSpans` belongs to exactly one application), and send the whole batch with one API key.

  ```python theme={null}
  import gzip
  import httpx
  from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
  from opentelemetry.proto.common.v1.common_pb2 import AnyValue, InstrumentationScope, KeyValue
  from opentelemetry.proto.resource.v1.resource_pb2 import Resource
  from opentelemetry.proto.trace.v1.trace_pb2 import ResourceSpans, ScopeSpans, Span, Status

  FIDDLER_URL   = "https://your-instance.fiddler.ai"
  FIDDLER_API_KEY = "your-api-key"
  APP_A = "11111111-1111-1111-1111-111111111111"
  APP_B = "22222222-2222-2222-2222-222222222222"


  def llm_span(name: str) -> Span:
      return Span(
          trace_id=bytes.fromhex("4bf92f3577b34da6a3ce929d0e0e4736"),
          span_id=bytes.fromhex("00f067aa0ba902b7"),
          name=name,
          kind=Span.SPAN_KIND_INTERNAL,
          start_time_unix_nano=1_700_000_000_000_000_000,
          end_time_unix_nano=1_700_000_001_000_000_000,
          status=Status(code=Status.STATUS_CODE_OK),
          attributes=[
              KeyValue(key="fiddler.span.type",    value=AnyValue(string_value="llm")),
              KeyValue(key="gen_ai.request.model", value=AnyValue(string_value="gpt-4o")),
          ],
      )


  def app_resource_spans(app_uuid: str, span_name: str) -> ResourceSpans:
      return ResourceSpans(
          resource=Resource(attributes=[
              KeyValue(key="application.id", value=AnyValue(string_value=app_uuid)),
          ]),
          scope_spans=[ScopeSpans(
              scope=InstrumentationScope(name="my-tracer", version="1.0.0"),
              spans=[llm_span(span_name)],
          )],
      )


  # One batch, two applications
  payload = ExportTraceServiceRequest(resource_spans=[
      app_resource_spans(APP_A, "chat-app-a"),
      app_resource_spans(APP_B, "chat-app-b"),
  ]).SerializeToString()

  response = httpx.post(
      f"{FIDDLER_URL}/v1/traces",
      content=gzip.compress(payload),
      headers={
          "Authorization": f"Bearer {FIDDLER_API_KEY}",
          "Content-Type": "application/x-protobuf",
          "Content-Encoding": "gzip",
      },
      timeout=30.0,
  )
  response.raise_for_status()  # 200 even when some spans are rejected — inspect partial_success
  ```

  For the full span/attribute schema (span types, LLM attributes, typing) and a value-to-`AnyValue` helper, see [Exporting OTel Traces to Fiddler](/integrations/agentic-ai/otel-trace-export) and [Span and Resource Attributes](/integrations/agentic-ai/attributes).
</Accordion>

***

## Producer Support

Any producer that can set `application.id` **per span** can drive multi-application routing. Today that is:

| Producer          | Per-request multi-application routing | How                                                                   |
| ----------------- | ------------------------------------- | --------------------------------------------------------------------- |
| OpenTelemetry SDK | ✅ Supported                           | `span.set_attribute("application.id", ...)` per span                  |
| AgentGateway      | ✅ Supported                           | CEL maps a request header to the `application.id` span attribute      |
| LiteLLM           | ⚠️ Single application per proxy       | Set one `application.id` for the proxy via `OTEL_RESOURCE_ATTRIBUTES` |

<Note>
  **LiteLLM does not currently support per-request application routing.** A caller's per-request `application.id` (whether sent as a header or in request metadata) is captured only inside namespaced metadata attributes such as `metadata.requester_custom_headers`, which Fiddler resolves *after* the routing decision has already been made. Routing keys off the bare `application.id` at ingest, and LiteLLM never sets that per request — so a per-request application ID cannot influence which application a span lands in. (This is the same reason per-request `gen_ai.conversation.id` works but `application.id` does not: a conversation ID is a display field resolved late in the pipeline, while an application ID is a routing field needed at ingest — same transport, different timing.) Use LiteLLM for single-application export: set one `application.id` for the proxy via `OTEL_RESOURCE_ATTRIBUTES`. Per-request routing through LiteLLM would require a Fiddler-supplied callback that sets `application.id` at emit time — contact your Fiddler Customer Success Manager if you need it.
</Note>

***

## Migrating from the Single-Application Header

<Warning>
  **Once multi-application routing is enabled, the `fiddler-application-id` header is ignored.** Routing is driven entirely by each span's `application.id` attribute. A client that identified its application **only** through that header — with no `application.id` on the resource or span — will have every span fail closed (rejected as unauthorized).

  Fiddler's documented integrations already set `application.id` alongside the header: both the [LiteLLM](/integrations/agentic-ai/litellm-integration) and [AgentGateway](/integrations/agentic-ai/agentgateway-integration) guides set the `application.id` attribute as well, so they keep working. Only a client that was trimmed down to header-only is affected. Before enabling multi-application routing, confirm every producer sets `application.id` on the resource or span.
</Warning>

***

## Trust Model and Caller-Supplied Application IDs

Multi-application routing authorizes the **API key**, not the caller. Fiddler checks each span's `application.id` against the applications your API key is permitted to write to, and rejects the rest. This protects Fiddler's boundary between tenants: a caller cannot route a span to an application your API key has no access to.

It does **not**, on its own, isolate a shared gateway's callers from one another. The `application.id` on a span is caller-supplied input, and an API key that can write to many applications can write to **any** of them — so any caller of that gateway can tag a span as any application the gateway's API key can reach. If a shared gateway must keep its own callers in separate applications, enforce that at the gateway — for example, derive the application ID from an authenticated caller identity rather than trusting a raw client header. Per-span authorization protects Fiddler's tenancy boundary, not the gateway's internal one.

***

## The partialSuccess Response

When every span is authorized, the response is a plain `200` with an empty body. When one or more spans are rejected, Fiddler still returns `200` and populates the standard OTLP **`partialSuccess`** block:

| Field           | Meaning                                                                                          |
| --------------- | ------------------------------------------------------------------------------------------------ |
| `rejectedSpans` | How many spans were not stored because the API key was not authorized for their `application.id` |
| `errorMessage`  | A human-readable summary naming the unauthorized `application.id` values                         |

```json theme={null}
{
  "partialSuccess": {
    "rejectedSpans": 1,
    "errorMessage": "1 span(s) rejected: unauthorized application IDs [22222222-2222-2222-2222-222222222222]"
  }
}
```

<Warning>
  `partialSuccess` is **not** an error — the HTTP status is still `200` and the authorized spans in the batch are ingested. Do not treat a populated `partialSuccess` as a failed request; instead, read `rejectedSpans` and `errorMessage` to see which applications were rejected and why.
</Warning>

The most common cause of a rejection is an API key without write access to the named application. Grant the API key's user the appropriate role on that application (see below) and resend.

***

## Access Control

Authorization is per span: Fiddler compares each span's `application.id` against the set of applications the API key's user is allowed to write to. A span is stored only if its application is in that set; otherwise it is rejected and reported in `partialSuccess`.

To let an API key send to an application, ensure its user has a role that grants write access to that application's project. See [Role-Based Access Control](/reference/access-control/role-based-access) for the available roles, and [API Keys](/reference/administration/settings#credentials) for creating and managing API keys.

***

## Frequently Asked Questions

<AccordionGroup>
  <Accordion title="How does the application ID get onto a span through a gateway?">
    The caller sends it as a request header and the gateway copies it onto each span as the `application.id` attribute. For AgentGateway this is a one-line CEL expression — see [Many Applications from One Process](#many-applications-from-one-process). The gateway itself does not know which application a request belongs to; the caller does.
  </Accordion>

  <Accordion title="Do I still need the fiddler-application-id header?">
    No. In the multi-application flow, routing is driven by each span's `application.id` and the header is ignored. The single-application `fiddler-application-id` header still works for one-application-per-request exports on deployments without multi-application routing.
  </Accordion>

  <Accordion title="Resource-level or span-level application.id — which wins?">
    Resource-level takes precedence. If a `ResourceSpans` sets `application.id` on its resource, that value applies to every span under it; the span-level attribute is a fallback used only when the resource does not set one. From a shared emitter, set it per span so one process can route to many applications — see [the trap above](#resource-or-span-by-topology).
  </Accordion>

  <Accordion title="What happens to the authorized spans if some spans are rejected?">
    They are still ingested. Rejection is per span — authorized and unauthorized spans can share a batch, and only the unauthorized ones are dropped and reported in `partialSuccess`.
  </Accordion>

  <Accordion title="What if a span has no application.id at all?">
    It is treated as unauthorized (fail-closed): it is not stored and is counted in `rejectedSpans`.
  </Accordion>

  <Accordion title="What if application.id is malformed, or names an application that doesn't exist?">
    Both are rejected the same way as any unauthorized application — counted in `rejectedSpans` and named in `errorMessage`. Fiddler matches each span's `application.id` against the set of applications your API key can write to; a well-formed UUID that no application matches, or a value that isn't a valid UUID at all, is simply not in that set (Fiddler does not validate UUID format at ingest). Only a missing or empty `application.id` behaves differently — it fails closed (see the previous question).
  </Accordion>

  <Accordion title="Why is every span coming back rejected?">
    Common causes: the application was **just created** and hasn't propagated yet — Fiddler caches each API key's authorized-application list (the collector's default is 5 minutes), so a brand-new application is rejected until that cache refreshes; wait a few minutes and resend. Otherwise: the API key does not have write access to the target applications; multi-application routing is not enabled on your deployment; or a fixed resource-level `application.id` (or header-only setup) is overriding the per-span value. Verify the API key's roles, confirm the capability is enabled with your Fiddler Customer Success Manager, and check that `application.id` is set per span.
  </Accordion>
</AccordionGroup>

***

## Related Documentation

<CardGroup cols={2}>
  <Card title="AgentGateway Integration" icon="network-wired" href="/integrations/agentic-ai/agentgateway-integration">
    Configure AgentGateway's tracing, including the CEL header-to-attribute mapping.
  </Card>

  <Card title="Exporting OTel Traces to Fiddler" icon="upload" href="/integrations/agentic-ai/otel-trace-export">
    The single-application export flow, span schema, and attribute mapping.
  </Card>

  <Card title="OpenTelemetry Integration" icon="satellite-dish" href="/integrations/agentic-ai/opentelemetry-integration">
    Live agent instrumentation via the OTel SDK.
  </Card>

  <Card title="Span and Resource Attributes" icon="tags" href="/integrations/agentic-ai/attributes">
    Attribute typing, custom attributes, and how they flow into metrics.
  </Card>

  <Card title="Role-Based Access Control" icon="shield-check" href="/reference/access-control/role-based-access">
    Roles that govern which applications an API key can write to.
  </Card>

  <Card title="API Keys" icon="key" href="/reference/administration/settings#credentials">
    Create and manage the API keys used for trace ingestion.
  </Card>
</CardGroup>