> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fiddler.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Capture Traces During Experiments

> Link the OpenTelemetry traces your task emits to the experiment item that produced them, and score evaluators on the captured spans.

A score tells you *that* an item failed.
The trace tells you *why*.

When you run an experiment, the Fiddler Evals SDK captures the OpenTelemetry spans your task emits and links them to the experiment item that produced them.
You get per-item traces in the UI for debugging, and your evaluators can score on the execution itself — tool call order, retry counts, token usage, latency — not just the final output string.

## What You'll Learn

* How traces are captured during an experiment run, with no setup
* How to score an evaluator on a captured trace
* How to keep evaluation traces out of your production application

**Time to complete**: \~15 minutes

## Prerequisites

* A task instrumented with OpenTelemetry — either through a [Fiddler integration](/integrations/agentic-ai-and-llm-frameworks/agentic-ai) or your own spans
* A Fiddler API key from [**Settings** > **Credentials**](/reference/administration/settings#credentials)
* Python 3.10 or later
* **Fiddler Evals SDK**: `pip install fiddler-evals`

<Info>
  If you prefer a notebook, open the fully worked example in [Google Colab](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Experiment_Trace_Capture.ipynb) or download it from [GitHub](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_Experiment_Trace_Capture.ipynb).
</Info>

## There Is Nothing to Turn On

Trace capture is automatic.
`fiddler-otel` is a base dependency of the Evals SDK, so `evaluate()` sets capture up on its own as long as it can resolve a Fiddler application for the dataset.

```python theme={null}
from fiddler_evals import Application, Dataset, Project, evaluate, init
from fiddler_evals.evaluators import AnswerRelevance

init(url='https://your-org.fiddler.ai', token='your-api-key')

project = Project.get_by_name(name='support')
application = Application.get_by_name(
    name='support-agent-preprod',
    project_id=project.id,
)
dataset = Dataset.get_by_name(
    name='support-agent-eval-suite',
    application_id=application.id,
)


def my_agent(inputs, extras, metadata):
    """Your instrumented application. Signature must be (inputs, extras, metadata)."""
    user_query = inputs['user_query']
    answer = call_your_agent(user_query)
    return {'user_query': user_query, 'rag_response': answer}


result = evaluate(
    dataset=dataset,
    task=my_agent,
    evaluators=[AnswerRelevance(model='openai/gpt-4o-mini', credential='my-openai-cred')],
    name_prefix='support_agent_v3',
)
```

If your task emits spans, they are captured and linked.
If it does not, nothing changes.

<Warning>
  Evaluators bind to your task's **outputs** by parameter name.
  The `inputs` bucket is passed through as a single dict, not spread into the namespace, so an evaluator cannot read a dataset input directly.

  `AnswerRelevance.score()` takes `(user_query, rag_response)`, which is why the task above returns `user_query` as well as the answer.
  Omit it and the run fails with `ScoreFunctionInvalidArgs: Missing required arguments ... ['user_query']`.
  Alternatively, remap with [`score_fn_kwargs_mapping`](/sdk-api/evals/evaluate).
</Warning>

Capture is **best-effort by design**: evaluation must work for users who do no tracing at all.
If setup fails, the SDK logs a warning and the experiment runs normally — the task still executes and the scores are still published.
There is exactly one exception, covered in [Keep evaluation traces out of production](#keep-evaluation-traces-out-of-production).

## Score an Evaluator on the Trace

Declare a `session` parameter on your evaluator's score function and the runner passes the captured spans to it.
Parameter binding is by name, so evaluators that do not declare `session` are unaffected.

```python theme={null}
from fiddler_evals import EvalFn, evaluate


def tool_call_efficiency(session=None):
    """Score 1.0 when the agent used three or fewer tool calls.

    `session` is None when trace capture is unavailable, so always guard.
    Returning None marks the score SKIPPED rather than failing the item.
    """
    if session is None:
        return None

    tool_spans = [
        span for span in session.spans
        if span['attributes'].get('fiddler.span.type') == 'tool'
    ]
    return len(tool_spans) <= 3


result = evaluate(
    dataset=dataset,
    task=my_agent,
    evaluators=[EvalFn(tool_call_efficiency, score_name='tool_call_efficiency')],
    name_prefix='support_agent_v3',
)
```

Always default `session` to `None` and handle the `None` case.
Capture is best-effort, so an evaluator that assumes a session breaks the moment tracing is unavailable.

`EvalFn` converts your return value for you: `bool` becomes 1.0 or 0.0, `int` and `float` pass through, and `None` produces a `SKIPPED` score.
Return a `Score` directly when you want to control the reasoning text:

```python theme={null}
from fiddler_evals import Score


def tool_call_efficiency(session=None):
    if session is None:
        return None

    tool_spans = [
        span for span in session.spans
        if span['attributes'].get('fiddler.span.type') == 'tool'
    ]
    names = ', '.join(span['name'] for span in tool_spans) or 'none'
    return Score(
        name='tool_call_efficiency',
        evaluator_name='tool_call_efficiency',
        value=1.0 if len(tool_spans) <= 3 else 0.0,
        reasoning=f'{len(tool_spans)} tool calls: {names}',
    )
```

A `Score` you construct yourself requires both `name` and `evaluator_name`, and the explanation field is `reasoning`.

### What a Session Contains

Your evaluator receives a `Session` with two attributes:

<ParamField path="session_id" type="UUID">
  The experiment item's ID. Equal to `experiment_item.id`.
</ParamField>

<ParamField path="spans" type="list[dict]">
  The spans the task produced, in completion order.
</ParamField>

Each span is a plain dict — no OpenTelemetry objects to import:

| Key                  | Type          | Notes                                      |
| -------------------- | ------------- | ------------------------------------------ |
| `trace_id`           | `str`         | Hex, `0x`-prefixed                         |
| `span_id`            | `str`         | Hex, `0x`-prefixed                         |
| `parent_span_id`     | `str \| None` | `None` for root spans                      |
| `name`               | `str`         | Span name                                  |
| `kind`               | `str \| None` | `INTERNAL`, `CLIENT`, `SERVER`, …          |
| `start_time`         | `int`         | Nanoseconds                                |
| `end_time`           | `int`         | Nanoseconds                                |
| `status_code`        | `str \| None` | `OK`, `ERROR`, `UNSET`                     |
| `status_description` | `str \| None` |                                            |
| `attributes`         | `dict`        | Where instrumentations put payloads        |
| `events`             | `list[dict]`  | Each has `name`, `timestamp`, `attributes` |

Times are integer nanoseconds so duration is a plain subtraction with no precision loss:

```python theme={null}
def responded_within_3s(session=None):
    if session is None or not session.spans:
        return None

    root = next(
        (span for span in session.spans if span['parent_span_id'] is None),
        session.spans[-1],
    )
    duration_ms = (root['end_time'] - root['start_time']) / 1_000_000
    return duration_ms < 3000
```

Exceptions arrive as span events rather than attributes, which is where most instrumentations record them:

```python theme={null}
def no_swallowed_exceptions(session=None):
    if session is None:
        return None

    exceptions = [
        event
        for span in session.spans
        for event in span['events']
        if event['name'] == 'exception'
    ]
    return not exceptions
```

The span dict is deliberately a curated subset aimed at scoring.
OpenTelemetry infrastructure fields — `resource`, `links`, and instrumentation scope — are omitted because they are constant or empty for eval runs.
The full-fidelity span is still written to the trace store and visible in the UI, so nothing is lost.

## Keep Evaluation Traces Out of Production

Evaluation traces are real traces, so send them to a dedicated pre-production application rather than the one serving live traffic.

A process shares one global `FiddlerClient`. If one already exists and its `application_id` does not match the dataset's application, `evaluate()` raises `ValueError` instead of degrading quietly — this is the one capture failure that is deliberately not best-effort, because silently mixing evaluation and production traffic is worse than a failed run.

Point the dataset at its own application and the SDK handles the rest:

```python theme={null}
eval_application = Application.get_or_create(
    name='support-agent-preprod',
    project_id=project.id,
)

dataset = Dataset.get_or_create(
    name='support-agent-eval-suite',
    application_id=eval_application.id,
)
```

<Note>
  A golden dataset [promoted from production spans](/evaluate-and-test/golden-datasets) necessarily belongs to the application that served those spans —
  span promotion resolves spans through the dataset's application, and trace capture follows it.
  Replaying such a dataset writes its captured traces to that same application.
</Note>

## Next Steps

* [Build a golden dataset from production spans](/evaluate-and-test/golden-datasets)
* [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start)
* [Instrument your application with OpenTelemetry](/developers/quick-starts/opentelemetry-quick-start)
* [`Experiment` SDK reference](/sdk-api/evals/experiment)