Capture Traces During Experiments

A score tells you that an item failed. The trace tells you why. When you run an experiment, the Fiddler Evals SDK captures the OpenTelemetry spans your task emits and links them to the experiment item that produced them. You get per-item traces in the UI for debugging, and your evaluators can score on the execution itself — tool call order, retry counts, token usage, latency — not just the final output string.

What You’ll Learn

How traces are captured during an experiment run, with no setup
How to score an evaluator on a captured trace
How to keep evaluation traces out of your production application

Time to complete: ~15 minutes

Prerequisites

A task instrumented with OpenTelemetry — either through a Fiddler integration or your own spans
A Fiddler API key from Settings > Credentials
Python 3.10 or later
Fiddler Evals SDK: pip install fiddler-evals

If you prefer a notebook, open the fully worked example in Google Colab or download it from GitHub.

There Is Nothing to Turn On

Trace capture is automatic. fiddler-otel is a base dependency of the Evals SDK, so evaluate() sets capture up on its own as long as it can resolve a Fiddler application for the dataset.

from fiddler_evals import Application, Dataset, Project, evaluate, init
from fiddler_evals.evaluators import AnswerRelevance

init(url='https://your-org.fiddler.ai', token='your-api-key')

project = Project.get_by_name(name='support')
application = Application.get_by_name(
    name='support-agent-preprod',
    project_id=project.id,
)
dataset = Dataset.get_by_name(
    name='support-agent-eval-suite',
    application_id=application.id,
)


def my_agent(inputs, extras, metadata):
    """Your instrumented application. Signature must be (inputs, extras, metadata)."""
    user_query = inputs['user_query']
    answer = call_your_agent(user_query)
    return {'user_query': user_query, 'rag_response': answer}


result = evaluate(
    dataset=dataset,
    task=my_agent,
    evaluators=[AnswerRelevance(model='openai/gpt-4o-mini', credential='my-openai-cred')],
    name_prefix='support_agent_v3',
)

If your task emits spans, they are captured and linked. If it does not, nothing changes.

Evaluators bind to your task’s outputs by parameter name. The inputs bucket is passed through as a single dict, not spread into the namespace, so an evaluator cannot read a dataset input directly.AnswerRelevance.score() takes (user_query, rag_response), which is why the task above returns user_query as well as the answer. Omit it and the run fails with ScoreFunctionInvalidArgs: Missing required arguments ... ['user_query']. Alternatively, remap with score_fn_kwargs_mapping.

Capture is best-effort by design: evaluation must work for users who do no tracing at all. If setup fails, the SDK logs a warning and the experiment runs normally — the task still executes and the scores are still published. There is exactly one exception, covered in Keep evaluation traces out of production.

Score an Evaluator on the Trace

Declare a session parameter on your evaluator’s score function and the runner passes the captured spans to it. Parameter binding is by name, so evaluators that do not declare session are unaffected.

from fiddler_evals import EvalFn, evaluate


def tool_call_efficiency(session=None):
    """Score 1.0 when the agent used three or fewer tool calls.

    `session` is None when trace capture is unavailable, so always guard.
    Returning None marks the score SKIPPED rather than failing the item.
    """
    if session is None:
        return None

    tool_spans = [
        span for span in session.spans
        if span['attributes'].get('fiddler.span.type') == 'tool'
    ]
    return len(tool_spans) <= 3


result = evaluate(
    dataset=dataset,
    task=my_agent,
    evaluators=[EvalFn(tool_call_efficiency, score_name='tool_call_efficiency')],
    name_prefix='support_agent_v3',
)

Always default session to None and handle the None case. Capture is best-effort, so an evaluator that assumes a session breaks the moment tracing is unavailable. EvalFn converts your return value for you: bool becomes 1.0 or 0.0, int and float pass through, and None produces a SKIPPED score. Return a Score directly when you want to control the reasoning text:

from fiddler_evals import Score


def tool_call_efficiency(session=None):
    if session is None:
        return None

    tool_spans = [
        span for span in session.spans
        if span['attributes'].get('fiddler.span.type') == 'tool'
    ]
    names = ', '.join(span['name'] for span in tool_spans) or 'none'
    return Score(
        name='tool_call_efficiency',
        evaluator_name='tool_call_efficiency',
        value=1.0 if len(tool_spans) <= 3 else 0.0,
        reasoning=f'{len(tool_spans)} tool calls: {names}',
    )

A Score you construct yourself requires both name and evaluator_name, and the explanation field is reasoning.

What a Session Contains

Your evaluator receives a Session with two attributes:

UUID

The experiment item’s ID. Equal to experiment_item.id.

list[dict]

The spans the task produced, in completion order.

Each span is a plain dict — no OpenTelemetry objects to import:

Key	Type	Notes
`trace_id`	`str`	Hex, `0x`-prefixed
`span_id`	`str`	Hex, `0x`-prefixed
`parent_span_id`	`str \| None`	`None` for root spans
`name`	`str`	Span name
`kind`	`str \| None`	`INTERNAL`, `CLIENT`, `SERVER`, …
`start_time`	`int`	Nanoseconds
`end_time`	`int`	Nanoseconds
`status_code`	`str \| None`	`OK`, `ERROR`, `UNSET`
`status_description`	`str \| None`
`attributes`	`dict`	Where instrumentations put payloads
`events`	`list[dict]`	Each has `name`, `timestamp`, `attributes`

Times are integer nanoseconds so duration is a plain subtraction with no precision loss:

def responded_within_3s(session=None):
    if session is None or not session.spans:
        return None

    root = next(
        (span for span in session.spans if span['parent_span_id'] is None),
        session.spans[-1],
    )
    duration_ms = (root['end_time'] - root['start_time']) / 1_000_000
    return duration_ms < 3000

Exceptions arrive as span events rather than attributes, which is where most instrumentations record them:

def no_swallowed_exceptions(session=None):
    if session is None:
        return None

    exceptions = [
        event
        for span in session.spans
        for event in span['events']
        if event['name'] == 'exception'
    ]
    return not exceptions

The span dict is deliberately a curated subset aimed at scoring. OpenTelemetry infrastructure fields — resource, links, and instrumentation scope — are omitted because they are constant or empty for eval runs. The full-fidelity span is still written to the trace store and visible in the UI, so nothing is lost.

Keep Evaluation Traces Out of Production

Evaluation traces are real traces, so send them to a dedicated pre-production application rather than the one serving live traffic. A process shares one global FiddlerClient. If one already exists and its application_id does not match the dataset’s application, evaluate() raises ValueError instead of degrading quietly — this is the one capture failure that is deliberately not best-effort, because silently mixing evaluation and production traffic is worse than a failed run. Point the dataset at its own application and the SDK handles the rest:

eval_application = Application.get_or_create(
    name='support-agent-preprod',
    project_id=project.id,
)

dataset = Dataset.get_or_create(
    name='support-agent-eval-suite',
    application_id=eval_application.id,
)

A golden dataset promoted from production spans necessarily belongs to the application that served those spans — span promotion resolves spans through the dataset’s application, and trace capture follows it. Replaying such a dataset writes its captured traces to that same application.

Getting Started

Concepts

Evaluate & Test

Reference

What You’ll Learn

Prerequisites

There Is Nothing to Turn On

Score an Evaluator on the Trace

What a Session Contains

Keep Evaluation Traces Out of Production

Next Steps

​What You’ll Learn

​Prerequisites

​There Is Nothing to Turn On

​Score an Evaluator on the Trace

​What a Session Contains

​Keep Evaluation Traces Out of Production

​Next Steps

What You’ll Learn

Prerequisites

There Is Nothing to Turn On

Score an Evaluator on the Trace

What a Session Contains

Keep Evaluation Traces Out of Production

Next Steps