Fireworks AI Integration

You can integrate Fiddler with Fireworks AI to observe, evaluate, and improve AI agents and LLM applications that run on open models. Fireworks serves open chat models behind an OpenAI-compatible API, and Fiddler records every model call, tool call, and agent step as a trace, then scores your application’s quality and safety. This page explains how Fireworks AI and Fiddler fit together, then walks you through a complete tutorial. You learn how to trace Fireworks model calls with the Fiddler OTel SDK, trace a tool-calling agent across a multi-turn conversation, test your instrumentation locally, run experiments with a Fireworks model as an LLM-as-a-Judge evaluator, and instrument a LangChain agent that runs on Fireworks.

Background

Fireworks AI is an inference and training platform for open models, such as Kimi, DeepSeek, GLM, Qwen, MiniMax, and gpt-oss. You can call models on shared serverless capacity and pay per token, or run them on dedicated deployments. You can also fine-tune models on your own data and serve the tuned model on the same platform. Because the Fireworks API is compatible with the OpenAI API, you use the official OpenAI client libraries and point them at the Fireworks base URL, https://api.fireworks.ai/inference/v1. Frameworks that support OpenAI-compatible providers, such as LangChain and LangGraph, work the same way. Fiddler complements Fireworks as the AI Control Plane for your Fireworks-powered applications and agents, from development through production:

Agentic observability: Trace every request through your agent, including the prompts, completions, token usage, and tool calls of each step, and group traces into conversations so you can understand every agent decision.
Evaluation: Score every trace for quality and safety with Fiddler Centor Models, which route each input automatically to the right evaluation model and return results in under 80ms, inside your environment, with no per-evaluation API calls. For criteria unique to your use case, bring your own judge model served on Fireworks.
Experiments: Test your application against a dataset of test cases, and compare models, prompts, and fine-tuned versions before you ship.
Monitoring: Track quality, safety, latency, and cost metrics in production, diagnose root causes down to the span, and get alerted when metrics change.
Guardrails and governance: Block unsafe inputs and outputs, such as PII leakage and jailbreak attempts, in real time with guardrails powered by Centor Models, and keep an audit trail of every enforcement decision for compliance reporting.

Key Concepts

Concept	Fireworks AI	Fiddler
Model calls	Generates completions and tool calls with an open chat model, and returns token usage.	Records each call as a generation span with the model, messages, and token counts.
Tools	Decides which tools to call through OpenAI-compatible function calling.	Records each tool run as a tool span with its input and output.
Agents	Not involved beyond the model calls.	Groups the model and tool spans of one request under a chain span, and groups requests into conversations.
Evaluation	Serves the application under test, and can serve the judge model for custom evaluators.	Runs experiments over datasets and stores the scores.
Fine-tuning	Trains a model on your data and serves it.	Compares the tuned model with the base model in experiments and in production monitoring.

Supported Models

The following serverless models support function calling and work with this tutorial. Fireworks adds and retires models regularly — for the current list, including models that you can run on dedicated deployments, see the Fireworks model library.

Model ID	Used in this tutorial for
`accounts/fireworks/models/gpt-oss-120b`	Single model calls
`accounts/fireworks/models/kimi-k3`	The agent and the judge
`accounts/fireworks/models/glm-5p3`	An alternative for any step
`accounts/fireworks/models/deepseek-v4p1-flash`	An alternative for any step
`accounts/fireworks/models/minimax-m3`	An alternative for any step

Tutorial: Trace and Evaluate a Fireworks Agent

In this tutorial, you instrument a customer support agent that runs on a Fireworks model and looks up orders with a tool. Then you send its traces to Fiddler and evaluate its answers.

Prerequisites

To complete this tutorial, you need:

A Fiddler GenAI application. In Fiddler, go to GenAI Applications, click Add Application, and copy the application ID. Then go to Settings > Credentials and copy your API key. To learn more, see the Fiddler OTel SDK documentation.
A Fireworks AI API key. To create one, go to API Keys in the Fireworks console.
Python 3.11, 3.12, 3.13, or 3.14.

Set Up the Environment

Install the dependencies

pip install --quiet --upgrade fiddler-otel "fiddler-evals>=0.8.0" openai

Set your environment variables

export FIDDLER_URL="https://<your-instance>.fiddler.ai"
export FIDDLER_API_KEY="<fiddler-api-key>"
export FIDDLER_APPLICATION_ID="<fiddler-application-id>"
export FIREWORKS_API_KEY="<fireworks-api-key>"

Create the Fiddler and Fireworks clients

The Fiddler client exports spans to your application. The Fireworks client is the standard OpenAI client pointed at the Fireworks base URL.

import json
import os
import uuid

from fiddler_otel import FiddlerClient, get_current_span, set_conversation_id, trace
from openai import OpenAI

fiddler = FiddlerClient(
    application_id=os.environ["FIDDLER_APPLICATION_ID"],
    api_key=os.environ["FIDDLER_API_KEY"],
    url=os.environ["FIDDLER_URL"],
)
fireworks = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.environ["FIREWORKS_API_KEY"],
)

CHAT_MODEL = "accounts/fireworks/models/gpt-oss-120b"
AGENT_MODEL = "accounts/fireworks/models/kimi-k3"
JUDGE_MODEL = "accounts/fireworks/models/kimi-k3"

Trace a Fireworks Model Call

Decorate the function that calls Fireworks with @trace(as_type="generation"). The model and system arguments record the model ID and the provider on the span. Inside the function, get_current_span returns the span, so that you can record the prompts, the completion, and the token usage that Fireworks returns.

SUPPORT_PROMPT = "You are a concise, friendly customer support assistant for an online store."


@trace(
    as_type="generation",
    capture_input=False,  # Record the prompts explicitly below instead
    capture_output=False,
    model=CHAT_MODEL,
    system="fireworks",
)
def ask(question):
    span = get_current_span(as_type="generation")
    response = fireworks.chat.completions.create(
        model=CHAT_MODEL,
        messages=[
            {"role": "system", "content": SUPPORT_PROMPT},
            {"role": "user", "content": question},
        ],
    )
    answer = response.choices[0].message.content

    if span:
        span.set_system_prompt(SUPPORT_PROMPT)
        span.set_user_prompt(question)
        span.set_completion(answer)
        span.set_usage(
            input_tokens=response.usage.prompt_tokens,
            output_tokens=response.usage.completion_tokens,
        )
    return answer


print(ask("What is your return policy for opened electronics?"))

Trace a Tool-Calling Agent

An agent request contains several model calls and tool runs. In this section, you trace each model call as a generation span, each tool run as a tool span, and the whole request as a chain span. Fiddler nests the spans by call stack, so the trace shows every step that the agent took.

Define the tool and trace it

The model sees the tool’s name, description, and parameters, and returns a tool call when it wants to look up an order.

ORDERS = {
    "A1001": {"status": "shipped", "carrier": "UPS", "estimated_delivery": "2027-03-12"},
    "A1002": {"status": "processing", "estimated_ship_date": "2027-03-10"},
}

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order",
            "description": "Look up the status and delivery details of an order.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string", "description": "The order ID, such as A1001."}
                },
                "required": ["order_id"],
            },
        },
    }
]


@trace(as_type="tool", capture_input=False, capture_output=False)
def lookup_order(order_id):
    span = get_current_span(as_type="tool")
    result = ORDERS.get(order_id, {"error": f"No order with ID {order_id}."})
    if span:
        span.set_tool_name("lookup_order")
        span.set_tool_input({"order_id": order_id})
        span.set_tool_output(result)
    return result

Trace each model call of the agent

The span records the model with set_model, the full conversation with set_messages, the response with set_output_messages, and the available tools with set_tool_definitions. Unlike the model argument of @trace, which is fixed when Python defines the function, set_model reads AGENT_MODEL on every call, so the spans stay accurate when you switch models to compare them.set_messages expects each tool call as a flat object with id, name, and arguments fields, but the OpenAI client nests the name and arguments under function. The for_fiddler helper flattens them, so the tool names appear on the span.

def for_fiddler(messages):
    converted = []
    for message in messages:
        message = dict(message)
        if message.get("tool_calls"):
            message["tool_calls"] = [
                {
                    "id": call["id"],
                    "name": call["function"]["name"],
                    "arguments": call["function"]["arguments"],
                }
                for call in message["tool_calls"]
            ]
        converted.append(message)
    return converted


@trace(
    as_type="generation",
    capture_input=False,
    capture_output=False,
    system="fireworks",
)
def call_agent_model(messages):
    span = get_current_span(as_type="generation")
    response = fireworks.chat.completions.create(
        model=AGENT_MODEL, messages=messages, tools=TOOLS
    )
    message = response.choices[0].message.model_dump(exclude_none=True)

    if span:
        span.set_model(AGENT_MODEL)  # Read at call time, so model comparisons stay labeled correctly
        span.set_messages(for_fiddler(messages))
        span.set_output_messages(for_fiddler([message]))
        span.set_tool_definitions(TOOLS)
        span.set_usage(
            input_tokens=response.usage.prompt_tokens,
            output_tokens=response.usage.completion_tokens,
        )
    return message

Run the agent loop inside a chain span

The agent calls the model, runs any tools that the model requests, sends the tool results back, and repeats until the model returns a final answer.

AGENT_PROMPT = (
    "You are a customer support agent for an online store. Always call "
    "lookup_order before you answer a question about an order."
)


@trace(as_type="chain", capture_input=False, capture_output=False)
def run_agent(history, user_message, max_steps=5):
    span = get_current_span(as_type="chain")
    if span:
        span.set_agent_name("support_agent")
        span.set_input(user_message)

    history.append({"role": "user", "content": user_message})
    for _ in range(max_steps):
        message = call_agent_model([{"role": "system", "content": AGENT_PROMPT}] + history)
        history.append(message)

        if not message.get("tool_calls"):
            if span:
                span.set_output(message["content"])
            return message["content"]

        for call in message["tool_calls"]:
            args = json.loads(call["function"]["arguments"])
            history.append(
                {
                    "role": "tool",
                    "tool_call_id": call["id"],
                    "content": json.dumps(lookup_order(**args)),
                }
            )

    raise RuntimeError("The agent didn't finish within max_steps.")

Run a two-turn conversation

set_conversation_id tags every span that follows with the same conversation ID, so that Fiddler shows both turns as one conversation.

set_conversation_id(str(uuid.uuid4()))
history = []
print(run_agent(history, "Where is my order A1001?"))
print(run_agent(history, "And what about A1002?"))

fiddler.force_flush()

The client exports spans in the background and flushes them when your program exits. Short scripts and notebooks should call force_flush to export the spans immediately.

View the traces in Fiddler

In Fiddler, open your application from GenAI Applications to see the traces. Each turn produces one trace with this structure:

run_agent                  support_agent handles the turn
├── call_agent_model       kimi-k3 returns a lookup_order tool call
├── lookup_order           {"order_id": "A1001"} returns the order status
└── call_agent_model       kimi-k3 returns the final answer

Each generation span shows the messages, the tool definitions, and the token usage of that Fireworks call. Both turns appear under the same conversation.

Test Your Instrumentation Locally

To check your spans before you send them to Fiddler, create the client in offline mode instead of the client from Set Up the Environment. In offline mode, the client doesn’t need an API key or URL, and writes spans as OpenTelemetry Protocol (OTLP) JSON files to a local directory instead of exporting them.

fiddler = FiddlerClient(
    application_id=os.environ["FIDDLER_APPLICATION_ID"],
    otlp_enabled=False,                       # Don't export to Fiddler
    otlp_json_capture_enabled=True,           # Write OTLP JSON files instead
    otlp_json_output_dir="./fiddler_traces",
)

Fiddler can also ingest these files from Amazon S3. To learn more, see S3 Trace Ingestion.

Run an Experiment with the Fiddler Evals SDK

Before you change a prompt, a model, or a fine-tuned version, run an experiment to measure the effect. An experiment runs your application on each test case in a dataset and scores the outputs with evaluators. This section uses the Fiddler Evals SDK.

Connect to Fiddler and create a dataset of test cases

The dataset belongs to the same application that receives your traces. A Python process shares one FiddlerClient, and the Evals SDK raises a ValueError if that client and the dataset belong to different applications.

from fiddler_evals import Application, Dataset, evaluate, init
from fiddler_evals.pydantic_models.dataset import NewDatasetItem

init(url=os.environ["FIDDLER_URL"], token=os.environ["FIDDLER_API_KEY"])

application = Application.get_by_id(id_=os.environ["FIDDLER_APPLICATION_ID"])
dataset = Dataset.create(
    name="order_questions",
    application_id=application.id,
    description="Order status questions for the support agent",
)
dataset.insert(
    [
        NewDatasetItem(
            inputs={"question": "Has order A1001 shipped yet?"},
            expected_outputs={"answer": "Yes. It shipped with UPS and arrives on 2027-03-12."},
        ),
        NewDatasetItem(
            inputs={"question": "When will order A1002 ship?"},
            expected_outputs={"answer": "It's processing and ships on 2027-03-10."},
        ),
        NewDatasetItem(
            inputs={"question": "What's the status of order Z9999?"},
            expected_outputs={"answer": "There's no order with that ID."},
        ),
    ]
)

Experiment runs produce real traces. Outside this tutorial, run experiments against a dedicated pre-production application so that they don’t mix with production traffic. See Keep Evaluation Traces Out of Production.

Create an LLM-as-a-Judge evaluator that runs on a Fireworks model

A custom evaluator subclasses Evaluator and returns a Score from its score method. This evaluator asks the judge model whether the agent’s answer agrees with the expected answer.

from fiddler_evals.evaluators.base import Evaluator
from fiddler_evals.pydantic_models.score import Score

JUDGE_PROMPT = (
    "You grade a customer support agent. Compare the agent's answer with the "
    "expected answer. Reply with a JSON object with two keys: score, which is 1 "
    "if the answer agrees with the expected answer on every fact and 0 "
    "otherwise, and reasoning, which explains the score in one sentence."
)


class FireworksCorrectness(Evaluator):
    def score(self, question: str, answer: str, expected_answer: str) -> Score:
        response = fireworks.chat.completions.create(
            model=JUDGE_MODEL,
            messages=[
                {"role": "system", "content": JUDGE_PROMPT},
                {
                    "role": "user",
                    "content": f"Question: {question}\n"
                    f"Expected answer: {expected_answer}\n"
                    f"Agent's answer: {answer}",
                },
            ],
            response_format={"type": "json_object"},
        )
        verdict = json.loads(response.choices[0].message.content)
        return Score(
            name="fireworks_correctness",
            evaluator_name=self.name,
            value=float(verdict["score"]),
            reasoning=verdict["reasoning"],
        )

Run the experiment

The task function runs the agent on each test case. The Evals SDK gives each test case its own conversation ID and links its traces to the test case, so experiment runs don’t merge into the tutorial’s two-turn conversation. score_fn_kwargs_mapping tells each evaluator where to find its arguments.

def support_task(inputs, extras, metadata):
    return {"answer": run_agent([], inputs["question"])}


results = evaluate(
    dataset=dataset,
    task=support_task,
    evaluators=[FireworksCorrectness()],
    name_prefix="kimi_k3_support",
    score_fn_kwargs_mapping={
        "question": lambda x: x["inputs"]["question"],
        "answer": "answer",
        "expected_answer": lambda x: x["expected_outputs"]["answer"],
    },
)

for result in results.results:
    for score in result.scores:
        print(f"{score.name}: {score.value} ({score.reasoning})")

To learn how the Evals SDK captures these traces and how evaluators can score them, see Capture Traces During Experiments.You can add Fiddler’s built-in evaluators, such as AnswerRelevance and Coherence, to the evaluators list. To learn which arguments and models they need, see the Fiddler Evals SDK.

Review and compare experiments

In Fiddler, open the experiment to review the score of each test case. To compare models, change AGENT_MODEL, run the experiment again with a different name_prefix, and compare the two experiments.

Use Fireworks with the Fiddler LangChain SDK

If you build agents with LangChain’s create_agent, use the Fiddler LangChain SDK to trace them automatically, and the langchain-fireworks package for the chat model. The instrumentor records the model calls, tool calls, and token usage of every agent that you create after you call instrument. ChatFireworks reads your Fireworks API key from the FIREWORKS_API_KEY environment variable.

pip install --quiet --upgrade fiddler-langchain langchain langchain-fireworks

import os
import uuid

import langchain.agents
from fiddler_langchain import FiddlerLangChainInstrumentor, set_conversation_id
from fiddler_otel import FiddlerClient
from langchain_core.tools import tool
from langchain_fireworks import ChatFireworks

fiddler = FiddlerClient(
    application_id=os.environ["FIDDLER_APPLICATION_ID"],
    api_key=os.environ["FIDDLER_API_KEY"],
    url=os.environ["FIDDLER_URL"],
)
FiddlerLangChainInstrumentor(client=fiddler).instrument()


@tool
def lookup_order(order_id: str) -> dict:
    """Look up the status and delivery details of an order."""
    orders = {"A1001": {"status": "shipped", "carrier": "UPS", "estimated_delivery": "2027-03-12"}}
    return orders.get(order_id, {"error": f"No order with ID {order_id}."})


agent = langchain.agents.create_agent(
    model=ChatFireworks(model="accounts/fireworks/models/kimi-k3"),
    tools=[lookup_order],
    system_prompt="You are a customer support agent. Call lookup_order for order questions.",
    name="support_agent",
)

set_conversation_id(str(uuid.uuid4()))
result = agent.invoke({"messages": [{"role": "user", "content": "Where is order A1001?"}]})
print(result["messages"][-1].content)

fiddler.force_flush()

Troubleshooting

Traces Are Missing in Fiddler

Call fiddler.force_flush() before a short script or notebook cell finishes. Check that FIDDLER_APPLICATION_ID is the UUID copied from GenAI Applications and that FIDDLER_URL has no trailing path. For more checks, see Fiddler OTel SDK troubleshooting.

Tool Calls Appear on Generation Spans Without a Name

Pass the messages through for_fiddler before you call set_messages or set_output_messages. The SDK reads the tool name from a flat name field, not from the OpenAI client’s nested function.name field.

Fireworks Returns a Model Not Found Error

Check the model ID against the Fireworks model library, and confirm that the model is available on serverless capacity. Models on a dedicated deployment use your deployment’s model ID instead.

Next Steps

Try other Fireworks models in the agent by changing AGENT_MODEL, and compare them in Fiddler experiments. See the Fireworks model library.
Fine-tune a model on Fireworks with conversations that score well in Fiddler, then compare the tuned model with the base model. To learn more, see Fine-tuning models.
Score your production traces automatically with Centor Models and evaluator rules.
Add more tools to the agent. To learn more about tool definitions, see Fireworks tool calling.

Agentic AI & LLM Frameworks

Data Platforms & Pipelines

ML Platforms & Tools

Cloud Platforms & Deployment

Monitoring & Alerting

Background

Key Concepts

Supported Models

Tutorial: Trace and Evaluate a Fireworks Agent

Prerequisites

Set Up the Environment

Trace a Fireworks Model Call

Trace a Tool-Calling Agent

Test Your Instrumentation Locally

Run an Experiment with the Fiddler Evals SDK

Use Fireworks with the Fiddler LangChain SDK

Troubleshooting

Traces Are Missing in Fiddler

Tool Calls Appear on Generation Spans Without a Name

Fireworks Returns a Model Not Found Error

Next Steps

​Background

​Key Concepts

​Supported Models

​Tutorial: Trace and Evaluate a Fireworks Agent

​Prerequisites

​Set Up the Environment

​Trace a Fireworks Model Call

​Trace a Tool-Calling Agent

​Test Your Instrumentation Locally

​Run an Experiment with the Fiddler Evals SDK

​Use Fireworks with the Fiddler LangChain SDK

​Troubleshooting

​Traces Are Missing in Fiddler

​Tool Calls Appear on Generation Spans Without a Name

​Fireworks Returns a Model Not Found Error

​Next Steps

​Related Resources

Background

Key Concepts

Supported Models

Tutorial: Trace and Evaluate a Fireworks Agent

Prerequisites

Set Up the Environment

Trace a Fireworks Model Call

Trace a Tool-Calling Agent

Test Your Instrumentation Locally

Run an Experiment with the Fiddler Evals SDK

Use Fireworks with the Fiddler LangChain SDK

Troubleshooting

Traces Are Missing in Fiddler

Tool Calls Appear on Generation Spans Without a Name

Fireworks Returns a Model Not Found Error

Next Steps

Related Resources