> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fiddler.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Fireworks AI Integration

> Monitor and evaluate Fireworks AI applications with Fiddler — trace open-model calls, tool calls, and agent steps, run experiments, and use a Fireworks model as an LLM-as-a-Judge evaluator.

You can integrate Fiddler with [Fireworks AI](https://fireworks.ai) to observe, evaluate, and improve AI agents and LLM applications that run on open models.
Fireworks serves open chat models behind an OpenAI-compatible API,
and Fiddler records every model call, tool call, and agent step as a trace,
then scores your application's quality and safety.

This page explains how Fireworks AI and Fiddler fit together, then walks you through a complete tutorial.
You learn how to trace Fireworks model calls with the [Fiddler OTel SDK](https://docs.fiddler.ai/integrations/agentic-ai/fiddler-otel-sdk.md),
trace a tool-calling agent across a multi-turn conversation,
test your instrumentation locally,
run [experiments](https://docs.fiddler.ai/glossary/experiments.md) with a Fireworks model as an LLM-as-a-Judge evaluator,
and instrument a LangChain agent that runs on Fireworks.

## Background

Fireworks AI is an inference and training platform for open models, such as Kimi, DeepSeek, GLM, Qwen, MiniMax, and gpt-oss.
You can call models on shared serverless capacity and pay per token, or run them on dedicated deployments.
You can also fine-tune models on your own data and serve the tuned model on the same platform.

Because the Fireworks API is compatible with the OpenAI API,
you use the official OpenAI client libraries and point them at the Fireworks base URL, `https://api.fireworks.ai/inference/v1`.
Frameworks that support OpenAI-compatible providers, such as LangChain and LangGraph, work the same way.

Fiddler complements Fireworks as the AI Control Plane for your Fireworks-powered applications and agents,
from development through production:

* **[Agentic observability](https://docs.fiddler.ai/glossary/agentic-observability.md):** Trace every request through your agent,
  including the prompts, completions, token usage, and tool calls of each step,
  and group traces into conversations so you can understand every agent decision.
* **Evaluation:** Score every trace for quality and safety with [Fiddler Centor Models](https://docs.fiddler.ai/glossary/centor-models.md),
  which route each input automatically to the right evaluation model and return results in under 80ms,
  inside your environment, with no per-evaluation API calls.
  For criteria unique to your use case, bring your own judge model served on Fireworks.
* **Experiments:** Test your application against a dataset of test cases,
  and compare models, prompts, and fine-tuned versions before you ship.
* **Monitoring:** Track quality, safety, latency, and cost metrics in production,
  diagnose root causes down to the span, and get alerted when metrics change.
* **[Guardrails](https://docs.fiddler.ai/protection/guardrails.md) and governance:** Block unsafe inputs and outputs,
  such as PII leakage and jailbreak attempts, in real time with guardrails powered by Centor Models,
  and keep an audit trail of every enforcement decision for compliance reporting.

## Key Concepts

| Concept | Fireworks AI | Fiddler |
| - | - | - |
| Model calls | Generates completions and tool calls with an open chat model, and returns token usage. | Records each call as a generation span with the model, messages, and token counts. |
| Tools | Decides which tools to call through OpenAI-compatible function calling. | Records each tool run as a tool span with its input and output. |
| Agents | Not involved beyond the model calls. | Groups the model and tool spans of one request under a chain span, and groups requests into conversations. |
| Evaluation | Serves the application under test, and can serve the judge model for custom evaluators. | Runs experiments over datasets and stores the scores. |
| Fine-tuning | Trains a model on your data and serves it. | Compares the tuned model with the base model in experiments and in production monitoring. |

## Supported Models

The following serverless models support function calling and work with this tutorial.
Fireworks adds and retires models regularly —
for the current list, including models that you can run on dedicated deployments, see the [Fireworks model library](https://fireworks.ai/models).

| Model ID | Used in this tutorial for |
| - | - |
| `accounts/fireworks/models/gpt-oss-120b` | Single model calls |
| `accounts/fireworks/models/kimi-k3` | The agent and the judge |
| `accounts/fireworks/models/glm-5p3` | An alternative for any step |
| `accounts/fireworks/models/deepseek-v4p1-flash` | An alternative for any step |
| `accounts/fireworks/models/minimax-m3` | An alternative for any step |

## Tutorial: Trace and Evaluate a Fireworks Agent

In this tutorial, you instrument a customer support agent that runs on a Fireworks model and looks up orders with a tool.
Then you send its traces to Fiddler and evaluate its answers.

### Prerequisites

To complete this tutorial, you need:

* **A Fiddler GenAI application.** In Fiddler, go to **GenAI Applications**, click **Add Application**, and copy the application ID.
  Then go to **Settings** > **Credentials** and copy your API key.
  To learn more, see the [Fiddler OTel SDK](https://docs.fiddler.ai/integrations/agentic-ai/fiddler-otel-sdk.md#application-setup) documentation.
* **A Fireworks AI API key.** To create one, go to [API Keys](https://app.fireworks.ai/settings/users/api-keys) in the Fireworks console.
* **Python 3.11, 3.12, 3.13, or 3.14.**

### Set Up the Environment

**1. Install the dependencies**

```bash theme={null}
pip install --quiet --upgrade fiddler-otel "fiddler-evals>=0.8.0" openai
```

**2. Set your environment variables**

```bash theme={null}
export FIDDLER_URL="https://<your-instance>.fiddler.ai"
export FIDDLER_API_KEY="<fiddler-api-key>"
export FIDDLER_APPLICATION_ID="<fiddler-application-id>"
export FIREWORKS_API_KEY="<fireworks-api-key>"
```

**3. Create the Fiddler and Fireworks clients**

The Fiddler client exports spans to your application.
The Fireworks client is the standard OpenAI client pointed at the Fireworks base URL.

```python theme={null}
import json
import os
import uuid

from fiddler_otel import FiddlerClient, get_current_span, set_conversation_id, trace
from openai import OpenAI

fiddler = FiddlerClient(
    application_id=os.environ["FIDDLER_APPLICATION_ID"],
    api_key=os.environ["FIDDLER_API_KEY"],
    url=os.environ["FIDDLER_URL"],
)
fireworks = OpenAI(
    base_url="https://api.fireworks.ai/inference/v1",
    api_key=os.environ["FIREWORKS_API_KEY"],
)

CHAT_MODEL = "accounts/fireworks/models/gpt-oss-120b"
AGENT_MODEL = "accounts/fireworks/models/kimi-k3"
JUDGE_MODEL = "accounts/fireworks/models/kimi-k3"
```

### Trace a Fireworks Model Call

Decorate the function that calls Fireworks with `@trace(as_type="generation")`.
The `model` and `system` arguments record the model ID and the provider on the span.
Inside the function, `get_current_span` returns the span,
so that you can record the prompts, the completion, and the token usage that Fireworks returns.

```python theme={null}
SUPPORT_PROMPT = "You are a concise, friendly customer support assistant for an online store."


@trace(
    as_type="generation",
    capture_input=False,  # Record the prompts explicitly below instead
    capture_output=False,
    model=CHAT_MODEL,
    system="fireworks",
)
def ask(question):
    span = get_current_span(as_type="generation")
    response = fireworks.chat.completions.create(
        model=CHAT_MODEL,
        messages=[
            {"role": "system", "content": SUPPORT_PROMPT},
            {"role": "user", "content": question},
        ],
    )
    answer = response.choices[0].message.content

    if span:
        span.set_system_prompt(SUPPORT_PROMPT)
        span.set_user_prompt(question)
        span.set_completion(answer)
        span.set_usage(
            input_tokens=response.usage.prompt_tokens,
            output_tokens=response.usage.completion_tokens,
        )
    return answer


print(ask("What is your return policy for opened electronics?"))
```

### Trace a Tool-Calling Agent

An agent request contains several model calls and tool runs.
In this section, you trace each model call as a generation span, each tool run as a tool span, and the whole request as a chain span.
Fiddler nests the spans by call stack, so the trace shows every step that the agent took.

**1. Define the tool and trace it**

The model sees the tool's name, description, and parameters,
and returns a tool call when it wants to look up an order.

```python theme={null}
ORDERS = {
    "A1001": {"status": "shipped", "carrier": "UPS", "estimated_delivery": "2027-03-12"},
    "A1002": {"status": "processing", "estimated_ship_date": "2027-03-10"},
}

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order",
            "description": "Look up the status and delivery details of an order.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string", "description": "The order ID, such as A1001."}
                },
                "required": ["order_id"],
            },
        },
    }
]


@trace(as_type="tool", capture_input=False, capture_output=False)
def lookup_order(order_id):
    span = get_current_span(as_type="tool")
    result = ORDERS.get(order_id, {"error": f"No order with ID {order_id}."})
    if span:
        span.set_tool_name("lookup_order")
        span.set_tool_input({"order_id": order_id})
        span.set_tool_output(result)
    return result
```

**2. Trace each model call of the agent**

The span records the model with `set_model`, the full conversation with `set_messages`,
the response with `set_output_messages`, and the available tools with `set_tool_definitions`.
Unlike the `model` argument of `@trace`, which is fixed when Python defines the function,
`set_model` reads `AGENT_MODEL` on every call, so the spans stay accurate when you switch models to compare them.

`set_messages` expects each tool call as a flat object with `id`, `name`, and `arguments` fields,
but the OpenAI client nests the name and arguments under `function`.
The `for_fiddler` helper flattens them, so the tool names appear on the span.

```python theme={null}
def for_fiddler(messages):
    converted = []
    for message in messages:
        message = dict(message)
        if message.get("tool_calls"):
            message["tool_calls"] = [
                {
                    "id": call["id"],
                    "name": call["function"]["name"],
                    "arguments": call["function"]["arguments"],
                }
                for call in message["tool_calls"]
            ]
        converted.append(message)
    return converted


@trace(
    as_type="generation",
    capture_input=False,
    capture_output=False,
    system="fireworks",
)
def call_agent_model(messages):
    span = get_current_span(as_type="generation")
    response = fireworks.chat.completions.create(
        model=AGENT_MODEL, messages=messages, tools=TOOLS
    )
    message = response.choices[0].message.model_dump(exclude_none=True)

    if span:
        span.set_model(AGENT_MODEL)  # Read at call time, so model comparisons stay labeled correctly
        span.set_messages(for_fiddler(messages))
        span.set_output_messages(for_fiddler([message]))
        span.set_tool_definitions(TOOLS)
        span.set_usage(
            input_tokens=response.usage.prompt_tokens,
            output_tokens=response.usage.completion_tokens,
        )
    return message
```

**3. Run the agent loop inside a chain span**

The agent calls the model, runs any tools that the model requests, sends the tool results back,
and repeats until the model returns a final answer.

```python theme={null}
AGENT_PROMPT = (
    "You are a customer support agent for an online store. Always call "
    "lookup_order before you answer a question about an order."
)


@trace(as_type="chain", capture_input=False, capture_output=False)
def run_agent(history, user_message, max_steps=5):
    span = get_current_span(as_type="chain")
    if span:
        span.set_agent_name("support_agent")
        span.set_input(user_message)

    history.append({"role": "user", "content": user_message})
    for _ in range(max_steps):
        message = call_agent_model([{"role": "system", "content": AGENT_PROMPT}] + history)
        history.append(message)

        if not message.get("tool_calls"):
            if span:
                span.set_output(message["content"])
            return message["content"]

        for call in message["tool_calls"]:
            args = json.loads(call["function"]["arguments"])
            history.append(
                {
                    "role": "tool",
                    "tool_call_id": call["id"],
                    "content": json.dumps(lookup_order(**args)),
                }
            )

    raise RuntimeError("The agent didn't finish within max_steps.")
```

**4. Run a two-turn conversation**

`set_conversation_id` tags every span that follows with the same conversation ID,
so that Fiddler shows both turns as one conversation.

```python theme={null}
set_conversation_id(str(uuid.uuid4()))
history = []
print(run_agent(history, "Where is my order A1001?"))
print(run_agent(history, "And what about A1002?"))

fiddler.force_flush()
```

The client exports spans in the background and flushes them when your program exits.
Short scripts and notebooks should call `force_flush` to export the spans immediately.

**5. View the traces in Fiddler**

In Fiddler, open your application from **GenAI Applications** to see the traces.
Each turn produces one trace with this structure:

```text theme={null}
run_agent                  support_agent handles the turn
├── call_agent_model       kimi-k3 returns a lookup_order tool call
├── lookup_order           {"order_id": "A1001"} returns the order status
└── call_agent_model       kimi-k3 returns the final answer
```

Each generation span shows the messages, the tool definitions, and the token usage of that Fireworks call.
Both turns appear under the same conversation.

### Test Your Instrumentation Locally

To check your spans before you send them to Fiddler, create the client in offline mode instead of the client from [Set Up the Environment](#set-up-the-environment).
In offline mode, the client doesn't need an API key or URL,
and writes spans as OpenTelemetry Protocol (OTLP) JSON files to a local directory instead of exporting them.

```python theme={null}
fiddler = FiddlerClient(
    application_id=os.environ["FIDDLER_APPLICATION_ID"],
    otlp_enabled=False,                       # Don't export to Fiddler
    otlp_json_capture_enabled=True,           # Write OTLP JSON files instead
    otlp_json_output_dir="./fiddler_traces",
)
```

Fiddler can also ingest these files from Amazon S3.
To learn more, see [S3 Trace Ingestion](https://docs.fiddler.ai/integrations/agentic-ai/s3-trace-ingestion.md).

### Run an Experiment with the Fiddler Evals SDK

Before you change a prompt, a model, or a fine-tuned version, run an experiment to measure the effect.
An experiment runs your application on each test case in a dataset and scores the outputs with evaluators.
This section uses the [Fiddler Evals SDK](https://docs.fiddler.ai/integrations/agentic-ai/evals-sdk.md).

**1. Connect to Fiddler and create a dataset of test cases**

The dataset belongs to the same application that receives your traces.
A Python process shares one `FiddlerClient`,
and the Evals SDK raises a `ValueError` if that client and the dataset belong to different applications.

```python theme={null}
from fiddler_evals import Application, Dataset, evaluate, init
from fiddler_evals.pydantic_models.dataset import NewDatasetItem

init(url=os.environ["FIDDLER_URL"], token=os.environ["FIDDLER_API_KEY"])

application = Application.get_by_id(id_=os.environ["FIDDLER_APPLICATION_ID"])
dataset = Dataset.create(
    name="order_questions",
    application_id=application.id,
    description="Order status questions for the support agent",
)
dataset.insert(
    [
        NewDatasetItem(
            inputs={"question": "Has order A1001 shipped yet?"},
            expected_outputs={"answer": "Yes. It shipped with UPS and arrives on 2027-03-12."},
        ),
        NewDatasetItem(
            inputs={"question": "When will order A1002 ship?"},
            expected_outputs={"answer": "It's processing and ships on 2027-03-10."},
        ),
        NewDatasetItem(
            inputs={"question": "What's the status of order Z9999?"},
            expected_outputs={"answer": "There's no order with that ID."},
        ),
    ]
)
```

> [!TIP]
>
> Experiment runs produce real traces.
> Outside this tutorial, run experiments against a dedicated pre-production application so that they don't mix with production traffic.
> See [Keep Evaluation Traces Out of Production](https://docs.fiddler.ai/evaluate-and-test/experiment-trace-capture.md#keep-evaluation-traces-out-of-production).

**2. Create an LLM-as-a-Judge evaluator that runs on a Fireworks model**

A custom evaluator subclasses `Evaluator` and returns a `Score` from its `score` method.
This evaluator asks the judge model whether the agent's answer agrees with the expected answer.

```python theme={null}
from fiddler_evals.evaluators.base import Evaluator
from fiddler_evals.pydantic_models.score import Score

JUDGE_PROMPT = (
    "You grade a customer support agent. Compare the agent's answer with the "
    "expected answer. Reply with a JSON object with two keys: score, which is 1 "
    "if the answer agrees with the expected answer on every fact and 0 "
    "otherwise, and reasoning, which explains the score in one sentence."
)


class FireworksCorrectness(Evaluator):
    def score(self, question: str, answer: str, expected_answer: str) -> Score:
        response = fireworks.chat.completions.create(
            model=JUDGE_MODEL,
            messages=[
                {"role": "system", "content": JUDGE_PROMPT},
                {
                    "role": "user",
                    "content": f"Question: {question}\n"
                    f"Expected answer: {expected_answer}\n"
                    f"Agent's answer: {answer}",
                },
            ],
            response_format={"type": "json_object"},
        )
        verdict = json.loads(response.choices[0].message.content)
        return Score(
            name="fireworks_correctness",
            evaluator_name=self.name,
            value=float(verdict["score"]),
            reasoning=verdict["reasoning"],
        )
```

**3. Run the experiment**

The task function runs the agent on each test case.
The Evals SDK gives each test case its own conversation ID and links its traces to the test case,
so experiment runs don't merge into the tutorial's two-turn conversation.
`score_fn_kwargs_mapping` tells each evaluator where to find its arguments.

```python theme={null}
def support_task(inputs, extras, metadata):
    return {"answer": run_agent([], inputs["question"])}


results = evaluate(
    dataset=dataset,
    task=support_task,
    evaluators=[FireworksCorrectness()],
    name_prefix="kimi_k3_support",
    score_fn_kwargs_mapping={
        "question": lambda x: x["inputs"]["question"],
        "answer": "answer",
        "expected_answer": lambda x: x["expected_outputs"]["answer"],
    },
)

for result in results.results:
    for score in result.scores:
        print(f"{score.name}: {score.value} ({score.reasoning})")
```

To learn how the Evals SDK captures these traces and how evaluators can score them, see [Capture Traces During Experiments](https://docs.fiddler.ai/evaluate-and-test/experiment-trace-capture.md).

You can add Fiddler's built-in evaluators, such as `AnswerRelevance` and `Coherence`, to the `evaluators` list.
To learn which arguments and models they need, see the [Fiddler Evals SDK](https://docs.fiddler.ai/integrations/agentic-ai/evals-sdk.md).

**4. Review and compare experiments**

In Fiddler, open the experiment to review the score of each test case.
To compare models, change `AGENT_MODEL`, run the experiment again with a different `name_prefix`,
and compare the two experiments.

## Use Fireworks with the Fiddler LangChain SDK

If you build agents with LangChain's `create_agent`,
use the [Fiddler LangChain SDK](https://docs.fiddler.ai/integrations/agentic-ai/langchain-sdk.md) to trace them automatically,
and the [`langchain-fireworks`](https://docs.langchain.com/oss/python/integrations/chat/fireworks) package for the chat model.
The instrumentor records the model calls, tool calls, and token usage of every agent that you create after you call `instrument`.
`ChatFireworks` reads your Fireworks API key from the `FIREWORKS_API_KEY` environment variable.

```bash theme={null}
pip install --quiet --upgrade fiddler-langchain langchain langchain-fireworks
```

```python theme={null}
import os
import uuid

import langchain.agents
from fiddler_langchain import FiddlerLangChainInstrumentor, set_conversation_id
from fiddler_otel import FiddlerClient
from langchain_core.tools import tool
from langchain_fireworks import ChatFireworks

fiddler = FiddlerClient(
    application_id=os.environ["FIDDLER_APPLICATION_ID"],
    api_key=os.environ["FIDDLER_API_KEY"],
    url=os.environ["FIDDLER_URL"],
)
FiddlerLangChainInstrumentor(client=fiddler).instrument()


@tool
def lookup_order(order_id: str) -> dict:
    """Look up the status and delivery details of an order."""
    orders = {"A1001": {"status": "shipped", "carrier": "UPS", "estimated_delivery": "2027-03-12"}}
    return orders.get(order_id, {"error": f"No order with ID {order_id}."})


agent = langchain.agents.create_agent(
    model=ChatFireworks(model="accounts/fireworks/models/kimi-k3"),
    tools=[lookup_order],
    system_prompt="You are a customer support agent. Call lookup_order for order questions.",
    name="support_agent",
)

set_conversation_id(str(uuid.uuid4()))
result = agent.invoke({"messages": [{"role": "user", "content": "Where is order A1001?"}]})
print(result["messages"][-1].content)

fiddler.force_flush()
```

## Troubleshooting

### Traces Are Missing in Fiddler

Call `fiddler.force_flush()` before a short script or notebook cell finishes.
Check that `FIDDLER_APPLICATION_ID` is the UUID copied from **GenAI Applications** and that `FIDDLER_URL` has no trailing path.
For more checks, see [Fiddler OTel SDK troubleshooting](https://docs.fiddler.ai/integrations/agentic-ai/fiddler-otel-sdk.md#troubleshooting).

### Tool Calls Appear on Generation Spans Without a Name

Pass the messages through `for_fiddler` before you call `set_messages` or `set_output_messages`.
The SDK reads the tool name from a flat `name` field, not from the OpenAI client's nested `function.name` field.

### Fireworks Returns a Model Not Found Error

Check the model ID against the [Fireworks model library](https://fireworks.ai/models),
and confirm that the model is available on serverless capacity.
Models on a dedicated deployment use your deployment's model ID instead.

## Next Steps

* Try other Fireworks models in the agent by changing `AGENT_MODEL`, and compare them in Fiddler experiments.
  See the [Fireworks model library](https://fireworks.ai/models).
* Fine-tune a model on Fireworks with conversations that score well in Fiddler, then compare the tuned model with the base model.
  To learn more, see [Fine-tuning models](https://docs.fireworks.ai/fine-tuning/fine-tuning-models).
* Score your production traces automatically with Centor Models and [evaluator rules](https://docs.fiddler.ai/evaluate-and-test/evaluator-rules.md).
* Add more tools to the agent.
  To learn more about tool definitions, see [Fireworks tool calling](https://docs.fireworks.ai/guides/function-calling).

## Related Resources

* [Fireworks AI documentation](https://docs.fireworks.ai)
* [Fiddler OTel SDK](https://docs.fiddler.ai/integrations/agentic-ai/fiddler-otel-sdk.md)
* [Fiddler Evals SDK](https://docs.fiddler.ai/integrations/agentic-ai/evals-sdk.md)
* [Fiddler LangChain SDK](https://docs.fiddler.ai/integrations/agentic-ai/langchain-sdk.md)