Skip to main content
PyPI

​
0.8

​
Fiddler Evals SDK
October 27, 2026
  • Breaking Changes
    • Python 3.10 support removed: fiddler-evals now requires Python 3.11 or newer. Python 3.10 reached end of life on October 1, 2026. See the Python Version Support Policy.
  • New Features
    • Dataset versions: A Dataset object now stands for one version of the dataset, available as Dataset.version. Load a past version with Dataset.get_by_id(..., version=N) or Dataset.get_by_name(..., version=N). get_items() reads the version of the dataset object you call it on, and evaluate() runs on the version of the dataset you pass it, so edits made after you load the dataset don’t change the experiment. Experiment.list() accepts dataset_version to list the experiments that ran on one version, and Experiment.create() and Experiment.get_or_create() accept dataset_version to pin the experiment to one version. Requires Fiddler Release 26.25 or later. On earlier releases, Dataset.version is None, and get_items() and evaluate() use the newest version of the dataset at the time they run.
    • Edit items in one write: Dataset.update_items(items=..., delete_ids=...) adds, replaces, and deletes items in a single write that creates one new dataset version. It returns an UpdateItemsResult with the dataset at the new version and the created, updated, deleted, and unchanged item IDs. Dataset.delete_items(ids) deletes items the same way. update_items() and delete_items() require Fiddler Release 26.25 or later. See the Release 26.25 notes. Build items from a pandas DataFrame, a CSV file, or a JSON Lines file with the new items_from_pandas(), items_from_csv(), and items_from_jsonl() functions.
  • Enhancements
    • Version conflict errors: update_items() raises StaleDatasetVersion when an item it names changed since the version you loaded, and writes nothing. It retries up to three times, waiting 1, 2, and then 4 seconds, when the dataset is busy (DatasetBusy) or temporarily unavailable for writes (DatasetWriteUnavailable). If the last retry also fails, it raises that error. insert() raises ItemIdCollision when an item ID already exists with different content. Each new error is a subclass of the error raised before, so existing except blocks keep working.
    • Security floors: Requires urllib3 2.8.0 or newer and anyio 4.14.2 or newer.
  • Deprecations
    • Deprecated insert methods: Version 0.8.0 deprecates insert(), its aliases add_items() and add_testcases(), and insert_from_pandas(), insert_from_csv_file(), and insert_from_jsonl_file(). They still work and now emit a DeprecationWarning. Use update_items(items=...), with items_from_pandas(), items_from_csv(), or items_from_jsonl() to build the items. update_items() requires Fiddler Release 26.25 or later; on earlier releases, keep using insert(). Unlike insert(), update_items() doesn’t change the Dataset object you call it on. Keep the dataset from the result it returns, for example dataset = dataset.update_items(items=items).dataset, so later steps such as evaluate() use the new version.
  • Fixes
    • Experiment.get_or_create() raises ValueError, instead of returning the existing experiment, when the name already exists on another dataset. When you pass dataset_version, it also raises ValueError if the existing experiment is pinned to another version, or if the server can’t report its version.
    • Importing items with a blank ID cell assigns a new ID, and a blank source_name or source_id becomes None instead of "nan".
    • Paged list methods such as Dataset.get_items() and Experiment.list() raise SDK errors such as NotFound instead of a raw requests.HTTPError.

​
0.7

​
Fiddler Evals SDK
October 19, 2026
  • New Evaluators
    • Centor Safety: Dynamic Model Escalation (New)
      This capability is currently in private preview and is not yet available to all customers. Contact your Fiddler Customer Success Manager to request access.
      Available under fiddler_evals.preview. See the Centor Safety DME release notes for details.

​
0.6

​
Fiddler Evals SDK
July 30, 2026
  • New Features
    • Golden Datasets (New): Build evaluation datasets from real production traffic by promoting spans into a dataset. Application.get_span_fields() discovers which attribute keys your spans carry, with coverage counts and any evaluator outputs recorded on them. Dataset.add_items_from_spans() resolves span references server-side, applies a FieldMapping, and writes one dataset item per span. Dataset.get_schema() introspects an existing dataset’s fields so a new mapping stays consistent with it. Writes are all-or-nothing: if any span cannot be resolved, nothing is written and the call returns 422 with a SpanNotFound entry per unresolved span. Adds the SpanReference, FieldMapping, SearchFilter, and SearchScope models; structured span filtering reuses the existing QueryCondition, QueryRule, and OperatorType models.
    • Experiment Trace Capture (New): evaluate() now captures the OpenTelemetry spans your task emits and links them to the experiment item that produced them, with no configuration. Evaluators opt in to reading them by declaring a session parameter, which receives the item’s captured spans and enables scoring on execution — tool call counts, latency, retries, span exception events — rather than only on the output string. Capture is best-effort: if it is unavailable, session is None and the experiment runs normally.
  • Enhancements
    • fiddler-otel is now a base dependency: Required by trace capture, so it installs with the SDK rather than as an extra.
    • Pre-production application guard: If an existing FiddlerClient is scoped to a different application than the dataset’s, evaluate() raises ValueError instead of sending evaluation traces to that application.
  • Fixes
    • evaluate() returns an up-to-date experiment: ExperimentResult.experiment reflected create()-time state, so status read PENDING and duration_ms was None even after a completed run. The runner now reassigns the hydrated entity at each status transition.

​
0.5

​
Fiddler Evals SDK
June 11, 2026
  • Fixes
    • Experiment runner threading: Fixed an issue where the experiment runner could fail when executed inside a thread.

​
0.4

​
Fiddler Evals SDK
June 9, 2026
  • Breaking Changes
    • Experiment result models renamed: The experiment item data models were refactored. ExperimentItem is now ExperimentResultItem and ExperimentItemResult is now ScoredExperimentItem (with a new ExperimentResultScore model). Experiment.get_items() now returns Iterator[ExperimentResultItem]. Update imports from fiddler_evals.pydantic_models.experiment accordingly.
  • Enhancements
    • CustomJudge alignment: CustomJudge now aligns with Fiddler’s internal PromptSpecV2 model for consistent custom LLM-as-a-Judge evaluation.
  • Fixes
    • Fixed a URL construction issue affecting API requests.

​
0.3

​
Fiddler Evals SDK
February 5, 2026
  • New Evaluators
    • Context Relevance (New): Measures whether retrieved documents are relevant to the user query. Ordinal scoring — High (1.0), Medium (0.5), Low (0.0) with detailed reasoning.
    • RAG Faithfulness (New): LLM-as-a-Judge evaluator that assesses whether the response is grounded in the retrieved documents. Binary scoring — Yes (1.0) / No (0.0) with detailed reasoning.
    • CustomJudge (New): Build custom LLM-as-a-Judge evaluators using prompt_template with Jinja {{ placeholder }} syntax and output_fields for structured evaluation results.
  • Enhancements
    • Answer Relevance 2.0: Upgraded from binary to ordinal scoring — High (1.0), Medium (0.5), Low (0.0) with detailed reasoning.
    • Ordinal Score Bounding: Ordinal scores from the scoring API are now bounded to [0, 1].
​
Fiddler Evals SDK
November 19, 2025
  • Enhancements
    • Model and Credential Parameters: model and credential are now parameters on LLM-as-a-Judge evaluators, enabling configuration of the LLM used for evaluation.
    • Evaluator-Level Score Function Mapping: Evaluators now support score_fn_kwargs_mapping at the evaluator level for more flexible parameter binding.
    • Score Name Prefix: Added support for custom score name prefixes on evaluators.
    • Evals API Error Handling: Improved error handling and messaging for Evals API responses.
    • Coherence Prompt Input Required: The prompt input for the Coherence evaluator is now required.
    • Removed Pandas Core Dependency: Pandas moved from core to optional dependency, reducing install footprint.
    • Docstring Standardization: Fixed docstring errors and standardized documentation format across all evaluators.
  • Removals
    • Toxicity Evaluator Removed: The Toxicity evaluator has been removed from the SDK.
​
Fiddler Evals SDK
October 8, 2025
  • Initial Release
    • Core SDK with HTTP client, entity management (Project, Application, Dataset, Experiment), and the evaluate() function for running experiments.
    • Evaluators: AnswerRelevance, Coherence, Conciseness, Sentiment, TopicClassification, FTLPromptSafety, FTLResponseFaithfulness, RegexSearch, and support for user-defined function evaluators.
    • Data Input: Load test cases from pandas DataFrames, CSV files, or JSONL files.
    • Concurrent Processing: Parallel evaluation with ThreadPoolExecutor and tqdm progress tracking.
    • PyPI Publishing: Available as pip install fiddler-evals.