Skip to main content
PyPI

​
0.6

​
Fiddler Evals SDK
July 30, 2026
  • New Features
    • Golden Datasets (New): Build evaluation datasets from real production traffic by promoting spans into a dataset. Application.get_span_fields() discovers which attribute keys your spans carry, with coverage counts and any evaluator outputs recorded on them. Dataset.add_items_from_spans() resolves span references server-side, applies a FieldMapping, and writes one dataset item per span. Dataset.get_schema() introspects an existing dataset’s fields so a new mapping stays consistent with it. Writes are all-or-nothing: if any span cannot be resolved, nothing is written and the call returns 422 with a SpanNotFound entry per unresolved span. Adds the SpanReference, FieldMapping, SearchFilter, and SearchScope models; structured span filtering reuses the existing QueryCondition, QueryRule, and OperatorType models.
    • Experiment Trace Capture (New): evaluate() now captures the OpenTelemetry spans your task emits and links them to the experiment item that produced them, with no configuration. Evaluators opt in to reading them by declaring a session parameter, which receives the item’s captured spans and enables scoring on execution — tool call counts, latency, retries, span exception events — rather than only on the output string. Capture is best-effort: if it is unavailable, session is None and the experiment runs normally.
  • Enhancements
    • fiddler-otel is now a base dependency: Required by trace capture, so it installs with the SDK rather than as an extra.
    • Pre-production application guard: If an existing FiddlerClient is scoped to a different application than the dataset’s, evaluate() raises ValueError instead of sending evaluation traces to that application.
  • Fixes
    • evaluate() returns an up-to-date experiment: ExperimentResult.experiment reflected create()-time state, so status read PENDING and duration_ms was None even after a completed run. The runner now reassigns the hydrated entity at each status transition.

​
0.5

​
Fiddler Evals SDK
June 11, 2026
  • Fixes
    • Experiment runner threading: Fixed an issue where the experiment runner could fail when executed inside a thread.

​
0.4

​
Fiddler Evals SDK
June 9, 2026
  • Breaking Changes
    • Experiment result models renamed: The experiment item data models were refactored. ExperimentItem is now ExperimentResultItem and ExperimentItemResult is now ScoredExperimentItem (with a new ExperimentResultScore model). Experiment.get_items() now returns Iterator[ExperimentResultItem]. Update imports from fiddler_evals.pydantic_models.experiment accordingly.
  • Enhancements
    • CustomJudge alignment: CustomJudge now aligns with Fiddler’s internal PromptSpecV2 model for consistent custom LLM-as-a-Judge evaluation.
  • Fixes
    • Fixed a URL construction issue affecting API requests.

​
0.3

​
Fiddler Evals SDK
February 5, 2026
  • New Evaluators
    • Context Relevance (New): Measures whether retrieved documents are relevant to the user query. Ordinal scoring — High (1.0), Medium (0.5), Low (0.0) with detailed reasoning.
    • RAG Faithfulness (New): LLM-as-a-Judge evaluator that assesses whether the response is grounded in the retrieved documents. Binary scoring — Yes (1.0) / No (0.0) with detailed reasoning.
    • CustomJudge (New): Build custom LLM-as-a-Judge evaluators using prompt_template with Jinja {{ placeholder }} syntax and output_fields for structured evaluation results.
  • Enhancements
    • Answer Relevance 2.0: Upgraded from binary to ordinal scoring — High (1.0), Medium (0.5), Low (0.0) with detailed reasoning.
    • Ordinal Score Bounding: Ordinal scores from the scoring API are now bounded to [0, 1].
​
Fiddler Evals SDK
November 19, 2025
  • Enhancements
    • Model and Credential Parameters: model and credential are now parameters on LLM-as-a-Judge evaluators, enabling configuration of the LLM used for evaluation.
    • Evaluator-Level Score Function Mapping: Evaluators now support score_fn_kwargs_mapping at the evaluator level for more flexible parameter binding.
    • Score Name Prefix: Added support for custom score name prefixes on evaluators.
    • Evals API Error Handling: Improved error handling and messaging for Evals API responses.
    • Coherence Prompt Input Required: The prompt input for the Coherence evaluator is now required.
    • Removed Pandas Core Dependency: Pandas moved from core to optional dependency, reducing install footprint.
    • Docstring Standardization: Fixed docstring errors and standardized documentation format across all evaluators.
  • Removals
    • Toxicity Evaluator Removed: The Toxicity evaluator has been removed from the SDK.
​
Fiddler Evals SDK
October 8, 2025
  • Initial Release
    • Core SDK with HTTP client, entity management (Project, Application, Dataset, Experiment), and the evaluate() function for running experiments.
    • Evaluators: AnswerRelevance, Coherence, Conciseness, Sentiment, TopicClassification, FTLPromptSafety, FTLResponseFaithfulness, RegexSearch, and support for user-defined function evaluators.
    • Data Input: Load test cases from pandas DataFrames, CSV files, or JSONL files.
    • Concurrent Processing: Parallel evaluation with ThreadPoolExecutor and tqdm progress tracking.
    • PyPI Publishing: Available as pip install fiddler-evals.