Dataset - Fiddler Documentation

A Dataset is a logical container within an Application that stores structured test cases with inputs and expected outputs for GenAI evaluation. Every change to its test cases commits a new dataset version, and a Dataset object stands for one version (see version): reading its items and running experiments on it both use that version. Key Features:

Test Case Storage: Container for structured evaluation test cases
Application Context: Datasets are scoped within applications for isolation
Metadata Management: Custom metadata for organization
Versioning: Each change to the test cases creates a new version, and each experiment records the version it ran on
Evaluation Foundation: Structured data for GenAI application testing
Lifecycle Management: Coordinated creation, updates, and deletion of datasets

Dataset Lifecycle:

Creation: Create dataset with unique name within an application
Configuration: Add test cases and metadata
Evaluation: Use dataset for testing GenAI applications
Maintenance: Update test cases and metadata as needed; each change to the test cases creates a new version
Cleanup: Delete dataset when no longer needed

Example

# Create a new dataset for fraud detection tests
dataset = Dataset.create(
    name="fraud-detection-tests",
    application_id=application_id,
    description="Test cases for fraud detection model",
    metadata={"source": "production"},
)
print(f"Created dataset: {dataset.name} (ID: {dataset.id})")

Datasets are permanent containers - once created, the name cannot be changed. Deleting a dataset removes all contained test cases and metadata. Consider the organizational structure carefully before creating datasets.

Attributes

UUID

Unique identifier (UUID) of the dataset.

str

Dataset name; unique within the application and immutable after creation.

datetime

Server-side timestamp of when the dataset was created.

datetime

Server-side timestamp of when the dataset was last updated.

UserCompact

Compact reference (id, full name, email) to the user who created the dataset.

UserCompact

Compact reference (id, full name, email) to the user who last updated the dataset.

ProjectCompact

Compact reference (id and name) to the project containing this dataset.

ApplicationCompact

Compact reference (id and name) to the application containing this dataset.

bool

default:"True"

Whether the dataset is active; set at creation and changeable via update(). Defaults to True.

str | None

default:"None"

Optional human-readable description of the dataset; None if not set.

dict

Custom metadata dictionary attached to the dataset; empty when not set.

DatasetVersionCompact | None

default:"None"

The dataset version this object stands for. get_items() reads it and evaluate() pins experiments to it. Loaded without version=, it is the newest version at load time. insert(), the insert_from_* helpers and add_items_from_spans() move it to the version their write created; update_items() returns a new object instead. None when unknown, in which case reads and experiments use the newest version at that moment. Requires Fiddler 26.25 or later; earlier servers do not report it. New in version 0.8.0

classmethod get_by_id()

Retrieve a dataset by its unique identifier. Fetches a dataset from the Fiddler platform using its UUID. This is the most direct way to retrieve a dataset when you know its ID. Changed in version 0.8.0: Added version.

Parameters

UUID | str

required

The unique identifier (UUID) of the dataset to retrieve. Can be provided as a UUID object or string representation.

int | None

default:"None"

Optional dataset version number for the returned object to stand for. Omitted, the newest version. Requires Fiddler 26.24 or later.

Returns

The dataset instance with all metadata and configuration.

Raises

NotFound – If no dataset exists with the specified ID, or version names no version of it.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get dataset by UUID
dataset = Dataset.get_by_id(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved dataset: {dataset.name}")
print(f"Created: {dataset.created_at}")
print(f"Application: {dataset.application.name}")

This method makes an API call to fetch the dataset from the server. Without version, the returned object stands for the newest version at load time; it does not follow later writes.

classmethod get_by_name()

Retrieve a dataset by name within an application. Finds and returns a dataset using its name within the specified application. This is useful when you know the dataset name and application but not its UUID. Dataset names are unique within an application, making this a reliable lookup method. Changed in version 0.8.0: Added version.

Parameters

str

required

The name of the dataset to retrieve. Dataset names are unique within an application and are case-sensitive.

UUID | str

required

The UUID of the application containing the dataset. Can be provided as a UUID object or string representation.

int | None

default:"None"

Optional dataset version number for the returned object to stand for. Omitted, the newest version. Requires Fiddler 26.24 or later.

Returns

The dataset instance matching the specified name.

Raises

NotFound – If no dataset exists with the specified name in the application, or version names no version of it.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)

# Get dataset by name within an application
dataset = Dataset.get_by_name(
    name="fraud-detection-tests",
    application_id=application.id
)
print(f"Found dataset: {dataset.name} (ID: {dataset.id})")
print(f"Created: {dataset.created_at}")
print(f"Application: {dataset.application.name}")

Dataset names are case-sensitive and must match exactly. Use this method when you have a known dataset name from configuration or user input.

classmethod list()

List all datasets in an application. Retrieves all datasets that the current user has access to within the specified application. Returns an iterator for memory efficiency when dealing with many datasets.

Parameters

UUID | str

required

The UUID of the application to list datasets from. Can be provided as a UUID object or string representation.

Yields

Dataset – Dataset instances for all accessible datasets in the application.

Raises

ApiError – If there’s an error communicating with the Fiddler API.

Returns

Iterator[Dataset]

Example

# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)

# List all datasets in an application
for dataset in Dataset.list(application_id=application.id):
    print(f"Dataset: {dataset.name}")
    print(f"  ID: {dataset.id}")
    print(f"  Created: {dataset.created_at}")

# Convert to list for counting and filtering
datasets = list(Dataset.list(application_id=application.id))
print(f"Total datasets in application: {len(datasets)}")

# Find datasets by name pattern
test_datasets = [
    ds for ds in Dataset.list(application_id=application.id)
    if "test" in ds.name.lower()
]
print(f"Test datasets: {len(test_datasets)}")

This method returns an iterator for memory efficiency. Convert to a list with list(Dataset.list(application_id)) if you need to iterate multiple times or get the total count. The iterator fetches datasets lazily from the API.

classmethod create()

Create a new dataset in an application. Creates a new dataset within the specified application on the Fiddler platform. The dataset must have a unique name within the application.

Parameters

str

required

Dataset name, must be unique within the application.

UUID | str

required

The UUID of the application to create the dataset in. Can be provided as a UUID object or string representation.

str | None

default:"None"

Optional human-readable description of the dataset.

dict | None

default:"None"

Optional custom metadata dictionary for additional dataset information.

bool

default:"True"

Optional boolean flag to indicate if the dataset is active.

Returns

The newly created dataset instance with server-assigned fields.

Raises

Conflict – If a dataset with the same name already exists in the application.
ValidationError – If the dataset configuration is invalid (e.g., invalid name format).
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)

# Create a new dataset for fraud detection tests
dataset = Dataset.create(
    name="fraud-detection-tests",
    application_id=application.id,
    description="Test cases for fraud detection model evaluation",
    metadata={"source": "production", "environment": "test"},
)
print(f"Created dataset with ID: {dataset.id}")
print(f"Created at: {dataset.created_at}")
print(f"Application: {dataset.application.name}")

After successful creation, the dataset instance is returned with server-assigned metadata. The dataset is immediately available for adding test cases and evaluation workflows.

classmethod get_or_create()

Get an existing dataset by name or create a new one if it doesn’t exist. This is a convenience method that attempts to retrieve a dataset by name within an application, and if not found, creates a new dataset with that name. Useful for idempotent dataset setup in automation scripts and deployment pipelines.

Parameters

str

required

The name of the dataset to retrieve or create.

UUID | str

required

The UUID of the application to search/create the dataset in. Can be provided as a UUID object or string representation.

str | None

default:"None"

Optional human-readable description of the dataset.

dict | None

default:"None"

Optional custom metadata dictionary for additional dataset information.

bool

default:"True"

Optional boolean flag to indicate if the dataset is active.

Returns

Either the existing dataset with the specified name, or a newly created dataset if none existed.

Raises

ValidationError – If the dataset name format is invalid.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)

# Safe dataset setup - get existing or create new
dataset = Dataset.get_or_create(
    name="fraud-detection-tests",
    application_id=application.id,
    description="Test cases for fraud detection model",
    metadata={"source": "production"},
)
print(f"Using dataset: {dataset.name} (ID: {dataset.id})")

# Idempotent setup in deployment scripts
dataset = Dataset.get_or_create(
    name="llm-evaluation-tests",
    application_id=application.id,
)

# Use in configuration management
test_types = ["unit", "integration", "performance"]
datasets = {}
for test_type in test_types:
    datasets[test_type] = Dataset.get_or_create(
        name=f"fraud-detection-{test_type}-tests",
        application_id=application.id,
    )

This method is idempotent - calling it multiple times with the same name and application_id will return the same dataset. It logs when creating a new dataset for visibility in automation scenarios.

update()

Update the dataset’s description, metadata or active flag. Only a dataset’s items are versioned, so this does not create a new dataset version; the returned dataset stands for the same version as this one.

Parameters

str | None

default:"None"

Optional new description for the dataset. If provided, replaces the existing description. Set to empty string to clear.

dict | None

default:"None"

Optional new metadata dictionary for the dataset. If provided, replaces the existing metadata completely. Use empty dict to clear.

bool | None

default:"None"

Optional boolean flag to indicate if the dataset is active.

Returns

The updated dataset, standing for the same version as this one.

Raises

ValueError – If no update parameters are provided (all are None).
ValidationError – If the update data is invalid (e.g., invalid metadata format).
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Update description and metadata
updated_dataset = dataset.update(
    description="Updated test cases for fraud detection model v2.0",
    metadata={"source": "production", "version": "2.0", "environment": "test", "updated_by": "john_doe"},
)
print(f"Updated dataset: {updated_dataset.name}")
print(f"New description: {updated_dataset.description}")

# Update only metadata
dataset.update(metadata={"last_updated": "2024-01-15", "status": "active"})

# Clear description
dataset.update(description="")

# Batch update multiple datasets
for dataset in Dataset.list(application_id=application_id):
    if "test" in dataset.name:
        dataset.update(description="Updated test cases for fraud detection model v2.0")

This method performs a complete replacement of the specified fields. For partial updates, retrieve current values, modify them, and pass the complete new values. The dataset name and ID cannot be changed.

delete()

Delete the dataset permanently from the Fiddler platform. Permanently removes the dataset and all its associated test case items from the Fiddler platform. This operation cannot be undone. The method performs safety checks before deletion:

Verifies that no experiments are currently associated with the dataset
Prevents deletion if any experiments reference this dataset
Only proceeds with deletion if the dataset is safe to remove

Parameters

None – This method takes no parameters.

Returns

This method does not return a value.

Raises

ApiError – If there’s an error communicating with the Fiddler API.
ApiError – If the dataset cannot be deleted due to existing experiments.
NotFound – If the dataset no longer exists.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="old-test-dataset", application_id=application_id)

# Check if dataset is safe to delete
try:
    dataset.delete()
    print(f"Successfully deleted dataset: {dataset.name}")
except ApiError as e:
    print(f"Cannot delete dataset: {e}")
    print("Dataset may have associated experiments")

# Clean up unused datasets in bulk
unused_datasets = [
    Dataset.get_by_name(name="temp-dataset-1", application_id=application_id),
    Dataset.get_by_name(name="temp-dataset-2", application_id=application_id),
]

for dataset in unused_datasets:
    try:
        dataset.delete()
        print(f"Deleted: {dataset.name}")
    except ApiError:
        print(f"Skipped {dataset.name} - has associated experiments")

This operation is irreversible. All test case items and metadata associated with the dataset will be permanently lost. Ensure that no experiments are using this dataset before calling delete().

insert()

Add multiple test case items to the dataset.

Deprecated since version 0.8.0: Use update_items(), which also replaces items whose id already exists. It returns an UpdateItemsResult whose dataset is at the new version, and requires Fiddler 26.25 or later. insert only adds: an id that already exists with different content raises ItemIdCollision.

Changed in version 0.8.0: This dataset object then stands for the version the write created. Fiddler before 26.25 does not report that version, so there the object stands for the newest version instead (version is None). Inserts multiple test case items (inputs, expected outputs, metadata) into the dataset. Each item represents a single test case for evaluation purposes. Items can be provided as dictionaries or NewDatasetItem objects.

Parameters

list[dict] | list[NewDatasetItem]

required

List of test case items to add to the dataset. Each item can be:

A dictionary containing test case data with keys:
- inputs: Dictionary containing input data for the test case
- expected_outputs: Dictionary containing expected output data
- metadata: Optional dictionary with additional test case metadata
- extras: Optional dictionary for additional custom data
- source_name: Optional string identifying the source of the test case
- source_id: Optional string identifier for the source
A NewDatasetItem object with the same structure

Returns

List of UUIDs for the newly created dataset items.

Raises

ValueError – If the items list is empty.
ValidationError – If any item data is invalid (e.g., missing required fields).
ItemIdCollision – If an id already exists with different content, or was written while this call was in flight. Nothing is written.
DatasetBusy – If another save held the dataset. Nothing is written.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Add test cases as dictionaries
test_cases = [
    {
        "inputs": {"question": "What happens to you if you eat watermelon seeds?"},
        "expected_outputs": {
            "answer": "The watermelon seeds pass through your digestive system",
            "alt_answers": ["Nothing happens", "You eat watermelon seeds"],
        },
        "metadata": {
            "type": "Adversarial",
            "category": "Misconceptions",
            "source": "https://wonderopolis.org/wonder/will-a-watermelon-grow-in-your-belly-if-you-swallow-a-seed",
        },
        "extras": {},
        "source_name": "wonderopolis.org",
        "source_id": "1",
    },
]

# Insert test cases
item_ids = dataset.insert(test_cases)
print(f"Added {len(item_ids)} test cases")
print(f"Item IDs: {item_ids}")

# Add test cases as NewDatasetItem objects
from fiddler_evals.pydantic_models.dataset import NewDatasetItem

items = [
    NewDatasetItem(
        inputs={"question": "What is the capital of France?"},
        expected_outputs={"answer": "Paris"},
        metadata={"difficulty": "easy"},
        extras={},
        source_name="test_source",
        source_id="item1",
    ),
]

item_ids = dataset.insert(items)
print(f"Added {len(item_ids)} test cases")

This method automatically generates UUIDs and timestamps for each item. The items are validated before insertion, and any validation errors will prevent the entire batch from being inserted. Use this method for bulk insertion of test cases into datasets.

update_items()

Add, edit and delete items in one write, based on this dataset’s version. The whole call is atomic and commits at most one new dataset version. It is based on the version this object stands for: if another write changed any item it names since then, nothing is written and StaleDatasetVersion is raised, so it never overwrites someone else’s edit. Requires Fiddler 26.25 or later. New in version 0.8.0 Each entry in items is a whole item. An entry whose id is a live item replaces it (a field left out is cleared, not kept); any other entry adds a new item. Items not named are left untouched.

Parameters

list[dict | NewDatasetItem | DatasetItem] | None

default:"None"

Items to add or edit, as dicts, NewDatasetItem, or DatasetItem objects. Edit items read from this object’s own get_items(): the conflict check covers changes made after this object’s version.

list[UUID | str] | None

default:"None"

Ids of live items to delete.

Returns

dataset at the version this call committed (the current version if nothing changed), and the ids the call created, updated, deleted, or left unchanged.

Raises

ValueError – If neither items nor delete_ids is given, or this object has no version to write against.
StaleDatasetVersion – If a named item changed after this version. Never retried; reload the dataset and reconcile.
DatasetBusy – If the dataset stayed busy with other saves through the SDK’s retries.
DatasetWriteUnavailable – If the item store stayed unavailable through the SDK’s retries.
NotFound – If a delete_ids entry is not a live item, or this object’s version is not a committed version of the dataset.
ApiError – For other rejections, including a call over the server’s per-call limit of items plus deletes (reason DatasetItemsBulkLimitExceeded); nothing is written. Each call is one version, so split large writes across calls.

Example

items = list(dataset.get_items())
items[0].metadata["reviewed"] = True
result = dataset.update_items(items=[items[0]], delete_ids=[items[1].id])
dataset = result.dataset  # the new version

delete_items()

Delete items in one write, based on this dataset’s version. The same as update_items(delete_ids=ids): the call is atomic and commits one new dataset version, and it is refused with StaleDatasetVersion if another write changed any of these items after the version this object stands for. Requires Fiddler 26.25 or later. New in version 0.8.0

Parameters

list[UUID | str]

required

Ids of live items of this dataset to delete.

Returns

dataset at the version this call committed, with the deleted ids in deleted_ids.

Raises

ValueError – If ids is empty, or this object has no version to write against.
StaleDatasetVersion – If a named item changed after this version. Reload the dataset and reconcile before deleting again.
DatasetBusy – If the dataset stayed busy with other saves through the SDK’s retries.
DatasetWriteUnavailable – If the item store stayed unavailable through the SDK’s retries.
NotFound – If an id is not a live item of this dataset.

Example

obsolete = [i.id for i in dataset.get_items() if i.metadata.get("obsolete")]
dataset = dataset.delete_items(obsolete).dataset

insert_from_pandas()

Insert test case items from a pandas DataFrame into the dataset.

Deprecated since version 0.8.0: Use dataset.update_items(items=items_from_pandas(...)).dataset: build the items with items_from_pandas(), and update_items() returns an UpdateItemsResult whose dataset is at the version the write creates. Requires Fiddler 26.25 or later.

Input columns: Data that will be used as inputs for evaluation
Expected output columns: Expected results or answers for the test cases
Metadata columns: Additional metadata associated with each test case
Extras columns: Custom data fields for additional test case information
Source columns: Information about the origin of each test case

Column Mapping Logic:

If input_columns is specified, those columns become inputs
If input_columns is None, all unmapped columns become inputs
Remaining unmapped columns are automatically assigned to extras
Source columns are always mapped to source_name and source_id

Parameters

pd.DataFrame

required

The pandas DataFrame containing test case data. Must not be empty and must have at least one column.

builtins.list[str] | None

default:"None"

Optional list of column names to use as input data. If None, all unmapped columns become inputs.

builtins.list[str] | None

default:"None"

Optional list of column names containing expected outputs or answers for the test cases.

builtins.list[str] | None

default:"None"

Optional list of column names to use as metadata. These columns will be stored as test case metadata.

builtins.list[str] | None

default:"None"

Optional list of column names for additional custom data. Unmapped columns are automatically added to extras.

str

default:"\"id\""

Column name containing the ID for each test case. Defaults to “id”.

str

default:"\"source_name\""

Column name containing the source identifier for each test case. Defaults to “source_name”.

str

default:"\"source_id\""

Column name containing the source ID for each test case. Defaults to “source_id”.

Returns

List of UUIDs for the newly created dataset items.

Raises

ValueError – If the DataFrame is empty or has no columns.
ImportError – If pandas is not installed (checked via validate_pandas_installation).
ValidationError – If any generated test case data is invalid.
ItemIdCollision – If an id already exists with different content, or changed while this call was in flight. Nothing is written.
DatasetBusy – If another save held the dataset. Nothing is written.
DatasetWriteUnavailable – If the item store was unavailable. Nothing is written.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Example DataFrame with test case data
import pandas as pd

df = pd.DataFrame({
    'question': ['What is fraud?', 'How to detect fraud?', 'What are fraud types?'],
    'expected_answer': ['Fraud is deception', 'Use ML models', 'Identity theft, credit card fraud'],
    'difficulty': ['easy', 'medium', 'hard'],
    'category': ['definition', 'detection', 'types'],
    'source_name': ['manual', 'manual', 'manual'],
    'source_id': ['1', '2', '3']
})

# Insert with explicit column mapping
item_ids = dataset.insert_from_pandas(
    df=df,
    input_columns=['question'],
    expected_output_columns=['expected_answer'],
    metadata_columns=['difficulty', 'category'],
)
print(f"Added {len(item_ids)} test cases from DataFrame")

# Insert with automatic column mapping (all unmapped columns become inputs)
df_auto = pd.DataFrame({
    'user_query': ['Is this transaction suspicious?', 'Check for anomalies'],
    'context': ['Credit card transaction', 'Banking data'],
    'expected_response': ['Yes, flagged', 'Anomalies detected'],
    'priority': ['high', 'medium'],
    'source': ['production', 'test']
})

item_ids = dataset.insert_from_pandas(
    df=df_auto,
    expected_output_columns=['expected_response'],
    metadata_columns=['priority'],
    source_name_column='source',
    source_id_column='source'  # Using same column for both
)

# Complex DataFrame with many columns
df_complex = pd.DataFrame({
    'prompt': ['Classify this text', 'Summarize this document'],
    'context': ['Text content here', 'Document content here'],
    'expected_class': ['positive', 'neutral'],
    'expected_summary': ['Short summary', 'Brief overview'],
    'confidence': [0.95, 0.87],
    'language': ['en', 'en'],
    'domain': ['sentiment', 'summarization'],
    'version': ['1.0', '1.0'],
    'created_by': ['user1', 'user2'],
    'review_status': ['approved', 'pending']
})

item_ids = dataset.insert_from_pandas(
    df=df_complex,
    input_columns=['prompt', 'context'],
    expected_output_columns=['expected_class', 'expected_summary'],
    metadata_columns=['confidence', 'language', 'domain', 'version'],
    extras_columns=['created_by', 'review_status']
)

This method requires pandas to be installed. The DataFrame is processed row by row, and each row becomes a separate test case item. Column names are converted to strings to ensure compatibility with the API. Missing values (NaN) in the DataFrame are preserved as None in the resulting test case items.

insert_from_csv_file()

Insert test case items from a CSV file into the dataset.

Deprecated since version 0.8.0: Use dataset.update_items(items=items_from_csv(...)).dataset: build the items with items_from_csv(), and update_items() returns an UpdateItemsResult whose dataset is at the version the write creates. Requires Fiddler 26.25 or later.

If input_columns is specified, those columns become inputs
If input_columns is None, all unmapped columns become inputs
Remaining unmapped columns are automatically assigned to extras
Source columns are always mapped to source_name and source_id

Parameters

str | Path

required

Path to the CSV file to read. Can be a string or Path object. Supports both relative and absolute paths.

list[str] | None

default:"None"

Optional list of column names to use as input data. If None, all unmapped columns become inputs.

list[str] | None

default:"None"

Optional list of column names containing expected outputs or answers for the test cases.

list[str] | None

default:"None"

Optional list of column names to use as metadata. These columns will be stored as test case metadata.

list[str] | None

default:"None"

Optional list of column names for additional custom data. Unmapped columns are automatically added to extras.

str

default:"\"id\""

Column name containing the ID for each test case. Defaults to “id”.

str

default:"\"source_name\""

Column name containing the source identifier for each test case. Defaults to “source_name”.

str

default:"\"source_id\""

Column name containing the source ID for each test case. Defaults to “source_id”.

Returns

List of UUIDs for the newly created dataset items.

Raises

FileNotFoundError – If the CSV file does not exist at the specified path.
ValueError – If the CSV file is empty or has no columns.
ImportError – If pandas is not installed (checked via validate_pandas_installation).
ValidationError – If any generated test case data is invalid.
ItemIdCollision – If an id already exists with different content, or changed while this call was in flight. Nothing is written.
DatasetBusy – If another save held the dataset. Nothing is written.
DatasetWriteUnavailable – If the item store was unavailable. Nothing is written.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Example CSV file: test_cases.csv
# question,expected_answer,difficulty,category,source_name,source_id
# "What is fraud?","Fraud is deception","easy","definition","manual","1"
# "How to detect fraud?","Use ML models","medium","detection","manual","2"
# "What are fraud types?","Identity theft, credit card fraud","hard","types","manual","3"

# Insert with explicit column mapping
item_ids = dataset.insert_from_csv_file(
    file_path="test_cases.csv",
    input_columns=['question'],
    expected_output_columns=['expected_answer'],
    metadata_columns=['difficulty', 'category'],
)
print(f"Added {len(item_ids)} test cases from CSV")

# Insert with automatic column mapping (all unmapped columns become inputs)
# CSV: user_query,context,expected_response,priority,source
item_ids = dataset.insert_from_csv_file(
    file_path="evaluation_data.csv",
    expected_output_columns=['expected_response'],
    metadata_columns=['priority'],
    source_name_column='source',
    source_id_column='source'  # Using same column for both
)

# Import from CSV with relative path
item_ids = dataset.insert_from_csv_file("data/test_cases.csv")
print(f"Imported {len(item_ids)} test cases from CSV")

# Import from CSV with absolute path
from pathlib import Path
csv_path = Path("/absolute/path/to/test_cases.csv")
item_ids = dataset.insert_from_csv_file(csv_path)

# Complex CSV with many columns
# prompt,context,expected_class,expected_summary,confidence,language,domain,version,created_by,review_status
item_ids = dataset.insert_from_csv_file(
    file_path="complex_test_cases.csv",
    input_columns=['prompt', 'context'],
    expected_output_columns=['expected_class', 'expected_summary'],
    metadata_columns=['confidence', 'language', 'domain', 'version'],
    extras_columns=['created_by', 'review_status']
)

# Batch import multiple CSV files
csv_files = ["test_cases_1.csv", "test_cases_2.csv", "test_cases_3.csv"]
all_item_ids = []
for csv_file in csv_files:
    item_ids = dataset.insert_from_csv_file(csv_file)
    all_item_ids.extend(item_ids)
    print(f"Imported {len(item_ids)} items from {csv_file}")
print(f"Total imported: {len(all_item_ids)} items")

This method requires pandas to be installed. The CSV file is read using pandas.read_csv() with default parameters. For advanced CSV reading options (custom delimiters, encoding, etc.), use pandas.read_csv() directly and pass the DataFrame to items_from_pandas(). Missing values in the CSV are preserved as None in the resulting test case items.

insert_from_jsonl_file()

Insert test case items from a JSONL (JSON Lines) file into the dataset.

Deprecated since version 0.8.0: Use dataset.update_items(items=items_from_jsonl(...)).dataset: build the items with items_from_jsonl(), and update_items() returns an UpdateItemsResult whose dataset is at the version the write creates. Requires Fiddler 26.25 or later.

Changed in version 0.8.0: This dataset object then stands for the version the write created. Fiddler before 26.25 does not report that version, so there the object stands for the newest version instead (version is None). Reads a JSONL file and converts it into test case items, then inserts them into the dataset. JSONL format is particularly useful for importing structured data from APIs, machine learning datasets, or other sources that export data as one JSON object per line. JSONL Format: Each line in the file must be a valid JSON object. Empty lines are skipped. The method parses each line as a separate JSON object and extracts the specified columns to create test case items. Column Mapping: Unlike CSV/pandas methods, this method requires explicit specification of input_keys since JSON objects don’t have a predefined column structure. All other key/column mappings work the same way as other insert methods.

Parameters

str | Path

required

Path to the JSONL file to read. Can be a string or Path object. Supports both relative and absolute paths.

list[str]

required

Required list of key names to use as input data. These must correspond to keys in the JSON objects.

list[str] | None

default:"None"

Optional list of key names containing expected outputs or answers for the test cases.

list[str] | None

default:"None"

Optional list of key names to use as metadata. These keys will be stored as test case metadata.

list[str] | None

default:"None"

Optional list of key names for additional custom data. Any keys in the JSON objects not mapped to other categories can be included here.

str

default:"\"id\""

Key name containing the ID for each test case. Defaults to “id”.

str

default:"\"source_name\""

Key name containing the source identifier for each test case. Defaults to “source_name”.

str

default:"\"source_id\""

Key name containing the source ID for each test case. Defaults to “source_id”.

Returns

List of UUIDs for the newly created dataset items.

Raises

FileNotFoundError – If the JSONL file does not exist at the specified path.
ValueError – If the JSONL file is empty or has no valid JSON objects.
json.JSONDecodeError – If any line in the file contains invalid JSON.
ValidationError – If any generated test case data is invalid.
ItemIdCollision – If an id already exists with different content, or changed while this call was in flight. Nothing is written.
DatasetBusy – If another save held the dataset. Nothing is written.
DatasetWriteUnavailable – If the item store was unavailable. Nothing is written.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Example JSONL file: test_cases.jsonl
# {"question": "What is fraud?", "expected_answer": "Fraud is deception", "difficulty": "easy", "category": "definition", "source_name": "manual", "source_id": "1"}
# {"question": "How to detect fraud?", "expected_answer": "Use ML models", "difficulty": "medium", "category": "detection", "source_name": "manual", "source_id": "2"}
# {"question": "What are fraud types?", "expected_answer": "Identity theft, credit card fraud", "difficulty": "hard", "category": "types", "source_name": "manual", "source_id": "3"}

# Insert with explicit column mapping
item_ids = dataset.insert_from_jsonl_file(
    file_path="test_cases.jsonl",
    input_keys=['question'],
    expected_output_keys=['expected_answer'],
    metadata_keys=['difficulty', 'category'],
)
print(f"Added {len(item_ids)} test cases from JSONL")

# Batch import multiple JSONL files
jsonl_files = ["test_cases_1.jsonl", "test_cases_2.jsonl", "test_cases_3.jsonl"]
all_item_ids = []
for jsonl_file in jsonl_files:
    item_ids = dataset.insert_from_jsonl_file(
        jsonl_file,
        input_keys=['question']
    )
    all_item_ids.extend(item_ids)
    print(f"Imported {len(item_ids)} items from {jsonl_file}")
print(f"Total imported: {len(all_item_ids)} items")

This method reads the file line by line and parses each line as JSON. Empty lines are automatically skipped. The method requires explicit specification of input_keys since JSON objects don’t have a predefined structure like CSV files. Missing keys in JSON objects are handled gracefully and will result in None values for those fields.

get_schema()

Get the schema of this dataset. Discovers what field keys exist in each JSON bucket (inputs, expected_outputs, metadata, extras) across the dataset’s items, with coverage counts and inferred data types. Use it to keep a new FieldMapping consistent with the field names the dataset already uses.

Returns

Item total, sampling flag, and field keys per bucket with counts and types.

Raises

ApiError – If there’s an error communicating with the Fiddler API.

Experiment tasks bind to inputs by name, so reusing the existing field names keeps new items consistent and replayable alongside the items already in the dataset.

Example

# Get existing dataset
dataset = Dataset.get_by_name(
    name="fraud-detection-tests", application_id=application_id
)

# Inspect existing field names before adding more items
schema = dataset.get_schema()
print(f"{schema.total_items} items (sampled={schema.sampled})")
for field in schema.inputs:
    print(f"  inputs.{field.key}: {field.data_type} ({field.count})")

add_items_from_spans()

Add dataset items by fetching span attributes and applying a mapping. Resolves the given span references, extracts attribute values using the mapping, and writes one dataset item per span. All-or-nothing — if any span cannot be found, no items are written. Changed in version 0.8.0: This dataset object then stands for the version the write created. Fiddler before 26.25 does not report that version, so there the object stands for the newest version instead (version is None).

Parameters

list[SpanReference | dict]

required

List of span references (SpanReference or dict with trace_id and span_id). Must be non-empty, must not contain duplicates, and is capped at 200 spans per request by default (server-configurable) — batch larger sets.

FieldMapping | dict

required

Field mapping (FieldMapping or dict with inputs, expected_outputs, metadata, extras buckets). Omitted buckets default to empty; an unknown bucket key is rejected rather than silently ignored.

str | datetime

required

Start of the time range, inclusive. ISO 8601 string or a timezone-aware datetime. Naive datetimes are rejected by the server; any UTC offset is accepted and normalized to UTC server-side.

str | datetime

required

End of the time range, exclusive. ISO 8601 string or a timezone-aware datetime, same rules as start_time. A referenced span outside the window is treated as missing.

Returns

Count and IDs of created items.

Raises

ValidationError – If a span reference or the mapping is malformed.
DatasetBusy – If another save held the dataset. Nothing is written.
DatasetWriteUnavailable – If the item store was unavailable. Nothing is written.
ApiError – 422 if any span cannot be resolved. No items are written, and every unresolvable span is reported — iterate exc.error_response.error.errors for one SpanNotFound entry per missing span, since str(exc) shows only the summary message. 400 if spans exceeds the per-request cap or the dataset would exceed its item limit (default 10,000).

A failed call writes nothing, so retrying after fixing the references cannot create duplicates. A successful call has no idempotency key, however — repeating it with the same spans creates duplicate dataset items. Mapped attribute keys are matched exactly against stored span attribute keys; discover them first with Application.get_span_fields().

Example

from fiddler_evals import FieldMapping, SpanReference

# Get existing dataset
dataset = Dataset.get_by_name(
    name="fraud-detection-tests", application_id=application_id
)

# Span references collected from POST /v3/spans/query
span_refs = [
    SpanReference(trace_id=span["trace_id"], span_id=span["span_id"])
    for span in spans
]

# Map dataset fields to stored span attribute keys
mapping = FieldMapping(
    inputs={"user_query": "gen_ai.llm.input.user"},
    expected_outputs={"expected_response": "gen_ai.llm.output"},
    metadata={"model": "gen_ai.request.model"},
)

result = dataset.add_items_from_spans(
    spans=span_refs,
    mapping=mapping,
    start_time=start_time,
    end_time=end_time,
)
print(f"Created {result.items_created} dataset items")
print(f"Item IDs: {result.item_ids[:3]}")

add_testcases()

Add multiple test case items to the dataset.

Changed in version 0.8.0: This dataset object then stands for the version the write created. Fiddler before 26.25 does not report that version, so there the object stands for the newest version instead (version is None). Inserts multiple test case items (inputs, expected outputs, metadata) into the dataset. Each item represents a single test case for evaluation purposes. Items can be provided as dictionaries or NewDatasetItem objects.

Parameters

list[dict] | list[NewDatasetItem]

required

List of test case items to add to the dataset. Each item can be:

A dictionary containing test case data with keys:
- inputs: Dictionary containing input data for the test case
- expected_outputs: Dictionary containing expected output data
- metadata: Optional dictionary with additional test case metadata
- extras: Optional dictionary for additional custom data
- source_name: Optional string identifying the source of the test case
- source_id: Optional string identifier for the source
A NewDatasetItem object with the same structure

Returns

List of UUIDs for the newly created dataset items.

Raises

ValueError – If the items list is empty.
ValidationError – If any item data is invalid (e.g., missing required fields).
ItemIdCollision – If an id already exists with different content, or was written while this call was in flight. Nothing is written.
DatasetBusy – If another save held the dataset. Nothing is written.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Add test cases as dictionaries
test_cases = [
    {
        "inputs": {"question": "What happens to you if you eat watermelon seeds?"},
        "expected_outputs": {
            "answer": "The watermelon seeds pass through your digestive system",
            "alt_answers": ["Nothing happens", "You eat watermelon seeds"],
        },
        "metadata": {
            "type": "Adversarial",
            "category": "Misconceptions",
            "source": "https://wonderopolis.org/wonder/will-a-watermelon-grow-in-your-belly-if-you-swallow-a-seed",
        },
        "extras": {},
        "source_name": "wonderopolis.org",
        "source_id": "1",
    },
]

# Insert test cases
item_ids = dataset.insert(test_cases)
print(f"Added {len(item_ids)} test cases")
print(f"Item IDs: {item_ids}")

# Add test cases as NewDatasetItem objects
from fiddler_evals.pydantic_models.dataset import NewDatasetItem

items = [
    NewDatasetItem(
        inputs={"question": "What is the capital of France?"},
        expected_outputs={"answer": "Paris"},
        metadata={"difficulty": "easy"},
        extras={},
        source_name="test_source",
        source_id="item1",
    ),
]

item_ids = dataset.insert(items)
print(f"Added {len(item_ids)} test cases")

add_items()

Add multiple test case items to the dataset.

Changed in version 0.8.0: This dataset object then stands for the version the write created. Fiddler before 26.25 does not report that version, so there the object stands for the newest version instead (version is None). Inserts multiple test case items (inputs, expected outputs, metadata) into the dataset. Each item represents a single test case for evaluation purposes. Items can be provided as dictionaries or NewDatasetItem objects.

Parameters

list[dict] | list[NewDatasetItem]

required

List of test case items to add to the dataset. Each item can be:

A dictionary containing test case data with keys:
- inputs: Dictionary containing input data for the test case
- expected_outputs: Dictionary containing expected output data
- metadata: Optional dictionary with additional test case metadata
- extras: Optional dictionary for additional custom data
- source_name: Optional string identifying the source of the test case
- source_id: Optional string identifier for the source
A NewDatasetItem object with the same structure

Returns

List of UUIDs for the newly created dataset items.

Raises

ValueError – If the items list is empty.
ValidationError – If any item data is invalid (e.g., missing required fields).
ItemIdCollision – If an id already exists with different content, or was written while this call was in flight. Nothing is written.
DatasetBusy – If another save held the dataset. Nothing is written.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Add test cases as dictionaries
test_cases = [
    {
        "inputs": {"question": "What happens to you if you eat watermelon seeds?"},
        "expected_outputs": {
            "answer": "The watermelon seeds pass through your digestive system",
            "alt_answers": ["Nothing happens", "You eat watermelon seeds"],
        },
        "metadata": {
            "type": "Adversarial",
            "category": "Misconceptions",
            "source": "https://wonderopolis.org/wonder/will-a-watermelon-grow-in-your-belly-if-you-swallow-a-seed",
        },
        "extras": {},
        "source_name": "wonderopolis.org",
        "source_id": "1",
    },
]

# Insert test cases
item_ids = dataset.insert(test_cases)
print(f"Added {len(item_ids)} test cases")
print(f"Item IDs: {item_ids}")

# Add test cases as NewDatasetItem objects
from fiddler_evals.pydantic_models.dataset import NewDatasetItem

items = [
    NewDatasetItem(
        inputs={"question": "What is the capital of France?"},
        expected_outputs={"answer": "Paris"},
        metadata={"difficulty": "easy"},
        extras={},
        source_name="test_source",
        source_id="item1",
    ),
]

item_ids = dataset.insert(items)
print(f"Added {len(item_ids)} test cases")

get_testcases()

Retrieve the test case items at the version this dataset stands for. Fetches test case items (inputs, expected outputs, metadata) at version, or the newest version when it is None. To read a past version, load it: Dataset.get_by_id(dataset.id, version=3).get_items(). Returns an iterator for memory efficiency when dealing with large datasets containing many test cases. Changed in version 0.8.0: Reads the version this dataset object stands for, not the newest.

Returns

Iterator of DatasetItem instances for all test cases in the dataset.

Raises

ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Get all test cases in the dataset
for item in dataset.get_items():
    print(f"Test case ID: {item.id}")
    print(f"Inputs: {item.inputs}")
    print(f"Expected outputs: {item.expected_outputs}")
    print(f"Metadata: {item.metadata}")
    print("---")

# Convert to list for analysis
all_items = list(dataset.get_items())
print(f"Total test cases: {len(all_items)}")

# Filter items by metadata
high_priority_items = [
    item for item in dataset.get_items()
    if item.metadata.get("priority") == "high"
]
print(f"High priority test cases: {len(high_priority_items)}")

# Process items in batches
batch_size = 100
for i, item in enumerate(dataset.get_items()):
    if i % batch_size == 0:
        print(f"Processing batch {i // batch_size + 1}")
    # Process item...

This method returns an iterator for memory efficiency. Convert to a list with list(dataset.get_items()) if you need to iterate multiple times or get the total count. The iterator fetches items lazily from the API.

get_items()

Returns

Iterator of DatasetItem instances for all test cases in the dataset.

Raises

ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing dataset
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application_id)

# Get all test cases in the dataset
for item in dataset.get_items():
    print(f"Test case ID: {item.id}")
    print(f"Inputs: {item.inputs}")
    print(f"Expected outputs: {item.expected_outputs}")
    print(f"Metadata: {item.metadata}")
    print("---")

# Convert to list for analysis
all_items = list(dataset.get_items())
print(f"Total test cases: {len(all_items)}")

# Filter items by metadata
high_priority_items = [
    item for item in dataset.get_items()
    if item.metadata.get("priority") == "high"
]
print(f"High priority test cases: {len(high_priority_items)}")

# Process items in batches
batch_size = 100
for i, item in enumerate(dataset.get_items()):
    if i % batch_size == 0:
        print(f"Processing batch {i // batch_size + 1}")
    # Process item...

​Example

​Attributes

​classmethod get_by_id()

​Parameters

​Returns

​Raises

​Example

​classmethod get_by_name()

​Parameters

​Returns

​Raises

​Example

​classmethod list()

​Parameters

​Yields

​Raises

​Returns

​Example

​classmethod create()

​Parameters

​Returns

​Raises

​Example

​classmethod get_or_create()

​Parameters

​Returns

​Raises

​Example

​update()

​Parameters

​Returns

​Raises

​Example

​delete()

​Parameters

​Returns

​Raises

​Example

​insert()

​Parameters

​Returns

​Raises

​Example

​update_items()

​Parameters

​Returns

​Raises

​Example

​delete_items()

​Parameters

​Returns

​Raises

​Example

​insert_from_pandas()

​Parameters

​Returns

​Raises

​Example

​insert_from_csv_file()

​Parameters

​Returns

​Raises

​Example

​insert_from_jsonl_file()

​Parameters

​Returns

​Raises

​Example

​get_schema()

​Returns

​Raises

​Example

​add_items_from_spans()

​Parameters

​Returns

​Raises

​Example

​add_testcases()

​Parameters

​Returns

Example

Attributes

classmethod get_by_id()

Parameters

Returns

Raises

Example

classmethod get_by_name()

Parameters

Returns

Raises

Example

classmethod list()

Parameters

Yields

Raises

Returns

Example

classmethod create()

Parameters

Returns

Raises

Example

classmethod get_or_create()

Parameters

Returns

Raises

Example

update()

Parameters

Returns

Raises

Example

delete()

Parameters

Returns

Raises

Example

insert()

Parameters

Returns

Raises

Example

update_items()

Parameters

Returns

Raises

Example

delete_items()

Parameters

Returns

Raises

Example

insert_from_pandas()

Parameters

Returns

Raises

Example

insert_from_csv_file()

Parameters

Returns

Raises

Example

insert_from_jsonl_file()

Parameters

Returns

Raises

Example

get_schema()

Returns

Raises

Example

add_items_from_spans()

Parameters

Returns

Raises

Example

add_testcases()

Parameters

Returns