Experiment - Fiddler Documentation

Represents an Experiment for tracking evaluation runs and results. An Experiment is a single evaluation run of a test suite against a specific application/LLM/Agent version and evaluators. Experiments provide comprehensive tracking, monitoring, and result management for GenAI evaluation workflows, enabling systematic testing and performance analysis. Key Features:

Evaluation Tracking: Complete lifecycle tracking of evaluation runs
Status Management: Real-time status updates (PENDING, IN_PROGRESS, COMPLETED, etc.)
Dataset Integration: Linked to specific datasets for evaluation
Result Storage: Comprehensive storage of results, metrics, and error information
Error Handling: Detailed error tracking with traceback information

Experiment Lifecycle:

Creation: Create experiment with dataset and application references
Execution: Experiment runs evaluation against the dataset
Monitoring: Track status and progress in real-time
Completion: Retrieve results, metrics, and analysis
Cleanup: Archive or delete completed experiments

Example

# Use this class to list
experiments = Experiment.list(
    application_id=application_id,
    dataset_id=dataset_id,
)

Experiments are permanent records of evaluation runs. Once created, the name cannot be changed, but metadata and description can be updated. Failed experiments retain error information for debugging and analysis.

id

name

status

created_at

updated_at

created_by

updated_by

project

application

dataset

description

error_reason

error_message

traceback

duration_ms

metadata

get_app_url()

Get the application URL for this experiment

Returns

str

classmethod get_by_id()

Retrieve an experiment by its unique identifier. Fetches an experiment from the Fiddler platform using its UUID. This is the most direct way to retrieve an experiment when you know its ID.

Parameters

id_

UUID | str

required

The unique identifier (UUID) of the experiment to retrieve. Can be provided as a UUID object or string representation.

Returns

The experiment instance with all metadata and configuration.

Raises

NotFound – If no experiment exists with the specified ID.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get experiment by UUID
experiment = Experiment.get_by_id(id_="550e8400-e29b-41d4-a716-446655440000")
print(f"Retrieved experiment: {experiment.name}")
print(f"Status: {experiment.status}")
print(f"Created: {experiment.created_at}")
print(f"Application: {experiment.application.name}")

This method makes an API call to fetch the latest experiment state from the server. The returned experiment instance reflects the current state in Fiddler.

classmethod get_by_name()

Retrieve an experiment by name within an application. Finds and returns an experiment using its name within the specified application. This is useful when you know the experiment name and application but not its UUID. Experiment names are unique within an application, making this a reliable lookup method.

Parameters

name

str

required

The name of the experiment to retrieve. Experiment names are unique within an application and are case-sensitive.

application_id

UUID | str

required

The UUID of the application containing the experiment. Can be provided as a UUID object or string representation.

Returns

The experiment instance matching the specified name.

Raises

NotFound – If no experiment exists with the specified name in the application.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)

# Get experiment by name within an application
experiment = Experiment.get_by_name(
    name="fraud-detection-eval-v1",
    application_id=application.id
)
print(f"Found experiment: {experiment.name} (ID: {experiment.id})")
print(f"Status: {experiment.status}")
print(f"Created: {experiment.created_at}")
print(f"Dataset: {experiment.dataset.name}")

Experiment names are case-sensitive and must match exactly. Use this method when you have a known experiment name from configuration or user input.

classmethod list()

List all experiments in an application. Retrieves all experiments that the current user has access to within the specified application. Returns an iterator for memory efficiency when dealing with many experiments.

Parameters

application_id

UUID | str

required

The UUID of the application to list experiments from. Can be provided as a UUID object or string representation.

dataset_id

UUID | str | None

default:"None"

The UUID of the dataset to list experiments from. Can be provided as a UUID object or string representation.

Yields

Experiment – Experiment instances for all accessible experiments in the application.

Raises

ApiError – If there’s an error communicating with the Fiddler API.

Returns

Iterator[Experiment]

Example

# Get application instance
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application.id)

# List all experiments in an application
for experiment in Experiment.list(application_id=application.id, dataset_id=dataset.id):
    print(f"Experiment: {experiment.name}")
    print(f"  ID: {experiment.id}")
    print(f"  Status: {experiment.status}")
    print(f"  Created: {experiment.created_at}")
    print(f"  Dataset: {experiment.dataset.name}")

# Convert to list for counting and filtering
experiments = list(Experiment.list(application_id=application.id, dataset_id=dataset.id ))
print(f"Total experiments in application: {len(experiments)}")

# Find experiments by status
completed_experiments = [
    exp for exp in Experiment.list(application_id=application.id, dataset_id=dataset.id)
    if exp.status == ExperimentStatus.COMPLETED
]
print(f"Completed experiments: {len(completed_experiments)}")

# Find experiments by name pattern
eval_experiments = [
    exp for exp in Experiment.list(application_id=application.id, dataset_id=dataset.id)
    if "eval" in exp.name.lower()
]
print(f"Evaluation experiments: {len(eval_experiments)}")

This method returns an iterator for memory efficiency. Convert to a list with list(Experiment.list(application_id)) if you need to iterate multiple times or get the total count. The iterator fetches experiments lazily from the API.

classmethod create()

Create a new experiment in an application. Creates a new experiment within the specified application on the Fiddler platform. The experiment must have a unique name within the application and will be linked to the specified dataset for evaluation. Note: It is not recommended to use this method directly. Instead, use the evaluate method. Creating and managing an experiment without evaluate wrapper is extremely advance usecase and should be avoided.

Parameters

name

str

required

Experiment name, must be unique within the application.

application_id

UUID | str

required

The UUID of the application to create the experiment in. Can be provided as a UUID object or string representation.

dataset_id

UUID | str

required

The UUID of the dataset to use for evaluation. Can be provided as a UUID object or string representation.

description

str | None

default:"None"

Optional human-readable description of the experiment.

metadata

dict | None

default:"None"

Optional custom metadata dictionary for additional experiment information.

Returns

The newly created experiment instance with server-assigned fields.

Raises

Conflict – If an experiment with the same name already exists in the application.
ValidationError – If the experiment configuration is invalid (e.g., invalid name format).
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get application and dataset instances
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application.id)

# Create a new experiment for fraud detection evaluation
experiment = Experiment.create(
    name="fraud-detection-eval-v1",
    application_id=application.id,
    dataset_id=dataset.id,
    description="Comprehensive evaluation of fraud detection model v1.0",
    metadata={"model_version": "1.0", "evaluation_type": "comprehensive", "baseline": "true"}
)
print(f"Created experiment with ID: {experiment.id}")
print(f"Status: {experiment.status}")
print(f"Created at: {experiment.created_at}")
print(f"Application: {experiment.application.name}")
print(f"Dataset: {experiment.dataset.name}")

After successful creation, the experiment instance is returned with server-assigned metadata. The experiment is immediately available for execution and monitoring. The initial status will be PENDING.

classmethod get_or_create()

Get an existing experiment by name or create a new one if it doesn’t exist. This is a convenience method that attempts to retrieve an experiment by name within an application, and if not found, creates a new experiment with that name. Useful for idempotent experiment setup in automation scripts and deployment pipelines.

Parameters

name

str

required

The name of the experiment to retrieve or create.

application_id

UUID | str

required

The UUID of the application to search/create the experiment in. Can be provided as a UUID object or string representation.

dataset_id

UUID | str

required

The UUID of the dataset to use for evaluation. Can be provided as a UUID object or string representation.

description

str | None

default:"None"

Optional human-readable description of the experiment.

metadata

dict | None

default:"None"

Optional custom metadata dictionary for additional experiment information.

Returns

Either the existing experiment with the specified name, or a newly created experiment if none existed.

Raises

ValidationError – If the experiment name format is invalid.
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get application and dataset instances
application = Application.get_by_name(name="fraud-detection-app", project_id=project_id)
dataset = Dataset.get_by_name(name="fraud-detection-tests", application_id=application.id)

# Safe experiment setup - get existing or create new
experiment = Experiment.get_or_create(
    name="fraud-detection-eval-v1",
    application_id=application.id,
    dataset_id=dataset.id,
    description="Comprehensive evaluation of fraud detection model v1.0",
    metadata={"model_version": "1.0", "evaluation_type": "comprehensive"}
)
print(f"Using experiment: {experiment.name} (ID: {experiment.id})")

# Idempotent setup in deployment scripts
experiment = Experiment.get_or_create(
    name="llm-benchmark-eval",
    application_id=application.id,
    dataset_id=dataset.id,
    metadata={"baseline": "true"}
)

# Use in configuration management
model_versions = ["v1.0", "v1.1", "v2.0"]
experiments = {}
for version in model_versions:
    experiments[version] = Experiment.get_or_create(
        name=f"fraud-detection-eval-{version}",
        application_id=application.id,
        dataset_id=dataset.id,
        metadata={"model_version": version}
    )

This method is idempotent - calling it multiple times with the same name and application_id will return the same experiment. It logs when creating a new experiment for visibility in automation scenarios.

update()

Update experiment description, metadata, and status. Updates the experiment’s description, metadata, and/or status. This method allows you to modify the experiment’s configuration after creation, including updating the experiment status and error information for failed experiments.

Parameters

description

str | None

default:"None"

Optional new description for the experiment. If provided, replaces the existing description. Set to empty string to clear.

metadata

dict | None

default:"None"

Optional new metadata dictionary for the experiment. If provided, replaces the existing metadata completely. Use empty dict to clear.

status

ExperimentStatus | None

default:"None"

Optional new status for the experiment. Can be used to update experiment status (e.g., PENDING, RUNNING, COMPLETED, FAILED).

error_reason

str | None

default:"None"

Required when status is FAILED. The reason for the experiment failure.

error_message

str | None

default:"None"

Required when status is FAILED. Detailed error message for the failure.

traceback

str | None

default:"None"

Required when status is FAILED. Stack trace information for debugging.

duration_ms

int | None

default:"None"

Optional duration in milliseconds for the experiment execution

Returns

The updated experiment instance with new metadata and configuration.

Raises

ValueError – If no update parameters are provided (all are None) or if status is FAILED but error_reason, error_message, or traceback are missing.
ValidationError – If the update data is invalid (e.g., invalid metadata format).
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)

# Update description and metadata
updated_experiment = experiment.update(
    description="Updated comprehensive evaluation of fraud detection model v1.1",
    metadata={"model_version": "1.1", "evaluation_type": "comprehensive", "updated_by": "john_doe"}
)
print(f"Updated experiment: {updated_experiment.name}")
print(f"New description: {updated_experiment.description}")

# Update only metadata
experiment.update(metadata={"last_updated": "2024-01-15", "status": "active"})

# Update experiment status to completed
experiment.update(status=ExperimentStatus.COMPLETED)

# Mark experiment as failed with error details
experiment.update(
    status=ExperimentStatus.FAILED,
    error_reason="Evaluation timeout",
    error_message="The evaluation process exceeded the maximum allowed time",
    traceback="Traceback (most recent call last): File evaluate.py, line 42..."
)

# Clear description
experiment.update(description="")

# Batch update multiple experiments
for experiment in Experiment.list(application_id=application_id):
    if experiment.status == ExperimentStatus.COMPLETED:
        experiment.update(metadata={"archived": "true"})

This method performs a complete replacement of the specified fields. For partial updates, retrieve current values, modify them, and pass the complete new values. The experiment name and ID cannot be changed. When updating status to FAILED, all error-related parameters are required.

delete()

Delete the experiment. Permanently deletes the experiment and all associated data from the Fiddler platform. This action cannot be undone and will remove all experiment results, metrics, and metadata.

Raises

ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)

# Delete the experiment
experiment.delete()
print("Experiment deleted successfully")

# Delete multiple experiments
for experiment in Experiment.list(application_id=application_id):
    if experiment.status == ExperimentStatus.FAILED:
        print(f"Deleting failed experiment: {experiment.name}")
        experiment.delete()

This operation is irreversible. Once deleted, the experiment and all its associated data cannot be recovered. Consider archiving experiments instead of deleting them if you need to preserve historical data.

add_items()

Add outputs of LLM/Agent/Application against dataset items to the experiment. Adds outputs of LLM/Agent/Application (task or target function) against dataset items to the experiment, representing individual test case outcomes. Each item contains the outputs of LLM/Agent/Application results, timing information, and status for a specific dataset item.

Parameters

items

list[NewExperimentItem]

required

List of NewExperimentItem instances containing outputs of LLM/Agent/Application against dataset items. Each item should include:

dataset_item_id: UUID of the dataset item being evaluated
outputs: Dictionary containing the outputs of the task function against dataset item
duration_ms: Duration of the execution in milliseconds:
status: Status of the outputs of the task function / scoring against dataset item (PENDING, COMPLETED, FAILED, etc.)
error_reason: Reason for failure, if applicable
error_message: Detailed error message, if applicable

Returns

List of UUIDs for the newly created experiment items.

Raises

ValueError – If the items list is empty.
ValidationError – If any item data is invalid (e.g., missing required fields).
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)

# Create evaluation result items
from fiddler_evals.pydantic_models.experiment import NewExperimentItem
from datetime import datetime, timezone

items = [
    NewExperimentItem(
        dataset_item_id=dataset_item_id_1,
        outputs={"answer": "The watermelon seeds pass through your digestive system"},
        duration_ms=1000,
        end_time=datetime.now(tz=timezone.utc),
        status="COMPLETED",
        error_reason=None,
        error_message=None
    ),
    NewExperimentItem(
        dataset_item_id=dataset_item_id_2,
        outputs={"answer": "The precise origin of fortune cookies is unclear"},
        duration_ms=1000,
        end_time=datetime.now(tz=timezone.utc),
        status="COMPLETED",
        error_reason=None,
        error_message=None
    )
]

# Add items to experiment
item_ids = experiment.add_items(items)
print(f"Added {len(item_ids)} evaluation result items")
print(f"Item IDs: {item_ids}")

# Add items from evaluation results
items = [
    {
        "dataset_item_id": str(dataset_item_id),
        "outputs": {"answer": result["answer"]},
        "duration_ms": result["duration_ms"],
        "end_time": result["end_time"],
        "status": "COMPLETED"
    }
    for result in items
]
item_ids = experiment.add_items([NewExperimentItem(**item) for item in items])

# Batch add items with error handling
try:
    item_ids = experiment.add_items(items)
    print(f"Successfully added {len(item_ids)} items")
except ValueError as e:
    print(f"Validation error: {e}")
except Exception as e:
    print(f"Failed to add items: {e}")

This method is typically used after running evaluations to store the results in the experiment. Each item represents the evaluation of a single dataset item and contains all relevant timing, output, and status information.

get_items()

Retrieve all experiment result items from the experiment. Fetches all experiment result items (outputs, timing, status) that were generated by the task function against dataset items. Returns an iterator for memory efficiency when dealing with large experiments containing many result items.

Returns

Iterator of ExperimentItem instances for all result items in the experiment.

Raises

ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)

# Get all result items from the experiment
for item in experiment.get_items():
    print(f"Item ID: {item.id}")
    print(f"Dataset Item ID: {item.dataset_item_id}")
    print(f"Outputs: {item.outputs}")
    print(f"Status: {item.status}")
    print(f"Duration: {item.duration_ms}")
    if item.error_reason:
        print(f"Error: {item.error_reason} - {item.error_message}")
    print("---")

# Convert to list for analysis
all_items = list(experiment.get_items())
print(f"Total result items: {len(all_items)}")

# Filter items by status
completed_items = [
    item for item in experiment.get_items()
    if item.status == "COMPLETED"
]
print(f"Completed items: {len(completed_items)}")

# Filter items by error status
failed_items = [
    item for item in experiment.get_items()
    if item.status == "FAILED"
]
print(f"Failed items: {len(failed_items)}")

# Process items in batches
batch_size = 100
for i, item in enumerate(experiment.get_items()):
    if i % batch_size == 0:
        print(f"Processing batch {i // batch_size + 1}")
    # Process item...

# Analyze outputs
for item in experiment.get_items():
    if item.outputs.get("confidence", 0) < 0.8:
        print(f"Low confidence item: {item.id}")

This method returns an iterator for memory efficiency. Convert to a list with list(experiment.get_items()) if you need to iterate multiple times or get the total count. The iterator fetches items lazily from the API.

add_results()

Add evaluation results to the experiment. Adds complete evaluation results to the experiment, including both the experiment item data (outputs, timing, status) and all associated evaluator scores. This method is typically used after running evaluations to store the complete results of the evaluation process for a batch of dataset items. This method will only append the results to the experiment. Note: It is not recommended to use this method directly. Instead, use the evaluate method. Creating and managing an experiment without evaluate wrapper is extremely advance usecase and should be avoided.

Parameters

items

list[ExperimentItemResult]

required

List of ExperimentItemResult instances containing:

experiment_item: NewExperimentItem with outputs, timing, and status
scores: List of Score objects from evaluators for this item

Returns

Results are added to the experiment on the server.

Raises

ValueError – If the items list is empty.
ValidationError – If any item data is invalid (e.g., missing required fields).
ApiError – If there’s an error communicating with the Fiddler API.

Example

# Get existing experiment
experiment = Experiment.get_by_name(name="fraud-detection-eval-v1", application_id=application_id)

# Create experiment item with outputs
experiment_item = NewExperimentItem(
    dataset_item_id=dataset_item.id,
    outputs={"prediction": "fraud", "confidence": 0.95},
    duration_ms=1000,
    end_time=datetime.now(tz=timezone.utc),
    status="COMPLETED"
)

# Create scores from evaluators
scores = [
    Score(
        name="accuracy",
        evaluator_name="AccuracyEvaluator",
        value=1.0,
        label="Correct",
        reasoning="Prediction matches ground truth"
    ),
    Score(
        name="confidence",
        evaluator_name="ConfidenceEvaluator",
        value=0.95,
        label="High",
        reasoning="High confidence in prediction"
    )
]

# Create result combining item and scores
result = ExperimentItemResult(
    experiment_item=experiment_item,
    scores=scores
)

# Add results to experiment
experiment.add_results([result])

This method is typically called after running evaluations to store complete results. The results include both the experiment item data and all evaluator scores, providing a complete record of the evaluation process.

​Example

​id

​name

​status

​created_at

​updated_at

​created_by

​updated_by

​project

​application

​dataset

​description

​error_reason

​error_message

​traceback

​duration_ms

​metadata

​get_app_url()

​Returns

​classmethod get_by_id()

​Parameters

​Returns

​Raises

​Example

​classmethod get_by_name()

​Parameters

​Returns

​Raises

​Example

​classmethod list()

​Parameters

​Yields

​Raises

​Returns

​Example

​classmethod create()

​Parameters

​Returns

​Raises

​Example

​classmethod get_or_create()

​Parameters

​Returns

​Raises

​Example

​update()

​Parameters

​Returns

​Raises

​Example

​delete()

​Raises

​Example

​add_items()

​Parameters

​Returns

​Raises

​Example

​get_items()

​Returns

​Raises

​Example

​add_results()

​Parameters

​Returns

​Raises

​Example

Example

id

name

status

created_at

updated_at

created_by

updated_by

project

application

dataset

description

error_reason

error_message

traceback

duration_ms

metadata

get_app_url()

Returns

classmethod get_by_id()

Parameters

Returns

Raises

Example

classmethod get_by_name()

Parameters

Returns

Raises

Example

classmethod list()

Parameters

Yields

Raises

Returns

Example

classmethod create()

Parameters

Returns

Raises

Example

classmethod get_or_create()

Parameters

Returns

Raises

Example

update()

Parameters

Returns

Raises

Example

delete()

Raises

Example

add_items()

Parameters

Returns

Raises

Example

get_items()

Returns

Raises

Example

add_results()

Parameters

Returns

Raises

Example