CustomJudgeSpec

Provides a structured, validated way to define evaluation prompts with input/output schemas, transforms, and multi-message templates. A CustomJudgeSpec can be defined once and reused across multiple evaluator instances or shared across a codebase.

Parameters

str | list[Message]

required

The evaluation prompt. Can be a plain string (wrapped in a single user message) or a list of Message dicts.

Dict[str, OutputField]

required

Schema defining the expected output fields.

Dict[str, InputFieldSpec] | None

default:"None"

Optional metadata for template variables.

Dict[str, OutputField] | None

default:"None"

Optional schema for the LLM response before transformation. Required when output fields use transform.

Example

Defining a reusable faithfulness evaluator spec:

from fiddler_evals.evaluators.custom_judge import (
    CustomJudge, CustomJudgeSpec, Message, InputFieldSpec,
    OutputFieldTransform,
)

FAITHFULNESS_SPEC = CustomJudgeSpec(
    prompt_template=[
        Message(role='system', content='Judge faithfulness.'),
        Message(role='user', content=(
            'Question: {{ query }}\n'
            'Documents: {{ docs }}\n'
            'Answer: {{ answer }}'
        )),
    ],
    inputs={
        'query': InputFieldSpec(required=True),
        'docs': InputFieldSpec(required=True),
        'answer': InputFieldSpec(required=True),
    },
    llm_response_fields={
        'is_faithful': {
            'type': 'string',
            'choices': ['faithful', 'not_faithful'],
        },
    },
    output_fields={
        'label': {
            'type': 'string',
            'choices': ['yes', 'no'],
            'transform': OutputFieldTransform(
                source_field='is_faithful',
                value_map={
                    'faithful': 'yes',
                    'not_faithful': 'no',
                },
            ),
        },
    },
)

evaluator = CustomJudge(
    prompt_spec=FAITHFULNESS_SPEC,
    model='openai/gpt-4o',
)

Attributes

The evaluation prompt — a plain string (wrapped in a single user message) or a list of Message dicts.

Schema defining the final output fields; each field produces one Score.

Optional metadata for the {{ placeholder }} template variables, keyed by placeholder name.

Optional schema for the raw LLM response before transformation; required when any output field uses transform.

Fiddler Python Client SDK

Fiddler Evals SDK

Fiddler OTel SDK

Fiddler LangGraph SDK

Fiddler LangChain SDK

Fiddler Google ADK SDK

Fiddler Strands Agent SDK

Fiddler OTel JS SDK

Fiddler LangGraph JS SDK

Fiddler LangChain JS SDK

Parameters

Example

Attributes

​Parameters

​Example

​Attributes

Parameters

Example

Attributes