Evaluator Rules

​

Overview

​

What Are Evaluator Rules?

1. Evaluator Configuration - The evaluator definition, including provider, model, and prompt
2. Input Field Mapping - How span data is passed to the evaluator’s input variables
3. Application Rules - Conditions that determine which spans qualify for evaluation
4. Backfill Configuration - Whether to apply evaluations to historical data

​

How Evaluator Rules Work

1. The system checks all active Evaluator Rules
2. Each rule evaluates whether its Application Rules match the span’s attributes
3. If a match is found, the system extracts data from the span using Input Field Mappings
4. The evaluator runs with the mapped data as input
5. Results are stored and made available in dashboards and analytics

​

Key Concepts

​

Evaluators

• Provider - The LLM provider (OpenAI, Anthropic, Gemini, Fiddler)
• Model - The specific model to use for evaluation
• Credentials - Authentication to the provider (configured via LLM Gateway)
• Prompt or Logic - The evaluation instructions or function

​

Input Mappings

​

Application Rules

• AND condition across categories - A span must match ALL rule categories
• OR condition within a category - A span can match ANY value within a single category

Rule 1: SpanType = llm
Rule 2: Region = us-east OR us-west

Result: Evaluates spans that are type "llm" AND in either "us-east" or "us-west"

​

Backfill

​

Rule Limit

​

Create an Evaluator Rule

​

Prerequisites

• Active Application - A GenAI application with span data
• Configured Evaluators - Organization-level evaluators ready to use
• LLM Gateway Credentials - If using custom LLM-based evaluators (see LLM Gateway Configuration)

​

Step-by-Step Guide

1

Select an Evaluator

Navigate to your application in the Fiddler UI and access the evaluator configuration:

1. Click the Evaluator Rules tab
2. Click Add Rule in the top-right corner
3. The Add Evaluator Rule dialog opens with available evaluators

Choose an evaluator from the list:Fiddler-Provided Evaluators:

• Topic Classification
• Embedding
• Token Count
• Answer Relevance
• Coherence
• Conciseness
• Context Relevance
• RAG Faithfulness
• PII Detection
• Sentiment Analysis
• F# Prompt Safety
• F# Response Faithfulness
• Llm As A Judge (custom evaluator)

Configure Custom Evaluator (Llm As A Judge)If you select Llm As A Judge, you’ll need to configure the evaluator:a. Evaluator Name

• Enter a descriptive name (e.g., saddestpuppynoises)

b. Provider

• Select the LLM provider (e.g., fiddler)

c. Credential

• Choose the API credential for authentication (e.g., dummy)

d. Model

• Select the specific model (e.g., llama3.1-8b)

e. Prompt Template

• Enter evaluation instructions with input variables using curly braces: {{variableName}}
• Example: sad {{puppynoises}}

f. Outputs

• Define the expected response format in JSON

Example Output Configuration:

{
  "name": "sadnoises",
  "description": "sad puppy noises",
  "type": "categorical",
  "choices": ["sad", "not sad"]
}

Tip: For Fiddler-provided evaluators, the evaluation method and fields are predetermined. You only need to map inputs and configure application rules.

Click Next to continue.

2

Map Input Fields

Map each evaluator input variable to a span attribute.

1. In the Map Evaluator step, you’ll see all required input variables
2. For each variable (e.g., puppynoises):
   • Click the Select an attribute or enter a custom path dropdown
   • Choose from available span attributes or enter a custom path manually

Common Span Attributes:

• fiddler.span.user.pirate_completion_score
• fiddler.contents.gen_ai.llm.context
• fiddler.span.system.gen_ai.usage.output_tokens
• fiddler.session.user.region
• fiddler.span.system.gen_ai.usage.input_tokens
• fiddler.session.user.max_conversation_turns
• gen_ai.system
• fiddler.contents.gen_ai.llm.input.user
• fiddler.contents.gen_ai.tool.input
• And many more…

3. Repeat for all input variables
4. Click Next to continue

All required input variables must be mapped. The evaluator cannot run without complete input mappings.

3

Define Application Rules

Specify which spans to evaluate by setting filter conditions.

1. In the Apply Rules step, you’ll see the current rule conditions
2. The info box shows: “This evaluator will apply to spans that match ALL of the following conditions:”
3. Click Add Rule to add a new condition category

For each rule category:a. Rule Category

• Select the attribute type (e.g., Span Type)

b. Values

• Choose which values to match:
  • chain
  • llm ✓
  • tool

c. Custom Values

• (Optional) Add specific custom values to match

Understanding Rule Logic

• AND condition across categories - A span must match ALL rule categories
• OR condition within a category - A span can match ANY value within a single category

Example:

Rule 1: SpanType = llm
Rule 2: Region = us-east OR us-west

Result: Evaluates spans that are type "llm" AND in either "us-east" or "us-west"

5. Add multiple rule categories as needed
6. Click Next to continue

4

Configure Backfill and Review

Determine whether to apply the evaluator to existing historical data and review your configuration.Backfill ConfigurationChoose one of three options:Option 1: Apply to all past data

• Evaluates all existing spans in the dataset
• Use when: You need complete historical coverage
• Warning: May take significant time for large datasets

Option 2: Apply from a specific past date

• Evaluates spans created after a chosen date
• Use when: You want partial historical coverage
• Select the start date using the date picker

Option 3: No backfill (Default)

• Evaluates only new spans created after activation
• Use when: You only need a forward-looking evaluation
• Best for: Testing new evaluators or reducing processing time

Review Configuration SummaryEvaluator Configuration

• Evaluator name, model, provider, credential
• Prompt template and expected outputs

Input Field Mapping

• Variable → Span attribute mappings

Application Rules

• Span matching conditions

Performance Tip: Start with “No backfill” to test your evaluator configuration. Once validated, you can create a new rule with backfill enabled.

5

Save and Activate

Complete the configuration and activate your evaluator rule.

1. Configuration Name
   • Enter a descriptive name for this evaluator rule (e.g., puppyjudge)
   • This name identifies the rule in your application’s Evaluator Rules list
2. Finalize:
   • Click Save to activate the rule
   • Or click Back to modify any settings
   • Or click Cancel to discard the configuration

Once saved, the evaluator rule becomes active and begins evaluating spans that match your criteria.

---

​

Manage Evaluator Rules

​

View Active Rules

​

Activate or Deactivate a Rule

1. Locate the rule in the Evaluator Rules table
2. Click the Status toggle to activate or deactivate
   • Active - Rule is running on matching spans
   • Inactive - Rule is paused and not evaluating new spans

​

Delete a Rule

1. Locate the rule in the Evaluator Rules table
2. Click the delete icon (trash can) at the end of the row
3. Confirm the deletion when prompted

​

Best Practices

​

Evaluator Configuration

• Use Descriptive Names - Name evaluators and rules clearly (e.g., rag_faithfulness_prod instead of rule1)
• Test Before Backfill - Create rules without backfill first, validate results, then create a new rule with backfill if needed
• Version Your Prompts - Include version identifiers in custom judge names (e.g., topic_classifier_v2)

​

Input Mapping

• Validate Paths - Ensure span attributes exist before mapping
• Use Consistent Paths - Standardize attribute naming across your application
• Document Custom Paths - Keep a reference of custom attribute paths for your team

​

Application Rules

• Start Broad, Refine Later - Begin with simple rules, add complexity as needed
• Avoid Over-Filtering - Don’t create rules so specific that they match too few spans
• Test Rule Logic - Verify spans are matching as expected using span search

​

Performance Optimization

• Limit Backfill Scope - Use date-based backfill instead of “all past data” for large datasets
• Monitor Evaluation Latency - Track how long evaluations take and optimize prompts if needed
• Batch Similar Rules - Group related evaluations to reduce overhead

​

Troubleshooting

​

Evaluator Not Running

• Verify Application Rules match actual span attributes
• Check that all input mappings point to valid span fields
• Ensure LLM Gateway credentials are valid and not expired
• Review span data to confirm matching spans exist

​

Missing Input Data

• Verify the span attribute path is correct
• Check that the attribute exists in your span schema
• Ensure spans contain data for the mapped field
• Use a different attribute or add the field to your instrumentation

​

Backfill Taking Too Long

• Use date-based backfill instead of all past data
• Start with recent data and expand the date range gradually
• Consider creating multiple rules for different time periods
• Deactivate unnecessary rules to free up processing capacity

​

Unexpected Evaluator Results

• Review the evaluator prompt template for clarity
• Verify input mappings are passing the correct data
• Test the evaluator with sample data outside Fiddler
• Check for prompt ambiguity or missing context
• Adjust the prompt and create a new rule version

​

Evaluator Rule Limit Reached

• Delete evaluator rules you no longer need — each application supports up to 100 rules
• Remember that deactivating a rule does not free a slot; disabled rules still count toward the limit
• Contact Fiddler support if your use case requires a higher limit

​

Related Documentation

• LLM Gateway Configuration - Configure LLM provider credentials
• Configure Evaluator Downsampling - Reduce evaluation cost at scale with per-rule sampling
• Evaluator Downsampling Quick Start - Configure a downsampled rule from the UI or REST API
• Fiddler Evals SDK - Create and manage evaluators programmatically
• Custom Evaluators - Build custom evaluation logic
• Application Monitoring - Monitor your GenAI applications