Evaluator Downsampling Quick Start - Fiddler Documentation

Evaluator downsampling lets an Evaluator Rule score only a fraction of the spans it matches instead of every one — cutting LLM-as-a-Judge evaluation cost at scale. You control it with one per-rule field, sampling_rate, a number in (0.0, 1.0] that defaults to 1.0 (evaluate everything). This guide walks you through configuring it from the UI and the REST API.

Traces are always persisted, even with downsampling enabled. Downsampling only controls whether an evaluator runs on a matching span — it never affects ingestion or storage. Sampled-out traces stay fully queryable in the Explorer.

Availability. Evaluator downsampling is rolling out behind a feature flag. Setting a sampling_rate requires it to be enabled for your deployment; until then, the field is unavailable in the UI and the API rejects requests that include it with a 400. Contact Fiddler support to enable it.

What you’ll learn

Set a rule’s sampling rate from the UI when creating or editing a rule
Create and tune a downsampled rule end-to-end with the REST API
Pause evaluation the right way (and the guardrails on sampling_rate)

Time to complete: ~10 minutes

Prerequisites

Before you begin, ensure you have:

Evaluator downsampling enabled for your deployment (see the availability note above)
A GenAI application with span data (see Evaluator Rules)
A Fiddler API key — from Settings > Credentials (for the API walkthrough)
jq installed locally (optional, used to pretty-print API responses)

Configure downsampling in the UI

When evaluator downsampling is enabled for your deployment, you set a rule’s sampling rate from the Evaluator Rules tab. The UI uses a percentage; the REST API uses a fraction — enter 10 in the UI for the same coverage as sampling_rate: 0.1 over the API.

In your application, open the Evaluator Rules tab and click Add Rule. Configure the evaluator, input mappings, and application rules as usual (see Evaluator Rules).
In the Add Evaluator Rule dialog, set the Sampling rate to the percentage of matching traces you want evaluated — for example, enter 10 for 10%. Leave it at the default (100) to evaluate everything.
Save the rule. It begins evaluating the configured fraction of matching spans.
To change the rate later, edit the Sampling rate for an existing rule directly in its Sampling rate cell in the Evaluator Rules table. To stop evaluation entirely, use the row’s Pause action — do not set a near-zero rate.

The Sampling rate control is rolling out behind a feature flag and may not yet be visible in your environment. If downsampling is enabled for your deployment but the control hasn’t rolled out, use the REST API below in the meantime.

Configure downsampling with the REST API

This walkthrough creates a GenAI project, an application, an evaluator, and a downsampled rule, then tunes and tears it down. All endpoints are under $FIDDLER_URL/v3/... and authenticate with a bearer token (Authorization: Bearer $FIDDLER_API_KEY).

Resource naming rule. Resource names must start with a letter and contain only letters, numbers, and underscores (_) — no hyphens or spaces.

1. Set up your environment

export FIDDLER_URL="https://YOUR_NAMESPACE.fiddler.ai"   # no trailing slash, no /v3
export FIDDLER_API_KEY="YOUR_API_KEY"

# reusable auth headers
auth=(-H "Authorization: Bearer $FIDDLER_API_KEY" -H "Content-Type: application/json")

2. Confirm connectivity and auth

curl -s "${auth[@]}" "$FIDDLER_URL/v3/projects?limit=1" | jq '.kind, .data.total'

Expect "NORMAL" (or "PAGINATED") and a number. A 401 means the URL or key is wrong.

3. Create a GenAI project and application

A project with asset_type=GEN_AI_APP is the container for GenAI applications; the application is what traces are sent to and what rules bind to.

export PROJECT_ID=$(curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/projects" \
  -d '{"name": "downsampling_demo", "asset_type": "GEN_AI_APP"}' | jq -r '.data.id')

export APPLICATION_ID=$(curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/applications" \
  -d "{\"name\": \"chat_assistant\", \"project_id\": \"$PROJECT_ID\"}" | jq -r '.data.id')

echo "PROJECT_ID=$PROJECT_ID  APPLICATION_ID=$APPLICATION_ID"

4. Create an evaluator

An evaluator is the scorer definition; the rule (next step) binds it to the application and is where sampling_rate lives. The enrichment_name field selects the evaluator type. This example uses a built-in PII check (least setup):

export EVALUATOR_ID=$(curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/evaluators" \
  -d '{"name": "demo_pii_check", "enrichment_name": "pii_detection", "enrichment_config": {}}' \
  | jq -r '.data.id')

echo "EVALUATOR_ID=$EVALUATOR_ID"

For the cost-savings story, use an LLM-as-a-Judge evaluator instead (enrichment_name: "llm_as_a_judge", with a model_id and prompt_spec in enrichment_config). Discover the available types and their config schemas with GET /v3/evaluators/get-config-options.

5. Discover the input keys to map

A rule maps the evaluator’s template variables to span attribute paths via mapped_input_keys. This call returns the keys the evaluator expects:

curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/evaluator-rules/map-input-keys" \
  -d "{\"application_id\": \"$APPLICATION_ID\", \"evaluator_id\": \"$EVALUATOR_ID\"}" \
  | jq '.data'

Match each value’s type to the key’s type in rule_input_keys: a string-typed key takes a single attribute path (e.g. "fiddler.contents.gen_ai.llm.output"); an array-typed key takes a list of paths.

6. Create the rule with a sampling rate

This is the feature. sampling_rate: 0.1 means “score ~10% of matching traces; store 100%.”

export RULE_ID=$(curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/evaluator-rules" \
  -d "{
    \"name\": \"downsampled_rule\",
    \"application_id\": \"$APPLICATION_ID\",
    \"evaluator_id\": \"$EVALUATOR_ID\",
    \"mapped_input_keys\": {\"text\": \"fiddler.contents.gen_ai.llm.output\"},
    \"sampling_rate\": 0.1
  }" | jq -r '.data.id')

echo "RULE_ID=$RULE_ID"

If this returns 400 ... "Evaluator downsampling is not enabled for this deployment", the feature flag is off for your deployment — see the availability note at the top of this page.

7. Read the rule back

Confirm the rate persisted. The list endpoint filters via a filter query param; build it with -g -G --data-urlencode so curl encodes the JSON and doesn’t treat [/] as glob characters:

curl -s -g -G "${auth[@]}" "$FIDDLER_URL/v3/evaluator-rules" \
  --data-urlencode "filter={\"condition\":\"AND\",\"rules\":[{\"field\":\"application_id\",\"operator\":\"equal\",\"value\":\"$APPLICATION_ID\"}]}" \
  | jq '.data.items[] | {id, name, enabled, sampling_rate}'

You should see "sampling_rate": 0.1. The traces are all still stored and queryable — only evaluation is sampled.

8. Tune the sampling rate live

Dial coverage up or down on a live rule without recreating it:

curl -s "${auth[@]}" -X PATCH "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" \
  -d '{"sampling_rate": 0.5}' | jq '.data | {id, name, sampling_rate}'

A new rate takes effect at the workers within the rule-cache TTL (~60 seconds).

9. Pause evaluation the right way

sampling_rate cannot be 0.0. To pause a rule entirely, set enabled=false (this preserves the configured rate, so re-enabling restores it):

# Pause:
curl -s "${auth[@]}" -X PATCH "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" \
  -d '{"enabled": false}' | jq '.data | {id, enabled, sampling_rate}'

# Re-enable:
curl -s "${auth[@]}" -X PATCH "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" \
  -d '{"enabled": true}' | jq '.data | {id, enabled, sampling_rate}'

Both sampling_rate: 0.0 and sampling_rate > 1.0 are rejected with a 400:

curl -s -o /dev/null -w "%{http_code}\n" "${auth[@]}" -X PATCH \
  "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" -d '{"sampling_rate": 0.0}'   # -> 400

10. Clean up

curl -s "${auth[@]}" -X DELETE "$FIDDLER_URL/v3/evaluator-rules/$RULE_ID" | jq '.kind'
# Optional: delete the project (cascades to the application)
curl -s "${auth[@]}" -X DELETE "$FIDDLER_URL/v3/projects/$PROJECT_ID" | jq '.kind'

Quick reference

Action	Method & path
Create project	`POST /v3/projects`
Create application	`POST /v3/applications`
List evaluator config options	`GET /v3/evaluators/get-config-options`
Create evaluator	`POST /v3/evaluators`
Discover input keys	`POST /v3/evaluator-rules/map-input-keys`
Create rule (+`sampling_rate`)	`POST /v3/evaluator-rules`
List rules	`GET /v3/evaluator-rules?filter=...`
Update rule (`sampling_rate`/`enabled`)	`PATCH /v3/evaluator-rules/{id}`
Delete rule	`DELETE /v3/evaluator-rules/{id}`

sampling_rate rules: range (0.0, 1.0]; default 1.0 (score everything); 0.0 rejected (pause with enabled=false instead); values above 1.0 rejected; resolution is 4 decimal places (1 basis point).

Units differ by surface. The UI uses a percentage (10 = 10%); the REST API uses a fraction (sampling_rate: 0.1).

Seeing sampling in action (optional, advanced)

Demonstrating a visible ~10% sample requires ingesting many traces and waiting for async enrichment. GenAI trace ingestion is OTLP-based (POST /v1/traces with an Authorization: Bearer header and a fiddler-application-id header) — it is not part of the v3 REST surface used above. After ingesting traces, query them back to confirm that sampled-out traces are still stored (just not scored):

curl -s "${auth[@]}" -X POST "$FIDDLER_URL/v3/spans/query" \
  -d "{
    \"application_id\": \"$APPLICATION_ID\",
    \"start_time\": \"2026-01-01T00:00:00Z\",
    \"end_time\": \"2026-12-31T23:59:59Z\",
    \"page_size\": 50
  }" | jq '.data.items[] | {trace_id, evaluators}'

Spans with a non-empty evaluators array were scored (sampled in); stored spans with an empty array were sampled out — same data retained, evaluation skipped.

Next steps

Configure Evaluator Downsampling — the full guide: when to use it, recommended minimum trace volume, sampling_rate vs deactivating a rule, and how sampling correlates across rules
Evaluator Rules — create and manage Evaluator Rules
Evaluator Rules REST API — full create/update API contract, including the sampling_rate field
Explorer — inspect ingested traces, including sampled-out traces

​What you’ll learn

​Prerequisites

​Configure downsampling in the UI

​Configure downsampling with the REST API

​1. Set up your environment

​2. Confirm connectivity and auth

​3. Create a GenAI project and application

​4. Create an evaluator

​5. Discover the input keys to map

​6. Create the rule with a sampling rate

​7. Read the rule back

​8. Tune the sampling rate live

​9. Pause evaluation the right way

​10. Clean up

​Quick reference

​Seeing sampling in action (optional, advanced)

​Next steps

What you’ll learn

Prerequisites

Configure downsampling in the UI

Configure downsampling with the REST API

1. Set up your environment

2. Confirm connectivity and auth

3. Create a GenAI project and application

4. Create an evaluator

5. Discover the input keys to map

6. Create the rule with a sampling rate

7. Read the rule back

8. Tune the sampling rate live

9. Pause evaluation the right way

10. Clean up

Quick reference

Seeing sampling in action (optional, advanced)

Next steps