Enhance ML and LLM Insights with Custom Metrics
Overview
Custom metrics offer the capability to define metrics that align precisely with your machine learning requirements. Whether it's tracking business KPIs, crafting specialized performance assessments, or computing weighted averages, custom metrics empower you to tailor measurements to your specific needs. Seamlessly integrate these custom metrics throughout Fiddler, leveraging them in dashboards, alerting, and performance tracking.
Create user-defined metrics by employing a simple query language we call Fiddler Query Language (FQL). FQL enables you to leverage your model's features, metadata, predictions, and outcomes for new data fields using a rich array of aggregations, operators, and metric functions, thereby expanding the depth of your analytical insights.
How to Define a Custom Metric
Build custom metrics effortlessly with Fiddler's intuitive Excel-formula-like syntax. Once a custom metric is defined, Fiddler distinguishes itself by seamlessly managing time granularity and ranges within the charting, dashboarding, and analytics experience. This empowers you to effortlessly adjust time range and granularity without the need to modify your query, ensuring a smooth and efficient analytical experience.
Adding a Custom Metric
Note: To add a Custom Metric using the Python client, see fdl.CustomMetric
From the model schema page, you can access the model's custom metrics by clicking the Custom Metrics tab at the top of the page. Then click Add Custom Metric to add a new Custom Metric. Finally, enter the name, description, and FQL definition for your custom metric and click Save.

Accessing Custom Metrics in Charts and Alerts
After your custom metric is saved, it can be used in your chart and alert definitions.
Charts
Set Metric Type
to Custom Metric
and select your desired custom metric.

Alerts
When creating a new alert rule, set Metric Type
to Custom Metric
, and under the Metric
field select your desired custom metric or author a new metric to use.

Modifying Custom Metrics
Since alerts can be set on Custom Metrics, making modifications to a metric may introduce inconsistencies in alerts.
🚧 Therefore, custom metrics cannot be modified once they are created.
If you'd like to try out a new metric, you can create a new one with a different name and definition.
Deleting Custom Metrics
To delete a custom metric using the Python client, see CustomMetric.delete(). Alternatively, from the custom metrics tab, you can delete a metric by clicking the trash icon next to the metric record.

Examples
📘 Custom metrics must return either:
an aggregate (produced by aggregate functions or built-in metric functions)
a combination of aggregates
Simple Metric
Given this example use case:
If an event is a false negative, assign a value of -40. If the event is a false positive, assign a value of -400. If the event is a true positive or true negative, then assign a value of 250.
Create a new Custom Metric with the following FQL formula:
average(if(fn(), -40, if(fp(), -400, 250)))
Fiddler offers many convenience functions such as fp()
and fn()
.
Alternatively, we could also identify false positives and false negatives the old fashioned way.
average(if(Prediction < 0.5 and Target == 1, -40, if(Prediction >= 0.5 and Target == 0, -400, 250)))
Here, we assume Prediction
is the name of the output column for a binary classifier and Target
is the name of our label column.
Tweedie Loss
In our next example, we provide an example implementation of the Tweedie Loss Function. Here, Target
is the name of the target column and Prediction
is the name of the prediction/output column.
average((Target \* Prediction ^ (1 - 0.5)) / (1 - 0.5) + Prediction ^ (2 - 0.5) / (2 - 0.5))
Modifying Custom Metrics
Since alerts can be set on Custom Metrics, making modifications to a metric may introduce inconsistencies in alerts.
🚧 Therefore, custom metrics cannot be modified once they are created.
If you'd like to try out a new metric, you can create a new one with a different name and definition.
❓ Questions? Talk to a product expert or request a demo.
💡 Need help? Contact us at [email protected].
Last updated
Was this helpful?