LogoLogo
👨‍💻 API Reference📣 Release Notes📺 Request a Demo
  • Introduction to Fiddler
    • Monitor, Analyze, and Protect your ML Models and Gen AI Applications
  • Fiddler Doc Chatbot
  • First Steps
    • Getting Started With Fiddler Guardrails
    • Getting Started with LLM Monitoring
    • Getting Started with ML Model Observability
  • Tutorials & Quick Starts
    • LLM and GenAI
      • LLM Evaluation - Compare Outputs
      • LLM Monitoring - Simple
    • Fiddler Free Guardrails
      • Guardrails - Quick Start Guide
      • Guardrails - Faithfulness
      • Guardrails - Safety
      • Guardrails FAQ
    • ML Observability
      • ML Monitoring - Simple
      • ML Monitoring - NLP Inputs
      • ML Monitoring - Class Imbalance
      • ML Monitoring - Model Versions
      • ML Monitoring - Ranking
      • ML Monitoring - Regression
      • ML Monitoring - Feature Impact
      • ML Monitoring - CV Inputs
  • Glossary
    • Product Concepts
      • Baseline
      • Custom Metric
      • Data Drift
      • Embedding Visualization
      • Fiddler Guardrails
      • Fiddler Trust Service
      • LLM and GenAI Observability
      • Metric
      • Model Drift
      • Model Performance
      • ML Observability
      • Trust Score
  • Product Guide
    • LLM Application Monitoring & Protection
      • LLM-Based Metrics
      • Embedding Visualizations for LLM Monitoring and Analysis
      • Selecting Enrichments
      • Enrichments (Private Preview)
      • Guardrails for Proactive Application Protection
    • Optimize Your ML Models and LLMs with Fiddler's Comprehensive Monitoring
      • Alerts
      • Package-Based Alerts (Private Preview)
      • Class Imbalanced Data
      • Enhance ML and LLM Insights with Custom Metrics
      • Data Drift: Monitor Model Performance Changes with Fiddler's Insights
      • Ensuring Data Integrity in ML Models And LLMs
      • Embedding Visualization With UMAP
      • Fiddler Query Language
      • Model Versions
      • How to Effectively Use the Monitoring Chart UI
      • Performance Tracking
      • Model Segments: Analyze Cohorts for Performance Insights and Bias Detection
      • Statistics
      • Monitoring ML Model and LLM Traffic
      • Vector Monitoring
    • Enhance Model Insights with Fiddler's Slice and Explain
      • Events Table in RCA
      • Feature Analytics Creation
      • Metric Card Creation
      • Performance Charts Creation
      • Performance Charts Visualization
    • Master AI Monitoring: Create, Customize, and Compare Dashboards
      • Creating Dashboards
      • Dashboard Interactions
      • Dashboard Utilities
    • Adding and Editing Models in the UI
      • Model Editor UI
      • Model Schema Editing Guide
    • Fairness
    • Explainability
      • Model: Artifacts, Package, Surrogate
      • Global Explainability: Visualize Feature Impact and Importance in Fiddler
      • Point Explainability
      • Flexible Model Deployment
        • On Prem Manual Flexible Model Deployment XAI
  • Technical Reference
    • Python Client API Reference
    • Python Client Guides
      • Installation and Setup
      • Model Onboarding
        • Create a Project and Onboard a Model for Observation
        • Model Task Types
        • Customizing your Model Schema
        • Specifying Custom Missing Value Representations
      • Publishing Inference Data
        • Creating a Baseline Dataset
        • Publishing Batches Of Events
        • Publishing Ranking Events
        • Streaming Live Events
        • Updating Already Published Events
        • Deleting Events From Fiddler
      • Creating and Managing Alerts
      • Explainability Examples
        • Adding a Surrogate Model
        • Uploading Model Artifacts
        • Updating Model Artifacts
        • ML Framework Examples
          • Scikit Learn
          • Tensorflow HDF5
          • Tensorflow Savedmodel
          • Xgboost
        • Model Task Examples
          • Binary Classification
          • Multiclass Classification
          • Regression
          • Uploading A Ranking Model Artifact
    • Integrations
      • Data Pipeline Integrations
        • Airflow Integration
        • BigQuery Integration
        • Integration With S3
        • Kafka Integration
        • Sagemaker Integration
        • Snowflake Integration
      • ML Platform Integrations
        • Integrate Fiddler with Databricks for Model Monitoring and Explainability
        • Datadog Integration
        • ML Flow Integration
      • Alerting Integrations
        • PagerDuty Integration
    • Comprehensive REST API Reference
      • Projects REST API Guide
      • Model REST API Guide
      • File Upload REST API Guide
      • Custom Metrics REST API Guide
      • Segments REST API Guide
      • Baselines REST API Guide
      • Jobs REST API Guide
      • Alert Rules REST API Guide
      • Environments REST API Guide
      • Explainability REST API Guide
      • Server Info REST API Guide
      • Events REST API Guide
      • Fiddler Trust Service REST API Guide
    • Fiddler Free Guardrails Documentation
  • Configuration Guide
    • Authentication & Authorization
      • Adding Users
      • Overview of Role-Based Access Control
      • Email Authentication
      • Okta Integration
      • SSO with Azure AD
      • Ping Identity SAML SSO Integration
      • Mapping LDAP Groups & Users to Fiddler Teams
    • Application Settings
    • Supported Browsers
  • History
    • Release Notes
    • Python Client History
    • Compatibility Matrix
    • Product Maturity Definitions
Powered by GitBook

© 2024 Fiddler Labs, Inc.

On this page
  • Overview
  • Definitions
  • FQL Rules
  • Data Types
  • Constants
  • Operators
  • Constant functions
  • Row-level functions
  • Aggregate functions
  • Built-in metric functions

Was this helpful?

  1. Product Guide
  2. Optimize Your ML Models and LLMs with Fiddler's Comprehensive Monitoring

Fiddler Query Language

PreviousEmbedding Visualization With UMAPNextModel Versions

Last updated 1 month ago

Was this helpful?

Overview

and are defined using the Fiddler Query Language (FQL), a flexible set of constants, operators, and functions which can accommodate a large variety of metrics.

Definitions

Term
Definition

Row-level function

A function which executes row-wise for a set of data. Returns a value for each row.

Aggregate function

A function which executes across rows. Returns a single value for a given set of rows.

FQL Rules

  • names can be referenced by name either with double quotes ("my_column") or with no quotes (my_column).

  • Single quotes (') are used to represent string values.

Data Types

FQL distinguishes between three data types:

Data type
Supported values
Examples
Supported Model Schema Data Types

Number

Any numeric value (integers and floats are both included)

10 2.34

Boolean

Only true and false

true false

String

Any value wrapped in single quotes (')

'This is a string.' '200.0'

Constants

Symbol
Description

true

Boolean constant for true expressions

false

Boolean constant for false expressions

Operators

Symbol
Description
Syntax
Returns
Examples

^

Exponentiation

Number ^ Number

Number

2.5 ^ 4 (column1 - column2)^2

-

Unary negation

-Number

Number

-column1

*

Multiplication

Number * Number

Number

2 * 10 2 * column1 column1 * column2 sum(column1) * 10

/

Division

Number / Number

Number

2 / 10 2 / column1 column1 / column2 sum(column1) / 10

%

Modulo

Number % Number

Number

2 % 10 2 % column1 column1 % column2 sum(column1) % 10

+

Addition

Number + Number

Number

2 + 2 2 + column1 column1 + column2 average(column1) + 2

-

Subtraction

Number - Number

Number

2 - 2 2 - column1 column1 - column2 average(column1) - 2

<

Less than

Number < Number

Boolean

10 < 20 column1 < 10 column1 < column2 average(column2) < 5

<=

Less than or equal to

Number <= Number

Boolean

10 <= 20 column1 <= 10 column1 <= column2 average(column2) <= 5

>

Greater than

Number > Number

Boolean

10 > 20 column1 > 10 column1 > column2 average(column2) > 5

>=

Greater than or equal to

Number >= Number

Boolean

10 >= 20 column1 >= 10 column1 >= column2 average(column2) >= 5

==

Equals

Number == Number

Boolean

10 == 20 column1 == 10 column1 == column2 average(column2) == 5

!=

Does not equal

Number != Number

Boolean

10 != 20 column1 != 10 column1 != column2 average(column2) != 5

not

Logical NOT

not Boolean

Boolean

not true not column1

and

Logical AND

Boolean and Boolean

Boolean

true and false column1 and column2

or

Logical OR

Boolean or Boolean

Boolean

true or false column1 or column2

Constant functions

Symbol
Description
Syntax
Returns
Examples

e()

Base of the natural logarithm

e()

Number

e() == 2.718281828459045

pi()

The ratio of a circle's circumference to its diameter

pi()

Number

pi() == 3.141592653589793

Row-level functions

Row-level functions can be applied either to a single value or to a column/row expression (in which case they are mapped element-wise to each value in the column/row expression).

Symbol
Description
Syntax
Returns
Examples

if(condition, value1, value2)

Evaluates condition and returns value1 if true, otherwise returns value2. value1 and value2 must have the same type.

if(Boolean, Any, Any)

Any

if(false, 'yes', 'no') == 'no' if(column1 == 1, 'yes', 'no')

length(x)

Returns the length of string x.

length(String)

Number

length('Hello world') == 11

to_string(x)

Converts a value x to a string.

to_string(Any)

String

to_string(42) == '42' to_string(true) == 'true'

startswith(str, prefix)

Returns true if str starts with prefix.

startswith(String, String)

Boolean

startswith('abcde', 'abc')

substring(str, offset, length)

Returns a substring of str of length length from offset offset. The first character has an offset of 1.

substring(String, Number, Number)

String

substring('abcde', 2, 3) == 'bcd'

match(str, regex)

match(String, String)

Boolean

match('abcde', 'a.c.*e')

is_null(x)

Returns true if x is null, otherwise returns false.

is_null(Any)

Boolean

is_null('') == true is_null("column1")

is_not_null(x)

Returns true if x is not null, otherwise returns false.

is_null(Any)

Boolean

is_not_null('') == false `is_not_null("column1")

abs(x)

Returns the absolute value of number x.

abs(Number)

Number

abs(-3) == 3

exp(x)

Returns e^x, where e is the base of the natural logarithm.

exp(Number)

Number

exp(1) == 2.718281828459045

log(x)

Returns the natural logarithm (base e) of number x.

log(Number)

Number

log(e) == 1

log2(x)

Returns the binary logarithm (base 2) of number x.

log2(Number)

Number

log2(16) == 4

log10(x)

Returns the binary logarithm (base 10) of number x.

log10(Number)

Number

log10(1000) == 3

sqrt(x)

Returns the positive square root of number x.

sqrt(Number)

Number

sqrt(144) == 12

Aggregate functions

Every Custom Metric must be wrapped in an aggregate function or be a combination of aggregate functions.

Symbol
Description
Syntax
Returns
Examples

sum(x)

Returns the sum of a numeric column or row expression x.

sum(Number)

Number

sum(column1 + column2)

average(x)

Returns the arithmetic mean/average value of a numeric column or row expression x.

average(Number)

Number

average(2 * column1)

count(x)

Returns the number of non-null rows of a column or row expression x.

count(Any)

Number

count(column1)

Built-in metric functions

Symbol
Description
Syntax
Returns
Examples

jsd(column, baseline)

The Jensen-Shannon distance of column column with respect to baseline baseline.

jsd(Any, String)

Number

jsd(column1, 'my_baseline')

psi(column, baseline)

The population stability index of column column with respect to baseline baseline.

psi(Any, String)

Number

psi(column1, 'my_baseline')

null_violation_count(column)

Number of rows with null values in column column.

null_violation_count(Any)

Number

null_violation_count(column1)

range_violation_count(column)

Number of rows with out-of-range values in column column.

range_violation_count(Any)

Number

range_violation_count(column1)

type_violation_count(column)

Number of rows with invalid data types in column column.

type_violation_count(Any)

Number

type_violation_count(column1)

any_violation_count(column)

Number of rows with at least one Data Integrity violation in column.

any_violation_count(Any)

Number

any_violation_count(column1)

traffic()

Total row count. Includes null rows.

traffic()

Number

traffic()

tp(class)

True positive count. Available for binary classification and multiclass classification models. For multiclass, class is used to specify the positive class.

tp(class=Optional[String])

Number

tp() tp(class='class1')

tn(class)

True negative count. Available for binary classification and multiclass classification models. For multiclass, class is used to specify the positive class.

tn(class=Optional[String])

Number

tn() tn(class='class1')

fp(class)

False positive count. Available for binary classification and multiclass classification models. For multiclass, class is used to specify the positive class.

fp(class=Optional[String])

Number

fp() fp(class='class1')

fn(class)

False negative count. Available for binary classification and multiclass classification models. For multiclass, class is used to specify the positive class.

fn(class=Optional[String])

Number

fn() fn(class='class1')

precision(target, threshold)

Precision between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

precision(target=Optional[Any], threshold=Optional[Number])

Number

precision() precision(target=column1) precision(threshold=0.5) precision(target=column1, threshold=0.5)

recall(target, threshold)

Recall between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

recall(target=Optional[Any], threshold=Optional[Number])

Number

recall() recall(target=column1) recall(threshold=0.5) recall(target=column1, threshold=0.5)

f1_score(target, threshold)

F1 score between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

f1_score(target=Optional[Any], threshold=Optional[Number])

Number

f1_score() f1_score(target=column1) f1_score(threshold=0.5) f1_score(target=column1, threshold=0.5)

fpr(target, threshold)

False positive rate between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

fpr(target=Optional[Any], threshold=Optional[Number])

Number

fpr() fpr(target=column1) fpr(threshold=0.5) fpr(target=column1, threshold=0.5)

auroc(target)

Area under the ROC curve between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

auroc(target=Optional[Any])

Number

auroc() auroc(target=column1)

geometric_mean(target, threshold)

Geometric mean score between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

geometric_mean(target=Optional[Any], threshold=Optional[Number])

Number

geometric_mean() geometric_mean(target=column1) geometric_mean(threshold=0.5) geometric_mean(target=column1, threshold=0.5)

expected_calibration_error(target)

Expected calibration error between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

expected_calibration_error(target=Optional[Any])

Number

expected_calibration_error() expected_calibration_error(target=column1)

log_loss(target)

Log loss (binary cross entropy) between target and output. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

log_loss(target=Optional[Any])

Number

log_loss() log_loss(target=column1)

calibrated_threshold(target)

Optimal threshold value for a high TPR and a low FPR. Available for binary classification model tasks. If target is specified, it will be used in place of the default target column.

calibrated_threshold(target=Optional[Any])

Number

calibrated_threshold() calibrated_threshold(target=column1)

accuracy(target, threshold)

Accuracy score between target and outputs. Available for multiclass classification model tasks. If target is specified, it will be used in place of the default target column.

accuracy(target=Optional[Any], threshold=Optional[Number])

Number

accuracy() accuracy(target=column1) accuracy(threshold=0.5) accuracy(target=column1, threshold=0.5)

log_loss(target)

Log loss score between target and outputs. Available for multiclass classification model tasks. If target is specified, it will be used in place of the default target column.

log_loss(target=Optional[Any])

Number

log_loss() log_loss(target=column1)

r2(target)

R-squared score between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.

r2(target=Optional[Any])

Number

r2() r2(target=column1)

mse(target)

Mean squared error between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.

mse(target=Optional[Any])

Number

mse() mse(target=column1)

mae(target)

Mean absolute error between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.

mae(target=Optional[Any])

Number

mae() mae(target=column1)

mape(target)

Mean absolute percentage error between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.

mape(target=Optional[Any])

Number

mape() mape(target=column1)

wmape(target)

Weighted mean absolute percentage error between target and output. Available for regression model tasks. If target is specified, it will be used in place of the default target column.

wmape(target=Optional[Any])

Number

wmape() wmape(target=column1)

map(target)

Mean average precision score. Available for ranking model tasks. If target is specified, it will be used in place of the default target column.

map(target=Optional[Any])

Number

map() map(target=column1)

ndcg_mean(target)

Mean normalized discounted cumulative gain score. Available for ranking model tasks. If target is specified, it will be used in place of the default target column.

ndcg_mean(target=Optional[Any])

Number

ndcg_mean() ndcg_mean(target=column1)

query_count(target)

Count of ranking queries. Available for ranking model tasks. If target is specified, it will be used in place of the default target column.

query_count(target=Optional[Any])

Number

query_count() query_count(target=column1)

Returns true if str matches the pattern regex in .

re2 regular syntax
Custom Metrics
Segments

Need help? Contact us at .

💡
help@fiddler.ai
Column
DataType.INTEGER
DataType.FLOAT
DataType.BOOLEAN
DataType.CATEGORY
DataType.STRING