Baseline datasets are used for making comparisons with production data.
A baseline dataset should be sampled from your model's training set, so it can serve as a representation of what the model expects to see in production.
A Model is a representation of your machine learning model which can be used for monitoring, explainability, and more. You do not need to upload your model artifact in order to onboard your model, but doing so will significantly improve the quality of explanations generated by Fiddler.
Model
Model object contains the below parameters.
constructor()
Initialize a new model instance.
Parameters
from_data()
Build model instance from the given dataframe or file(csv/parquet).
from_data will not create a model entry on Fiddler Platform.
Instead this method only returns a model instance which can be edited, call .create() to onboard the model to
Fiddler Platform.
spec is optional to from_data method. However, a spec with at least inputs is required for model onboarding.
Make sure spec is passed to from_data method if model requires custom features. This method generates centroids
which are needed for custom feature drift computation
If version is not explicitly passed, Fiddler Platform will treat it as v1 version of the model.
Since Model contains a lot of information, list operations does not return all the fields of a model.
Instead this method returns ModelCompact objects on which .fetch() can be called to get the complete Model
instance.
For most of the use-cases, ModelCompact objects are sufficient.
update()
Update an existing model. Only following fields are allowed to be updated, backend will ignore if any other
field is updated on the instance.
Parameters
version parameter is available from fiddler-client==3.1 onwards
Duplicate the model instance with the given version name.
This call will not save the model on Fiddler Platform. After making changes to the model instance, call .create() to add the model version to Fiddler Platform.
This method is available from fiddler-client==3.1 onwards.
Model deletion is an async process, hence a job object is returned on delete() call.
Call job.wait() to wait for the job to complete. If you are planning to create a model with the same
name, please wait for the job to complete, otherwise backend will not allow new model with same name.
# Before publishing, make sure you set up the necessary fields of the model(if any).# If you set the fields to non-empty value, We expect them passed in the source.model.event_ts_col ='timestamp'model.event_id_col ='event_id'model.update()
Publish dataset (pre-production data) from file
# Publish FileFILE_PATH ='PATH_TO_DATASET_FILE'job = model.publish( source=FILE_PATH, environment=fdl.EnvType.PRE_PRODUCTION, dataset_name='training_dataset')# The publish() method is asynchronous by default. Use the publish job's wait() method # if sychronous behavior is desired.# job.wait()
Publish dataset (pre-production data) from dataframe
df = pd.DataFrame(np.random.randint(0, 100, size=(10, 4)), columns=list('ABCD'))job = model.publish( source=df, environment=fdl.EnvType.PRE_PRODUCTION, dataset_name='training_dataset')# The publish() method is asynchronous by default. Use the publish job's wait() method # if sychronous behavior is desired.# job.wait()
Publish production events from list
List is only supported for production data but not for pre-production.
Events are published as a stream. This mode is recommended If you have a high volume of continuous real-time traffic of events, as it allows for more efficient processing on our backend.
It returns a list of event_id for each of the published events.
# Publish list of dictionary objectsevents = [{'A':56,'B':68,'C':67,'D':27,'event_id':'A1','timestamp':'2024-05-01 00:00:00'},{'A':43,'B':59,'C':64,'D':18,'event_id':'A2','timestamp':'2024-05-01 00:00:00'}, ...]event_ids = model.publish( source=events, environment=fdl.EnvType.PRODUCTION)
Notes
In this example where model.event_id_col=event_id, we expect event_id as the required key of the dictionary. Otherwise if you keep model.event_id_col=None, our backend will generate unique event ids and return these back to you. Same for model.event_ts_col, we assign current time as event timestamp in case of None.
Publish production events from file
Batch events is faster if you want to publish a large-scale set of historical data.
# Publish FileFILE_PATH ='PATH_TO_EVENTS_FILE'job = model.publish( source=FILE_PATH, environment=fdl.EnvType.PRODUCTION,)# The publish() method is asynchronous by default. Use the publish job's wait() method # if sychronous behavior is desired.# job.wait()
if you need to update the target or metadata columns for a previously published production event, set update=True. For more details please refer to Updating Events. Note only production events can be updated.
Update production events from list
events_update = [{'A': [0],# suppose 'A' is the target'B': [0],# suppose 'B' is the medadata'event_id': ['A1'],# required model.event_id_col},{'A': [1],# suppose 'A' is the target'B': [1],# suppose 'B' is the medadata'event_id': ['A2'],# required model.event_id_col},]event_ids = model.publish( source=events_update, environment=fdl.EnvType.PRODUCTION, update=True,)
Update production events from dataframe
df_update = pd.DataFrame(
{
'A': [0, 1], # suppose 'A' is the target
'B': [0, 1], # suppose 'B' is the medadata
'event_id': ['A1', 'A2'], # required model.event_id_col
}
)
event_ids = model.publish(
source=df_update,
environment=fdl.EnvType.PRODUCTION,
update=True,
)
Returns
In case of streaming publish
In case of batch publish
Model Compact
Model object contains the below parameters.
fetch()
Fetch the model instance from Fiddler Platform.
Parameters
No
Returns
Raises
Model deployment
Get model deployment object of a particular model.
Model deployment:
Model deployment object contains the below parameters.
Update model deployment
Update an existing model deployment.
Parameters
Usage
# Update CPU allocation and activate the model pod
model_id = 'a920ddb6-edb7-473b-a5f7-035f91e1d53a'
model = fdl.Model.get(model_id)
model_deployment = model.deployment
model_deployment.cpu = 300
model_deployment.active = True
model_deployment.update()
Returns
No
Raises
Organizations
Organization in which all the projects, models are present.
Organization:
Organization object contains the below parameters.
Projects
Projects are used to organize your models and datasets. Each project can represent a machine learning task (e.g. predicting house prices, assessing creditworthiness, or detecting fraud).
A project can contain one or more models (e.g. lin_reg_house_predict, random_forest_house_predict).
WEBHOOK_NAME = "YOUR_WEBHOOK_NAME"
webhook_list = fdl.Webhook.list()
webhook_instance = None
for webhook in webhook_list:
if WEBHOOK_NAME == webhook.name:
webhook_instance = webhook
webhook_instance.name = 'NEW_WEBHOOK_NAME'
webhook_instance.update()
Returns
None
Raises
delete()
Delete a webhook.
Parameters
Usage
WEBHOOK_NAME = "YOUR_WEBHOOK_NAME"
webhook_instance = None
for webhook in webhook_list:
if WEBHOOK_NAME == webhook.name:
webhook_instance = webhook
webhook_instance.delete()
Returns
None
Explainability
Explainability methods for models.
precompute_feature_importance
Pre-compute feature importance for a model on a dataset. This is used in various places in the UI.
A single feature importance can be precomputed (computed and cached) for a model.
Pre-compute feature impact for a model on a dataset. This is used in various places in the UI.
A single feature impact can be precomputed (computed and cached) for a model.
Upload a custom feature impact for a model of input type TABULAR. All input features need to be passed for the method to run successfully. Partial upload of feature impacts are not supported.
Fetch data with slice query. 1M rows is the max size that can be fetched.
Parameters
Usage
MODEL_NAME = 'YOUR_MODEL_NAME'
DATASET_NAME = 'YOUR_DATASET_NAME'
model.get_slice(
query=f"SELECT * FROM {DATASET_NAME}.{MODEL_NAME} WHERE geography='France' order by balance desc LIMIT 3",
columns=['Age'],
max_rows=2,
)
Returns
Raises
📘 Info
Only read-only SQL operations are supported. Certain SQL operations like aggregations and joins might not result in a valid slice.
download_slice()
Download data with slice query to parquet file. 10M rows is the max size that can be downloaded.
Parameters
Usage
MODEL_NAME = 'YOUR_MODEL_NAME'
DATASET_NAME = 'YOUR_DATASET_NAME'
model.download_slice(
output_dir=parquet_output,
query=f'SELECT * FROM {DATASET_NAME}.{MODEL_NAME} WHERE age>=20 order by balance desc',
columns=['Age'],
)
Returns
Parquet file with slice query contents downloaded to the Path mentioned in output_dir.
The Mutual information measures the dependency between two random variables. It's a non-negative value. If two random variables are independent MI is equal to zero. Higher MI values means higher dependency.
Enrichments are disabled by default. To enable them, contact your administrator. Failing to do so will result in an error during the add_model call.
Embedding (Private Preview)
Supported Models:
fiddler_custom_features = [
fdl.Enrichment(
name='Question Embedding', # name of the enrichment, will be the vector col
enrichment='embedding',
columns=['question'], # only one allowed per embedding enrichment, must be a text column in dataframe
config={ # optional
'model_name': ... # default: 'thenlper/gte-base'
'pooling_method': ... # choose from '{first/last/mean}_token'. Only required if NOT using a sentence transformer
}
),
fdl.TextEmbedding(
name='question_cf', # name of the text embedding custom feature
source_column='question', # source - raw text
column='Question Embedding', # the name of the vector - outpiut of the embedding enrichment
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
\
The above example will lead to generation of new column:
\
Note
In the context of Hugging Face models, particularly transformer-based models used for generating embeddings, the pooling_method determines how the model processes the output of its layers to produce a single vector representation for input sequences (like sentences or documents). This is crucial when using these models for tasks like sentence or document embedding, where you need a fixed-size vector representation regardless of the input length.
The above example will lead to generation of new column:
Note
Does not calculate membership for preproduction data, so you cannot calculate drift. Centroid Distance is automatically added if the TextEmbedding enrichment is created for any given model.
Personally Identifiable Information (Private Preview)
List of PII entities
fiddler_custom_features = [
fdl.Enrichment(
name='Rag PII',
enrichment='pii',
columns=['question'], # one or more columns
allow_list=['fiddler'], # Optional: list of strings that are white listed
score_threshold=0.85, # Optional: float value for minimum possible confidence
),
]
model_spec = fdl.ModelSpec(
inputs=['question'],
custom_features=fiddler_custom_features,
)
The above example will lead to generation of new columns:
The above example leads to creation of two columns -
Fast Safety (Private Preview)
The Fast safety enrichment evaluates the safety of the text along ten different dimensions: illegal, hateful, harassing, racist, sexist, violent, sexual, harmful, unethical, jailbreaking. These dimensions are all returned by default, but can be selectively chosen as needed. Fast safety is generated through the Fast Trust Models.
The above example leads to creation of a column for each dimension. -
Fast Faithfulness (Private Preview)
The Fast faithfulness enrichment is designed to evaluate the accuracy and reliability of facts presented in AI-generated text responses. Fast safety is generated through the Fast Trust Models.
List of 1 or more column names for the rule to evaluate. Use ['__ANY__'] to evaluate all columns.
baseline_id
Optional[UUID]
None
UUID of the baseline for the alert.
segment_id
Optional[UUID]
None
UUID of segment for the alert
compare_bin_delta
Optional[int]
None
Indicates previous period for comparison e.g. for fdl.BinSize.DAY, compare_bin_delta=1 will compare 1 day back, compare_bin_delta=7 will compare 7 days back.
warning_threshold
Optional[float]
None
Warning alert is triggered when this value satisfies the condition to the selected metric_id.
created_at
datetime
-
The creation timestamp.
updated_at
datetime
-
The timestampe of most recent update.
evaluation_delay
int
0
Specifies a delay in hours before AlertRule is evaluated. The delay period must not exceed one year(8760 hours).
name
str
-
Unique name of the model
model_id
UUID
-
Details of the model.
metric_id
Union[str, UUID]
-
Type of alert metric UUID or enum.
columns
Optional[List[str]]
None
List of column names on which AlertRule is to be created. It can take ['__ANY__'] to check for all columns.
Source can be:
1. Path or str path: path for data file.
2. list[dict]: list of event dicts. EnvType.PRE_PRODUCTION not supported.
3. dataframe: events dataframe.
environment
EnvType
EnvType.PRODUCTION
Either EnvType.PRE_PRODUCTION or EnvType.PRODUCTION
dataset_name
Optional[str]
None
Name of the dataset. Not supported for EnvType.PRODUCTION
update
Optional[bool]
False
If True, the events data passed in the publish call will be used to update previously published event records matched by their event_ids (note that only updating target and metadata columns is supported). For more details refer to Updating Events
DataSource for the input dataset to compute feature importance on (DatasetDataSource or SqlSliceQueryDataSource).
num_iterations
Optional[int]
None
The maximum number of ablated model inferences per feature.
num_refs
Optional[int]
None
The number of reference points used in the explanation.
ci_level
Optional[float]
None
The confidence level (between 0 and 1).
Tuple
A named tuple with the feature importance results .
BadRequest
If dataset id is not specified.
dataset_id
UUID
-
The unique identifier of the dataset.
num_samples
Optional[int]
None
The number of samples used.
num_iterations
Optional[int]
None
The maximum number of ablated model inferences per feature.
num_refs
Optional[int]
None
The number of reference points used in the explanation.
ci_level
Optional[float]
None
The confidence level (between 0 and 1).
min_support
Optional[int]
15
Only used for NLP (TEXT inputs) models. Specify a minimum support (number of times a specific word was present in the sample data) to retrieve top words. Default to 15.
update
Optional[bool]
False
Flag to indicate whether the precomputed feature impact should be recomputed and updated.
DataSource for the input dataset to compute feature importance on (DatasetDataSource or SqlSliceQueryDataSource).
num_iterations
Optional[int]
None
The maximum number of ablated model inferences per feature.
num_refs
Optional[int]
None
The number of reference points used in the explanation.
ci_level
Optional[float]
None
The confidence level (between 0 and 1).
min_support
Optional[int]
15
Only used for NLP (TEXT inputs) models. Specify a minimum support (number of times a specific word was present in the sample data)to retrieve top words. Default to 15.
output_columns
Optional[list[str]]
None
Only used for NLP (TEXT inputs) models. Output column names to compute feature impact on.
Tuple
A named tuple with the feature impact results .
BadRequest
If dataset id is not specified or query is not valid.
dataset_id
UUID
-
Unique identifier of the dataset used for prediction.
chunk_size
Optional[int]
None
Chunk size for fetching predictions.
update
Optional[bool]
False
Flag to indicate whether the pre-computed predictions should be re-computed and updated for this dataset.
DataSource for the reference data to compute explanationon (DatasetDataSource, SqlSliceQueryDataSource).
Only used for non-text models and the following methods:
'SHAP', 'FIDDLER_SHAP', 'PERMUTE', 'MEAN_RESET'.
Explanation method name. Could be your custom explanation method or one of the following method:
'SHAP', 'FIDDLER_SHAP', 'IG', 'PERMUTE', 'MEAN_RESET', 'ZERO_RESET'.
num_permutations
Optional[int]
None
For Fiddler SHAP, that corresponds to the number of coalitions to sample to estimate the Shapley values of each single-reference game.
For the permutation algorithms, this corresponds to the number of permutations from the dataset to use for the computation.
ci_level
Optional[float]
None
The confidence level (between 0 and 1) to use for the confidence intervals in Fiddler SHAP. Not used for other methods.
top_n_class
Optional[int]
None
For multiclass classification models only, specifying if only the n top classes are computed or all classes (when parameter is None).
Tuple
A named tuple with the explanation results.
NotSupported
If specified source type is not supported.
query
str
-
An SQL query that begins with the keyword 'SELECT'.
sample
Optional[bool]
False
Whether rows should be sample or not from the database.
max_rows
Optional[int]
None
Number of maximum rows to fetch.
columns
Optional[list[str]]
None
Allows caller to explicitly specify list of columns to select overriding columns selected in the query.
Dataframe
A pandas DataFrame containing the slice returned by the query.
BadRequest
If given query is wrong.
output_dir
Union[Path, str]
-
Path to download the file.
query
str
-
An SQL query that begins with the keyword 'SELECT'.
sample
Optional[bool]
False
Allows caller to explicitly specify list of columns to select overriding columns selected in the query.
max_rows
Optional[int]
None
Number of maximum rows to fetch.
columns
Optional[list[str]]
None
Whether rows should be sample or not from the database.
BadRequest
If given query is wrong.
df
pd.DataFrame
None
Feature dataframe.
chunk_size
Optional[int]
None
Chunk size for fetching predictions.
Dataframe
A pandas DataFrame of the predictions.
query
str
-
Slice query to compute Mutual information on.
column_name
str
-
Column name to compute mutual information with respect to all the variables in the dataset.
num_samples
Optional[int]
None
Number of samples to select for computation.
normalized
Optional[bool]
False
If set to True, it will compute Normalized Mutual Information.
Dictionary
Contains mutual information w.r.t the given feature for each column given.
BadRequest
If given query is wrong.
ModelInputType.TABULAR
For tabular models.
ModelInputType.TEXT
For text models.
ModelInputType.MIXED
For models which can be a mixture of text and tabular.
ModelTask.REGRESSION
For regression models.
ModelTask.BINARY_CLASSIFICATION
For binary classification models.
ModelTask.MULTICLASS_CLASSIFICATION
For multiclass classification models.
ModelTask.RANKING
For ranking classification models.
ModelTask.LLM
For LLM models.
ModelTask.NOT_SET
For other model tasks or no model task specified.
DataType.FLOAT
For floats.
DataType.INTEGER
For integers.
DataType.BOOLEAN
For booleans.
DataType.STRING
For strings.
DataType.CATEGORY
For categorical types.
DataType.TIMESTAMP
For timestamps.
DataType.VECTOR
For vector types
CustomFeatureType.FROM_COLUMNS
Represents custom features derived directly from columns.
CustomFeatureType.FROM_VECTOR
Represents custom features derived from a vector column.
CustomFeatureType.FROM_TEXT_EMBEDDING
Represents custom features derived from text embeddings.
CustomFeatureType.FROM_IMAGE_EMBEDDING
Represents custom features derived from image embeddings.
CustomFeatureType.ENRICHMENT
Represents custom features derived from an enrichment.
ArtifactType.SURROGATE
For surrogates.
ArtifactType.PYTHON_PACKAGE
For python package.
DeploymentType.BASE_CONTAINER
For base containers.
DeploymentType.MANUAL
For manual deployment.
EnvType.PRODUCTION
For production events.
EnvType.PRE_PRODUCTION
For pre production events.
BaselineType.STATIC
For static production baseline.
BaselineType.ROLLING
For rolling production baseline.
WindowBinSize.HOUR
For rolling window to be 1 hour.
WindowBinSize.DAY
For rolling window to be 1 day.
WindowBinSize.WEEK
For rolling window to be 1 week.
WindowBinSize.MONTH
For rolling window to be 1 month.
WebhookProvider.SLACK
For slack.
WebhookProvider.OTHER
For any other app.
AlertCondition.GREATER
The greater than operator.
AlertCondition.LESSER
the less than operator.
BinSize.HOUR
The 1 hour bin.
BinSize.DAY
the 1 day bin.
BinSize.WEEK
The 7 day bin.
BinSize.MONTH
The 30 day bin.
CompareTo.RAW_VALUE
For an absolute comparison of a specified value to the alert metric
CompareTo.TIME_PERIOD
For a relative comparison of the alert metric to the same metric from a previous time period.
Priority.LOW
The low priority label.
Priority.MEDIUM
The medium priority label.
Priority.HIGH
The high priority label.
Severity.DEFAULT
For AlertRule when none of the thresholds have passed.
Severity.WARNING
For AlertRule when alert crossed the warning_threshold but not the critical_threshold.
Severity.CRITICAL
For AlertRule when alert crossed the critical_raw_threshold.
A Crypto wallet number. Currently only Bitcoin address is supported
Pattern match, context and checksum
1BoatSLRHtKNngkdXEeobR76b53LETtpyT
DATE_TIME
Absolute or relative dates or periods or times smaller than a day.
Pattern match and context
../2024
EMAIL_ADDRESS
An email address identifies an email box to which email messages are delivered
Pattern match, context and RFC-822 validation
trust@fiddler.ai
IBAN_CODE
The International Bank Account Number (IBAN) is an internationally agreed system of identifying bank accounts across national borders to facilitate the communication and processing of cross border transactions with a reduced risk of transcription errors.
Pattern match, context and checksum
DE89 3704 0044 0532 0130 00
IP_ADDRESS
An Internet Protocol (IP) address (either IPv4 or IPv6).
US Individual Taxpayer Identification Number (ITIN). Nine digits that start with a "9" and contain a "7" or "8" as the 4 digit.
Pattern match and context
912-34-1234
US_PASSPORT
A US passport number begins with a letter, followed by eight numbers
Pattern match and context
L12345678
FDL Rag PII (question)
bool
Whether any PII was detected.
FDL Rag PII (question) Matches
str
What matches in raw text were flagged as potential PII (ex. ‘Douglas MacArthur,Korean’)?
FDL Rag PII (question) Entities
str
What entites these matches were tagged as (ex. 'PERSON')?
bleu
Measures precision of word n-grams between generated and reference texts
Simple, fast, widely used
Ignores recall, meaning, and word order
rouge
Measures recall of word n-grams and longest common sequences
Captures more information than BLEU
Still relies on word matching, not semantic similarity
meteor
Incorporates recall, precision, and additional semantic matching based on stems and paraphrasing
More robust and flexible than BLEU and ROUGE
Requires linguistic resources and alignment algorithms
FDL QA Evaluate (bleu)
float
FDL QA Evaluate (rouge1)
float
FDL QA Evaluate (rouge2)
float
FDL QA Evaluate (rougel)
float
FDL QA Evaluate (rougelsum)
float
FDL QA Evaluate (meteor)
floa
char_count
Total number of characters in text, including everything.
Assessing text length, useful for platforms with character limits.
letter_count
Total number of letters only, excluding numbers, punctuation, spaces.
Gauging text complexity, used in readability formulas.
miniword_count
Count of small words (usually 1-3 letters).
Specific readability analyses, especially for simplistic texts.
words_per_sentence
Average number of words in each sentence.
Understanding sentence complexity and structure.
polysyllabcount
Number of words with more than three syllables.
Analyzing text complexity, used in some readability scores.
lexicon_count
Total number of words in the text.
General text analysis, assessing overall word count.
syllable_count
Total number of syllables in the text.
Used in readability formulas, measures text complexity.
sentence_count
Total number of sentences in the text.
Analyzing text structure, used in readability scores.
flesch_reading_ease
Readability score indicating how easy a text is to read (higher scores = easier).
Assessing readability for a general audience.
smog_index
Measures years of education needed to understand a text.
Evaluating text complexity, especially for higher education texts.
flesch_kincaid_grade
Grade level associated with the complexity of the text.
Educational settings, determining appropriate grade level for texts.
coleman_liau_index
Grade level needed to understand the text based on sentence length and letter count.
Assessing readability for educational purposes.
automated_readability_index
Estimates the grade level needed to comprehend the text.
Evaluating text difficulty for educational materials.
dale_chall_readability_score
Assesses text difficulty based on a list of familiar words for average American readers.
Determining text suitability for average readers.
difficult_words
Number of words not on a list of commonly understood words.
Analyzing text difficulty, especially for non-native speakers.
linsear_write_formula
Readability formula estimating grade level of text based on sentence length and easy word count.
Simplifying texts, especially for lower reading levels.
gunning_fog
Estimates the years of formal education needed to understand the text.
Assessing text complexity, often for business or professional texts.
long_word_count
Number of words longer than a certain length (often 6 or 7 letters).
Evaluating complexity and sophistication of language used.
monosyllabcount
Count of words with only one syllable.
Readability assessments, particularly for simpler texts.
FDL Text Statistics (question) char_count
int
Character count of string in questioncolumn.
FDL Text Statistics (question) dale_chall_readability_score
float
Readability score of string in questioncolumn.
FDL Question Sentiment (question) compound
float
Raw score of sentiment.
FDL Question Sentiment (question) sentiment
string
One of positive, negative and `neutral.
FDL Profanity (prompt) contains_profanity
bool
To indicate if input contains profanity in the value of the prompt column.
FDL Profanity (response) contains_profanity
bool
To indicate if input contains profanity in the value of the response column.
FDL Answer Relevance
bool
Binary metric, which is True if response is relevant to the prompt.
FDL Faithfulness
bool
Binary metric, which is True if the facts used inresponse is correctly used from the context columns.
FDL Coherence
bool
Binary metric, which is True ifresponse makes coherent arguments which flow well.
FDL Conciseness
Binary metric, which is True ifresponse is concise, and not overly verbose.
Toxic-Chat
0.4
0.64
0.24
FDL Toxicity (prompt) toxicity_prob
float
Model prediction probability between 0-1.
FDL Toxicity (prompt) contains_toxicity
bool
Model prediction either 0 or 1.
FDL Regex - only digits
category
Match or No Match, depending on the regex specified in config matching in the string.
FDL Topics (response) topic_model_scores
list[float]
Indicating probability of the given column in each of the topics specified in the Enrichment config. Each float value indicate probability of the given input classified in the corresponding topic, in the same order as topics. Each value will be between 0 and 1. The sum of values does not equal to 1, as each classification is performed independently of other topics.
FDL Topics (response) max_score_topic
string
Topic with the maximum score from the list of topic names specified in the Enrichment config.
FDL Banned KW (prompt) contains_banned_kw
bool
To indicate if input contains one of the specified banned keywords in the value of the prompt column.
FDL Banned KW (response) contains_banned_kw
bool
To indicate if input contains one of the specified banned keywords in the value of the response column.
FDL Language (prompt) language
string
Language prediction for input text
FDL Language (prompt) language_probability
float
To indicate the confidence probabillity of language prediction
FDL Prompt Safety (prompt) dimension
bool
Binary metric, which is True if the input is deemed unsafe, False otherwise
FDL Prompt Safety (prompt) dimension score
float
To indicate the confidence probabillity of safety prediction
FDL Faithfulness faithful
bool
Binary metric, which is True if the facts used inresponse is correctly used from the context columns.
FDL Faithfulness faithful score
float
To indicate the confidence probabillity of faithfulness prediction
JSON Validator valid
bool
String is valid JSON.
JSON Validator errors
str
If the string failed to parse to JSON any parsing errors will be returned as a serialized json list of dictionaries,
binary_classification_threshold
float
None
Threshold for labels.
target_class_order
list
None
Order of target classes.
group_by
str
None
Query/session id column for ranking models.
top_k
int
None
Top k results to consider when computing ranking metrics.
class_weights
list[float]
None
Weight of each class.
weighted_ref_histograms
bool
None
Whether baseline histograms must be weighted or not when calculating drift metrics.
