LLM Based Metrics

LLM-based metrics use large language models to evaluate the quality of text generated by AI. This approach is much closer to how humans judge text, making these metrics particularly useful for evaluating AI-generated content for use cases such as chatbots, writing assistants, or content creation tools.

LLM-based metrics can adapt to different topics and types of text because LLMS have been trained on a wide range of information, making them a valuable tool for developers and researchers looking to enhance the quality of AI-generated text.

Currently, Fiddler supports two types of LLM-based metrics - OpenAI-based metrics and Fiddler Fast Trust Model metrics.

OpenAI-based metrics

• These metrics are generated through the OpenAI API, which may introduce latency due to network communication and processing time.

• OpenAI API access token MUST BE provided by the user, which will be configured during onboarding.

• The specific model to be used for these metrics will also be chosen during onboarding.

Currently, the below metrics are OpenAI-based:

• Answer Relevance

• Faithfulness

• Coherence

• Conciseness

Fiddler Fast Trust metrics

• These metrics are generated through Fiddler's in-house, purpose-built LLMs.

• These metrics can be generated in airgapped environments and do not rely on any over-the-network connection to generate such scores.

Currently, the below metrics are Fiddler Fast Trust Model-based:

• Fast Safety

• Fast Jailbreak

• Fast Faithfulness