LLM Monitoring

Monitoring LLM Applications with Fiddler

Monitoring Large Language Model (LLM) applications with Fiddler requires publishing the LLM application's inputs and outputs, including prompts, prompt context, responses, and the source documents retrieved (for RAG-based applications). Fiddler will then generate enrichments, which are LLM trust and safety metrics, for alerting, analysis, or debugging purposes.

Fiddler is a pioneer in the AI Trust domain and, as such, offers the most extensive set of AI safety and trust metrics available today.

Selecting Enrichments to Enhance Monitoring

Fiddler offers many enrichments that each measure different aspects of an LLM application. For detailed information about which enrichment to select for any specific use case, visit this page. Some enrichments use Fast Trust Models to generate these scores.

Generating Enrichments with Fiddler

LLM application owners must specify the enrichments to be generated by Fiddler during model onboarding. The enrichment pipeline then generates enrichments for the LLM application's inputs and outputs as events are published to Fiddler.

Fiddler Enrichment Framework diagram displaying sample inputs and outputs flowing into the Fiddler enrichment pipeline.

Figure 1. The Fiddler Enrichment Framework

After the LLM application's raw, unstructured inputs and outputs are published to Fiddler, the enrichment framework augments them with various AI trust and safety metrics. These metrics can monitor the application's overall health and alert users to any performance degradation.

Fiddler dashboard showing LLM application performance using enrichment metrics.

Figure 2. A Fiddler dashboard showing LLM application performance

Using the metrics produced by the enrichment framework, users can monitor LLM application performance over time and conduct root-cause analysis when problematic trends are identified.

During model onboarding, application owners can opt in to the various, ever-expanding Fiddler enrichments by specifying Enrichments as custom features in the Fiddler ModelSpec object.