Embedding Visualizations for LLM Monitoring

Introduction to embedding visualization

Embedding visualization is a powerful technique used to understand and interpret complex relationships in high-dimensional data. Reducing the dimensionality of custom features into a 2D or 3D space makes it easier to identify patterns, clusters, and outliers.

In Fiddler, high-dimensional data like embeddings and vectors are ingested as a Custom feature.

Our goal in this document is to visualize these custom features.

UMAP Technique for embedding visualization

We utilize the UMAP (Uniform Manifold Approximation and Projection) technique for embedding visualizations. UMAP is a dimension reduction technique that is particularly good at preserving the local structure of the data, making it ideal for visualizing embeddings. We reduce the high-dimensional embeddings to a 3D space.

UMAP is supported for both Text and Image embeddings in Custom feature.

UMAP understanding Generative AI applications

Specifically for GenAI applications, UMAP embedding visualizations are extremely helpful in understanding common themes and topics present in the data corpus. When evaluating prompts and responses, it is paramount to see which concepts clusters are emerging and which clusters are exhibiting the most problems. By further coloring these clusters with a variety of LLM and GenAI correctness and safety metrics, users can quickly be drawn to the clusters with the most issues.

📘

To create an embedding visualization chart

Follow the UI Guide on creating the embedding visualization chart here.