> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fiddler.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Compare LLM Outputs
> Learn how to systematically compare outputs from different LLM models (GPT-3.5, Claude, etc.) using Fiddler's pre-production evaluation environment to make data-driven model selection decisions.
## Overview
Making the right LLM model choice is critical for your application's success, but comparing models effectively requires more than intuition. This quick start demonstrates how to use Fiddler's pre-production evaluation environment to perform **systematic, side-by-side comparisons** of different LLM models using the same prompts and consistent evaluation metrics.
## What You'll Learn
In this hands-on notebook guide, you'll learn how to:
* **Upload model outputs** from different LLMs (GPT-3.5 and Claude) to Fiddler's pre-production environment
* **Define a consistent evaluation schema** that works across multiple models
* **Apply Fiddler enrichments** for automated quality assessment:
* **Faithfulness (Centor Model)** - Detect hallucinations in RAG responses using Fiddler Centor Models
* **Sentiment Analysis** - Understand response tone
* **PII Detection** - Identify privacy risks
* **Embeddings** - Track semantic patterns and outliers
**Looking for RAG-specific evaluation?** For comprehensive RAG pipeline diagnostics using Answer Relevance 2.0, Context Relevance, and RAG Faithfulness, see the [RAG Health Metrics Tutorial](/developers/tutorials/experiments/rag-health-metrics-tutorial) and [RAG Evaluation Fundamentals Cookbook](/developers/cookbooks/rag-evaluation-fundamentals).
\* \*\*Build comparison dashboards\*\* using metric cards to visualize differences \* \*\*Make data-driven decisions\*\* about which model best fits your needs
## Why Compare in Pre-Production?
Pre-production evaluation lets you test models **before committing to production deployment**:
* **Apples-to-apples comparison** - Same prompts, same metrics, consistent evaluation
* **Cost optimization** - Identify the most cost-effective model that meets quality requirements
* **Risk assessment** - Understand safety and quality trade-offs before production
* **Informed decisions** - Replace guesswork with quantitative evidence
## What You'll Build
By the end of this guide, you'll have:
1. **Two datasets** uploaded to Fiddler (GPT-3.5 and Claude outputs)
2. **Consistent enrichments** applied to both datasets for fair comparison
3. **Metric card dashboards** showing side-by-side model performance
4. **Clear insights** into which model performs better for your use case
## Prerequisites
Before you begin:
* **Fiddler account** with access to create projects
* **Python environment** with Jupyter notebook support
* **Fiddler Python client** (installed via pip)
* **Sample data** (provided in the notebook from Fiddler examples repository)
## Time to Complete
**\~15 minutes** - Follow the interactive notebook for a guided experience
## Get Started
Choose your preferred environment:
### Option 1: Google Colab (Recommended)
Run the notebook directly in your browser with zero setup:
[Open in Google Colab →](https://colab.research.google.com/github/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Comparison.ipynb)
\\
### Option 2: Local Jupyter Notebook
Download and run locally in your own environment:
[Download from GitHub →](https://github.com/fiddler-labs/fiddler-examples/blob/main/quickstart/latest/Fiddler_Quickstart_LLM_Comparison.ipynb)
## Workflow Overview
The notebook walks through this experiment workflow:
Initialize the Python client with your credentials.
Set up a project to organize your evaluation.
Read sample datasets containing GPT-3.5 and Claude responses.
Enable automated quality metrics:
* Text embeddings for semantic analysis
* Faithfulness scoring for hallucination detection
* Sentiment analysis for response tone
* PII detection for privacy compliance
Specify input/output columns and metadata.
Upload both model outputs to the pre-production environment.
Create metric cards for visual comparison.
Analyze results to choose the best model.
## Key Features Demonstrated
### Fiddler Enrichments
The notebook demonstrates several powerful Fiddler enrichments:
**Text Embeddings:**
* Generate semantic representations of prompts, responses, and source documents
* Track outliers and drift in your experiment datasets
* Enable similarity-based analysis
**Faithfulness Assessment:**
* Automatically detect hallucinations in RAG-based responses
* Compare how well each model grounds responses in source documents
* Critical for applications requiring factual accuracy
**Sentiment & Safety:**
* Analyze response tone and sentiment
* Detect PII leakage across models
* Assess safety and compliance risks
### Pre-Production Comparison
**Side-by-Side Analysis:**
* Upload multiple model outputs to the same project
* Apply identical enrichments for fair comparison
* Visualize differences with metric cards
**Data-Driven Decisions:**
* Quantitative metrics replace subjective judgment
* Compare quality, safety, and consistency across models
* Balance performance with cost considerations
## Use Cases
This comparison approach works for:
* **Model selection** - Choose between GPT-4, Claude, Llama, or other LLMs
* **Prompt optimization** - Compare different prompt strategies on the same model
* **Version testing** - Evaluate new model versions before upgrading
* **Cost optimization** - Find the most cost-effective model meeting quality standards
* **RAG system tuning** - Compare retrieval strategies and grounding effectiveness
## What Happens Next?
After completing this quick start:
1. **Analyze your results** - Use metric cards to understand performance differences
2. **Iterate on your evaluation** - Add more test cases or different models
3. **Scale to production** - Deploy the winning model with confidence
4. **Continue monitoring** - Use Fiddler's production monitoring for ongoing quality tracking
## Related Resources
**Expand Your Experiment Capabilities:**
* [Evals SDK Quick Start](/evaluate-and-test/evals-sdk-quick-start) - Build custom experiment workflows
* [Prompt Specs Quick Start](/evaluate-and-test/prompt-specs-quick-start) - Create custom LLM-as-a-Judge evaluators
* [Experiments Overview](/getting-started/experiments) - Comprehensive guide to Fiddler Experiments
**Production Deployment:**
* [Agentic Monitoring](/getting-started/agentic-monitoring) - Monitor LLM applications in production
* [LLM Monitoring](/getting-started/llm-monitoring) - Track production LLM performance
* [Guardrails](/getting-started/guardrails) - Add real-time safety validation