> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fiddler.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Gateway

> Configure LLM provider credentials to enable AI-powered features in Fiddler using your own API keys from OpenAI, Anthropic, Gemini, and other providers.

The **LLM Gateway** allows you to manage and securely configure credentials for multiple Large Language Model (LLM) providers, enabling you to use your own API keys for AI-powered features throughout the Fiddler platform.

## Overview

LLM Gateway provides centralized management of LLM provider credentials, giving you control over which models and API keys power Fiddler's AI features such as:

* **Custom Evaluators** - Use your preferred LLM to evaluate model outputs
* **LLM Enrichments** - Generate AI-powered insights on your monitoring data
* **Content Analysis** - Assess response quality, detect hallucinations, and monitor trust metrics

### Key Capabilities

* **Multiple Provider Support** - Configure credentials for OpenAI, Anthropic, Gemini, Vertex AI, Databricks, AWS Bedrock, Azure OpenAI, Azure AI, and Fiddler
* **Credential Redundancy** - Add multiple API keys per provider for failover and load balancing
* **Flexible Key Management** - Organize credentials by team, environment, or purpose
* **Secure Storage** - API keys are encrypted and stored securely

***

## Prerequisites

Before configuring LLM Gateway, ensure you have:

* **Admin Permissions** - Only administrators can access the LLM Gateway settings
* **Provider API Keys** - Valid API credentials from your chosen LLM providers (OpenAI, Anthropic, Gemini, Vertex AI, Databricks, AWS Bedrock, Azure OpenAI, Azure AI, or Fiddler)

<Info>
  **Note:** Each provider requires a separate API key. Obtain keys from your provider's developer portal before proceeding.
</Info>

***

## Configure LLM Providers

### Add a New Provider

Follow these steps to add an LLM provider to your Fiddler organization:

<Steps>
  <Step title="Navigate to the LLM Gateway Settings">
    From the top navigation bar, click the **Settings** icon (gear icon) and select the **LLM Gateway** tab.
  </Step>

  <Step title="Add a Provider">
    Click the **Add Provider** button to open the provider configuration dialog.
  </Step>

  <Step title="Select the Provider Type">
    Choose your LLM provider from the dropdown menu:

    * **OpenAI** - OpenAI GPT models
    * **Anthropic** - Anthropic Claude models
    * **Gemini** - Google Gemini models
    * **Vertex AI** - Google Cloud Vertex AI Model Garden (Gemini and Anthropic Claude)
    * **AWS Bedrock** - Bedrock-hosted models (Anthropic, Amazon Nova, Meta, Mistral AI)
    * **Azure OpenAI** - OpenAI models on Azure infrastructure
    * **Azure AI** - Anthropic Claude models via the Azure AI catalog
    * **Databricks** - Foundation models served through Databricks Model Serving
    * **Fiddler** - Fiddler-hosted LLM services
  </Step>

  <Step title="Add Your First Credential">
    a. Click **Add Credential**

    b. Enter a **Nickname** for the credential (for example, `Production Team Key` or `Test Environment`)

    <Info>
      **Tip:** Use clear, descriptive nicknames to differentiate between test and production keys or to identify which team owns the credential.
    </Info>

    c. Paste your **API Key** into the credential field

    <Info>
      Some providers use authentication methods other than — or in addition to — an API key. For example, **Microsoft Entra ID** (Azure), **AWS Access Key** (Bedrock), **GCP Service Account** (Vertex AI), and **OAuth Client Credentials** (Databricks). Vertex AI, for instance, uses a GCP Service Account instead of an API key. See [Supported Providers](#supported-providers) for the credential fields each provider requires.
    </Info>

    d. The provider's available models will be automatically populated
  </Step>

  <Step title="Save the Provider">
    Click **Update Provider** to save your configuration, or **Cancel** to discard changes.

    Your provider is now configured and ready to use with Fiddler's AI-powered features.
  </Step>
</Steps>

### Add Multiple Credentials to a Provider

You can add multiple API keys to a single provider for redundancy, load balancing, or to separate keys by environment or team.

**Why Use Multiple Credentials?**

* **Redundancy** - Automatic failover if one key reaches rate limits or expires
* **Load Balancing** - Distribute API calls across multiple keys to improve throughput
* **Key Rotation** - Safely test new credentials before removing old ones
* **Environment Separation** - Use different keys for development, staging, and production

**To Add Additional Credentials:**

1. Navigate to **Settings → LLM Gateway**
2. Click the **edit** icon next to the provider you want to modify
3. In the Edit Provider dialog, click **Add Credential**
4. Enter a nickname and paste the new API key
5. Click **Update Provider** to save

All credentials for a provider will be available for use across Fiddler features. The platform handles credential selection and failover automatically.

### Edit an Existing Provider

You can modify provider configurations, update credentials, or rename existing keys.

**To Edit a Provider:**

1. Navigate to **Settings → LLM Gateway**
2. Locate the provider you want to edit
3. Click the **edit** icon next to the provider name
4. Make your changes:
   * **Update Credentials** - Click the edit icon next to a credential to modify the API key
   * **Rename Credentials** - Update the nickname to reflect the key's purpose
   * **Add More Credentials** - Click **Add Credential** to add another API key
   * **Remove Credentials** - Delete individual credentials that are no longer needed
5. Click **Update Provider** to save your changes

<Warning>
  Removing a credential that is actively in use may impact running evaluations or enrichments. Ensure you have alternate credentials configured before removing a key.
</Warning>

***

## Supported Providers

The LLM Gateway supports the following providers:

### OpenAI

* **Models Available:** GPT-5 (including mini and nano), GPT-4.1 (including mini and nano), GPT-4o, and GPT-4o mini
* **API Key Location:** [OpenAI Platform - API Keys](https://platform.openai.com/api-keys)
* **Use Cases:** Custom evaluators, content generation, response quality assessment

### Anthropic

* **Models Available:** Claude Opus 4.1, Claude Opus 4, Claude Sonnet 4, and Claude 3.5 Haiku
* **API Key Location:** [Anthropic Console](https://console.anthropic.com/)
* **Use Cases:** Advanced reasoning, content analysis, evaluation tasks

### Gemini

* **Models Available:** Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite
* **API Key Location:** [Google AI Studio](https://makersuite.google.com/app/apikey)
* **Use Cases:** Multimodal analysis, content generation, embeddings

### Vertex AI

Access models from Google Cloud's Vertex AI Model Garden. Requests are routed through your GCP project and region.

* **Models Available:** Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, Claude Sonnet 4.5, and Claude Opus 4.5
* **Authentication**:
  * **GCP Service Account** — Provide a service account JSON key, plus the Vertex AI project (the GCP project where requests are sent) and Vertex AI location (the region, for example `us-central1`). The project and location may differ from the project embedded in the service account key.
* **Use cases:** Organizations with existing GCP / Vertex AI infrastructure or Google-managed inference environments

### AWS Bedrock

Access LLM models hosted on Amazon Web Services through the Bedrock managed service. All requests are routed through your AWS account.

* **Supported model families** (custom model IDs for fine-tuned and newly released models are also supported):
  * **Anthropic** (`bedrock/anthropic.*`) — Claude models for advanced reasoning and long-context tasks
  * **Amazon** (`bedrock/amazon.nova-*`) — Nova family for multimodal and text generation
  * **Meta** (`bedrock/meta.llama*`) — Open-weights Llama models across multiple sizes
  * **Mistral AI** (`bedrock/mistral.*`) — High-performance models optimized for efficiency
* **Authentication**:
  * **API Key** — Use Fiddler-managed AWS credentials
  * **AWS Access Key** — Provide your own AWS access key ID, secret access key, and AWS region
* **Custom endpoint** (optional) — Set a custom **API Base URL** to route requests through an AWS VPC endpoint instead of the public Bedrock endpoint (for example, `https://vpce-xxx.bedrock-runtime.us-east-1.vpce.amazonaws.com`)
* **Use cases:** Evaluators and LLM-as-a-Judge workflows requiring inference traffic to stay within AWS-managed cloud boundaries

### Azure OpenAI

Access OpenAI models deployed on Microsoft Azure. When configuring this provider, specify your **Azure deployment name**, which may differ from the base model name.

* **Supported models** (custom fine-tuned models are also supported):
  * **GPT-5** and **GPT-5 mini** — Latest-generation reasoning and general-purpose models
  * **GPT-4o** and **GPT-4o mini** — Multimodal, cost-optimized models
* **Authentication**:
  * **API Key** — Azure deployment API key for simple authentication
  * **Microsoft Entra ID** (formerly Azure AD) — OAuth 2.0 client credentials flow using tenant ID, client ID, and client secret; for enterprise SSO integration
* **Custom endpoint** — Set a custom **API Base URL** to target a specific Azure region or resource endpoint (for example, `https://<your-resource>.openai.azure.com/`)
* **Use cases:** Organizations with existing Azure OpenAI deployments, enterprise compliance requirements, or Microsoft-managed inference environments

### Azure AI

Access models from the Azure AI Model Catalog. When configuring this provider, specify your **Azure AI deployment name**.

* **Supported models**:
  * **Anthropic** — Claude Sonnet 4.5, Claude Haiku 4.5, and Claude Opus 4.1
* **Authentication**:
  * **API Key** — Azure deployment API key for simple authentication
  * **Microsoft Entra ID** (formerly Azure AD) — OAuth 2.0 client credentials flow using tenant ID, client ID, and client secret; for enterprise SSO integration
* **Custom endpoint** — Set a custom **API Base URL** to target a specific Azure AI resource (for example, `https://<your-hub>.services.ai.azure.com/`)
* **Use cases:** Organizations with existing Azure AI infrastructure or multi-provider model access through Azure's unified catalog

### Databricks

Access foundation models served through Databricks Model Serving. This provider requires a custom **API Base URL** pointing to your workspace serving endpoint.

* **Supported model families:**
  * **OpenAI GPT-OSS** — `databricks-gpt-oss-120b`, `databricks-gpt-oss-20b`
  * **Meta Llama** — `databricks-meta-llama-3-1-8b-instruct`, `databricks-meta-llama-3-3-70b-instruct`, `databricks-meta-llama-3-1-405b-instruct`, `databricks-llama-4-maverick`
  * **Google Gemma** — `databricks-gemma-3-12b`
* **Authentication**:
  * **API Key** — Databricks personal access token for simple authentication
  * **OAuth Client Credentials** — OAuth 2.0 client credentials (machine-to-machine) flow using a Databricks service principal's token URL, client ID, client secret, and scope (for example, `all-apis`). Fiddler exchanges these for a short-lived bearer token and refreshes it automatically — no long-lived tokens to manage or rotate.
* **Custom endpoint** — Set the **API Base URL** to your Databricks workspace serving endpoint (for example, `https://adb-xxxx.azuredatabricks.net/serving-endpoints`)
* **Use cases:** Organizations serving foundation models on Databricks, or requiring enterprise machine-to-machine authentication via service principals

<Accordion title="Set up a Databricks service principal">
  1. In Databricks, create a **service principal** and generate an **OAuth secret** for it, following Databricks' OAuth machine-to-machine (M2M) guide for your cloud ([AWS](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m), [Azure](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m), [GCP](https://docs.databricks.com/gcp/en/dev-tools/auth/oauth-m2m)). This gives you the **Client ID** (the service principal's application ID) and **Client Secret** (shown only once).
  2. Enter these values when adding the Databricks credential in Fiddler:
     * **Client ID** and **Client Secret** — from step 1.
     * **Token URL** — `https://<your-workspace-host>/oidc/v1/token`
     * **API Base URL** — `https://<your-workspace-host>/serving-endpoints` (same workspace host)
     * **Scope** — `all-apis`
</Accordion>

### Fiddler

* **Models Available:** Fiddler-managed LLM services
* **API Key Location:** Provided by Fiddler
* **Use Cases:** Pre-configured evaluators, platform-optimized features

***

## Best Practices

### Credential Management

* **Use Descriptive Nicknames** - Label credentials by team, environment, or purpose (for example, `ML Team - Production`, `Data Science - Test`)
* **Rotate Keys Regularly** - Add new credentials before removing old ones to avoid service interruption
* **Separate Environments** - Use different API keys for development, staging, and production
* **Monitor Usage** - Track API consumption through your provider's dashboard to avoid unexpected costs

### Security

* **Restrict Access** - Only grant Admin permissions to users who need to manage LLM credentials
* **Avoid Sharing Keys** - Each team should have their own credentials rather than sharing a single key
* **Revoke Compromised Keys** - If a key is exposed, immediately revoke it in your provider's console and remove it from Fiddler

### Performance Optimization

* **Add Multiple Credentials** - Configure 2-3 keys per provider for redundancy and better throughput
* **Test Before Production** - Validate new credentials in a test environment before using them in production
* **Monitor Rate Limits** - Be aware of your provider's rate limits and adjust your credential count accordingly

***

## Related Features

Once you've configured LLM providers in the Gateway, you can use them with these Fiddler features:

* **Custom Evaluators** - Create LLM-based evaluators to assess model outputs
* **LLM Enrichments** - Generate AI-powered metrics for your LLM applications
* **Content Safety** - Use LLM providers for advanced content analysis

***

## Troubleshooting

### Provider Not Appearing in Feature Selection

**Issue:** After adding a provider, it doesn't appear in evaluator or enrichment configuration.

**Solution:**

* Verify the provider was saved successfully (check the LLM Gateway tab)
* Ensure at least one credential was added to the provider
* Refresh your browser page
* Contact your Fiddler administrator to verify permissions

### Invalid API Key Error

**Issue:** Receiving authentication errors when using a configured provider.

**Solution:**

* Verify the API key is correct in your provider's console
* Check that the key hasn't expired or been revoked
* Ensure the key has the necessary permissions for the models you're using
* Update the credential in Settings → LLM Gateway

### Rate Limit Warnings

**Issue:** Receiving rate limit errors from a provider.

**Solution:**

* Add additional credentials to the provider for load balancing
* Check your provider's dashboard for current usage and limits
* Consider upgrading your provider plan for higher limits
* Temporarily reduce the number of concurrent evaluations or enrichments