# LLM Gateway

The **LLM Gateway** allows you to manage and securely configure credentials for multiple Large Language Model (LLM) providers, enabling you to use your own API keys for AI-powered features throughout the Fiddler platform.

## Overview

LLM Gateway provides centralized management of LLM provider credentials, giving you control over which models and API keys power Fiddler's AI features such as:

* **Custom Evaluators** - Use your preferred LLM to evaluate model outputs
* **LLM Enrichments** - Generate AI-powered insights on your monitoring data
* **Content Analysis** - Assess response quality, detect hallucinations, and monitor trust metrics

### Key Capabilities

* **Multiple Provider Support** - Configure credentials for OpenAI, Anthropic, Gemini, AWS Bedrock, Azure OpenAI, Azure AI, and Fiddler
* **Credential Redundancy** - Add multiple API keys per provider for failover and load balancing
* **Flexible Key Management** - Organize credentials by team, environment, or purpose
* **Secure Storage** - API keys are encrypted and stored securely

***

## Prerequisites

Before configuring LLM Gateway, ensure you have:

* **Admin Permissions** - Only administrators can access the LLM Gateway settings
* **Provider API Keys** - Valid API credentials from your chosen LLM providers (OpenAI, Anthropic, Gemini, AWS Bedrock, Azure OpenAI, Azure AI, or Fiddler)

{% hint style="info" %}
**Note:** Each provider requires a separate API key. Obtain keys from your provider's developer portal before proceeding.
{% endhint %}

***

## Configure LLM Providers

### Add a New Provider

Follow these steps to add an LLM provider to your Fiddler organization:

{% stepper %}
{% step %}
**Navigate to the LLM Gateway Settings**

From the top navigation bar, click the **Settings** icon (gear icon) and select the **LLM Gateway** tab.
{% endstep %}

{% step %}
**Add a Provider**

Click the **Add Provider** button to open the provider configuration dialog.
{% endstep %}

{% step %}
**Select the Provider Type**

Choose your LLM provider from the dropdown menu:

* **OpenAI** - GPT-4, GPT-3.5, and other OpenAI models
* **Anthropic** - Claude models (Sonnet, Opus, Haiku)
* **Gemini** - Google's Gemini models
* **AWS Bedrock** - Bedrock-hosted models including Anthropic, Amazon Nova, Meta, and Mistral AI
* **Azure OpenAI** - GPT-4, GPT-4o, and other OpenAI models on Azure infrastructure
* **Azure AI** - Multi-provider model catalog including Anthropic, Meta, Mistral AI, and Cohere
* **Fiddler** - Fiddler-hosted LLM services
  {% endstep %}

{% step %}
**Add Your First Credential**

a. Click **Add Credential**

b. Enter a **Nickname** for the credential (for example, `Production Team Key` or `Test Environment`)

{% hint style="info" %}
**Tip:** Use clear, descriptive nicknames to differentiate between test and production keys or to identify which team owns the credential.
{% endhint %}

c. Paste your **API Key** into the credential field

d. The provider's available models will be automatically populated
{% endstep %}

{% step %}
**Save the Provider**

Click **Update Provider** to save your configuration, or **Cancel** to discard changes.

Your provider is now configured and ready to use with Fiddler's AI-powered features.
{% endstep %}
{% endstepper %}

***

### Add Multiple Credentials to a Provider

You can add multiple API keys to a single provider for redundancy, load balancing, or to separate keys by environment or team.

**Why Use Multiple Credentials?**

* **Redundancy** - Automatic failover if one key reaches rate limits or expires
* **Load Balancing** - Distribute API calls across multiple keys to improve throughput
* **Key Rotation** - Safely test new credentials before removing old ones
* **Environment Separation** - Use different keys for development, staging, and production

**To Add Additional Credentials:**

1. Navigate to **Settings → LLM Gateway**
2. Click the **edit** icon next to the provider you want to modify
3. In the Edit Provider dialog, click **Add Credential**
4. Enter a nickname and paste the new API key
5. Click **Update Provider** to save

All credentials for a provider will be available for use across Fiddler features. The platform handles credential selection and failover automatically.

***

### Edit an Existing Provider

You can modify provider configurations, update credentials, or rename existing keys.

**To Edit a Provider:**

1. Navigate to **Settings → LLM Gateway**
2. Locate the provider you want to edit
3. Click the **edit** icon next to the provider name
4. Make your changes:
   * **Update Credentials** - Click the edit icon next to a credential to modify the API key
   * **Rename Credentials** - Update the nickname to reflect the key's purpose
   * **Add More Credentials** - Click **Add Credential** to add another API key
   * **Remove Credentials** - Delete individual credentials that are no longer needed
5. Click **Update Provider** to save your changes

{% hint style="warning" %}
Removing a credential that is actively in use may impact running evaluations or enrichments. Ensure you have alternate credentials configured before removing a key.
{% endhint %}

***

## Supported Providers

The LLM Gateway supports the following providers:

### OpenAI

* **Models Available:** GPT-4, GPT-4 Turbo, GPT-3.5 Turbo, and other OpenAI models
* **API Key Location:** [OpenAI Platform - API Keys](https://platform.openai.com/api-keys)
* **Use Cases:** Custom evaluators, content generation, response quality assessment

### Anthropic

* **Models Available:** Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku, and other Claude models
* **API Key Location:** [Anthropic Console](https://console.anthropic.com/)
* **Use Cases:** Advanced reasoning, content analysis, evaluation tasks

### Gemini

* **Models Available:** Gemini Pro, Gemini Ultra, and other Google AI models
* **API Key Location:** [Google AI Studio](https://makersuite.google.com/app/apikey)
* **Use Cases:** Multimodal analysis, content generation, embeddings

### Fiddler

* **Models Available:** Fiddler-managed LLM services
* **API Key Location:** Provided by Fiddler
* **Use Cases:** Pre-configured evaluators, platform-optimized features

### AWS Bedrock

Access LLM models hosted on Amazon Web Services through the Bedrock managed service. All requests are routed through your AWS account.

* **Supported model families** (custom model IDs for fine-tuned and newly released models are also supported):
  * **Anthropic** (`bedrock/anthropic.*`) — Claude models for advanced reasoning and long-context tasks
  * **Amazon** (`bedrock/amazon.nova-*`) — Nova family for multimodal and text generation
  * **Meta** (`bedrock/meta.llama*`) — Open-weights Llama models across multiple sizes
  * **Mistral AI** (`bedrock/mistral.*`) — High-performance models optimized for efficiency
* **Authentication**:
  * **API Key** — Use Fiddler-managed AWS credentials
  * **AWS Access Key** — Provide your own AWS access key ID, secret access key, and AWS region
* **Custom endpoint** (optional) — Set a custom **API Base URL** to route requests through an AWS VPC endpoint instead of the public Bedrock endpoint (for example, `https://vpce-xxx.bedrock-runtime.us-east-1.vpce.amazonaws.com`)
* **Use cases:** Evaluators and LLM-as-a-Judge workflows requiring inference traffic to stay within AWS-managed cloud boundaries

### Azure OpenAI

Access OpenAI models deployed on Microsoft Azure. When configuring this provider, specify your **Azure deployment name**, which may differ from the base model name.

* **Supported models** (custom fine-tuned models are also supported):
  * **GPT-4** and **GPT-4 Turbo** — Advanced reasoning and long-context tasks
  * **GPT-3.5 Turbo** — Fast, cost-effective completions
  * **GPT-4o** — Multimodal capabilities
* **Authentication**:
  * **API Key** — Azure deployment API key for simple authentication
  * **Microsoft Entra ID** (formerly Azure AD) — OAuth 2.0 client credentials flow using tenant ID, client ID, and client secret; for enterprise SSO integration
* **Custom endpoint** — Set a custom **API Base URL** to target a specific Azure region or resource endpoint (for example, `https://<your-resource>.openai.azure.com/`)
* **Use cases:** Organizations with existing Azure OpenAI deployments, enterprise compliance requirements, or Microsoft-managed inference environments

### Azure AI

Access models from the Azure AI Model Catalog. When configuring this provider, specify your **Azure AI deployment name**.

* **Supported model providers**:
  * **Anthropic** — Claude models (Opus, Sonnet, Haiku)
  * **Meta** — Llama models (8B to 70B parameters)
  * **Mistral AI** — Mistral and Mixtral models
  * **Cohere** — Command and Embed models
  * Other foundation models available in the Azure AI catalog
* **Authentication**:
  * **API Key** — Azure deployment API key for simple authentication
  * **Microsoft Entra ID** (formerly Azure AD) — OAuth 2.0 client credentials flow using tenant ID, client ID, and client secret; for enterprise SSO integration
* **Custom endpoint** — Set a custom **API Base URL** to target a specific Azure AI resource (for example, `https://<your-hub>.services.ai.azure.com/`)
* **Use cases:** Organizations with existing Azure AI infrastructure or multi-provider model access through Azure's unified catalog

***

## Best Practices

### Credential Management

* **Use Descriptive Nicknames** - Label credentials by team, environment, or purpose (for example, `ML Team - Production`, `Data Science - Test`)
* **Rotate Keys Regularly** - Add new credentials before removing old ones to avoid service interruption
* **Separate Environments** - Use different API keys for development, staging, and production
* **Monitor Usage** - Track API consumption through your provider's dashboard to avoid unexpected costs

### Security

* **Restrict Access** - Only grant Admin permissions to users who need to manage LLM credentials
* **Avoid Sharing Keys** - Each team should have their own credentials rather than sharing a single key
* **Revoke Compromised Keys** - If a key is exposed, immediately revoke it in your provider's console and remove it from Fiddler

### Performance Optimization

* **Add Multiple Credentials** - Configure 2-3 keys per provider for redundancy and better throughput
* **Test Before Production** - Validate new credentials in a test environment before using them in production
* **Monitor Rate Limits** - Be aware of your provider's rate limits and adjust your credential count accordingly

***

## Related Features

Once you've configured LLM providers in the Gateway, you can use them with these Fiddler features:

* **Custom Evaluators** - Create LLM-based evaluators to assess model outputs
* **LLM Enrichments** - Generate AI-powered metrics for your LLM applications
* **Content Safety** - Use LLM providers for advanced content analysis

***

## Troubleshooting

### Provider Not Appearing in Feature Selection

**Issue:** After adding a provider, it doesn't appear in evaluator or enrichment configuration.

**Solution:**

* Verify the provider was saved successfully (check the LLM Gateway tab)
* Ensure at least one credential was added to the provider
* Refresh your browser page
* Contact your Fiddler administrator to verify permissions

### Invalid API Key Error

**Issue:** Receiving authentication errors when using a configured provider.

**Solution:**

* Verify the API key is correct in your provider's console
* Check that the key hasn't expired or been revoked
* Ensure the key has the necessary permissions for the models you're using
* Update the credential in Settings → LLM Gateway

### Rate Limit Warnings

**Issue:** Receiving rate limit errors from a provider.

**Solution:**

* Add additional credentials to the provider for load balancing
* Check your provider's dashboard for current usage and limits
* Consider upgrading your provider plan for higher limits
* Temporarily reduce the number of concurrent evaluations or enrichments

***


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fiddler.ai/reference/settings/llm-gateway.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.