LLM Gateway

The LLM Gateway allows you to manage and securely configure credentials for multiple Large Language Model (LLM) providers, enabling you to use your own API keys for AI-powered features throughout the Fiddler platform. Not to be confused with a third-party AI gateway (such as Kong AI Gateway, AgentGateway, or LiteLLM), which proxies your own application’s LLM traffic — this page configures credentials for Fiddler’s own AI features instead.

​

Overview

LLM Gateway provides centralized management of LLM provider credentials, giving you control over which models and API keys power Fiddler’s AI features such as:

• Custom Evaluators - Use your preferred LLM to evaluate model outputs
• LLM Enrichments - Generate AI-powered insights on your monitoring data
• Content Analysis - Assess response quality, detect hallucinations, and monitor trust metrics

​

Key Capabilities

• Multiple Provider Support - Configure credentials for OpenAI, Anthropic, Gemini, Vertex AI, Databricks, AWS Bedrock, Azure OpenAI, Azure AI, and Fiddler
• Credential Redundancy - Add multiple API keys per provider for failover and load balancing
• Flexible Key Management - Organize credentials by team, environment, or purpose
• Secure Storage - API keys are encrypted and stored securely

---

​

Prerequisites

Before configuring LLM Gateway, ensure you have:

• Admin Permissions - Only administrators can access the LLM Gateway settings
• Provider API Keys - Valid API credentials from your chosen LLM providers (OpenAI, Anthropic, Gemini, Vertex AI, Databricks, AWS Bedrock, Azure OpenAI, Azure AI, or Fiddler)

---

​

Configure LLM Providers

​

Add a New Provider

Follow these steps to add an LLM provider to your Fiddler organization:

​

Add Multiple Credentials to a Provider

You can add multiple API keys to a single provider for redundancy, load balancing, or to separate keys by environment or team. Why Use Multiple Credentials?

• Redundancy - Automatic failover if one key reaches rate limits or expires
• Load Balancing - Distribute API calls across multiple keys to improve throughput
• Key Rotation - Safely test new credentials before removing old ones
• Environment Separation - Use different keys for development, staging, and production

To Add Additional Credentials:

1. Navigate to Settings → LLM Gateway
2. Click the edit icon next to the provider you want to modify
3. In the Edit Provider dialog, click Add Credential
4. Enter a nickname and paste the new API key
5. Click Update Provider to save

All credentials for a provider will be available for use across Fiddler features. The platform handles credential selection and failover automatically.

​

Edit an Existing Provider

You can modify provider configurations, update credentials, or rename existing keys. To Edit a Provider:

1. Navigate to Settings → LLM Gateway
2. Locate the provider you want to edit
3. Click the edit icon next to the provider name
4. Make your changes:
   • Update Credentials - Click the edit icon next to a credential to modify the API key
   • Rename Credentials - Update the nickname to reflect the key’s purpose
   • Add More Credentials - Click Add Credential to add another API key
   • Remove Credentials - Delete individual credentials that are no longer needed
5. Click Update Provider to save your changes

---

​

Supported Providers

The LLM Gateway supports the following providers:

​

OpenAI

• Models Available: GPT-5 (including mini and nano), GPT-4.1 (including mini and nano), GPT-4o, and GPT-4o mini
• API Key Location: OpenAI Platform - API Keys
• Use Cases: Custom evaluators, content generation, response quality assessment

​

Anthropic

• Models Available: Claude Opus 4.1, Claude Opus 4, Claude Sonnet 4, and Claude 3.5 Haiku
• API Key Location: Anthropic Console
• Use Cases: Advanced reasoning, content analysis, evaluation tasks

​

Gemini

• Models Available: Gemini 2.5 Pro, Gemini 2.5 Flash, and Gemini 2.5 Flash-Lite
• API Key Location: Google AI Studio
• Use Cases: Multimodal analysis, content generation, embeddings

​

Vertex AI

Access models from Google Cloud’s Vertex AI Model Garden. Requests are routed through your GCP project and region.

• Models Available: Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, Claude Sonnet 4.5, and Claude Opus 4.5
• Authentication:
  • GCP Service Account — Provide a service account JSON key, plus the Vertex AI project (the GCP project where requests are sent) and Vertex AI location (the region, for example us-central1). The project and location may differ from the project embedded in the service account key.
• Use cases: Organizations with existing GCP / Vertex AI infrastructure or Google-managed inference environments

​

AWS Bedrock

Access LLM models hosted on Amazon Web Services through the Bedrock managed service. All requests are routed through your AWS account.

• Supported model families (custom model IDs for fine-tuned and newly released models are also supported):
  • Anthropic (bedrock/anthropic.*) — Claude models for advanced reasoning and long-context tasks
  • Amazon (bedrock/amazon.nova-*) — Nova family for multimodal and text generation
  • Meta (bedrock/meta.llama*) — Open-weights Llama models across multiple sizes
  • Mistral AI (bedrock/mistral.*) — High-performance models optimized for efficiency
• Authentication:
  • API Key — Use Fiddler-managed AWS credentials
  • AWS Access Key — Provide your own AWS access key ID, secret access key, and AWS region
• Custom endpoint (optional) — Set a custom API Base URL to route requests through an AWS VPC endpoint instead of the public Bedrock endpoint (for example, https://vpce-xxx.bedrock-runtime.us-east-1.vpce.amazonaws.com)
• Use cases: Evaluators and LLM-as-a-Judge workflows requiring inference traffic to stay within AWS-managed cloud boundaries

​

Azure OpenAI

Access OpenAI models deployed on Microsoft Azure. When configuring this provider, specify your Azure deployment name, which may differ from the base model name.

• Supported models (custom fine-tuned models are also supported):
  • GPT-5 and GPT-5 mini — Latest-generation reasoning and general-purpose models
  • GPT-4o and GPT-4o mini — Multimodal, cost-optimized models
• Authentication:
  • API Key — Azure deployment API key for simple authentication
  • Microsoft Entra ID (formerly Azure AD) — OAuth 2.0 client credentials flow using tenant ID, client ID, and client secret; for enterprise SSO integration
• Custom endpoint — Set a custom API Base URL to target a specific Azure region or resource endpoint (for example, https://<your-resource>.openai.azure.com/)
• Use cases: Organizations with existing Azure OpenAI deployments, enterprise compliance requirements, or Microsoft-managed inference environments

​

Azure AI

Access models from the Azure AI Model Catalog. When configuring this provider, specify your Azure AI deployment name.

• Supported models:
  • Anthropic — Claude Sonnet 4.5, Claude Haiku 4.5, and Claude Opus 4.1
• Authentication:
  • API Key — Azure deployment API key for simple authentication
  • Microsoft Entra ID (formerly Azure AD) — OAuth 2.0 client credentials flow using tenant ID, client ID, and client secret; for enterprise SSO integration
• Custom endpoint — Set a custom API Base URL to target a specific Azure AI resource (for example, https://<your-hub>.services.ai.azure.com/)
• Use cases: Organizations with existing Azure AI infrastructure or multi-provider model access through Azure’s unified catalog

​

Databricks

Access foundation models served through Databricks Model Serving. This provider requires a custom API Base URL pointing to your workspace serving endpoint.

• Supported model families:
  • OpenAI GPT-OSS — databricks-gpt-oss-120b, databricks-gpt-oss-20b
  • Meta Llama — databricks-meta-llama-3-1-8b-instruct, databricks-meta-llama-3-3-70b-instruct, databricks-meta-llama-3-1-405b-instruct, databricks-llama-4-maverick
  • Google Gemma — databricks-gemma-3-12b
• Authentication:
  • API Key — Databricks personal access token for simple authentication
  • OAuth Client Credentials — OAuth 2.0 client credentials (machine-to-machine) flow using a Databricks service principal’s token URL, client ID, client secret, and scope (for example, all-apis). Fiddler exchanges these for a short-lived bearer token and refreshes it automatically — no long-lived tokens to manage or rotate.
• Custom endpoint — Set the API Base URL to your Databricks workspace serving endpoint (for example, https://adb-xxxx.azuredatabricks.net/serving-endpoints)
• Use cases: Organizations serving foundation models on Databricks, or requiring enterprise machine-to-machine authentication via service principals

Set up a Databricks service principal

1. In Databricks, create a service principal and generate an OAuth secret for it, following Databricks’ OAuth machine-to-machine (M2M) guide for your cloud (AWS, Azure, GCP). This gives you the Client ID (the service principal’s application ID) and Client Secret (shown only once).
2. Enter these values when adding the Databricks credential in Fiddler:
   • Client ID and Client Secret — from step 1.
   • Token URL — https://<your-workspace-host>/oidc/v1/token
   • API Base URL — https://<your-workspace-host>/serving-endpoints (same workspace host)
   • Scope — all-apis

​

Fiddler

• Models Available: Fiddler-managed LLM services
• API Key Location: Provided by Fiddler
• Use Cases: Pre-configured evaluators, platform-optimized features

​

Fiddler-Hosted Ministral 3 8B

The gateway offers Mistral’s Ministral 3 8B Instruct (fiddler/ministral3-8b) as a Fiddler-hosted model for evaluators and LLM-as-a-Judge workflows. It is slower than fiddler/llama3.1-8b but better suited to complex or nuanced evaluations that benefit from stronger reasoning, and it runs without external provider credentials or per-token charges.

---

​

Best Practices

​

Credential Management

• Use Descriptive Nicknames - Label credentials by team, environment, or purpose (for example, ML Team - Production, Data Science - Test)
• Rotate Keys Regularly - Add new credentials before removing old ones to avoid service interruption
• Separate Environments - Use different API keys for development, staging, and production
• Monitor Usage - Track API consumption through your provider’s dashboard to avoid unexpected costs

​

Security

• Restrict Access - Only grant Admin permissions to users who need to manage LLM credentials
• Avoid Sharing Keys - Each team should have their own credentials rather than sharing a single key
• Revoke Compromised Keys - If a key is exposed, immediately revoke it in your provider’s console and remove it from Fiddler

​

Performance Optimization

• Add Multiple Credentials - Configure 2-3 keys per provider for redundancy and better throughput
• Test Before Production - Validate new credentials in a test environment before using them in production
• Monitor Rate Limits - Be aware of your provider’s rate limits and adjust your credential count accordingly

---

​

Related Features

Once you’ve configured LLM providers in the Gateway, you can use them with these Fiddler features:

• Custom Evaluators - Create LLM-based evaluators to assess model outputs
• LLM Enrichments - Generate AI-powered metrics for your LLM applications
• Content Safety - Use LLM providers for advanced content analysis

---

​

Troubleshooting

​

Provider Not Appearing in Feature Selection

Issue: After adding a provider, it doesn’t appear in evaluator or enrichment configuration. Solution:

• Verify the provider was saved successfully (check the LLM Gateway tab)
• Ensure at least one credential was added to the provider
• Refresh your browser page
• Contact your Fiddler administrator to verify permissions

​

Invalid API Key Error

Issue: Receiving authentication errors when using a configured provider. Solution:

• Verify the API key is correct in your provider’s console
• Check that the key hasn’t expired or been revoked
• Ensure the key has the necessary permissions for the models you’re using
• Update the credential in Settings → LLM Gateway

​

Rate Limit Warnings

Issue: Receiving rate limit errors from a provider. Solution:

• Add additional credentials to the provider for load balancing
• Check your provider’s dashboard for current usage and limits
• Consider upgrading your provider plan for higher limits
• Temporarily reduce the number of concurrent evaluations or enrichments