1. What LiteLLM Is

LiteLLM is an open-source LLM adapter: call 100+ APIs (OpenAI, Anthropic, Gemini, Bedrock, Azure, Ollama, vLLM, Qwen, and more) using the standard OpenAI format.

Two ways to use it:

• Python SDK — called directly from your code
• AI Gateway — a team-level proxy with authentication, billing, load balancing, and an admin console

It converges “a separate SDK, auth scheme, and request format for every model” into a single interface and a single config.

• Repository: https://github.com/BerriAI/litellm
• Docs: https://docs.litellm.ai/docs/simple_proxy

2. Why You Need It

3. Core Use Cases

1. Unified multi-model integration — one set of OpenAI-format code; just change the model name to switch between GPT, Claude, and Qwen
2. Failover — when GPT-4 is rate-limited or down, switch automatically to Claude, then to a local Llama
3. Load balancing — round-robin across multiple keys or endpoints to break through TPM/RPM limits
4. Enterprise AI gateway — unified keys, budget quotas, log auditing
5. Cost monitoring — a built-in price table, with support for pushing to Prometheus and Langfuse
6. Local/cloud switching — Ollama in development, GPT-4 in production, with no code changes

1
2
3
4
5

from litellm import completion

response1 = completion(model="gpt-4", messages=[{"content": "你好", "role": "user"}])
response2 = completion(model="claude-3-sonnet-20240229", messages=[{"content": "你好", "role": "user"}])
response3 = completion(model="qwen/qwen-max", messages=[{"content": "你好", "role": "user"}])

When you are building an AI product and need multi-model access or team-wide key management, LiteLLM is basically the standard; if you only reach for a single model occasionally, you may not need it.

4. Two Modes of Use

4.1 Python SDK

Suited to scripts and calls embedded inside a service. The model name format is provider/model, and the SDK handles the format conversion.

1
2
3
4

from litellm import completion

response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

4.2 AI Gateway

Suited to team sharing and multi-application access. Clients just point base_url at the gateway:

1
2
3
4

import openai

client = openai.OpenAI(api_key="sk-litellm-virtual-key", base_url="http://localhost:4000")
response = client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

Client → LiteLLM Gateway (:4000) → 100+ Providers
           Auth / Routing / Guardrails / Billing

5. Core Capabilities

• Unified endpoints: /v1/chat/completions, /v1/embeddings, image/speech/batch inference, and more
• Virtual Key: managed by a Master Key; a Virtual Key binds a budget, rate limit, and model allowlist
• Routing and fallback: multiple deployments map to the same model_name; when the primary provider fails, switch automatically
• Guardrails: PII detection, third-party guardrail services, custom hooks
• Observability: Langfuse, OpenTelemetry, Prometheus; the Admin UI is at http://localhost:4000/ui
• MCP: Agents call gateway-managed tools and models through MCP

6. Quick Start

6.1 Install and Start

1
2
3

pip install litellm          # SDK
pip install 'litellm[proxy]' # Gateway
litellm --model gpt-4o       # start with a single model, listening on :4000

6.2 Multi-model config.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: local-llama
    litellm_params:
      model: ollama/llama3
      api_base: http://localhost:11434

general_settings:
  master_key: sk-litellm-master-key

1
2
3

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
litellm --config config.yaml

6.3 Docker

1
2

docker run --security-opt apparmor=unconfined --security-opt seccomp=unconfined -e OPENAI_API_KEY=$OPENAI_API_KEY -p 4000:4000 \
  ghcr.io/berriai/litellm:main-latest --model gpt-4o

In production, mount config.yaml and use PostgreSQL to persist Virtual Keys and spend logs.

7. Configuration Highlights

7.1 Secret References

1
2

litellm_params:
  api_key: os.environ/OPENAI_API_KEY

AWS Secrets Manager, Azure Key Vault, and others are also supported.

7.2 Creating a Virtual Key

1
2
3
4

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-litellm-master-key" \
  -H "Content-Type: application/json" \
  -d '{"models": ["gpt-4o", "claude-sonnet"], "max_budget": 10.0, "duration": "30d"}'

8. LiteLLM vs. New API

LiteLLM and New API (a fork of One API) are both popular LLM relay tools, but they target different audiences: LiteLLM is aimed at developers, New API at administrators/relay providers.

Choose LiteLLM: writing AI applications, needing high-availability fallback, integrating with Prometheus/Langfuse, local/cloud switching.

Choose New API: key distribution and relaying, team quota management, not wanting to write YAML, mainly domestic models.

9. Common CLI Commands

1
2
3
4
5
6
7

litellm --model gpt-4o
litellm --config config.yaml
litellm --config config.yaml --detailed_debug

litellm-proxy keys list
litellm-proxy models list
litellm-proxy spend logs

10. Summary

LiteLLM unifies 100+ LLMs behind an OpenAI-compatible interface and offers two modes — SDK and AI Gateway — covering key management, cost tracking, load balancing, fallback, and enterprise governance. Compared with New API, it is better suited to developers who need logic control; New API is better suited to resource distribution.

The Python gateway runs at roughly 8ms P95 under 1k RPS, and since 2026 a Rust rewrite has been underway to further reduce latency.