What is LiteLLM?
LiteLLM is an open-source AI gateway and large language model (LLM) proxy for platform teams. It provides an OpenAI-compatible application programming interface (API) that centralizes access to LLM providers, agent tools, and Model Context Protocol (MCP) servers.
Key Capabilities
- Model access and routing: LiteLLM routes requests across LLM providers and deployments, allowing teams to change models without modifying application code.
- Access management: Virtual API keys, single sign-on, and team- or project-scoped permissions control access to models and services.
- Usage controls: The platform tracks requests and spending by user, key, team, model, agent, and MCP server. Teams can set budgets and rate limits that restrict requests when thresholds are reached.
- Reliability and cost controls: LiteLLM supports load balancing, fallback routing, response caching, and model selection policies.
- Security controls: Guardrails can apply PII masking, prompt-injection detection, secret detection, and audit logging to AI requests.
Audience & Use Cases
- Audience: AI platform engineers, machine learning teams, developers, security administrators, and FinOps teams.
- Use cases: Providing a shared AI access layer, managing model-provider credentials, enforcing spend controls, routing production LLM traffic, and deploying AI services in controlled environments.
Technical Specifications
- Deployment: LiteLLM can be self-hosted in a private cloud or air-gapped environment, using Docker, Helm, or Terraform.
- Infrastructure: The platform supports deployments using PostgreSQL, Redis, and Kubernetes-based autoscaling.
- Integration scope: The vendor documents one OpenAI-compatible API for more than 140 providers and 1,800 models, as well as agents and MCP servers.
Categories & Use Cases
Technical Details
| Mobile Application | No |
|---|
FAQs
What is LiteLLM?
LiteLLM is an open-source AI gateway and large language model (LLM) proxy for platform teams. It provides an OpenAI-compatible application programming interface (API) that centralizes access to LLM providers, agent tools, and Model Context Protocol (MCP) servers.