Envoy AI Gateway
What is Envoy AI Gateway?
Envoy AI Gateway is open-source AI Gateway software built on Envoy Gateway (itself on Envoy Proxy). It sits between application or agent clients and downstream generative AI services. It routes, authenticates, rate-limits, fails over, and observes LLM and related AI traffic. It does not load model weights or run inference.
Current docs line: 1.1 (1.0 was the first stable control-plane API).
Key Capabilities
- Unified LLM routing: One front door to hosted providers and self-hosted model clusters. Out-of-the-box providers include OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, Google Gemini, Vertex AI, Mistral, Cohere, Groq, Together AI, DeepSeek, DeepInfra, SambaNova, Grok, Hunyuan, and Tetrate Agent Router Service.
- Two-tier pattern: A tier-one gateway for authentication, top-level routing, and global rate limits. A tier-two gateway in front of a self-hosted serving cluster, including endpoint-picker style routing for inference optimization. The second tier still does not execute the model; the serving cluster does.
- Policy and usage limits: Backend rate limiting and usage policies so organizations can cap tokens, requests, or cost per route or tenant.
- Upstream authentication: Secure egress to external providers (API keys and related upstream auth) so applications do not hold raw provider credentials in every client.
- Failover: Automatic failover across providers or backends when a target is unavailable.
- MCP path: A Model Context Protocol (MCP) gateway, including per-user identity on tool calls via header forwarding.
- Observability: Traffic, usage, and cost-oriented metrics on the control plane (AIGatewayRoute resources).
- Kubernetes-native: Control-plane CRDs on Envoy Gateway; data plane is Envoy.
Audience & Use Cases
- Audience: Platform and site reliability (SRE) teams and security/platform architects who already run Kubernetes and Envoy and need a single policy point for many LLM backends.
- Use Case: Central egress to OpenAI/Bedrock/Gemini with org-wide rate limits; failover from a hosted API to an in-cluster vLLM/KServe pool; MCP tool calls with per-user identity; cost and latency dashboards without putting a serving stack in the gateway.
Technical Specifications
- Source: github.com/envoyproxy/ai-gateway. Apache-2.0. Site: aigateway.envoyproxy.io.
- Foundation: Envoy Gateway + Envoy Proxy. Not a fork of visitor-management Envoy and not a replacement for the existing Envoy Proxy service-mesh listing.
- Install: Kubernetes manifests; community Slack #envoy-ai-gateway.
- Vendor: CNCF/Envoy community project. Tetrate and others ship commercial distributions; those are separate products.
Categories & Use Cases
Technical Details
| Mobile Application | No |
|---|
FAQs
What is Envoy AI Gateway?
Envoy AI Gateway is open-source AI Gateway software built on Envoy Gateway (itself on Envoy Proxy). It sits between application or agent clients and downstream generative AI services. It routes, authenticates, rate-limits, fails over, and observes LLM and related AI traffic. It does not load model weights or run inference.