Envoy AI Gateway is open-source AI Gateway software built on Envoy Gateway (itself on Envoy Proxy). It sits between application or agent clients and downstream generative AI services. It routes, authenticates, rate-limits, fails over, and observes LLM and related AI traffic. It does not load model weights or run inference.

Current docs line: 1.1 (1.0 was the first stable control-plane API).

Key Capabilities

Unified LLM routing: One front door to hosted providers and self-hosted model clusters. Out-of-the-box providers include OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, Google Gemini, Vertex AI, Mistral, Cohere, Groq, Together AI, DeepSeek, DeepInfra, SambaNova, Grok, Hunyuan, and Tetrate Agent Router Service.
Two-tier pattern: A tier-one gateway for authentication, top-level routing, and global rate limits. A tier-two gateway in front of a self-hosted serving cluster, including endpoint-picker style routing for inference optimization. The second tier still does not execute the model; the serving cluster does.
Policy and usage limits: Backend rate limiting and usage policies so organizations can cap tokens, requests, or cost per route or tenant.
Upstream authentication: Secure egress to external providers (API keys and related upstream auth) so applications do not hold raw provider credentials in every client.
Failover: Automatic failover across providers or backends when a target is unavailable.
MCP path: A Model Context Protocol (MCP) gateway, including per-user identity on tool calls via header forwarding.
Observability: Traffic, usage, and cost-oriented metrics on the control plane (AIGatewayRoute resources).
Kubernetes-native: Control-plane CRDs on Envoy Gateway; data plane is Envoy.

Audience & Use Cases

Audience: Platform and site reliability (SRE) teams and security/platform architects who already run Kubernetes and Envoy and need a single policy point for many LLM backends.
Use Case: Central egress to OpenAI/Bedrock/Gemini with org-wide rate limits; failover from a hosted API to an in-cluster vLLM/KServe pool; MCP tool calls with per-user identity; cost and latency dashboards without putting a serving stack in the gateway.

Technical Specifications

Source: github.com/envoyproxy/ai-gateway. Apache-2.0. Site: aigateway.envoyproxy.io.
Foundation: Envoy Gateway + Envoy Proxy. Not a fork of visitor-management Envoy and not a replacement for the existing Envoy Proxy service-mesh listing.
Install: Kubernetes manifests; community Slack #envoy-ai-gateway.
Vendor: CNCF/Envoy community project. Tetrate and others ship commercial distributions; those are separate products.

Envoy AI Gateway

What is Envoy AI Gateway?

Categories & Use Cases