KServe is open-source AI Model Serving & Inference software for Kubernetes. Operators declare an InferenceService (and optional InferenceGraph) as custom resources. KServe then loads trained models from object storage or Hugging Face, starts a pluggable serving runtime, and exposes prediction or token APIs. It is a control plane for serving, not a training platform and not a hosted model catalog.

KServe is a CNCF incubating project in the Kubeflow ecosystem (formerly KFServing).

Key Capabilities

InferenceService CRD: YAML for model URI, runtime, resources, and traffic. Storage backends include S3, GCS, Azure Blob, Hugging Face, and PersistentVolumeClaims.
Pluggable ServingRuntime / ClusterServingRuntime: The process that executes weights is swapped per model: vLLM (and llm-d) for LLMs; NVIDIA Triton, TorchServe, TensorFlow Serving, scikit-learn, XGBoost, ONNX, MLflow, Hugging Face, or a custom container for predictive models.
Protocols: Open Inference Protocol (V1/V2) for predictive models. LLM paths expose OpenAI-compatible chat, streaming, and embeddings.
InferenceGraph: Chains predictors, transformers (pre/post-process), and explainers; supports ensembles and canary/A/B traffic splits.
Serverless path: Optional Knative install for request-based autoscaling, including scale-to-zero on CPU and GPU. A “raw” Kubernetes install is lighter and does not include canary or scale-to-zero.
LLM extras: LocalModelCache to keep large weights on node disk; KV-cache offload for long context; multi-node inference; metric-based autoscaling on token throughput, queue depth, and GPU utilization.
ModelMesh (optional): Separate high-density path for many small, frequently swapped models.
Envoy AI Gateway (optional): Ingress/API management in front of InferenceServices. That is a gateway in front of KServe, not KServe’s job.

Audience & Use Cases

Audience: Platform and site reliability (SRE) teams and ML engineers who already run Kubernetes and need a cluster-native way to serve both classical ML and LLMs.
Use Case: Production InferenceService for sklearn/XGBoost/ONNX; LLM endpoints via vLLM with an OpenAI-shaped API; multi-model graphs; scale-to-zero on idle GPU nodes.

Technical Specifications

Source: github.com/kserve. Apache-2.0. Helm/YAML install; Kubeflow add-on or standalone.
Data plane roles: Predictor (model), Transformer (pre/post), Explainer (Alibi-class explainability, drift/outlier optional).
Install modes: Raw Kubernetes, Knative serverless, ModelMesh, local quickstart.
Vendor: CNCF project (not a commercial SKU). Commercial Kubernetes distributions and clouds (including OpenShift and AWS Kubeflow guides) package it; those platforms stay in their own primaries.

KServe

What is KServe?

Categories & Use Cases