KServe
What is KServe?
KServe is open-source AI Model Serving & Inference software for Kubernetes. Operators declare an InferenceService (and optional InferenceGraph) as custom resources. KServe then loads trained models from object storage or Hugging Face, starts a pluggable serving runtime, and exposes prediction or token APIs. It is a control plane for serving, not a training platform and not a hosted model catalog.
KServe is a CNCF incubating project in the Kubeflow ecosystem (formerly KFServing).
Key Capabilities
- InferenceService CRD: YAML for model URI, runtime, resources, and traffic. Storage backends include S3, GCS, Azure Blob, Hugging Face, and PersistentVolumeClaims.
- Pluggable ServingRuntime / ClusterServingRuntime: The process that executes weights is swapped per model: vLLM (and llm-d) for LLMs; NVIDIA Triton, TorchServe, TensorFlow Serving, scikit-learn, XGBoost, ONNX, MLflow, Hugging Face, or a custom container for predictive models.
- Protocols: Open Inference Protocol (V1/V2) for predictive models. LLM paths expose OpenAI-compatible chat, streaming, and embeddings.
- InferenceGraph: Chains predictors, transformers (pre/post-process), and explainers; supports ensembles and canary/A/B traffic splits.
- Serverless path: Optional Knative install for request-based autoscaling, including scale-to-zero on CPU and GPU. A “raw” Kubernetes install is lighter and does not include canary or scale-to-zero.
- LLM extras: LocalModelCache to keep large weights on node disk; KV-cache offload for long context; multi-node inference; metric-based autoscaling on token throughput, queue depth, and GPU utilization.
- ModelMesh (optional): Separate high-density path for many small, frequently swapped models.
- Envoy AI Gateway (optional): Ingress/API management in front of InferenceServices. That is a gateway in front of KServe, not KServe’s job.
Audience & Use Cases
- Audience: Platform and site reliability (SRE) teams and ML engineers who already run Kubernetes and need a cluster-native way to serve both classical ML and LLMs.
- Use Case: Production InferenceService for sklearn/XGBoost/ONNX; LLM endpoints via vLLM with an OpenAI-shaped API; multi-model graphs; scale-to-zero on idle GPU nodes.
Technical Specifications
- Source: github.com/kserve. Apache-2.0. Helm/YAML install; Kubeflow add-on or standalone.
- Data plane roles: Predictor (model), Transformer (pre/post), Explainer (Alibi-class explainability, drift/outlier optional).
- Install modes: Raw Kubernetes, Knative serverless, ModelMesh, local quickstart.
- Vendor: CNCF project (not a commercial SKU). Commercial Kubernetes distributions and clouds (including OpenShift and AWS Kubeflow guides) package it; those platforms stay in their own primaries.
Categories & Use Cases
Technical Details
| Mobile Application | No |
|---|
FAQs
What is KServe?
KServe is open-source AI Model Serving & Inference software for Kubernetes. Operators declare an InferenceService (and optional InferenceGraph) as custom resources. KServe then loads trained models from object storage or Hugging Face, starts a pluggable serving runtime, and exposes prediction or token APIs. It is a control plane for serving, not a training platform and not a hosted model catalog.