Seldon Core
What is Seldon Core?
Seldon Core is a Kubernetes-native AI Model Serving & Inference runtime. It loads trained machine learning (ML) and large language model (LLM) weights onto inference servers, exposes REST and gRPC endpoints, and executes those models in production so applications can obtain predictions or generated tokens at scale. The current architecture (Core 2) is bring-your-own-weights serving software that the operator runs on a cluster; it is not a hosted model-lab API and not a training platform.
Key Capabilities
- Inference servers as a separate resource: Models and Servers are distinct objects. A Server is an inference engine such as Seldon MLServer or NVIDIA Triton Inference Server. Models are packed onto those servers rather than each model owning a dedicated replica set.
- Open Inference Protocol (OIP): Request and response payloads follow OIP over REST and gRPC, so different model types share one client contract.
- Split control plane and data plane: A Scheduler loads and unloads Models, Pipelines, and Experiments. Envoy is the data-plane ingress and load-balances inference traffic onto server pods. Inference can continue if the Scheduler is down.
- Kafka pipelines: Optional inference graphs stream tensors between models on Apache Kafka. Joins (inner, outer, trigger) compose multi-step serving paths without a central orchestrator in the request path.
- Multi-model serving and overcommit: Several models share one server process. Overcommit pages unused models from memory to disk so more models can be registered than fit in RAM at once.
- Experiments: Traffic can be split or shadowed across candidate models or pipelines (A/B and canary routing) at the serving layer.
- Custom components: Drift, outlier, and other pre-/post-processors (including Alibi modules) attach as Models on the same serving path.
Audience & Use Cases
- Audience: ML engineers, platform and site reliability (SRE) teams, and cluster operators who already run Kubernetes (or Docker Compose for local use) and need to execute their own weights.
- Use Case: Production inference for predictive models, LLMs, and multi-step inference graphs on-premises, in a single cloud, or across hybrid clusters, including packing many models onto shared GPUs or CPUs.
Technical Specifications
- Packaging: Open-source project at github.com/SeldonIO/seldon-core; Kubernetes operator (Controller) plus Scheduler, Agent, Envoy, and optional Kafka dataflow services.
- Protocols: REST and gRPC via the Open Inference Protocol; internal control-plane traffic is gRPC.
- Inference engines: MLServer, NVIDIA Triton, and plug-in servers that speak OIP.
- Ingress / routing: Envoy with weighted least-request load balancing.
- Pipeline backbone: Kafka (synchronous request/response is translated to produce/consume at the Pipeline Gateway).
- License: Business Source License (see the repository LICENSE file). Commercial terms are sold separately by Seldon.
Categories & Use Cases
Technical Details
| Mobile Application | No |
|---|
FAQs
What is Seldon Core?
Seldon Core is a Kubernetes-native AI Model Serving & Inference runtime. It loads trained machine learning (ML) and large language model (LLM) weights onto inference servers, exposes REST and gRPC endpoints, and executes those models in production so applications can obtain predictions or generated tokens at scale. The current architecture (Core 2) is bring-your-own-weights serving software that the operator runs on a cluster; it is not a hosted model-lab API and not a training platform.