NVIDIA Triton
What is NVIDIA Triton?
NVIDIA Triton Inference Server (Triton) is open-source AI Model Serving & Inference software. It loads trained models from a model repository, runs them on GPU, CPU, or other accelerators, and returns predictions or generated tokens over HTTP/REST and gRPC. Triton is a general inference server: one process can host many models, many frameworks, and multi-step ensembles. It is not a hosted model catalog (that is NVIDIA NIM) and not a multi-node LLM disaggregation fabric (that is NVIDIA Dynamo).
The marketing URL lives under Dynamo. The product on that page, the GitHub repo, and the NGC container are still Triton Inference Server.
Key Capabilities
- Multi-framework backends: TensorRT, TensorRT-LLM, PyTorch, ONNX Runtime, OpenVINO, Python, RAPIDS Forest Inference Library / XGBoost / scikit-learn, and custom C++ or Python backends. Models sit in a directory-based model repository with a config.pbtxt.
- Concurrent execution and batching: Several models run at once. Dynamic batching groups independent requests; sequence batching keeps state for stateful models.
- Ensembles and Business Logic Scripting (BLS): A DAG of models and pre-/post-processing steps can run in one request, with CPU and GPU stages mixed.
- KServe v2 protocols: HTTP/REST and gRPC follow the community KServe predict API (Open Inference Protocol family). An in-process C API and Java API exist for embedding Triton on device.
- Model Analyzer and Perf Analyzer: Profile batch size, precision, and instance counts against latency, throughput, and memory before production.
- PyTriton: A Python wrapper that starts Triton for local serving of functions or pipelines.
- Kubernetes and Prometheus: Helm examples and metrics for GPU utilization, throughput, and latency. Cloud ML platforms (Amazon SageMaker, Azure Machine Learning, Google Vertex AI, Oracle) can host Triton as their serving runtime; those platforms stay in Data Science.
Audience & Use Cases
- Audience: ML engineers and platform/SRE teams who already have trained artifacts and need a production inference process they operate, including mixed classical ML and deep learning on the same server.
- Use Case: Multi-model serving on a cluster or edge box; computer-vision and recommender pipelines; LLM serving via the TensorRT-LLM backend when the operator wants a general server rather than a per-model NIM container.
Technical Specifications
- Source / container: github.com/triton-inference-server/server (BSD-3-Clause). NGC image nvcr.io/nvidia/tritonserver (current public line: 2.71.0 / 26.07).
- Hardware: NVIDIA GPUs, x86 and Arm CPUs, AWS Inferentia; Jetson/JetPack for edge.
- Protocols: HTTP/REST and gRPC (KServe v2); optional OpenAI-compatible Python front-end in-tree.
- Licensing: Open source for development. Production support, security updates, and API stability are sold as part of NVIDIA AI Enterprise, which remains a separate suite listing.
Categories & Use Cases
Technical Details
| Mobile Application | No |
|---|
FAQs
What is NVIDIA Triton?
NVIDIA Triton Inference Server (Triton) is open-source AI Model Serving & Inference software. It loads trained models from a model repository, runs them on GPU, CPU, or other accelerators, and returns predictions or generated tokens over HTTP/REST and gRPC. Triton is a general inference server: one process can host many models, many frameworks, and multi-step ensembles. It is not a hosted model catalog (that is NVIDIA NIM) and not a multi-node LLM disaggregation fabric (that is NVIDIA Dynamo).