NVIDIA Triton

NVIDIA Triton Inference Server (Triton) is open-source AI Model Serving & Inference software. It loads trained models from a model repository, runs them on GPU, CPU, or other accelerators, and returns predictions or generated tokens over HTTP/REST and gRPC. Triton is a general inference server: one process can host many models, many frameworks, and multi-step ensembles. It is not a hosted model catalog (that is NVIDIA NIM) and not a multi-node LLM disaggregation fabric (that is NVIDIA Dynamo).