NVIDIA NIM
What is NVIDIA NIM?
NVIDIA NIM (NVIDIA Inference Microservices) is AI Model Serving & Inference software. Each NIM is a GPU-accelerated container that loads a specific trained model, runs an optimized inference engine, and exposes an HTTP API so applications can obtain predictions or generated tokens. Operators can call NVIDIA-hosted NIM endpoints for prototyping, or pull the same microservices from NVIDIA GPU Cloud (NGC) and run them on their own NVIDIA GPUs.
Key Capabilities
- Per-model inference containers: A NIM image packages one model or model family (for example meta/llama-3.1-8b-instruct) with its runtime. The container downloads weights from NGC into a local cache and serves them.
- Engine selection: On supported NVIDIA GPUs, large language model (LLM) NIMs run a TensorRT-LLM engine. On other NVIDIA GPUs they fall back to vLLM. Other NIMs use TensorRT, SGLang, or NVIDIA Triton depending on the domain.
- OpenAI-compatible LLM API: Language NIMs implement /v1/chat/completions and related OpenAI-spec routes, so existing OpenAI-style clients can point at a NIM base_url.
- Self-host or hosted: The same microservice is available as an NVIDIA-hosted API (NVIDIA API Catalog / build.nvidia.com) and as a downloadable container for workstations, data centers, air-gapped clusters, or cloud virtual machines with NVIDIA GPUs. Hugging Face and other partners also offer dedicated NIM endpoints.
- Kubernetes operations: Helm charts and the NIM Operator deploy and scale NIMs on Kubernetes. Observability metrics are exported for dashboards.
- Catalog breadth: NIMs are not limited to chat LLMs. The same packaging covers embeddings and rerankers, vision-language models, speech (Riva), visual generation, medical imaging, and BioNeMo scientific models.
- Fine-tuned weights: A NIM can load customer-tuned weights (including LoRA adapters) into a locally built TensorRT-LLM engine rather than only the catalog checkpoint.
Audience & Use Cases
- Audience: ML engineers, platform and site reliability (SRE) teams, and application developers who need a production inference endpoint on NVIDIA GPUs without assembling TensorRT-LLM, Triton, and an API layer by hand.
- Use Case: Private or on-premises LLM and multimodal inference; swapping a public model API for a base_url on owned GPUs; packing catalog or fine-tuned models into a supported, versioned container.
Technical Specifications
- Runtime: Docker/OCI containers from NGC; optional NIM Operator on Kubernetes; WSL2 and RTX workstation paths exist for local development.
- Hardware: NVIDIA GPUs with enough device memory for the chosen model. Some model/GPU pairs ship a prebuilt TensorRT engine; others run vLLM.
- Protocols: REST (OpenAI-spec for LLM NIMs); domain-specific APIs for speech, vision, and scientific NIMs.
- Distribution: NVIDIA API Catalog for hosted trial; NGC for self-host images.
- Licensing: NVIDIA Developer Program membership covers hosted APIs and downloadable NIMs for research, development, and testing. Production self-host and enterprise support require an NVIDIA AI Enterprise subscription (Feature Branch vs Production Branch, including CVE handling on Certified images). NIM is a serving SKU under that license, not a synonym for the full AI Enterprise suite.
Categories & Use Cases
Technical Details
| Mobile Application | No |
|---|
FAQs
What is NVIDIA NIM?
NVIDIA NIM (NVIDIA Inference Microservices) is AI Model Serving & Inference software. Each NIM is a GPU-accelerated container that loads a specific trained model, runs an optimized inference engine, and exposes an HTTP API so applications can obtain predictions or generated tokens. Operators can call NVIDIA-hosted NIM endpoints for prototyping, or pull the same microservices from NVIDIA GPU Cloud (NGC) and run them on their own NVIDIA GPUs.