NVIDIA’s new Nemotron 3 Nano Omni is a multimodal large language model that extends the capabilities of its predecessor by adding robust audio and video processing while maintaining strong performance on long documents. The architecture combines a 30B‑parameter language backbone with a C‑RADIOv4‑H vision encoder and a Parakeet‑TDT‑0.6B‑v2 audio encoder, all connected through lightweight projectors into a unified encoder‑projector‑decoder pipeline. The model incorporates 23 Mamba selective state‑space layers for efficient long‑context handling, 23 mixture‑of‑experts layers with 128 experts, and six grouped‑query attention layers to preserve global interaction. For vision, it replaces tiling with dynamic resolution processing, allowing each image to be represented by 1,024 to 13,312 16×16 patches, which is critical for high‑resolution documents and GUI layouts. Video input is handled via a Conv3D tubelet embedding that fuses consecutive frames, halving the token count, and an EVS pruning mechanism that removes redundant static tokens during inference, improving latency and throughput. Audio is sampled at 16 kHz, with training inputs up to 1,200 seconds and a context length supporting over five hours. The model is trained on NVIDIA H100 GPUs using Megatron‑LM, Transformer Engine, and Megatron Energon, with extensive parallelism and online sequence packing. Reinforcement learning stages use NeMo‑RL and NeMo Gym on Ray‑based clusters, and the model is further fine‑tuned with synthetic QA pairs generated from real PDFs, totaling 11.4 million pairs (~45 B tokens). Nemotron 3 Nano Omni is positioned for tasks such as OCR‑heavy document analysis, speech transcription, and joint audio‑visual reasoning, offering significant efficiency gains over other open‑weights omni models in multi‑document and video workloads.
Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents
NVIDIA has released the Nemotron 3 Nano Omni, a multimodal large language model that supports long documents, audio, and video inputs. The model builds on the Nemotron Nano V2 VL architecture, adding significant visual improvements and new audio/video capabilities while outperforming other open‑weights omni models such as Qwen3‑Omni in many domains. It is designed for high‑value, long‑form documents, speech transcription, and joint audio‑visual reasoning, and it can handle documents over 100 pages and audio up to 20 minutes with a context length of more than five hours.
Why it matters
The model’s ability to process long, multimodal contexts efficiently positions it as a powerful tool for enterprise AI workflows, enabling more accurate document analysis, speech transcription, and cross‑modal reasoning at scale.