Hugging Face has introduced NeoMME, a family of 260M and 800M multilingual multimodal encoders designed to process text tokens and raw image patches through a single bidirectional Transformer. Unlike many generative visual language models that rely on separately pretrained vision towers and causal language decoders, NeoMME is trained entirely from scratch using a masked discrete-diffusion objective. Pretraining mixes multilingual text, code, mathematics, natural images, and document images across a relatively compact token budget supported by the NorMuon optimizer. To demonstrate the backbone's utility, the developers fine-tuned NeoMME for visual document retrieval using ColPali's page-image approach. NeoMME-Retriever generates both dense and late-interaction embeddings in a single forward pass, bypassing traditional OCR preprocessing steps to preserve visual layouts, charts, and tables. Evaluations on the ViDoRe v3 benchmark place both the 260M and 800M model sizes on the Pareto frontier for nDCG@10 and model size, with the 260M variant achieving top scores among evaluated models below 800M parameters. Tested at a 2048x2048 image input size on an NVIDIA L40S GPU, the 260M model encodes approximately 51 pages per second, yielding nearly twice the throughput of ColModernVBERT. Furthermore, combining hierarchical token pooling and asymmetric quantization reduces late-interaction index storage from roughly 1.5 MB down to 6 kB per page while retaining over 95 percent of baseline nDCG@10 accuracy. All model checkpoints are released under the Apache 2.0 license on Hugging Face, alongside support for Sentence Transformers v6 integration.
NeoMME: an efficient Multimodal-native and Multilingual Encoder
Hugging Face introduces NeoMME, a family of 260M and 800M multilingual multimodal encoders that process text tokens and raw image patches using a single bidirectional Transformer. Trained from scratch with a masked discrete-diffusion objective, NeoMME avoids the parameter and compute overhead of separate vision towers and causal language models. Fine-tuned for visual document retrieval, NeoMME-Retriever delivers high performance, strong throughput, and efficient storage via hierarchical token pooling and asymmetric quantization.
Why it matters
NeoMME demonstrates that efficient multimodal encoders can bypass the parameter and compute overhead of traditional generative visual language models while maintaining high retrieval performance. Its strong throughput and aggressive index compression techniques offer significant practical advantages for large-scale document processing and vector storage.