Vision Language Models (Better, faster, stronger)

Vision Language Models (VLMs) are the talk of the town. In a previous blog post (from April 2024 ), we talked a lot about VLMs. A major chunk was about LLaVA , the first successful and easily reproducible open-source vision language model, along with tips on how to discover, evaluate, and fine-tune open models.

Since then, so much has changed. Models have become smaller yet more powerful . We've seen the rise of new architectures and capabilities (reasoning, agency, long video understanding, etc.). In parallel, entirely new paradigms, such as multimodal Retrieval Augmented Generation (RAG) and multimodal agents have taken shape.

In this blog post, we’ll take a look back and unpack everything that happened with vision language models the past year. You’ll discover key changes, emerging trends, and notable developments.

The latest and most capable any-to-any model, Qwen 2.5 Omni (figure below) is a good example to understand the architecture of an any-to-any model.

Qwen2.5-Omni employs a novel "Thinker-Talker" architecture, where the "Thinker" handles text generation, and the "Talker" produces natural speech responses in a streaming manner. MiniCPM-o 2.6 , an 8B parameter multimodal model is capable of understanding and generating content across vision, speech, and language modalities. Janus-Pro-7B , introduced by DeepSeek AI, is a unified multimodal model that excels in both understanding and generating content across modalities. It features a decoupled visual encoding architecture, separating the processes for understanding and generation.

This year there’s another player, Kimi-VL-A3B-Thinking by the Moonshot AI team. It consists of MoonViT (SigLIP-so-400M) as the image encoder and a Mixture-of-Experts (MoE) decoder with 16B total parameters and only 2.8B active parameters. The model is a long chain-of-thought fine-tuned and further aligned (reinforcement learning) version of the Kimi-VL base vision language model. You can try the model here .

The community used to scale intelligence through the number of parameters, and then high-quality synthetic data. After a certain point, the benchmarks saturated and scaling models had diminishing returns. The community went to shrink larger models through various methods, like distillation. This makes sense because it reduces compute costs, simplifies deployment, and unlocks use cases like local execution, enhancing data privacy.

Great examples of VLAs are π0 and π0-FAST, the first robotics foundation models by Physical Intelligence, ported to Hugging Face’s LeRobot library. These models are trained across 7 robotics platforms and 68 unique tasks. They show strong zero-shot and fine-tuned performance on complex, real-world activities such as laundry folding, table bussing, grocery bagging, box assembly, and object retrieval.

You can get started with using gemma-3-4b-it model in GGUF format with llama.cpp through CLI with this one-liner:

GR00T N1 is NVIDIA’s open VLA foundation model for generalist humanoid robots. It understands images and language, and turns them into actions, like moving its arms or following instructions, thanks to a system that combines smart reasoning with real-time movement control. GR00T N1 also builds on the LeRobot dataset format, the open standard created to simplify sharing and training on robot demonstrations.

We highly recommend reading the first blog post if you want a good primer on how vision language models work.

In this section, we will look at the new types of VLMs. While some are absolutely new, others are improved versions of previous research.

Any-to-any models, as the name suggests, are models that can take in any modality and output any modality (image, text, audio). They do it by aligning the modalities, where an input from one modality can be translated to another (e.g. the word “dog” would be associated with an image of a dog, or with the utterance of the word).