A Deepdive into Aya Vision: Advancing the Frontier of Multilingual Multimodality

With the release of the Aya Vision family , our new 8B and 32B parameter vision-language models (VLMs), we are addressing one of the biggest challenges in AI: bringing multilingual performance to multimodal models .

Aya Vision is Cohere For AI 's latest open-weight multilingual and multimodal model family, designed to be a strong foundation for language and vision understanding across 23 languages . It builds on the success of Aya Expanse , state-of-the-art multilingual language models, and extends it using a combination of advanced techniques. These include synthetic annotations, scaling up multilingual data through translation and rephrasing, and multimodal model merging – key methods that improve both language and vision understanding in a multilingual setting.

As a result, our models perform well in a variety of tasks, including image captioning, visual question answering, text generation, and translating both text and images into clear, natural-language text. We evaluated Aya Vision models on a set of datasets, including our new open-ended vision-language benchmark AyaVisionBench and a multilingual version of Wild Vision Bench ( mWildVision ) that is translated into 23 languages, which we release both of them for research.

Our compact and more efficient model Aya Vision 8B achieves the best performance in multilingual multimodal in its parameter class, outperforming leading models such as Qwen2.5-VL 7B, Pixtral 12B, Gemini Flash 1.5 8B, Llama-3.2 11B Vision, Molmo-D 7B, and Pangea 7B by up to 79% win-rates on AyaVisionBench and 81% on mWildBench.

We release both 8B and 32B models as open weights for the research community to further accelerate multilingual multimodal progress. In this blog post, we share the key technical details behind Aya Vision models

We trained Aya Vision models in 2 stages – vision-language alignment and supervised fine-tuning (SFT) . In the vision-language alignment stage, only the vision-language connector is trained, while the vision encoder and the language model weights are kept frozen. This enables rudimentary vision-language understanding by mapping the image encoder features to the language model embedding space. In the SFT stage, we train both the connector and the language model on a diverse set of multimodal tasks in 23 languages.

Download weights and datasets from the Aya Vision collection on Hugging Face. Try Aya Vision using our Hugging Face Space or text it on Whatsapp Build on Aya using our colab example .

One of the biggest challenges in developing a multilingual vision-language model is ensuring strong performance across underrepresented languages. To address this, we first gather synthetic annotations using a diverse pool of high-quality datasets in English, which lay the basis for our multilingual multimodal annotation. Following the synthetic annotations of English datasets, we translated a large volume of the data into 23 languages. To avoid translation artefacts and maintain fluent textual characteristics with high precision in answers, we then rephrased translated prompt/generation pairs by matching them with the original high-quality synthetic samples, expanding language coverage where real-world datasets are scarce. This improves both linguistic fluency and alignment between vision and text, allowing Aya Vision to exhibit superior image understanding in multiple languages.

Together with Aya Vision models, we also release a high-quality multilingual vision-language benchmark called AyaVisionBench , constructed based on real-world applications, covering 23 languages and 9 distinct task categories, with 135 image-question pairs per language.

To create this dataset, we first selected images from the Cauldron held-out test set, a large collection derived from 50 high-quality datasets, ensuring they had not been seen during training. For each image, we then generated a corresponding question that explicitly required visual context for an answer. These questions were synthetically generated and subsequently refined through a two-stage verification process. First, human annotators reviewed and validated each question to ensure it was clear, relevant, and truly dependent on the image. This rigorous selection and validation process ensures that the dataset serves as a robust benchmark for evaluating vision-language models in multilingual and real-world settings.

Aya Vision has a wide range of practical applications, where one notable example is its availability on WhatsApp , one of the most broadly used communications platforms in the world. This allows a massive audience of global citizens who speak a multitude of languages to utilize the capabilities of Aya Vision on a platform they use to communicate every single day.

In pair-wise comparison, Aya Vision 32B outperforms models more than 2x of its size, such as Llama-3.2 90B Vision, Molmo 72B, and Qwen2.5-VL 72B by win rates ranging from 50% to 64% on AyaVisionBench and 52% to 72% on mWildVision average across 23 languages.

For a high-performance vision-language model, it is important to process images with a rbitrary resolutions , especially high-resolution images. To enable this capability in Aya Vision, we dynamically resize and split any higher-resolution images into multiple tiles to generate rich image features from the image encoder. In Aya Vision models, we use the recently released SigLIP2-patch14-384 model as the initialization for the vision encoder.

While dynamic resizing enables processing high-resolution images, it also leads to a larger number of image tokens passing through the vision-language connector and LLM decoder. To improve latency and throughput, we use a downsampling method called Pixel Shuffle , to compress the number of image tokens by 4x. After downsampling, image tokens are aligned to the language model input embeddings through a vision-language connector and passed to an LLM decoder.