Opening article
The Language Technologies Lab has released Visual Salamandra, a 7‑billion‑parameter multimodal model that extends the Salamandra LLM to images and video. Built on the Salamandra Instructed 7B and Google’s SigLIP encoder, it uses a late‑fusion architecture and a four‑phase training pipeline to align visual and textual modalities. The model is trained on 6.1 million instruction‑tuning instances, including 842,000 text‑only samples, and emphasizes multilingual support, especially for European languages. Visual Salamandra offers capabilities such as VQA, OCR, document understanding, mathematical reasoning, and instruction‑based image interaction, while also supporting video inputs for future applications.
The Language Technologies Lab takes a major step forward in multimodal artificial intelligence with the release of Visual Salamandra , extending the capabilities of the Salamandra large language model (LLM) to both images and video. Visual Salamandra is based on the 7 billion parameters foundational model maintaining its compactness and efficiency while extending it to multimodal tasks.
Designed with vision-language alignment at its core, Visual Salamandra builds on top of the Salamandra Instructed 7B model by integrating Google’s SigLIP encoder (SigLIP-So400m), a 2-layer MLP projector, and advanced late-fusion techniques to bridge the gap between visual and textual modalities.
The resulting architecture enables Visual Salamandra to comprehend and generate contextually accurate responses from diverse inputs, ranging from single and multiple images and videos to purely textual instructions. This development reflects a broader commitment by the Lab to support robust, multilingual, and multimodal AI systems—especially those that prioritize European linguistic diversity.
To adapt Salamandra for visual inputs, the Lab implemented a four-phase training process centered on late-fusion architecture. In this setup, a pre-trained image encoder (SigLIP, 14 patches at 384x384 resolution) generates image embeddings, which are then aligned with the LLM via a custom-trained multilayer perceptron (MLP) projector.
Visual Salamandra is released under a Apache License, Version 2.0 , allowing for research and non-commercial use.
Data diversity played a crucial role throughout training. A total of 6.1 million instruction-tuning instances were used, including 842,000 text-only samples. The training corpus featured data from sources like AI2D, Cambrian, and LLaVA Next, chosen to enhance visual grounding, document understanding, mathematical reasoning, and OCR.
By integrating a state‑of‑the‑art vision encoder with a multilingual LLM, Visual Salamandra advances inclusive multimodal AI, enabling robust language‑vision tasks across diverse European languages and setting a benchmark for future multimodal instruction‑tuned models.