Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

Sentence Transformers, a Python library for embedding and reranker models, now offers full support for multimodal training and finetuning. The library’s Trainer can be used with datasets that include images, audio, or video alongside text, automatically preprocessing each modality through the model’s processor. A practical demonstration finetunes the Qwen/Qwen3‑VL‑Embedding‑2B model on a Visual Document Retrieval (VDR) task, where the goal is to retrieve relevant document screenshots from a large corpus based on a text query. Using the llamaindex‑vdr‑en‑train‑preprocessed dataset, which contains 53,512 English query‑image pairs, the finetuned model achieves an NDCG@10 of 0.947 compared to the base model’s 0.888, outperforming all tested VDR models, including those up to four times larger. The training pipeline employs CachedMultipleNegativesRankingLoss, which uses hard negatives to push query‑positive similarity up and query‑negative similarity down, and MatryoshkaLoss to ensure good performance across different embedding dimensionalities. The framework also supports Router modules for combining separate encoders per modality, allowing lightweight, specialized encoders to be used while aligning their embedding spaces during training. This approach enables practitioners to adapt large vision‑language models to domain‑specific tasks efficiently, achieving state‑of‑the‑art performance without increasing model size.