Multimodal Embedding & Reranker Models with Sentence Transformers

Sentence Transformers v5.4 expands the library to support multimodal embedding and reranker models that encode and compare text, images, audio, and video through a unified API. Multimodal embedding models map inputs from different modalities into a shared embedding space, allowing similarity calculations between, for example, a text query and an image document. Multimodal rerankers score relevance for pairs where one or both elements may be images, text, audio, or video, typically outperforming embeddings in quality at the cost of slower, pair‑wise processing. The update introduces encode_query() and encode_document() wrappers that automatically apply model‑specific prompts for retrieval tasks, while rank() and predict() methods provide relevance scoring for lists of documents. Models detect supported modalities automatically, and users can control image resolution or precision via processor_kwargs and model_kwargs. VLM‑based models such as Qwen3‑VL‑2B require GPUs with at least ~8 GB VRAM, whereas CLIP models run more efficiently on CPUs. The library also offers companion guides for training and fine‑tuning multimodal models on custom data. These capabilities enable new use cases such as visual document retrieval, cross‑modal search, and multimodal RAG pipelines, broadening the applicability of Sentence Transformers across diverse AI workloads.