Opening article
Sentence Transformers is a Python library that enables training and finetuning of dense, reranker, and sparse embedding models for tasks such as retrieval‑augmented generation, semantic search, and paraphrase mining. The blog post explains how to finetune a sparse encoder, resulting in a lightweight model that performs well in hybrid search or retrieve‑and‑rerank scenarios. It details the components involved—model, datasets, loss functions, training arguments, evaluators, and trainer class—and provides practical examples. The post also highlights a curated SPLADE Models collection on the Hugging Face Hub and discusses the advantages of neural sparse embeddings, including query expansion and interpretability.
Sentence Transformers is a Python library for using and training dense embedding, reranker (cross encoder), and sparse embedding models for a wide range of applications, such as retrieval augmented generation, semantic search, semantic textual similarity, paraphrase mining, and more. In this blogpost, I'll show you how to use it to finetune a sparse encoder/embedding model and explain why you might want to do so. This results in sparse-encoder/example-inference-free-splade-distilbert-base-uncased-nq , a cheap model that works especially well in hybrid search or retrieve and rerank scenarios.
Finetuning sparse embedding models involves several components: the model, datasets, loss functions, training arguments, evaluators, and the trainer class. I'll have a look at each of these components, accompanied by practical examples of how they can be used for finetuning strong sparse embedding models.
In addition to training your own models, you can choose from a wide range of pretrained sparse encoders available on the Hugging Face Hub . To help navigate this growing space, we’ve curated a SPLADE Models collection highlighting some of the most relevant models. We list the most prominent ones along with their benchmark results in Pretrained Models in the documentation.
Training Sentence Transformer models involves the following components:
Splade models use the MLMTransformer followed by a SpladePooling modules. The former loads a pretrained Masked Language Modeling transformer model (e.g. BERT , RoBERTa , DistilBERT , ModernBERT , etc.) and the latter pools the output of the MLMHead to produce a single sparse embedding of the size of the vocabulary.
This architecture is the default if you provide a fill-mask model architecture to SparseEncoder, so it's easier to use the shortcut:
Inference-free Splade uses a Router module with different modules for queries and documents. Usually for this type of architecture, the documents part is a traditional Splade architecture (a MLMTransformer followed by a SpladePooling module) and the query part is an SparseStaticEmbedding module, which just returns a pre-computed score for every token in the query.
It's crucial to ensure that your dataset format matches your chosen loss function . This involves checking two things:
The broader term "embedding models" refer to models that convert some input, usually text, into a vector representation (embedding) that captures the semantic meaning of the input. Unlike with the raw inputs, you can perform mathematical operations on these embeddings, resulting in similarity scores that can be used for various tasks, such as search, clustering, or classification.
With dense embedding models, i.e. the common variety, the embeddings are typically low-dimensional vectors (e.g., 384, 768, or 1024 dimensions) where most values are non-zero. Sparse embedding models, on the other hand, produce high-dimensional vectors (e.g., 30,000+ dimensions) where most values are zero. Usually, each active dimension (i.e. the dimension with a non-zero value) in a sparse embedding corresponds to a specific token in the model's vocabulary, allowing for interpretability.
Let's have a look at naver/splade-v3 , a state-of-the-art sparse embedding model, as an example:
Efficient sparse models enable faster, more interpretable search and retrieval in large‑scale AI systems, bridging the gap between lexical and dense approaches.