Training and Finetuning Sparse Embedding Models with Sentence Transformers

Sentence Transformers is a Python library for using and training dense embedding, reranker (cross encoder), and sparse embedding models for a wide range of applications, such as retrieval augmented generation, semantic search, semantic textual similarity, paraphrase mining, and more. In this blogpost, I'll show you how to use it to finetune a sparse encoder/embedding model and explain why you might want to do so. This results in sparse-encoder/example-inference-free-splade-distilbert-base-uncased-nq , a cheap model that works especially well in hybrid search or retrieve and rerank scenarios.

Finetuning sparse embedding models involves several components: the model, datasets, loss functions, training arguments, evaluators, and the trainer class. I'll have a look at each of these components, accompanied by practical examples of how they can be used for finetuning strong sparse embedding models.

In addition to training your own models, you can choose from a wide range of pretrained sparse encoders available on the Hugging Face Hub . To help navigate this growing space, we’ve curated a SPLADE Models collection highlighting some of the most relevant models. We list the most prominent ones along with their benchmark results in Pretrained Models in the documentation.

Training Sentence Transformer models involves the following components:

Splade models use the MLMTransformer followed by a SpladePooling modules. The former loads a pretrained Masked Language Modeling transformer model (e.g. BERT , RoBERTa , DistilBERT , ModernBERT , etc.) and the latter pools the output of the MLMHead to produce a single sparse embedding of the size of the vocabulary.

This architecture is the default if you provide a fill-mask model architecture to SparseEncoder, so it's easier to use the shortcut:

Inference-free Splade uses a Router module with different modules for queries and documents. Usually for this type of architecture, the documents part is a traditional Splade architecture (a MLMTransformer followed by a SpladePooling module) and the query part is an SparseStaticEmbedding module, which just returns a pre-computed score for every token in the query.

It's crucial to ensure that your dataset format matches your chosen loss function . This involves checking two things:

The broader term "embedding models" refer to models that convert some input, usually text, into a vector representation (embedding) that captures the semantic meaning of the input. Unlike with the raw inputs, you can perform mathematical operations on these embeddings, resulting in similarity scores that can be used for various tasks, such as search, clustering, or classification.

With dense embedding models, i.e. the common variety, the embeddings are typically low-dimensional vectors (e.g., 384, 768, or 1024 dimensions) where most values are non-zero. Sparse embedding models, on the other hand, produce high-dimensional vectors (e.g., 30,000+ dimensions) where most values are zero. Usually, each active dimension (i.e. the dimension with a non-zero value) in a sparse embedding corresponds to a specific token in the model's vocabulary, allowing for interpretability.

Let's have a look at naver/splade-v3 , a state-of-the-art sparse embedding model, as an example: