Ulysses Sequence Parallelism: Training with Million-Token Contexts

Training large language models on long sequences has become essential for building capable AI systems. As models are increasingly used for tasks like document analysis, code understanding, complex reasoning, and RAG workloads, the need to process sequences of hundreds of thousands—or even millions—of tokens has grown dramatically. To put this in perspective, an average book is roughly 250k tokens, so training on multi-document contexts or book-length inputs requires handling sequences well beyond what fits on a single GPU. However, training with such long contexts presents significant memory challenges: the attention computation scales quadratically with sequence length, quickly exceeding GPU memory for contexts beyond tens of thousands of tokens.

Ulysses Sequence Parallelism (part of the Arctic Long Sequence Training (ALST) protocol from Snowflake AI Research) provides an elegant solution by distributing the attention computation across multiple GPUs through attention head parallelism. In this post, we'll explore how Ulysses works and how it's been integrated across the Hugging Face ecosystem—from Accelerate to the Transformers Trainer and TRL's SFTTrainer.

The attention mechanism in transformers scales quadratically with sequence length. For a sequence of length , standard attention requires FLOPs and memory to compute and store the attention score matrix. Optimized implementations like FlashAttention reduce the memory to by tiling the computation and never materializing the full attention matrix—but the compute remains. For very long sequences (32k+ tokens), even with FlashAttention, training still pushes the limits of single-GPU memory.

If your desired model architecture is supported by Liger-Kernel , it is fully compatible with Ulysses SP and can be enabled with a single flag:

All-to-All Communication : Another all-to-all operation reverses the redistribution, returning to sequence-sharded format.

Traditional data parallelism doesn't help here—each GPU still needs to process the full sequence inside the attention block. We need a way to split the sequence itself across multiple devices.

Ulysses Sequence Parallelism (SP), introduced in the DeepSpeed Ulysses paper , takes a clever approach: in addition to splitting on the sequence dimension, it also partitions the attention heads across GPUs.

Sequence Sharding : The input sequence is split along the sequence dimension across GPUs. Each GPU holds tokens .