Opening article
Hugging Face has introduced a co-located vLLM feature in TRL v0.18.0 to improve GPU efficiency during online learning algorithms like GRPO. Previously, vLLM ran as a separate server process on different GPUs, causing idle time and communication overhead. By sharing the same GPUs for both training and inference within a single process group, the new setup eliminates HTTP communication and enhances throughput.
TRL supports training LLMs using GRPO, an online learning algorithm recently introduced in the DeepSeekMath paper . In GRPO, the model learns from its own outputs: it generates responses during training, receives feedback, and uses that feedback to improve itself over time.
This makes generation a critical step in the training loop — and also a major bottleneck. To speed up generation, TRL integrates with vLLM. This combination lets you train powerful models more efficiently in GRPO setup. However, there’s a catch.
Before TRL v0.18.0, vLLM was only supported in server mode , running as a separate process on different GPUs from the training job. It communicated with the training script over HTTP, which made the setup modular and easy to use — but also introduced GPU inefficiencies.
Note: Depending on the model size and the overall GPU memory requirements for training, you may need to adjust the vllm_gpu_memory_utilization parameter in GRPOConfig to avoid underutilization or out-of-memory errors.
Level 1 : Unloads model weights from GPU (keeps them in CPU memory) and clears the KV cache. Useful when the same model will be reused soon.
Level 2 : Unloads both model weights and KV cache entirely. Best for scenarios where the model will change or won’t be reused right away.
Level 2 Sleep Buffer Bug. Initially, level 2 sleep didn’t work correctly when we tried to reload weights using load_weights. This issue was tracked in Issue #16564 . The problem was that model buffers (like running mean/var in BatchNorm) weren’t restored after waking up from sleep. The fix came with PR #16889 , which added logic to explicitly restore buffers when waking up from level 2 sleep. We now keep a copy of the original buffers and manually reapply them after loading new weights.
Previously, this wasn’t possible in TRL, which relied on vLLM as an external HTTP server. That changed with our PR #3394 , which added support for vLLM’s external launcher and true integration into the training process.
TP and DP Support : Compatible with Tensor Parallelism and Data Parallelism, making it suitable for large-scale training runs.
When using co-located training, managing GPU memory is crucial so that both training and inference can run smoothly on the same devices. To support this, we added vLLM’s sleep() API into the GRPO training loop.
In online learning methods like GRPO — where generation happens constantly — this inefficiency becomes even more painful. You spend more on hardware, but don't get the performance you'd expect.
So, the key question becomes: Can we share the same GPUs for both training and generation, instead of separating them?
The main issue was that training and inference ran on separate GPUs, leading to idle time and underutilization. The natural solution? Run both on the same GPUs. Instead of having vLLM operate as a standalone server in its own process and devices, what if vLLM could run alongside the training code, within the same distributed process group? This would let us launch a single distributed job where training and inference share the same devices, switching between tasks efficiently without wasting resources.
Co-locating training and inference optimizes hardware utilization in resource-intensive online learning loops like GRPO. This approach reduces infrastructure costs and enables faster training for large-scale models without sacrificing performance.