Hugging Face has detailed an asynchronous training setup using TRL's AsyncGRPOTrainer and LoRA across Hugging Face Jobs without relying on NCCL or shared node clusters. Because LoRA adapters for models like a 1.5B variant are only a few megabytes compared to gigabyte-scale full models, training and inference can be completely decoupled. According to the post, the setup utilizes storage buckets provided by Hugging Face Jobs and mounted as a FUSE filesystem via hf-mount. Every few optimizer steps, the trainer saves the adapter directory, performs an atomic rename, and sends the path to vLLM's load lora adapter endpoint. This avoids the heavy communication overhead of full-weight synchronization across machines that cannot form a traditional cluster. To maintain training stability and prevent inference issues, the configuration uses versioned adapter names to ensure that prefix caching does not accidentally mix weights between rollout versions. Additionally, a proxy is deployed between the trainer and the vLLM jobs to handle authorization headers and coordinate multiple generation replicas, successfully enabling robust reinforcement learning workloads on ephemeral virtual machine infrastructure.
Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL
Hugging Face explains how to run asynchronous GRPO training with LoRA across separate Hugging Face Jobs without requiring a shared cluster or NCCL. By leveraging storage buckets mounted as FUSE filesystems, trainers and vLLM inference workers can synchronize lightweight LoRA adapters instead of massive full model weights.
Why it matters
This approach demonstrates that complex reinforcement learning loops like GRPO can be run asynchronously across ephemeral, single-node cloud jobs without the need for traditional dense cluster setups or NCCL network paths, drastically simplifying distributed RL infrastructure.