vLLM V0 to V1: Correctness Before Corrections in RL
PipelineRL migrated its inference engine from vLLM V0 to V1 for rollout generation in online reinforcement learning. The transition revealed train-inference mismatches caused by logprob differences and runtime defaults. Fixing backend correctness first restored parity with the V0 reference before pursuing objective-side corrections.
The Core Development
PipelineRL uses vLLM as the inference engine for rollout generation. The inference engine samples tokens and returns token logprobs; the trainer uses those logprobs to compute policy ratios, KL, clip rate, entropy, and reward. Any discrepancy in how those logprobs are computed can change the training dynamics. This is the train-inference mismatch we needed to eliminate during the vLLM V0 to V1 migration.
TL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 lm_head used for the final projection. We fixed the backend behavior before changing the RL objective.
The reference run used vLLM 0.8.5 ; the V1 runs used vLLM 0.18.1 . Figure 1 shows the final result. The red run is the initial V1 attempt, and the green run is the final V1 run after the fixes described below.
Strategic Perspective & Ecosystem Impact
vLLM V1 is a substantial rewrite of the V0 engine. Our migration target was therefore deliberately narrow:
Those metrics came from a GSPO training run, the objective used for this experiment. The same class of mismatch can surface in PPO, GRPO, or any online RL system that treats rollout-side logprobs as part of the optimization target.
The initial V1 run showed the problem clearly. The trainer-side logprobs and reward moved away from the V0 reference early in training.
Why it matters
Train-inference mismatches in reinforcement learning can quietly alter training dynamics and rewards. Highlighting backend correctness over premature objective adjustments offers a reliable migration blueprint for large-scale AI training systems.