Same Cluster, 33 Points More Utilization: What Changed Was the Order

The previous post argued that utilization, not intelligence, is where the next real constraint in enterprise AI is forming, and it closed by noting that no playbook has emerged yet for what a mature GPU Management practice looks like. This is ours.

We built a constraint-aware GPU allocator and benchmarked it against a FIFO scheduler across seven benchmark scenarios. On identical hardware, running identical workloads, GPU utilization rose by as much as 33 percentage points, and priority-weighted output rose in every one of them, by as much as 105%. Nothing about the hardware changed. What changed was the order in which allocation decisions get made.

One note on measurement before the numbers start. Every gain below is expressed as improvement over the FIFO result on the same scenario. Utilization is reported in percentage points; value is reported as a percentage increase in priority-weighted output.

The relative size of those weights is the entire service-level policy, expressed as one number. The real-time penalty weight is 5 to 10 times greater than the allocation weight. One unit of unmet real-time demand therefore costs what 5 to 10 GPU-timesteps of equal-priority batch work costs. The asymmetry is deliberate, and it means latency obligations are enforced inside the same optimization that places batch work, rather than by a separate autoscaler competing with the scheduler for the same GPUs.

The formal model defines what a legal, well-scored allocation looks like. Answering an incoming request is a separate job, and it belongs to a separate component. This is NP-hard combinatorial allocation, and the scheduler is re-invoked every time a job arrives, so the decision has to come back in the gap between two API requests. That latency budget is the fixed constraint the architecture is designed around, which is why a heuristic sits on the hot path and the formal model sits behind it as the specification the heuristic is built to satisfy.

The reservation. Real-time inference cannot wait for capacity; the GPUs have to be there the moment traffic needs them. A scheduler that places jobs in arrival order has no mechanism for releasing GPUs during a trough and reclaiming them before the next peak, so the only way to guarantee availability is to take each real-time application's maximum demand for the day and reserve that many GPUs for the whole day. The cost lands in every hour that is not the peak. An application needing six GPUs at midday and two at 4am holds all six for twenty-four hours, and the four idle GPUs are unavailable to any batch job for the entire day. They are not being used, and they are not free either. It is why the baseline sits near half the cluster in the two scenarios where reservation dominates: 51.6% in the mixed control and 53.6% in the training-heavy case. Roughly half a pool, with much of the idle half reserved rather than free. This cost is paid whether the cluster is contended or not — contention only makes it visible.

It is also what makes the elastic treatment of real-time demand safe. The allocator can hand a GPU to batch work during a trough because underserving real-time demand later is priced so far above whatever that batch work earns — the penalty, not a static reservation, is what protects availability.

"Keep the GPUs busy" is not a decision a system can execute. The decision is narrower and much harder: which GPU runs which job, in which timestep, at what priority. Formally it is one binary choice per combination of GPU, job and timestep, and the output is a grid — every GPU, across the whole scheduling horizon, with a job name in each cell or nothing at all.

Four workload types compete for that grid: training, real-time inference, batch inference, and quantization. They split into two allocation shapes, and the split is where the difficulty lives. Training, batch inference and quantization are batch-like: once started, each needs a contiguous block of GPUs held without interruption until the job finishes. Real-time inference is the opposite: elastic, driven by a demand curve that changes every timestep, growing and shrinking as traffic does.

Two incompatible shapes competing for the same hardware in the same timestep is the core problem. A second heterogeneity sits inside a single type: for the same base model, training jobs range from a few hours to several days, and from one GPU to dozens.