Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face

Intel and Hugging Face collaborated to benchmark the OpenAI GPT OSS large language model on Google Cloud’s new C4 virtual machine, which runs on Intel Xeon 6 processors codenamed Granite Rapids. The controlled, repeatable generation workload isolated architectural differences between the C4 VM and the previous‑generation C3 VM, which uses 4th‑generation Intel Xeon processors. The benchmark focused on steady‑state decoding per‑token latency and end‑to‑end normalized throughput with increasing batch size while keeping sequence lengths fixed, using static KV cache and SDPA attention for determinism. Across batch sizes up to 64, the C4 VM consistently outperformed the C3 VM, delivering 1.4× to 1.7× higher throughput per vCPU. At batch size 64, the C4 VM achieved 1.7× the per‑vCPU throughput of the C3, and with near‑parity in hourly price per vCPU, this translates to a 1.7× total cost of ownership advantage. The performance gains are attributed to an expert execution optimization that eliminates redundant computation by routing each expert only to the tokens it is assigned, reducing FLOPs waste and improving utilization. These results underline that, thanks to targeted framework optimizations from Intel and Hugging Face, large mixture‑of‑experts models can be efficiently served on next‑generation general‑purpose CPUs, offering both higher throughput and lower cost compared to previous‑generation hardware.