According to Hugging Face, the transformers library vLLM modeling backend is now as fast or even faster than custom vLLM implementations across many language model architectures. Model authors can automatically leverage their transformers implementations to obtain ultra-fast vLLM inference without writing custom ports. Last year, transformers was integrated as a modeling backend in vLLM, allowing LLMs and VLMs to run inside vLLM by utilizing transformers for modeling code and vLLM for optimized inference techniques like continuous batching and custom attention kernels. While previous iterations focused on attention as the bottleneck, the latest update applies dynamic inference-specific layer fusions at runtime to match custom code performance. The updated backend uses torch.fx for static analysis of the model graph to search for optimizable patterns. It then uses abstract syntax tree manipulation to rewrite operations in place. Consequently, compatible models achieve native vLLM inference speeds using a single flag without requiring additional optimization code. Models using linear attention and custom models with code living in Hub repositories are not currently supported.
Native-speed vLLM transformers modeling backend
The Hugging Face transformers library now features an updated vLLM modeling backend that matches or exceeds the speed of custom vLLM implementations for many LLM architectures. By using torch.fx for static analysis and abstract syntax trees to apply runtime layer fusions, model authors can achieve native-speed vLLM inference automatically. This capability allows users to run transformers models inside vLLM using a single flag without needing manual porting.
Why it matters
This development eliminates the need for model authors to write custom inference ports to achieve maximum performance, significantly streamlining the deployment of new LLM and VLM architectures.