Opening article
The Hugging Face blog details a recipe that speeds up LoRA inference for the Flux.1‑Dev text‑to‑image model by up to 2.3× on high‑end GPUs and 2.0× on consumer RTX 4090 cards. The approach combines torch.compile, Flash Attention 3, FP8 quantization of the Flux transformer, and NF4 quantization of the T5 text encoder, while enabling hotswapping of LoRAs without recompilation. The post includes code snippets and performance tables showing the trade‑offs between memory usage, latency, and quality.
LoRA adapters provide a great deal of customization for models of all shapes and sizes. When it comes to image generation, they can empower the models with different styles, different characters, and much more . Sometimes, they can also be leveraged to reduce inference latency . Hence, their importance is paramount, particularly when it comes to customizing and fine-tuning models.
In this post, we take the Flux.1-Dev model for text-to-image generation because of its widespread popularity and adoption, and how to optimize its inference speed when using LoRAs (~2.3x). It has over 30k adapters trained with it ( as reported on the Hugging Face Hub platform). Therefore, its importance to the community is significant.
Note that even though we demonstrate speedups with Flux, our belief is that our recipe is generic enough to be applied to other models as well.
When serving LoRAs, it is common to hotswap (swap in and swap out different LoRAs) them. A LoRA changes the base model architecture. Additionally, LoRAs can be different from one another – each one of them could have varying ranks and different layers they target for adaptation. To account for these dynamic properties of LoRAs, we must take necessary steps to ensure the optimizations we apply are robust.
One can also fuse the LoRA parameters into the base model parameters, run compilation, and unfuse the LoRA parameters when loading new ones. However, this approach will again encounter the problem of recompilation whenever inference is run, due to potential architecture-level changes.
The problems start surfacing when we try to swap in and swap out LoRAs into a compiled diffusion transformer ( pipe.transformer ) without triggering recompilation.
Normally, loading and unloading LoRAs will require recompilation, which defeats any speed advantage gained from compilation. Thankfully, there is a way to avoid the need for recompilation. By passing hotswap=True , diffusers will leave the model architecture unchanged and only exchange the weights of the LoRA adapter itself, which does not necessitate recompilation.
The benefits of this workflow become evident when we look at the inference latency without using compilation with hotswapping.
Hopefully, the recipes from this post will inspire you to optimize your LoRA-based use cases, benefitting from speedy inference.
If you cannot wait to get started with the code, please check out the accompanying code repository .
For example, we can apply torch.compile on a model loaded with a particular LoRA to obtain speedups on inference latency. However, the moment we swap out the LoRA with a different one (with a potentially different configuration), we will run into recompilation issues, causing slowdowns in inference.
Our optimization recipe takes into account the above-mentioned situations to be as realistic as possible. Below are the key components of our optimization recipe:
Fast LoRA inference for Flux with Diffusers and PEFT. Covered by Hugging Face on July 23, 2025.