Opening article
When operating with latent-space diffusion models for high-resolution image and video synthesis, the VAE decoder can consume quite a bit more memory. This makes it hard for the users to run these models on consumer GPUs without going through latency sacrifices and others alike. Therefore, we want to pilot an idea with the community — delegating the decoding process to a remote endpoint. Below, we cover three use cases where we think this remote VAE inference would be beneficial.
When operating with latent-space diffusion models for high-resolution image and video synthesis, the VAE decoder can consume quite a bit more memory. This makes it hard for the users to run these models on consumer GPUs without going through latency sacrifices and others alike.
For example, with offloading, there is a device transfer overhead, causing delays in the overall inference latency. Tiling is another solution that lets us operate on so-called “tiles” of inputs. However, it can have a negative impact on the quality of the final image.
Therefore, we want to pilot an idea with the community — delegating the decoding process to a remote endpoint.
Below, we cover three use cases where we think this remote VAE inference would be beneficial.
No data is stored or tracked, and code is open source. We made some changes to huggingface-inference-toolkit and use custom handlers .
Install diffusers from main to run the code. pip install git+https://github.com/huggingface/diffusers@main
Usage for Flux is slightly different. Flux latents are packed so we need to send the height and width .
Remote VAEs for decoding with Inference Endpoints 🤗. Covered by Hugging Face on February 24, 2025.