Opening article
OpenAI’s Sora demo sparked a surge in video generation, prompting major companies and open‑source projects to release models such as Google’s Veo2, Runway’s Gen3 Alpha, and LTX‑Video. Hugging Face’s Diffusers team is adding optimizations to make these large models more memory‑efficient and easier to run, while also offering tools for fine‑tuning and training. The post outlines current model capabilities, memory requirements, and upcoming features like Control LoRAs and distillation.
OpenAI’s Sora demo marked a striking advance in AI-generated video last year and gave us a glimpse of the potential capabilities of video generation models. The impact was immediate and since that demo, the video generation space has become increasingly competitive with major players and startups producing their own highly capable models such as Google’s Veo2, Haliluo’s Minimax, Runway’s Gen3 Alpha, Kling, Pika, and Luma Lab’s Dream Machine.
Open-source has also had its own surge of video generation models with CogVideoX, Mochi-1, Hunyuan, Allegro, and LTX Video. Is the video community having its “Stable Diffusion moment”?
This post will provide a brief overview of the state of video generation models, where we are with respect to open video generation models, and how the Diffusers team is planning to support their adoption at scale.
The denoising network itself builds on the DiT architecture developed by William Peebles and Saining Xie , while incorporating various design elements from PixArt .
Memory required by weights can be lowered via - quantization, downcasting to lower dtypes, or by offloading to CPU. Memory required for activations states can also be lowered but that is a more involved process, which is out of the scope of this blog.
With image generation models, we usually only care about the first three aspects. However, for video generation we now have to consider motion quality, coherence and consistency over time, potentially with multiple subjects. Finding the right balance between good data, right inductive priors, and training methodologies to suit these additional requirements has proved to be more challenging than other modalities.
Text-to-video generation models have similar components as their text-to-image counterparts:
The latest generation of video models shares a core feature where the denoising network processes 3D video tokens that capture both spatial and temporal information. The video encoder-decoder system, responsible for producing and decoding these tokens, employs both spatial and temporal compression. While decoding the latents typically demands the most memory, these models offer frame-by-frame decoding options to reduce memory usage.
Efficient video generation expands AI applications in media, entertainment, and communication, while open‑source tools democratize access and accelerate innovation.