π0 and π0-FAST: Vision-Language-Action Models for General Robot Control

We have ported the first robotics foundation models to Hugging Face LeRobot ! Both π0 and π0-FAST , developed by Physical Intelligence, are now available in the LeRobot repository , bringing generalist robotic intelligence to the Hugging Face ecosystem. If you are curious about how Vision-Language-Action (VLA) models differ from Vision-Language Models (VLMs) and how actions are represented, dive into this blog post to find out!

Explore the model collection and the PyTorch Version of the model in our repository: Huggingface collection of Pi0 models | Huggingface collection of Pi0+FAST models | LeRobot repo

Robert Heinlein suggests that a well-rounded person should be capable of handling a wide range of tasks—both intellectual and physical—rather than being narrowly specialized in one field. Drawing a parallel between a well-rounded person and machine intelligence: AI systems vary widely, but human intelligence excels in versatility—adapting to tasks, environments, and surprises. While large language and vision-language models (LLMs, VLMs) show promise, they lack interaction with the physical world. To bridge this gap, we need models trained on robotic data. Generalist robot models can enhance adaptability, using diverse data to improve generalization and robustness. Instead of training on isolated tasks, pre-training on varied robotic data—similar to LLMs—boosts efficiency and performance.

To address these issues, Frequency-space Action Sequence Tokenization (FAST) introduces a novel time-series compression approach using the Discrete Cosine Transform (DCT). FAST reduces redundancy, improves efficiency, and enhances action fidelity.

With this, we present π0-FAST , faster and autoregressive version of π0 also available in Lerobot repo, an extension of π0 , which leverages this new tokenizer for better action representation.

To fine-tune the π0 neural network with PaliGemma and Expert Gemma, which were pretrained using VLM default parameters before π0 fine-tuning, execute:

Since all operations are invertible, actions can be reconstructed efficiently and losslessly from tokens. The tokenization pipeline has only two hyperparameters: the scaling coefficient applied before rounding and the BPE vocabulary size. Both parameters remain robust across different datasets.

π0 is trained on data from 7 robotic platforms and 68 unique tasks , demonstrating strong zero-shot and fine-tuned performance on complex, real-world tasks such as laundry folding, table bussing, grocery bagging, box assembly, and object retrieval .

Unlike standard robotic policies, π0 employs flow matching to produce smooth, real-time action trajectories at 50Hz , making it highly efficient, precise, and adaptable for real-world deployment. Flow matching was used in continuous normalizing flows and improved generation quality in diffusion models. The denoising process π0 used works in the same way, starting with a random noise that progressively converges towards a sequence of motor actions that make sense.

The current implementation runs, and a WIP is to have it support torch.compile and leverage it to the fullest!

Developing generalist robot policies, or robot foundation models, presents three key challenges:

Designing model architectures that can integrate diverse data sources while capturing complex physical interactions. A key challenge in this regard is cross-embodiment training , where a model must learn from diverse robot types with varying configurations, control spaces, and action representations. Existing approaches tackle this by:

Crafting an effective training recipe , as recent advances in NLP and vision have heavily relied on careful pre-training and post-training strategies.