LeRobot Community Datasets: The “ImageNet” of Robotics — When and How?

Recent advances in Vision-Language-Action (VLA) models have enabled robots to perform a wide range of tasks—from simple commands like “grasp the cube” to more complex activities like folding laundry or cleaning a table. These models aim to achieve generalization : the ability to perform tasks in novel settings, with unseen objects, and in varying conditions.

“The biggest challenge in robotics isn’t dexterity, but generalization—across physical, visual, and semantic levels.” — Physical Intelligence

A robot must "figure out how to correctly perform even a simple task in a new setting or with new objects," and this requires both robust skills and common-sense understanding of the world. Yet, progress is often limited by the availability of diverse data for such robotic systems.

We frame generalization as a data-centric view : treating it as the process of abstracting broader patterns from data—essentially “zooming out” to reveal task-agnostic structures and principles. This shift in perspective emphasizes the role of dataset diversity , rather than model architecture alone, in driving generalization.

While foundation models in vision and language have thrived on massive, web-scale datasets, robotics lacks an “Internet of robots”—a vast, diverse corpus of real-world interactions. Instead, robotic data is fragmented across different embodiments, sensor setups, and control modes, forming isolated data islands .

Within this framework, efficient real-world data collection is indispensable—it anchors learned behaviors in actual robotic hardware and closes the sim-to-real gap , ultimately improving the generalization, adaptability, and performance of robotics foundation models.

By expanding the volume and diversity of real-world datasets , we reduce fragmentation between heterogeneous data sources. When datasets are disjoint in terms of environment, embodiment, or task distribution, models struggle to transfer knowledge across domains.

“Generalization must occur at many levels. At the low level, the robot must understand how to pick up a spoon (by the handle) or plate (by the edge), even if it has not seen these specific spoons or plates before, and even if they are placed in a pile of dirty dishes. At a higher level, the robot must understand the semantics of each task—where to put clothes and shoes (ideally in the laundry hamper or closet, not on the bed), and what kind of tool is appropriate for wiping down a spill. This generalization requires both robust physical skills and a common-sense understanding of the environment, so that the robot can generalize at many levels at the same time, from physical, to visual, to semantic. This is made even harder by the limited availability of diverse data for such robotic systems.” — Physical Intelligence

Different datasets use different action or state dimensions, even for the same robot (e.g., so100 ). Some datasets show inconsistencies in action/state format.

To simplify, the core of generalist policies lies in a simple idea: co-training on heterogeneous datasets . By exposing VLA models to a variety of environments, tasks, and robot embodiments, we can teach models not only how to act, but why —how to interpret a scene, understand a goal, and adapt skills across contexts.

💡 “Generalization is not just a model property—it’s a data phenomenon.” It emerges from the diversity, quality, and abstraction level of the training data.

Can a robot meaningfully respond to a completely novel prompt—say, *"set up a surprise birthday party"*—if it has never encountered anything remotely similar during training? Especially when most datasets are collected in academic labs, by a limited number of people, under well-controlled setups?