LeRobot goes to driving school: World’s largest open-source self-driving dataset

State-of-the art Vision Language Models and Large Language Models are trained on open-source image-text corpora sourced from the internet, which spearheaded the recent acceleration of open-source AI. Despite these breakthroughs, the adoption of end-to-end AI within the robotics and automotive community remains low, primarily due to a lack of high quality, large scale multimodal datasets like OXE . To unlock the potential for robotics AI, Yaak teamed up with the LeRobot team at 🤗 and is excited to announce Learning to Drive (L2D) to the robotics AI community. L2D is the world’s largest multimodal dataset aimed at building an open-sourced spatial intelligence for the automotive domain with first class support for 🤗’s LeRobot training pipeline and models. Drawing inspiration from the best practices of source version control , Yaak also invites the AI community to search and discover novel episodes in our entire dataset (> 1 PetaBytes), and queue their collection for review to be merged into future release ( R5+ ).

Table 1: Open source self-driving datasets (*excluding lidar and radar). Source

L2D was collected with identical sensor suites installed on 60 EVs operated by driving schools in 30 German cities over the span of 3 years. The policies in L2D are divided into two groups — expert policies executed by driving instructors and student policies by learner drivers. Both the policy groups include natural language instructions for the driving task. For example, “When you have the right of way, take the third exit from the roundabout, carefully driving over the pedestrian crossing ”.

L2D on 🤗 is converted to LeRobotDataset v2.1 and LeRobotDataset v3.0 format to fully leverage the current and future models supported within LeRobot . The AI community can now build end-to-end self-driving models leveraging the state-of-the-art imitation learning and reinforcement learning models for real world robotics like ACT , Diffusion Policy , and Pi0 .

Both groups cover all driving scenarios that are mandatory for completion to obtain a driving license within the EU (German version), for example, overtaking, roundabouts and train tracks. In the release (See below R3+ ), for suboptimal student policies, a natural language reasoning for sub-optimality will be included. F.ex “ incorrect/jerky handling of the steering wheel in the proximity of in-coming traffic ” (Fig 2)

For R0, R1 we recommend using LeRobotDataset , with revision=[R0|R1] , which can be used directly from the pypi release of LeRobot. For R2+, please follow installation outlined here or install from main as below, as we recommend using StreamingLeRobotDataset as R3 is is Dataset v3.0 format.

Fig 1: Visualization: Nutron (3 of 6 cameras shown for clarity) Instructions: “When you have the right of way, drive through the roundabout and take the third exit”.

Expert policies have zero driving mistakes and are considered as optimal, whereas student policies have known sub optimality (Fig 2).

Fig 2: Student policy with jerky steering to prevent going into lane of the incoming truck