Asynchronous Robot Inference: Decoupling Action Prediction and Execution

TL;DR Robotic policies are increasingly bulky, and predict chunks of future actions rather than a single next action. This results in the robot being idle while awaiting new actions to perform, introducing noticeable lags at execution, and lacking responsiveness. Asynchronous inference tightens the control loop, removing lags at runtime and resulting in more adaptive control by decoupling action prediction from action execution. In this blog post, we cover the basics behind async inference, and how it can be used to improve the performance of robotic policies in the real-world.

Get started with async inference by following our tutorial .

Sequential inference (first) versus async inference (second) . Allowing for replanning and a tighter control loop, async inference results in (1) attempts at recovery, and (2) a ~2x speedup in task completion. Sequential inference keeps acting out the current action chunk even after failure to grasp the object, while async inference can replan and act the new action chunk. Both setups use the same policy!

Queue monitoring : The client continuously monitors its action queue length against a chunk size threshold parameter. When the queue drops below this threshold, it signals that a new observation should be sent for processing.

Experiments (see plots below) show that offers a good trade-off when observations sent are not filtered out (they are all must-go). We recommend setting and following our documentation to tune this parameter to your needs.

To run your policy using async inference, you just need to follow our tutorial with your own custom parameters (e.g., the policy path or the chunk size threshold). Async inference comes with support for policies supporting action chunking!

The policy workflow , in which incoming observations are compared to the last one used for inference, and processed only if different enough, or `must_go`.

Action chunk aggregation : When a new action chunk arrives from the server, the client merges it with any remaining actions in the current queue over the overlapping portion. This is where custom aggregators come into play, handling overlapping sections between the current and incoming chunks differently. As of now, we support flexibly aggregation between the chunks via the specification of a custom aggregate_fn(chunk1: torch.Tensor, chunk2: torch.Tensor) -> torch.Tensor function, which is called for each overlapping timestep and can be user-provided. The overlapping portions (shown in light blue in the diagram) require careful handling. We can design different aggregation strategies:

With async inference, we decouple action execution from action prediction. This is particularly relevant considering the tendency of currently popular models like [ ACT ], [ OpenVLA ], [ PI0 ], and [ SmolVLA ] to be outputting chunks of actions rather than single actions given an observation . Convince yourself of this by running all these models using LeRobot .

Using chunks sequentially results in (1) lags at runtime, impacting task execution time and (2) lack of responsiveness, due to acting widely open-loop. Asynchronous inference mitigates both these limitations by decoupling action prediction from action execution . We introduced asynchronous inference in SmolVLA, and found it to result in a ~2x speed-up in task completion time with comparable task success rate.

In particular, we design a 2-component system where policy inference and action execution are performed in two different processes, possibly on two different machines connected through the network: