How to Build a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac for Healthcare

Simulation has been a cornerstone in medical imaging to address the data gap. However, in healthcare robotics until now, it's often been too slow, siloed, or difficult to translate into real-world systems. That’s now changing. With new advances in GPU-accelerated simulation and digital twins, developers can design, test, and validate robotic workflows entirely in virtual environments - reducing prototyping time from months to days, improving model accuracy, and enabling safer, faster innovation before a single device reaches the operating room.

That's why NVIDIA introduced Isaac for Healthcare earlier this year, a developer framework for AI healthcare robotics, that enables developers in solving these challenges via integrated data collection, training, and evaluation pipelines that work across both simulation and hardware. Specifically, the Isaac for Healthcare v0.4 release provides users with an end-to-end SO-ARM based starter workflow and the bring your own operating room tutorial . The SO-ARM starter workflow lowers the barrier for MedTech developers to experience the full workflow from simulation to training to deployment and start building and validating autonomously on real hardware right away.

In this post, we'll walk through the starter workflow and its technical implementation details to help you build a surgical assistant robot in less time than ever imaginable before.

The SO-ARM starter workflow introduces a new way to explore surgical assistance tasks, and provides developers with a complete end-to-end pipeline for autonomous surgical assistance:

This workflow gives developers a safe, repeatable environment to train and refine assistive skills before moving into the Operating Room.

The workflow implements a three-stage pipeline that integrates simulation and real hardware:

The workflow combines simulation and real-world data to address the fundamental challenge that training robots in the real world is expensive and limited, while pure simulation often fails to capture real-world complexities. The approach uses approximately 70 simulation episodes for diverse scenarios and environmental variations, combined with 10-20 real-world episodes for authenticity and grounding. This mixed training creates policies that generalize beyond either domain alone.

For real-world data collection with SO-ARM101 hardware or any other version supported in LeRobot:

Notably, over 93% of the data used for policy training was generated synthetically in simulation, underscoring the strength of simulation in bridging the robotic data gap.

Notably, developers could run all the simulation, training and deployment (3 computers needed for physical AI) on one DGX Spark .

The trained model processes natural language instructions such as "Prepare the scalpel for the surgeon" or "Hand me the forceps" and executes the corresponding robotic actions. With the latest LeRobot release (v0.4.0) you will be able to post-train GR00T N1.5 natively in LeRobot!