To teach robots new tasks, humans usually have to laboriously teleoperate them during training. Nvidia's new AI model GR00T 2 is designed to cut that effort by transferring human movements from videos to robotic arms.
To train a robot for new tasks, engineers usually spend hours remotely controlling the machines. This so-called teleoperation is expensive, slow, and needs its own data for every robot model. In a new research report on GR00T 2, Nvidia introduces a new AI foundation model designed to shorten this process.
Unlike previous approaches, including its predecessor GR00T 1, GR00T 2 doesn't translate camera images and language commands directly into motor movements. Nvidia says the software works as what the company calls a world-action model. It simultaneously calculates the next movement and a short video of the immediate future. The system predicts how the camera image should change when a robotic arm picks up an object, for example.
Unlike its predecessor, GR00T 2 is no longer a vision-language-action model. Nvidia calls it a world-action model (WAM) that also predicts what it will see in the future via diffusion. | Image: Nvidia
This video prediction should let the model transfer movement sequences between different robot form factors. Nvidia trained the system on a cluster with more than a thousand GB300 chips, including about 400,000 hours of video footage showing people performing various tasks while wearing head-mounted cameras.
To then teach a robot a new task, the researchers say it should be enough for a person to demonstrate the steps using data gloves, specialized hand grippers, or a finger exoskeleton. When these recordings are combined during training with general motion data from the target robot, the AI generates the robotic arm instead of the human arm in its video prediction and derives the matching control commands from that.
In lab tests, a robot with five-fingered hands was able to light a match, slice a banana, or toss packages onto a conveyor belt this way, without having been teleoperated for those specific tasks.
Success rates varied between 10 and 80 percent depending on the task. From a single video of a human hand movement, the system also learned to hammer nails by way of a computer simulation. For 50 selected tasks where the model already performed well after just a few examples, less than three minutes of conventional training was enough to raise the success rate from 40 to 95 percent.
A higher-level language model helps with errors but slows the robot down
For longer assembly work, Nvidia pairs the system with a higher-level AI language model like OpenAI's GPT-6 Astra. While GR00T 2 handles the gripping movements, the language model breaks the task into individual steps, color-codes grip points in the camera image, and launches helper routines when a part falls off.
When the researchers placed loose parts for toy models on a table in no particular order, the combined system assembled them in at least 97 percent of trials. Without the corrective language model and additional fine-tuning, the toy car assembly succeeded in only 23 out of 100 attempts. The team also trained precision tasks like plugging in network cables or chargers through repeated trial and error on the real device.
For bipedal robots, Nvidia developed the slimmed-down variant GR00T 2 Turbo, which processes fewer image details. It runs directly on the Jetson Thor onboard computer of a humanoid robot. In testing, the machine was able to scoop popcorn, clear tables, and wipe up liquids with a towel under its foot. That last task, though, succeeded in just under half of all attempts.
The research team identifies significant hurdles before the system is ready for reliable real-world use. The model doesn't yet process direct sensor data for touch and sometimes misses subtle errors like half-turned screws. The higher-level language model also slows things down considerably. During trash sorting, the robot stood idle for roughly a third of the total time because it had to wait more than seven seconds on average for the planning model's next computation.
Other labs are also looking for ways to teach robots new tasks with less teleoperation. We covered some of them in our AI Radar #5 on Neolabs.
原文版权归 The Decoder 及原作者所有，本站转载仅供阅读。如需删除，请联系 support@aisa.one。
Studio
