Understanding Physical AI
How robots learn new tasks from demonstrations
How recorded movements become a learned task, and which tests establish whether a robot can also handle unfamiliar situations.
Closing a zipped lunchbox sounds like a small task. For a robot, it combines several requirements: recognising the object, reaching the fastener and adapting its movement to the material. In 2025, Google DeepMind used this task as one example of how a robotics model can be adapted to new activities through additional demonstrations.
In its Gemini Robotics On-Device announcement of 24 June 2025, the research team reported using 50 to 100 demonstrations for the adaptations it studied. This does not mean a robot can master any task after watching a hundred arbitrary examples. The model already has capabilities from previous training. The new examples help transfer that existing knowledge to a particular activity.
Another task, chosen here purely for illustration, helps explain the distinction: a robot must move a cup from a table into a holder. The process does not begin with a text about cups. It begins with observations and movements recorded together. From these pairs, a model learns which action fits which observed situation.
From a guided grasp to an independent attempt
One way to collect this data is for a person to control the robot remotely. The person moves an input device while the robot performs the corresponding activity. Cameras capture the scene, while the system’s states and actions are recorded at the same time. The result is a sequence pairing what was visible with the movement performed.
The ALOHA research project from 2023 demonstrates this principle with two robot arms. The researchers combine a demonstration interface with a learning method for precise manipulation. Their method predicts connected sequences of movements. One aim is to prevent small errors from accumulating over many individually chosen steps.
For our cup example, a demonstration might proceed as follows: the gripper approaches the unobstructed side of the cup, closes, lifts it and places it in the holder. If the cup is farther to the left on the next attempt, the demonstrations should include that changed starting position. The movement then begins elsewhere but serves the same purpose. Such differences reveal what should remain consistent and what the system needs to respond to.
A fixed motion sequence could simply repeat recorded positions. A system learning from observations should instead derive its next action from the current situation. New trials establish whether it actually can. One successful run with a previously used cup in a familiar position is insufficient.
Vision-language-action models introduce language as another input. Their name describes the connection between images, verbal instructions and robot actions. In 2023, the researchers behind RT-2 demonstrated an approach in which a model pretrained on image and language data receives further training using robot trajectories. Actions are represented in a form the model can output.
In our example, “Put the cup in the holder” would be the instruction. The camera image supplies information about the current scene. The output must ultimately become an executable movement. Knowing the word “cup” does not guarantee a stable grasp. Recognition through language and physical success have to come together in the trial.
What a useful test needs to show
When learning from demonstrations, the number of examples tells only part of the story. A meaningful assessment should establish which starting conditions they contain. Fifty almost identical repetitions on an unchanged table address a different question from fifty demonstrations with different positions and objects. More data becomes informative when the variation it covers is clear.
For the cup task, a test could be organised into separate groups. In one, only positions change. In another, a different cup is used. In a third, an extra object appears on the table. This is an illustrative test design. It prevents a single overall result from obscuring which changes cause difficulties.
Success also needs a clear definition. If the cup is inside the holder and remains there, the task is complete. If the robot has merely lifted it, the task is still unfinished. If a person has to reposition it during the movement, that intervention should be recorded separately. A trial log stating the starting conditions, outcome and assistance is more informative than a collection of successful video clips.
A fictional result illustrates the point: out of 100 attempts, 90 succeed without assistance, six succeed after an intervention and four fail. The unassisted success rate is therefore 90 percent. Quietly including the six assisted attempts would create a different impression. It is also necessary to check how long each attempt took and whether the result remained stable over repeated runs.
Google describes Gemini Robotics On-Device as a model that runs locally on the robot and demonstrates adaptations to different robotic systems. This is a research example with defined tests. The reported demonstration counts are not a commitment for an arbitrary factory task. A new application would require its own specification of objects, workspace and success criteria.
Learning from demonstrations therefore changes how a workflow is specified: people provide examples from which the model learns relationships between observation and action. Work shifts towards choosing useful demonstrations and testing unfamiliar situations. Whether this produces a usable process is decided by the complete, repeatable operation on the real object.
Sources checked: 12 November 2026. This explainer draws on the cited research from 2023 and 2025. The cup example and its test figures are fictional.