Automation · Research insights
A camera learns the motion map needed to control very different robots
A Nature study learns visuomotor Jacobian fields from video and random commands, reducing dependence on hand-built robot models.
Study published: 25 June 2025 · Nature. Original research commentary by NPT Research.
A controller must translate a desired movement into actuator commands. That translation is straightforward only when the robot’s geometry and response are well understood. Soft, unusual or changing structures challenge conventional models, motivating a method that learns directly from observation.
Latest breakthrough
The researchers trained deep networks to infer a visuomotor Jacobian field from video. This field describes how points on the robot respond to actuator changes. The approach controlled a diverse set of manipulators using a single camera and observations of random commands, without assuming a particular material or actuation system.
Read the original study: Controlling diverse robots by inferring Jacobian fields with deep networks.
How the study worked
The robot explored movements while the camera recorded the resulting shapes. Learning related actuator inputs to local three-dimensional motion sensitivity. The inferred field then supported closed-loop control, in which fresh observations help correct the next action rather than relying on a single open-loop prediction.
The idea in everyday terms
When learning an unfamiliar game controller, small experimental movements reveal which button moves what. A Jacobian makes that relationship mathematical and local: it predicts how a small input change affects motion. The analogy explains learning a control map, not human-like understanding or unrestricted reasoning.
Why it matters
A generic observation-based controller could make unconventional hardware easier to use. It complements learning a robot’s own simulator, which focuses on predicting body shape and movement. Both approaches reduce some dependence on manually specified geometry.
What the evidence does not establish
The paper’s varied demonstrations do not establish reliable control of every robot or task. Camera visibility, training coverage, contact changes and timing can constrain deployment. A learned local motion map is not by itself a safety guarantee or a complete autonomous planning system.
Explore the device
The robotic-arm model shows the familiar chain of rigid joints for which geometric modelling is comparatively natural. The research becomes especially valuable when that simple description is inadequate. Contact-aware predictive control addresses the additional difficulty that appears when the moving robot touches its environment.
Explore the Industrial robotic arm model and its description, or read our earlier device-focused research explanation.
Primary citation: Sizhe Lester Li, Annan Zhang, Boyuan Chen et al.. Controlling diverse robots by inferring Jacobian fields with deep networks. Nature (2025). 10.1038/s41586-025-09170-0.
Related articles
A new representation makes changing robot contacts easier to predict
Robots learn a predictive body model by watching their own movements
A modular robot arm succeeds in 91.67% of 240 test trials
A robot can prevent slipping by changing its path instead of only squeezing harder
Dense liquid-metal coils combine soft actuation, sensing and communication
A DNA clutch lets a magnetic nanomotor engage and disengage its rotor
Elastic electromagnetic mechanisms give small robots muscle-like motion
Electric current drives a molecular rotor and changes its conductance
Nanotextured surfaces help graphite maintain an extremely low-friction contact
Projected light assembles and drives reconfigurable microscopic machines
Liquid-metal networks turn a flat elastomer into programmable 3D surfaces
Microscopy reveals which internal features hold a permanent magnet’s direction
Light drives microscopic gear trains through patterned metasurfaces
Soft robots steer their shape by combining local electromagnetic vectors
Friction consolidates brittle magnetic powders in a single processing step
A soft robot arm stays bendable while transmitting continuous torque
An engineered environment changes a microscopic engine’s power–efficiency trade-off
Magnetic measurements expose early fatigue changes in tested metals
Changing the character of noise shifts a microscopic engine’s best operating speed
Switching between unstable modes can stabilise a mechanical oscillator
Correcting encoder signals improves tiny-motion measurements
Gearbox vibrations reveal clues to efficiency during operation
A redesigned magnetic path cuts opening time in a solenoid studyExplore the models
Sources and further reading
- Controlling diverse robots by inferring Jacobian fields with deep networks Published 25 Jun 2025