RESEARCH / BLOG

Research,
Explained.

Read the research question, follow the authors’ methods and explore the evidence behind the article.

Research Blog

Automation · Research insights

A camera learns the motion map needed to control very different robots

A Nature study learns visuomotor Jacobian fields from video and random commands, reducing dependence on hand-built robot models.

By NPT Research2 min read

Study published: 25 June 2025 · Nature. Original research commentary by NPT Research.

A controller must translate a desired movement into actuator commands. That translation is straightforward only when the robot’s geometry and response are well understood. Soft, unusual or changing structures challenge conventional models, motivating a method that learns directly from observation.

Latest breakthrough

The researchers trained deep networks to infer a visuomotor Jacobian field from video. This field describes how points on the robot respond to actuator changes. The approach controlled a diverse set of manipulators using a single camera and observations of random commands, without assuming a particular material or actuation system.

Read the original study: Controlling diverse robots by inferring Jacobian fields with deep networks.

How the study worked

The robot explored movements while the camera recorded the resulting shapes. Learning related actuator inputs to local three-dimensional motion sensitivity. The inferred field then supported closed-loop control, in which fresh observations help correct the next action rather than relying on a single open-loop prediction.

The idea in everyday terms

When learning an unfamiliar game controller, small experimental movements reveal which button moves what. A Jacobian makes that relationship mathematical and local: it predicts how a small input change affects motion. The analogy explains learning a control map, not human-like understanding or unrestricted reasoning.

Why it matters

A generic observation-based controller could make unconventional hardware easier to use. It complements learning a robot’s own simulator, which focuses on predicting body shape and movement. Both approaches reduce some dependence on manually specified geometry.

What the evidence does not establish

The paper’s varied demonstrations do not establish reliable control of every robot or task. Camera visibility, training coverage, contact changes and timing can constrain deployment. A learned local motion map is not by itself a safety guarantee or a complete autonomous planning system.

Explore the device

The robotic-arm model shows the familiar chain of rigid joints for which geometric modelling is comparatively natural. The research becomes especially valuable when that simple description is inadequate. Contact-aware predictive control addresses the additional difficulty that appears when the moving robot touches its environment.

Explore the Industrial robotic arm model and its description, or read our earlier device-focused research explanation.

Primary citation: Sizhe Lester Li, Annan Zhang, Boyuan Chen et al.. Controlling diverse robots by inferring Jacobian fields with deep networks. Nature (2025). 10.1038/s41586-025-09170-0.

Sources and further reading

  1. Controlling diverse robots by inferring Jacobian fields with deep networks Published 25 Jun 2025