Depth perception
Dense depth maps improve spatial grounding and reduce the number of action rounds.
0%→0%
RGB → RGB + ground-truth depth
LIBERO-Plus subset · Table 1
Large Multimodal Models
for Direct Robot Control:
GPT-6 as a Beginning
How far can general-purpose intelligence go in the physical world?
A systematic study of GPT-6 Astra’s capabilities—and the boundaries of robotic manipulation.
1 Nanjing University2 Nanyang Technological University3 Peking University4 Institute of Automation of the Chinese Academy of Sciences5 Independent Researcher
* Equal contribution † Corresponding authors
Emerging capabilities.
Measurable boundaries.
Direct, closed-loop manipulation through a lightweight tool interface. No specialized action policy is used.
LIBERO-PLUS
0%
Generalization under variation40 tasks · 10 trials per task↗
ROBOTWIN 2.00%
Bimanual manipulation50 tasks · 10 trials per task↗
REAL-WORLD · ONE-EXAMPLE ICL0%
+40 pp over zero-shot20 / 25 successes · 5 tasks↗
LIBERO-GEOMOTION · ZERO-SHOT0%
Geometry remains a challenge4 / 36 scene variants↗
Semantic reasoning is a beginning.
Reliable physical execution is the next challenge.
Read the abstract +
We systematically evaluate GPT-6 Astra for closed-loop robotic manipulation through a lightweight agent execution framework. We probe depth perception, embodied in-context learning, and tactile understanding, then evaluate five manipulation benchmarks across six capability dimensions. The results reveal a gap between semantic reasoning and physical execution, especially in consequence-aware planning, precision, and complex motion coordination. We introduce LIBERO-GeoMotion to test geometry-constrained manipulation and validate the findings on a real robot. Contextual demonstrations improve execution, but motion feasibility, precise control, physical safety, and inference latency remain important limitations.
Observe. Reason. Act.
Then close the loop.
GPT-6 Astra generates tool calls and action arguments. The environment executes them and returns fresh observations for the next decision.
Perceive the scene
RGB observations, robot state, and optional depth or tactile feedback.
GPT-6 Astra
Instruction + interaction history + demonstration context.
Execute a command
End-effector position, orientation, and gripper commands through tools.
↳ Updated observations and tool results return to the interaction history ↰
TOOL INTERFACE
Observe()Reference()Act()Status()Finish()What helps intelligence
become embodied?
Controlled ablations isolate the contribution of spatial evidence, demonstration context, and contact feedback.
Learning in context
Actions, video frames, and textual descriptions together provide useful motion references.
0%→0%
No context → combined context
LIBERO-Plus subset · Table 2
Tactile understanding
Contact feedback supports corrective actions, with the representation of force making a difference.
0%→0%
RGB → RGB + tactile text
UniVTAC subset · Table 3
One general model.
Five different tests.
164 tasks evaluated with demonstration context. Strong results on some benchmarks coexist with difficult precision and dexterous-control failures.
GPT-6 Astra · task success
SUCCESS RATE (%) ↑Paper §4.5 / Figure 8. Reduced evaluation protocol: 10 trials per task on LIBERO-Plus, RoboTwin 2.0, and RoboDojo; 20 on RoboCasa GR1 and UniVTAC. Results are specific to these evaluated task sets.
Six capability dimensions
Generalization · Memory · Task understanding · Long-horizon execution · High-order operations · Precision
From understanding
to real-world action.
Five tasks on a ROKAE Helios robot. One related demonstration doubles the observed overall success rate, without parameter updates.
OVERALL SUCCESS
0%
→Zero-shot · 10/25
0%
One-example ICL · 20/25
One example. A measurable difference.
5 trials per task per condition · 50 trials total.
Task-level rates and counts follow Table 7.
Success by task
Zero-shotOne-example ICL
Watch the real-robot experiments
Understanding the goal
is only the beginning.
LIBERO-GeoMotion tests geometry-constrained manipulation across nine task families. Familiar objects and straightforward goals put the focus on pose alignment, coordinated motion, and physically feasible execution.
ZERO-SHOT0%4 / 36 scene variants
→WITH CONTEXTUAL DEMONSTRATIONS0%Overall success · Table 6
Context helps.
Geometry still constrains.
Look closer.
Every example tells a story.
BIBTEX
@misc{zhou2026directrobotcontrol,
title = {Large Multimodal Models for Direct Robot
Control: GPT-6 as a Beginning},
author = {Zhou, Youwei and Hou, Bohan and Fu, Yankai
and Lyu, Yaoxu and Ye, Yifan and Shan, Shilin
and Guo, Chen and Liu, Jiaming and He, Ran
and Shan, Caifeng and Zhang, Shanghang
and Yang, Jianfei and Lu, Zhihe},
year = {2026},
eprint = {2610.28259},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28259}
}