Zhiqin Yang*, Chenxin Li*†, Xiaomeng Hu*†, Yibin Liu, Weidong Huang, Jiankai Sun, Haitao Li, Zijian Wu, Yuzhi Huang, Fanding Huang, Hanwen Sun, Jiashun Liu, Jingqi Tong, Mingxin Huang, Shaoli Hu, Shijue Huang, Tianyi Bai, Xinyuan Wang, Yunlong Lin, Zhengyang Tang, Zhexin Zhang, Zhuo Chen, Xierui Song, Juntao Dai, Boyuan Chen, Jiaming Ji, Fangneng Zhan, Mengkang Hu, Wei Xue, Yonggang Zhang, Han Hu, Tsung-Yi Ho, Yike Guo
Preprint, 2026
Project Page
arXiv
Code
Video
Abstract
RobotWorld investigates how general-purpose multimodal agents transfer their digital tool-use capabilities to physical task execution through robot interfaces. Its 84 simulation tasks span manipulation, mobile manipulation, locomotion, driving, and aerial control, with explicit interaction budgets and executable success checks. Agents can construct sophisticated perception and control workflows involving segmentation, camera calibration, spatial estimation, and dynamics computation, yet these capabilities do not consistently compose into successful behavior. Execution-trace analysis exposes lost object states, ineffective action correction, late recovery, and mistaken completion judgments, providing concrete targets for more reliable physical-world agents.
A benchmark for multimodal robot use across 84 tasks and 18 embodiments, revealing challenges in state tracking, feedback correction, and reliable task completion.