Towards Physically-grounded Human-AI Collaboration

Abstract

Creating physically-grounded human-AI collaboration remains challenging because of continuous state-action spaces, constrained physical transitions, and diverse human behaviors. Successful collaboration in physical environments requires an agent to generalize their learned policies across three key collaboration modes: coordination, where agents must coordinate subtasks or movements and avoid collision; awareness, where agents need to recognize when another agent needs help and offer assistance; and action consistency, where agents must align their actions toward the same goals when engaging in joint actions. We designed Moving Out, a physical human-AI collaboration environment to illustrate these challenges and collaboration modes. We observe that existing AI agents often fail to assist appropriately, align actions, or generalize to unseen physical settings. Our findings suggest future research directions in physical reasoning, behavior adaptation, and reliable and scalable evaluation of human-AI collaboration.