Towards Continually-Evolving AI: Selective and Expandable Multimodal Memory System

Abstract

Our world is constantly evolving, and human beings continuously enhance their knowledge by learning from their experiences throughout life. Despite significant advancements in embodied AI, current agents struggle to operate reliably, robustly, and continually in complex, real-world multimodal environments due to complex problems that span large and diverse domains, especially when faced with out-of-distribution (OOD) scenarios they have not previously encountered. Our goal is to develop a multimodal, embodied AI system that continually enhances its capabilities and skills through safe and robust interactions with an ever-changing, multimodal world. This is enabled by a novel, adaptively expandable memory architecture that integrates both long- and short-term information across multiple modalities. The system selectively decides what to store and learn, filtering out adversarial or low-quality inputs to prevent negative transfer and distractions, while improving overall efficiency and effectiveness.