Learning Intuitive Physics with Multimodal Generative Models


  • Sahand Rezaei-Shoshtari Samsung AI Center Montreal McGill University
  • Francois R. Hogan Samsung AI Center Montreal
  • Michael Jenkin Samsung AI Center Montreal York University
  • David Meger Samsung AI Center Montreal McGill University
  • Gregory Dudek Samsung AI Center Montreal McGill University




Multimodal Perception & Sensor Fusion, Multimodal Learning, Multi-modal Vision, Representation Learning


Predicting the future interaction of objects when they come into contact with their environment is key for autonomous agents to take intelligent and anticipatory actions. This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes. Visual information captures object properties such as 3D shape and location, while tactile information provides critical cues about interaction forces and resulting object motion when it makes contact with the environment. Utilizing a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces, our system captures both the visual appearance and the tactile properties of objects. We interpret the dual stream signals from the sensor using a Multimodal Variational Autoencoder (MVAE), allowing us to capture both modalities of contacting objects and to develop a mapping from visual to tactile interaction and vice-versa. Additionally, the perceptual system can be used to infer the outcome of future physical interactions, which we validate through simulated and real-world experiments in which the resting state of an object is predicted from given initial conditions.




How to Cite

Rezaei-Shoshtari, S., Hogan, F. R., Jenkin, M., Meger, D., & Dudek, G. (2021). Learning Intuitive Physics with Multimodal Generative Models. Proceedings of the AAAI Conference on Artificial Intelligence, 35(7), 6110-6118. https://doi.org/10.1609/aaai.v35i7.16761



AAAI Technical Track on Intelligent Robots