Placing Objects in Gesture Space: Toward Incremental Interpretation of Multimodal Spatial Descriptions


  • Ting Han Bielefeld University
  • Casey Kennington Boise State University
  • David Schlangen Bielefeld University



Co-verbal gestures, abstract deixis, real time system, incremental processing, multimodal interface


When describing routes not in the current environment, a common strategy is to anchor the description in configurations of salient landmarks, complementing the verbal descriptions by "placing" the non-visible landmarks in the gesture space.  Understanding such multimodal descriptions and later locating the landmarks from real world is a challenging task for the hearer, who must interpret speech and gestures in parallel, fuse information from both modalities, build a mental representation of the description, and ground the knowledge to real world landmarks.  In this paper, we model the hearer's task, using a multimodal spatial description corpus we collected.  To reduce the variability of verbal descriptions, we simplified the setup to use simple objects as landmarks.  We describe a real-time system to  evaluate the separate and joint contribution of the modalities. We show that gestures not only help to improve the overall system performance, even if to a large extent they encode redundant information, but also result in earlier final correct interpretations. Being able to build and apply representations incrementally will be of use in more dialogical settings, we argue, where it can enable immediate clarification in cases of mismatch.




How to Cite

Han, T., Kennington, C., & Schlangen, D. (2018). Placing Objects in Gesture Space: Toward Incremental Interpretation of Multimodal Spatial Descriptions. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).