Speaking Louder than Words with Pictures Across Languages


  • Andrew Finch NICT
  • Wei Song Canon Inc.
  • Kumiko Tanaka-Ishii Kyushu University
  • Eiichiro Sumita NICT




In this article, we investigate the possibility of cross-language communication using a synergy of words and pictures on mobile devices. Communicating with only pictures is in itself a very powerful strategy, but is limited in expressiveness. On the other hand, words can express everything you could wish to say, but they are cumbersome to work with on mobile devices, and need to be translated in order for their meaning to be understood. Automatic translations can contain errors that pervert the communication process, and this may undermine the users’ confidence when expressing themselves across language barriers. Our idea is to create a user interface for cross-language communication that uses pictures as the primary mode of input, and words to express the detailed meaning. This interface creates a visual process of communication that occurs on two heterogeneous channels that can support each other. We implemented this user interface as application on the Apple iPad tablet, and performed a set of experiments to determine its usefulness as a translation aid for travellers.




How to Cite

Finch, A., Song, W., Tanaka-Ishii, K., & Sumita, E. (2013). Speaking Louder than Words with Pictures Across Languages. AI Magazine, 34(2), 31. https://doi.org/10.1609/aimag.v34i2.2471