Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models


  • Liqi He Wuhan University
  • Zuchao Li Wuhan University
  • Xiantao Cai Wuhan University
  • Ping Wang Wuhan University



NLP: Generation, CV: Language and Vision


Chain-of-thought (CoT) reasoning has exhibited impressive performance in language models for solving complex tasks and answering questions. However, many real-world questions require multi-modal information, such as text and images. Previous research on multi-modal CoT has primarily focused on extracting fixed image features from off-the-shelf vision models and then fusing them with text using attention mechanisms. This approach has limitations because these vision models were not designed for complex reasoning tasks and do not align well with language thoughts. To overcome this limitation, we introduce a novel approach for multi-modal CoT reasoning that utilizes latent space learning via diffusion processes to generate effective image features that align with language thoughts. Our method fuses image features and text representations at a deep level and improves the complex reasoning ability of multi-modal CoT. We demonstrate the efficacy of our proposed method on multi-modal ScienceQA and machine translation benchmarks, achieving state-of-the-art performance on ScienceQA. Overall, our approach offers a more robust and effective solution for multi-modal reasoning in language models, enhancing their ability to tackle complex real-world problems.



How to Cite

He, L., Li, Z., Cai, X., & Wang, P. (2024). Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16), 18180-18187.



AAAI Technical Track on Natural Language Processing I