Coupling RNN with LLM: Does Their Integration Improve Highly Order-Sensitive Language Understanding? - Rahman, Md Mostafizer; Shiplu, Ariful Islam; Watanobe, Yutaka; Naqvi, Syed Rameez; Peng, Lu (Proceedings of the AAAI Symposium Series, 2026)
Pretrained large language models (LLMs) have demonstratedremarkable success across various language modeling tasks.However, in domain-specific applications, particularlythose involving highly order-sensitive data, general LLMsexhibit limitations in achieving state-of-the-artperformance. One notable issue is that the contextualembeddings by LLMs still lack a strong positional inductivebias, especially for long and highly ordered sequences,leading to the lost in the middle problem. In this work, weutilized the potential of sequential models (RNNs) withLLMs to address the issue and investigate whether RNNintegration improves LLM performance. The LLM generatesrich contextual embeddings using the attention mechanism ofthe Transformer. The RNN further processes the LLMembeddings to capture the contextual semantics of long andorder-sensitive dependencies. The LLM-RNN model leveragesthe potential of both Transformer and recurrent structuresto enhance performance in domain-specific tasks. We performa wide range of experiments leveraging multiple types ofLLMs (encoder-only, encoder-decoder, and decoder-only) andRNNs (GRU, LSTM, BiGRU, and BiLSTM) across diverse publicand real-world datasets to investigate the potential(either positive or negative) of LLM-RNN models. Theexperimental results highlight the superiority of theLLM-RNN model, showing improvements in commonsensereasoning, code understanding, and biomedical reasoningtasks.