TY - JOUR AU - Li, Jian AU - Wang, Xing AU - Yang, Baosong AU - Shi, Shuming AU - Lyu, Michael R. AU - Tu, Zhaopeng PY - 2020/04/03 Y2 - 2024/03/28 TI - Neuron Interaction Based Representation Composition for Neural Machine Translation JF - Proceedings of the AAAI Conference on Artificial Intelligence JA - AAAI VL - 34 IS - 05 SE - AAAI Technical Track: Natural Language Processing DO - 10.1609/aaai.v34i05.6334 UR - https://ojs.aaai.org/index.php/AAAI/article/view/6334 SP - 8204-8211 AB - <p>Recent NLP studies reveal that substantial linguistic information can be attributed to single neurons, i.e., individual dimensions of the representation vectors. We hypothesize that modeling strong interactions among neurons helps to better capture complex information by composing the linguistic properties embedded in individual neurons. Starting from this intuition, we propose a novel approach to compose representations learned by different components in neural machine translation (e.g., multi-layer networks or multi-head attention), based on modeling strong interactions among neurons in the representation vectors. Specifically, we leverage bilinear pooling to model pairwise multiplicative interactions among individual neurons, and a low-rank approximation to make the model computationally feasible. We further propose extended bilinear pooling to incorporate first-order representations. Experiments on WMT14 English⇒German and English⇒French translation tasks show that our model consistently improves performances over the SOTA T<span style="font-variant: small-caps;">ransformer</span> baseline. Further analyses demonstrate that our approach indeed captures more syntactic and semantic information as expected.</p> ER -