Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement

Authors

  • Zi-Yi Dou Carnegie Mellon University
  • Zhaopeng Tu Tencent, Inc.
  • Xing Wang Tencent, Inc.
  • Longyue Wang Tencent, Inc.
  • Shuming Shi Tencent, Inc.
  • Tong Zhang Tencent, Inc.

DOI:

https://doi.org/10.1609/aaai.v33i01.330186

Abstract

With the promising progress of deep neural networks, layer aggregation has been used to fuse information across layers in various fields, such as computer vision and machine translation. However, most of the previous methods combine layers in a static fashion in that their aggregation strategy is independent of specific hidden states. Inspired by recent progress on capsule networks, in this paper we propose to use routing-by-agreement strategies to aggregate layers dynamically. Specifically, the algorithm learns the probability of a part (individual layer representations) assigned to a whole (aggregated representations) in an iterative way and combines parts accordingly. We implement our algorithm on top of the state-of-the-art neural machine translation model TRANSFORMER and conduct experiments on the widely-used WMT14 sh⇒German and WMT17 Chinese⇒English translation datasets. Experimental results across language pairs show that the proposed approach consistently outperforms the strong baseline model and a representative static aggregation model.

Downloads

Published

2019-07-17

How to Cite

Dou, Z.-Y., Tu, Z., Wang, X., Wang, L., Shi, S., & Zhang, T. (2019). Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 86-93. https://doi.org/10.1609/aaai.v33i01.330186

Issue

Section

AAAI Technical Track: AI and the Web