Implicit Kernel Attention

Authors

  • Kyungwoo Song University of Seoul
  • Yohan Jung KAIST
  • Dongjun Kim KAIST
  • Il-Chul Moon KAIST

Keywords:

(Deep) Neural Network Algorithms, (Deep) Neural Network Learning Theory, Kernel Methods, Probabilistic Graphical Models

Abstract

Attention computes the dependency between representations, and it encourages the model to focus on the important selective features. Attention-based models, such as Transformer and graph attention network (GAT), are widely utilized for sequential data and graph-structured data. This paper suggests a new interpretation and generalized structure of the attention in Transformer and GAT. For the attention in Transformer and GAT, we derive that the attention is a product of two parts: 1) the RBF kernel to measure the similarity of two instances and 2) the exponential of L2 norm to compute the importance of individual instances. From this decomposition, we generalize the attention in three ways. First, we propose implicit kernel attention with an implicit kernel function instead of manual kernel selection. Second, we generalize L2 norm as the Lp norm. Third, we extend our attention to structured multi-head attention. Our generalized attention shows better performance on classification, translation, and regression tasks.

Downloads

Published

2021-05-18

How to Cite

Song, K., Jung, Y., Kim, D., & Moon, I.-C. (2021). Implicit Kernel Attention. Proceedings of the AAAI Conference on Artificial Intelligence, 35(11), 9713-9721. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17168

Issue

Section

AAAI Technical Track on Machine Learning IV