TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning

Authors

  • Bingyan Liu Peking University
  • Yifeng Cai Peking University
  • Yao Guo Peking University
  • Xiangqun Chen Peking University

Keywords:

Transfer/Adaptation/Multi-task/Meta/Automated Learning

Abstract

The increasing of pre-trained models has significantly facilitated the performance on limited data tasks with transfer learning. However, progress on transfer learning mainly focuses on optimizing the weights of pre-trained models, which ignores the structure mismatch between the model and the target task. This paper aims to improve the transfer performance from another angle - in addition to tuning the weights, we tune the structure of pre-trained models, in order to better match the target task. To this end, we propose TransTailor, targeting at pruning the pre-trained model for improved transfer learning. Different from traditional pruning pipelines, we prune and fine-tune the pre-trained model according to the target-aware weight importance, generating an optimal sub-model tailored for a specific target task. In this way, we transfer a more suitable sub-structure that can be applied during fine-tuning to benefit the final performance. Extensive experiments on multiple pre-trained models and datasets demonstrate that TransTailor outperforms the traditional pruning methods and achieves competitive or even better performance than other state-of-the-art transfer learning methods while using a smaller model. Notably, on the Stanford Dogs dataset, TransTailor can achieve 2.7% accuracy improvement over other transfer methods with 20% fewer FLOPs.

Downloads

Published

2021-05-18

How to Cite

Liu, B., Cai, Y., Guo, Y., & Chen, X. (2021). TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(10), 8627-8634. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17046

Issue

Section

AAAI Technical Track on Machine Learning III