TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning
AbstractThe increasing of pre-trained models has significantly facilitated the performance on limited data tasks with transfer learning. However, progress on transfer learning mainly focuses on optimizing the weights of pre-trained models, which ignores the structure mismatch between the model and the target task. This paper aims to improve the transfer performance from another angle - in addition to tuning the weights, we tune the structure of pre-trained models, in order to better match the target task. To this end, we propose TransTailor, targeting at pruning the pre-trained model for improved transfer learning. Different from traditional pruning pipelines, we prune and fine-tune the pre-trained model according to the target-aware weight importance, generating an optimal sub-model tailored for a specific target task. In this way, we transfer a more suitable sub-structure that can be applied during fine-tuning to benefit the final performance. Extensive experiments on multiple pre-trained models and datasets demonstrate that TransTailor outperforms the traditional pruning methods and achieves competitive or even better performance than other state-of-the-art transfer learning methods while using a smaller model. Notably, on the Stanford Dogs dataset, TransTailor can achieve 2.7% accuracy improvement over other transfer methods with 20% fewer FLOPs.
How to Cite
Liu, B., Cai, Y., Guo, Y., & Chen, X. (2021). TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(10), 8627-8634. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/17046
AAAI Technical Track on Machine Learning III