AutoLR: Layer-wise Pruning and Auto-tuning of Learning Rates in Fine-tuning of Deep Networks

Authors

  • Youngmin Ro Seoul National University Samsung SDS
  • Jin Young Choi Seoul National University

Keywords:

Image and Video Retrieval

Abstract

Existing fine-tuning methods use a single learning rate over all layers. In this paper, first, we discuss that trends of layer-wise weight variations by fine-tuning using a single learning rate do not match the well-known notion that lower-level layers extract general features and higher-level layers extract specific features. Based on our discussion, we propose an algorithm that improves fine-tuning performance and reduces network complexity through layer-wise pruning and auto-tuning of layer-wise learning rates. The proposed algorithm has verified the effectiveness by achieving state-of-the-art performance on the image retrieval benchmark datasets (CUB-200, Cars-196, Stanford online product, and Inshop). Code is available at https://github.com/youngminPIL/AutoLR.

Downloads

Published

2021-05-18

How to Cite

Ro, Y., & Choi, J. Y. (2021). AutoLR: Layer-wise Pruning and Auto-tuning of Learning Rates in Fine-tuning of Deep Networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3), 2486-2494. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16350

Issue

Section

AAAI Technical Track on Computer Vision II