Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling
Keywords:Hyperparameter Tuning / Algorithm Configuration
AbstractTraining deep neural networks using a large batch size has shown promising results and benefits many real-world applications. Warmup is one of nontrivial techniques to stabilize the convergence of large batch training. However, warmup is an empirical method and it is still unknown whether there is a better algorithm with theoretical underpinnings. In this paper, we propose a novel Complete Layer-wise Adaptive Rate Scaling (CLARS) algorithm for large-batch training. We prove the convergence of our algorithm by introducing a new fine-grained analysis of gradient-based methods. Furthermore, the new analysis also helps to understand two other empirical tricks, layer-wise adaptive rate scaling and linear learning rate scaling. We conduct extensive experiments and demonstrate that the proposed algorithm outperforms gradual warmup technique by a large margin and defeats the convergence of the state-of-the-art large-batch optimizer in training advanced deep neural networks (ResNet, DenseNet, MobileNet) on ImageNet dataset.
How to Cite
Huo, Z., Gu, B., & Huang, H. (2021). Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9), 7883-7890. https://doi.org/10.1609/aaai.v35i9.16962
AAAI Technical Track on Machine Learning II