Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling

Authors

  • Zhouyuan Huo Google
  • Bin Gu MBZUAI JD Finance America Corporation
  • Heng Huang University of Pittsburgh JD Finance America Corporation

Keywords:

Hyperparameter Tuning / Algorithm Configuration

Abstract

Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. Warmup is one of nontrivial techniques to stabilize the convergence of large batch training. However, warmup is an empirical method and it is still unknown whether there is a better algorithm with theoretical underpinnings. In this paper, we propose a novel Complete Layer-wise Adaptive Rate Scaling (CLARS) algorithm for large-batch training. We prove the convergence of our algorithm by introducing a new fine-grained analysis of gradient-based methods. Furthermore, the new analysis also helps to understand two other empirical tricks, layer-wise adaptive rate scaling and linear learning rate scaling. We conduct extensive experiments and demonstrate that the proposed algorithm outperforms gradual warmup technique by a large margin and defeats the convergence of the state-of-the-art large-batch optimizer in training advanced deep neural networks (ResNet, DenseNet, MobileNet) on ImageNet dataset.

Downloads

Published

2021-05-18

How to Cite

Huo, Z., Gu, B., & Huang, H. (2021). Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9), 7883-7890. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16962

Issue

Section

AAAI Technical Track on Machine Learning II