THOR, Trace-based Hardware-driven Layer-Oriented Natural Gradient Descent Computation


  • Mengyun Chen Huawei Technologies Co. Ltd
  • Kaixin Gao Tianjin University
  • Xiaolei Liu Tianjin University
  • Zidong Wang Huawei Technologies Co. Ltd
  • Ningxi Ni Huawei Technologies Co. Ltd
  • Qian Zhang Beijing University of Technology
  • Lei Chen Hong Kong University of Science and Technology
  • Chao Ding Chinese Academy of Sciences
  • Zhenghai Huang Tianjin University
  • Min Wang Huawei Technologies Co. Ltd
  • Shuangling Wang Huawei Technologies Co. Ltd
  • Fan Yu Huawei Technologies Co. Ltd
  • Xinyuan Zhao Beijing University of Technology
  • Dachuan Xu Beijing University of Technology



(Deep) Neural Network Algorithms, Learning & Optimization for CV


It is well-known that second-order optimizer can accelerate the training of deep neural networks, however, the huge computation cost of second-order optimization makes it impractical to apply in real practice. In order to reduce the cost, many methods have been proposed to approximate a second-order matrix. Inspired by KFAC, we propose a novel Trace-based Hardware-driven layer-ORiented Natural Gradient Descent Computation method, called THOR, to make the second-order optimization applicable in the real application models. Specifically, we gradually increase the update interval and use the matrix trace to determine which blocks of Fisher Information Matrix (FIM) need to be updated. Moreover, by resorting the power of hardware, we have designed a Hardware-driven approximation method for computing FIM to achieve better performance. To demonstrate the effectiveness of THOR, we have conducted extensive experiments. The results show that training ResNet-50 on ImageNet with THOR only takes 66.7 minutes to achieve a top-1 accuracy of 75.9 % under an 8 Ascend 910 environment with MindSpore, a new deep learning computing framework. Moreover, with more computational resources, THOR can only takes 2.7 minutes to 75.9 % with 256 Ascend 910.




How to Cite

Chen, M., Gao, K., Liu, X., Wang, Z., Ni, N., Zhang, Q., Chen, L. ., Ding, C., Huang, Z., Wang, M., Wang, S., Yu, F., Zhao, X., & Xu, D. (2021). THOR, Trace-based Hardware-driven Layer-Oriented Natural Gradient Descent Computation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), 7046-7054.



AAAI Technical Track on Machine Learning I