Learning with Retrospection

Authors

  • Xiang Deng State University of New York at Binghamton
  • Zhongfei Zhang State University of New York at Binghamton

Keywords:

Classification and Regression, (Deep) Neural Network Algorithms

Abstract

Deep neural networks have been successfully deployed in various domains of artificial intelligence, including computer vision and natural language processing. We observe that the current standard procedure for training DNNs discards all the learned information in the past epochs except the current learned weights. An interesting question is: is this discarded information indeed useless? We argue that the discarded information can benefit the subsequent training. In this paper, we propose learning with retrospection (LWR) which makes use of the learned information in the past epochs to guide the subsequent training. LWR is a simple yet effective training framework to improve accuracies, calibration, and robustness of DNNs without introducing any additional network parameters or inference cost, but only with a negligible training overhead. Extensive experiments on several benchmark datasets demonstrate the superiority of LWR for training DNNs.

Downloads

Published

2021-05-18

How to Cite

Deng, X., & Zhang, Z. (2021). Learning with Retrospection. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), 7201-7209. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16885

Issue

Section

AAAI Technical Track on Machine Learning I