Learning with Retrospection


  • Xiang Deng State University of New York at Binghamton
  • Zhongfei Zhang State University of New York at Binghamton




Classification and Regression, (Deep) Neural Network Algorithms


Deep neural networks have been successfully deployed in various domains of artificial intelligence, including computer vision and natural language processing. We observe that the current standard procedure for training DNNs discards all the learned information in the past epochs except the current learned weights. An interesting question is: is this discarded information indeed useless? We argue that the discarded information can benefit the subsequent training. In this paper, we propose learning with retrospection (LWR) which makes use of the learned information in the past epochs to guide the subsequent training. LWR is a simple yet effective training framework to improve accuracies, calibration, and robustness of DNNs without introducing any additional network parameters or inference cost, but only with a negligible training overhead. Extensive experiments on several benchmark datasets demonstrate the superiority of LWR for training DNNs.




How to Cite

Deng, X., & Zhang, Z. (2021). Learning with Retrospection. Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), 7201-7209. https://doi.org/10.1609/aaai.v35i8.16885



AAAI Technical Track on Machine Learning I