Adaptive Orthogonal Projection for Batch and Online Continual Learning


  • Yiduo Guo Peking University
  • Wenpeng Hu Peking University
  • Dongyan Zhao Peking Univeristy
  • Bing Liu UIC



Machine Learning (ML), Computer Vision (CV), Cognitive Modeling & Cognitive Systems (CMS)


Catastrophic forgetting is a key obstacle to continual learning. One of the state-of-the-art approaches is orthogonal projection. The idea of this approach is to learn each task by updating the network parameters or weights only in the direction orthogonal to the subspace spanned by all previous task inputs. This ensures no interference with tasks that have been learned. The system OWM that uses the idea performs very well against other state-of-the-art systems. In this paper, we first discuss an issue that we discovered in the mathematical derivation of this approach and then propose a novel method, called AOP (Adaptive Orthogonal Projection), to resolve it, which results in significant accuracy gains in empirical evaluations in both the batch and online continual learning settings without saving any previous training data as in replay-based methods.




How to Cite

Guo, Y., Hu, W., Zhao, D., & Liu, B. (2022). Adaptive Orthogonal Projection for Batch and Online Continual Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6), 6783-6791.



AAAI Technical Track on Machine Learning I