Investigation into Training Dynamics of Learned Optimizers (Student Abstract)

Authors

  • Jan Sobotka Faculty of Information Technology, Czech Technical University in Prague
  • Petr Šimánek Faculty of Information Technology, Czech Technical University in Prague

DOI:

https://doi.org/10.1609/aaai.v38i21.30514

Keywords:

Optimization, Deep Learning, Applications Of AI

Abstract

Modern machine learning heavily relies on optimization, and as deep learning models grow more complex and data-hungry, the search for efficient learning becomes crucial. Learned optimizers disrupt traditional handcrafted methods such as SGD and Adam by learning the optimization strategy itself, potentially speeding up training. However, the learned optimizers' dynamics are still not well understood. To remedy this, our work explores their optimization trajectories from the perspective of network architecture symmetries and proposed parameter update distributions.

Published

2024-03-24

How to Cite

Sobotka, J., & Šimánek, P. (2024). Investigation into Training Dynamics of Learned Optimizers (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence, 38(21), 23657-23658. https://doi.org/10.1609/aaai.v38i21.30514