TY - JOUR
AU - Ren, Zhaolin
AU - Zhou, Zhengyuan
AU - Qiu, Linhai
AU - Deshpande, Ajay
AU - Kalagnanam, Jayant
PY - 2020/04/03
Y2 - 2021/04/12
TI - Delay-Adaptive Distributed Stochastic Optimization
JF - Proceedings of the AAAI Conference on Artificial Intelligence
JA - AAAI
VL - 34
IS - 04
SE - AAAI Technical Track: Machine Learning
DO - 10.1609/aaai.v34i04.6001
UR - https://ojs.aaai.org/index.php/AAAI/article/view/6001
SP - 5503-5510
AB - <p>In large-scale optimization problems, <em>distributed asynchronous stochastic gradient descent</em> (DASGD) is a commonly used algorithm. In most applications, there are often a large number of computing nodes asynchronously computing gradient information. As such, the gradient information received at a given iteration is often stale. In the presence of such delays, which can be unbounded, the convergence of DASGD is uncertain. The contribution of this paper is twofold. First, we propose a delay-adaptive variant of DASGD where we adjust each iteration's step-size based on the size of the delay, and prove asymptotic convergence of the algorithm on variationally coherent stochastic problems, a class of functions which properly includes convex, quasi-convex and star-convex functions. Second, we extend the convergence results of standard DASGD, used usually for problems with bounded domains, to problems with unbounded domains. In this way, we extend the frontier of theoretical guarantees for distributed asynchronous optimization, and provide new insights for practitioners working on large-scale optimization problems.</p>
ER -