Asynchronous Distributed Semi-Stochastic Gradient Optimization

Authors

  • Ruiliang Zhang Hong Kong University of Science and Technology
  • Shuai Zheng Hong Kong University of Science and Technology
  • James T. Kwok Hong Kong University of Science and Technology

DOI:

https://doi.org/10.1609/aaai.v30i1.10286

Keywords:

Asynchronous, Distributed Computing, Stochastic Gradient Decent, SGD, Optimization, Large Scale Learning, Big Data

Abstract

With the recent proliferation of large-scale learning problems, there have been a lot of interest on distributed machine learning algorithms, particularly those that are based on stochastic gradient descent (SGD) and its variants. However, existing algorithms either suffer from slow convergence due to the inherent variance of stochastic gradients, or have a fast linear convergence rate but at the expense of poorer solution quality. In this paper, we combine their merits by proposing a fast distributed asynchronous SGD-based algorithm with variance reduction. A constant learning rate can be used, and it is also guaranteed to converge linearly to the optimal solution. Experiments on the Google Cloud Computing Platform demonstrate that the proposed algorithm outperforms state-of-the-art distributed asynchronous algorithms in terms of both wall clock time and solution quality.

Downloads

Published

2016-03-02

How to Cite

Zhang, R., Zheng, S., & Kwok, J. T. (2016). Asynchronous Distributed Semi-Stochastic Gradient Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.10286

Issue

Section

Technical Papers: Machine Learning Methods