Online Adaptive Methods, Universality and Acceleration

We present a novel method for convex unconstrained optimization that, without any modifications, ensures: (i) accelerated convergence rate for smooth objectives, (ii) standard convergence rate in the general (non-smooth) setting, and (iii) standard convergence rate in the stochastic optimization setting. To the best of our knowledge, this is the first method that simultaneously applies to all of the above settings. At the heart of our method is an adaptive learning rate rule that employs importance weights, in the spirit of adaptive online learning algorithms, combined with an update that linearly couples two sequences. An empirical examination of our method demonstrates its applicability to the above mentioned scenarios and corroborates our theoretical findings.

Presented at:
32nd Conference on Neural Information Processing Systems conference (NIPS 2018), Montreal, Canada, December 3-8, 2018
Jul 04 2018

Note: The status of this file is: Anyone

 Record created 2018-07-04, last modified 2020-04-20

Download fulltext

Rate this document:

Rate this document:
(Not yet reviewed)