Repository logo

Infoscience

  • English
  • French
Log In
Logo EPFL, École polytechnique fédérale de Lausanne

Infoscience

  • English
  • French
Log In
  1. Home
  2. Academic and Research Output
  3. Conferences, Workshops, Symposiums, and Seminars
  4. ADAGRAD Avoids Saddle Points
 
conference paper

ADAGRAD Avoids Saddle Points

Antonakopoulos, Kimon  
•
Mertikopoulos, Panayotis
•
Piliouras, Georgios
Show more
January 1, 2022
International Conference On Machine Learning, Vol 162
38th International Conference on Machine Learning (ICML)

Adaptive first-order methods in optimization are prominent in machine learning and data science owing to their ability to automatically adapt to the landscape of the function being optimized. However, their convergence guarantees are typically stated in terms of vanishing gradient norms, which leaves open the issue of converging to undesirable saddle points (or even local maximizers). In this paper, we focus on the ADAGRAD family of algorithms - with scalar, diagonal or full-matrix preconditioning - and we examine the question of whether the method's trajectories avoid saddle points. A major challenge that arises here is that ADAGRAD's step-size (or, more accurately, the method's preconditioner) evolves over time in a filtration-dependent way, i.e., as a function of all gradients observed in earlier iterations; as a result, avoidance results for methods with a constant or vanishing step-size do not apply. We resolve this challenge by combining a series of step-size stabilization arguments with a recursive representation of the ADAGRAD preconditioner that allows us to employ stable manifold techniques and ultimately show that the induced trajectories avoid saddle points from almost any initial condition.

  • Details
  • Metrics
Type
conference paper
Web of Science ID

WOS:000899944900033

Author(s)
Antonakopoulos, Kimon  
Mertikopoulos, Panayotis
Piliouras, Georgios
Wang, Xiao
Date Issued

2022-01-01

Publisher

JMLR-JOURNAL MACHINE LEARNING RESEARCH

Publisher place

San Diego

Published in
International Conference On Machine Learning, Vol 162
Series title/Series vol.

Proceedings of Machine Learning Research

Start page

731

End page

771

Subjects

Computer Science, Artificial Intelligence

•

Computer Science

Editorial or Peer reviewed

REVIEWED

Written at

EPFL

EPFL units
LIONS  
Event nameEvent placeEvent date
38th International Conference on Machine Learning (ICML)

Baltimore, MD

Jul 17-23, 2022

Available on Infoscience
March 13, 2023
Use this identifier to reference this record
https://infoscience.epfl.ch/handle/20.500.14299/195876
Logo EPFL, École polytechnique fédérale de Lausanne
  • Contact
  • infoscience@epfl.ch

  • Follow us on Facebook
  • Follow us on Instagram
  • Follow us on LinkedIn
  • Follow us on X
  • Follow us on Youtube
AccessibilityLegal noticePrivacy policyCookie settingsEnd User AgreementGet helpFeedback

Infoscience is a service managed and provided by the Library and IT Services of EPFL. © EPFL, tous droits réservés