Repository logo

Infoscience

  • English
  • French
Log In
Logo EPFL, École polytechnique fédérale de Lausanne

Infoscience

  • English
  • French
Log In
  1. Home
  2. Academic and Research Output
  3. Conferences, Workshops, Symposiums, and Seminars
  4. Stable Nonconvex-Nonconcave Training via Linear Interpolation
 
conference paper not in proceedings

Stable Nonconvex-Nonconcave Training via Linear Interpolation

Pethick, Thomas Michaelsen  
•
Xie, Wanyun  
•
Cevher, Volkan  orcid-logo
September 21, 2023
Thirty-seventh Conference on Neural Information Processing Systems

This paper presents a theoretical analysis of linear interpolation as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear interpolation can help by leveraging the theory of nonexpansive operators. We construct a new optimization scheme called relaxed approximate proximal point (RAPP), which is the first 1-SCLI method to achieve last iterate convergence rates for $\rho$-comonotone problems while only requiring $\rho > -\tfrac{1}{2L}$. The construction extends to constrained and regularized settings. By replacing the inner optimizer in RAPP we rediscover the family of Lookahead algorithms for which we establish convergence in cohypomonotone problems even when the base optimizer is taken to be gradient descent ascent. The range of cohypomonotone problems in which Lookahead converges is further expanded by exploiting that Lookahead inherits the properties of the base optimizer. We corroborate the results with experiments on generative adversarial networks which demonstrates the benefits of the linear interpolation present in both RAPP and Lookahead.

  • Files
  • Details
  • Metrics
Type
conference paper not in proceedings
Author(s)
Pethick, Thomas Michaelsen  
Xie, Wanyun  
Cevher, Volkan  orcid-logo
Date Issued

2023-09-21

Subjects

Minimax optimization

•

Lookahead

•

Generative adversarial networks

•

Stability

•

Nonconvex-nonconcave

•

Cohypomonotone

Editorial or Peer reviewed

REVIEWED

Written at

EPFL

EPFL units
LIONS  
Event nameEvent placeEvent date
Thirty-seventh Conference on Neural Information Processing Systems

New Orleans, Louisiana, USA

December 10-16, 2023

Available on Infoscience
October 20, 2023
Use this identifier to reference this record
https://infoscience.epfl.ch/handle/20.500.14299/201675
Logo EPFL, École polytechnique fédérale de Lausanne
  • Contact
  • infoscience@epfl.ch

  • Follow us on Facebook
  • Follow us on Instagram
  • Follow us on LinkedIn
  • Follow us on X
  • Follow us on Youtube
AccessibilityLegal noticePrivacy policyCookie settingsEnd User AgreementGet helpFeedback

Infoscience is a service managed and provided by the Library and IT Services of EPFL. © EPFL, tous droits réservés