Mohtashami, AmirkeivanJaggi, MartinStich, Sebastian U.2022-11-072022-11-072022-11-072022-01-01https://infoscience.epfl.ch/handle/20.500.14299/191911WOS:000841852300012State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a subset of parameters for increased efficiency (such as meProp) or a combination of both (such as Dropout). However, the convergence of these methods is often not studied in theory.We propose a unified theoretical framework to study such SGD variants-encompassing the aforementioned algorithms and additionally a broad variety of methods used for communication efficient training or model compression. Our insights can be used as a guide to improve the efficiency of such methods and facilitate generalization to new applications. As an example, we tackle the task of jointly training networks, a version of which (limited to sub-networks) is used to create Slimmable Networks. By training a low-rank Transformer jointly with a standard one we obtain superior performance than when it is trained separately.Computer Science, Artificial IntelligenceStatistics & ProbabilityComputer ScienceMathematicstext::conference output::conference proceedings::conference paper