Liu, YueLin, TaoKoloskova, AnastasiiaStich, Sebastian U.2024-04-032024-04-032024-04-032024-03-1210.1080/10556788.2024.2322095https://infoscience.epfl.ch/handle/20.500.14299/206943WOS:001185046400001Gradient tracking (GT) is an algorithm designed for solving decentralized optimization problems over a network (such as training a machine learning model). A key feature of GT is a tracking mechanism that allows us to overcome data heterogeneity between nodes. We develop a novel decentralized tracking mechanism, K-GT, which enables communication-efficient local updates in GT while inheriting the data-independence property of GT. We prove a convergence rate for K-GT on smooth non-convex functions and prove that it reduces the communication overhead asymptotically by a linear factor K, where K denotes the number of local steps. We illustrate the robustness and effectiveness of this heterogeneity correction on convex and non-convex benchmark problems and a non-convex neural network training task with the MNIST dataset.TechnologyPhysical SciencesGradient TrackingDecentralized LearningConvergence RateLocal Stepstext::journal::journal article::research article