Galimberti, ClaraFurieri, LucaXu, LiangFerrari Trecate, Giancarlo2023-04-032023-04-032023-01-2410.1109/TAC.2023.3239430https://infoscience.epfl.ch/handle/20.500.14299/196677Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretization of continuous-time Hamiltonian systems and include several existing DNN architectures based on ordinary differential equations. Our main result is that a broad set of H-DNNs ensures non-vanishing gradients by design for an arbitrary network depth. This is obtained by proving that, using a semi-implicit Euler discretization scheme, the backward sensitivity matrices involved in gradient computations are symplectic. We also provide an upper-bound to the magnitude of sensitivity matrices and show that exploding gradients can be controlled through regularization. The good performance of H-DNNs is demonstrated on benchmark classification problems, including image classification with the MNIST dataset.Deep neural networksHamiltonian systemsODE discretizationtext::journal::journal article::research article