Recent ultrafast magnetic-sensitive measurements [Johnson et al., Phys. Rev. B 92, 184429 (2015); Bothschafter et al., Phys. Rev. B 96, 184414 (2017)] have revealed a delayed melting of the long-range cycloid spin order in TbMnO 3 following photoexcitation across the fundamental Mott-Hubbard gap. The microscopic mechanism behind this slow transfer of energy from the photoexcited carriers to the spin degrees of freedom is still elusive and not understood. Here, we address this problem by combining spectroscopic ellipsometry, ultrafast broadband optical spectroscopy, and ab initio calculations. Upon photoexcitation, we observe the emergence of a complex collective response, which is due to high-energy coherent optical phonons coupled to the out-of-equilibrium charge density. This response precedes the magnetic order melting and is interpreted as the fingerprint of the formation of anti-Jahn-Teller polarons. We propose that the charge localization in a long-lived self-trapped state hinders the emission of magnons and other spin-flip mechanisms, causing the energy transfer from the charge to the spin system to be mediated by the reorganization of the lattice. Furthermore, we provide evidence for the coherent excitation of a phonon mode associated with the ferroelectric phase transition.