We present a microscopic theory for the phonon-driven decay of the magnetization fluctuations in a wide class of nanomagnets where the dominant energy is set by isotropic exchange and/or uniaxial anisotropy. Based on the Zwanzig-Mori projection formalism, the theory reveals that the magnetization fluctuations are governed by a single decay rate omega(c), which we further identify with the zero-frequency portion of the associated self-energy. This dynamical decoupling from the remaining slow degrees of freedom is attributed to a conservation law and the discreteness of the energy spectrum and explains the omnipresent monoexponential decay of the magnetization over several decades in time, as observed experimentally. A physically transparent analytical expression for omega(c) is derived which highlights the three specific mechanisms of the slowing-down effect which are known so far in nanomagnets.