We investigate the spin relaxation of Ho single atom magnets on MgO/Ag(100) as a function of temperature and magnetic field. We find that the spin relaxation is thermally activated at low field, while it remains larger than 1000 s up to 30 K and 8 T. This behavior contrasts with that of single molecule magnets and bulk paramagnetic impurities, which relax faster at high field. Combining our results with density functional theory, we rationalize this unconventional behavior by showing that local vibrations activate a two-phonon Raman process with a relaxation rate that peaks near zero field and is suppressed at high field. Our work shows the importance of these excitations in the relaxation of axially coordinated magnetic atoms.