X-ray magnetic circular dichroism, atomic multiplet simulations, and density functional theory calculations are employed to identify criteria for the optimum combination of supporting alkaline earth oxide and adsorption site maximizing the spin lifetimes of lanthanide single-atom magnets. Dy and Ho atoms adsorbed on BaO(100) thin films on Pt(100) are characterized and compared with previous results for the same two elements on MgO/Ag(100). Dy shows hysteresis in magnetic fields up to approximate to 3.5 T and long spin lifetime, exceeding 300 s at 2.5 K and 0.5 T. Dy displays superior magnetic stability on the bridge site than on the top-O site. Surprisingly, Ho shows paramagnetism, as opposed to its long spin lifetime on MgO. These differences originate from the local surface distortions induced by the adatoms. On MgO, minimal distortions involve only the closest O atoms, while, on BaO, they affect both the closest anions and cations. This trend reflects the decrease of the lattice energy along the series of the alkaline earth oxides, going from MgO to BaO. This study represents a step ahead in the understanding of the factors determining the spin dynamics of surface-adsorbed single-atom magnets in order to achieve their operation as qubits and memories.