We present a systematic study of the crystal-field interactions in the LiRF4 (R = Gd, Ho, Er, Tm, and Yb) family of rare-earth magnets. Using detailed inelastic neutron scattering measurements, we have been able to quantify the transition energies and wave functions for each system. This allows us to quantitatively describe the high-temperature susceptibility measurements for the series of materials and make predictions based on a mean-field approach for the low-temperature thermal and quantum phase transitions. We show that coupling between crystal field and phonon states leads to line-shape broadening in LiTmF4 and level splitting in LiYbF4. Furthermore, using high-resolution neutron scattering from LiHoF4, we find anomalous broadening of crystal-field excitations which we attribute to magnetoelastic coupling.