We investigate the absorption spectra of the $F$ centers in MgO and CaO using material-specific hybrid functionals combined with time-dependent hybrid-functional theory calculations. Our study accounts for electron-hole interaction, zero-phonon renormalization, and finite-size effects, which are all found to contribute to the achieved accuracy. The spectra of the neutral states solely present absorption lines stemming from internal defect-to-defect transitions without involving conduction band states of the host. At variance, the spectra of the singly positively charged states show two lines, one analogous to those seen in the neutral states and an extra line corresponding to electron excitations from the valence band to the unoccupied defect level in the band gap. In MgO, the three transition energies pertaining to these two charged states are found to coincide, but are spread out in the case of CaO. The agreement with the energies of experimental lines is excellent, indicating that our scheme reaches an accuracy of 0.2 eV or better.