The insulating rare-earth magnet LiY1-xHoxF4 has received great attention because a laboratory field applied perpendicular to its crystallographic c axis converts the low-energy electronic spin Hamiltonian into the (dilute) transverse field Ising model. The mapping between the real magnet and the transverse field Ising model is strongly dependent on the exact nature of the low-energy Hamiltonian for the material, which can be determined by spectroscopy in the dilute limit. The energies of the eigenstates are in the difficult terahertz (THz) regime, and here we use THz time domain and Fourier transform spectroscopy to directly measure the lowest crystal-field levels of LiY1-xHoxF4 in the dilute limit, including nuclear hyperfine substructure. The high resolution of our measurements allows us to observe the nonequidistantly spaced Ho (I = 7/2) hyperfine transitions originating from dipolar and quadrupolar hyperfine interactions. We provide refined crystal-field parameters and extract the dipolar and quadrupolar hyperfine constants A(J) = 0.027 03 +/- 0.000 03 cm(-1) (810.3 +/- 0.9 MHz) and B = 0.04 +/- 0.01 cm(-1)(1.2 +/- 0.3 GHz), respectively. Thereupon we determine all crystal-field energy levels and magnetic moments of the I-5(8) ground-state manifold, including the (nonlinear) hyperfine corrections. The latter improve the prediction precision by a factor of 60 compared to previous crystal-field parameters. Additionally, we establish the far-infrared, low-temperature refractive index of LiY1-xHoxF4.