LiReF$_4$ ("Re" stands for the rare-earth element) and their doped derivatives have long been recognized as a family of compounds that exhibit rich phenomena in quantum magnetism, drawing wide attention to them from both fundamental researchers and industrial application designers. In particular, LiHoF$_4$ is considered one of the best real-world realizations of the 3D dipolar Ising model, which is of tremendous importance in facilitating our understanding of quantum phase transition, critical behaviors, and magnetism in general. However, despite of decades of study, LiHoF$_4$ is yet to be fully described and a thorough understanding of it is hindered by complications arising from its dominant long range interaction, many-body effect, and diverse inter-particle interactions. One particular example is the strong hyperfine interaction present in LiHoF$_4$ that has a plethora of physical consequences with unclear mechanisms in addition to the further split of electronic spin states. Partly due to the fact of that these finer electro-nuclear states lie above the operating frequency of conventional NMR, and below the energy resolution of most neutron scattering instrument, scarce reports exist on the investigation of the hyperfine splitting of electronic states in LiHoF$_4$. In the present text, we take advantage of a high-finesse re-entrant cavity resonator to resolve these electro-nuclear spin states. In addition, by deriving an adequate theoretical description of the cavity-spin hybrid system under existing frameworks, we also make an attempt to understand their effects on some of the magnetic properties of LiHoF$_4$.