This thesis presents results of studies of novel compounds modeling complex fundamental physics phenomena. Cu2OSO4 is a copper based magnetic Mott Insulator system, where spin half magnetic moments form a new type of lattice. These intrinsically quantum pins are exhibiting atypical magnetic order and spin dynamics. The recent success in the growth of large single crystals of Cu2OSO4 enabled to perform measurements probing its static and fluctuating properties. The peculiarity of this sample is that its atoms are forming layers, with a geometry close to the intensively studied Kagomé lattice, but with a third of its spins replaced by dimers. This quantum magnetism system has been probed in its bulk, by the means of heat capacity and DC-susceptibility measurements, revealing a transition to a magnetically long range ordered state upon cooling, the details of which are revealed by neutron scattering. Single crystal inelastic neutron scattering shed light on the spin-dynamics in the system, with clear spin waves appearing as fluctuations around the peculiar ground state of the system: a 120 degrees spin configuration where the magnetic moment of the spin-dimer causes the sample to be globally ferrimagnetic. The presented results indicate that Cu2OSO4 represents a new type of model lattice with frustrated interactions where interplay between magnetic order, thermal and quantum fluctuations can be explored. The magnetic excitations of the compound can be modeled by a yet-to-be-understood internal effective mean-field that no simple magnetic coupling seems to reproduce. K2Ni2(SO4)3 is another compound that allows for the existence of non-trivial topological phases. This thesis presents results of the study of the unusual magnetic behavior of K2Ni2(SO4)3. No clear sign of well-established magnetic long range order has been observed down to dilution temperatures. Neutron scattering reveals the details of the competition between frustration and fluctuations that prevent order from settling in. Low temperature spin excitations take the form of a continuum at 500 mK, but also of broad, energy independent continua at higher temperatures. Bulk and neutron scattering measurements are put in perspective and linked together with a view to building up a better understanding of how quantum spin liquids can be stabilized in general, and in particular in this model compound. Finally, the last contribution of this thesis to the field of condensed matter physics regards the establishment of a state-of-the-art technique to fit heat capacity and unit cell volume of samples to try and make the extraction of magnetic information from specific heat measurements more robust. This newly-developed technique consists in modeling lattice contributions with better accuracy by using data from multiple experimentally accessible quantities to consolidate the fitting scheme. This method has been cautiously applied to several compounds at the forefront of research in experimental physics.