In quantum mechanics, the Heisenberg uncertainty principle places a fundamental limit in the measurement precision for certain pairs of physical quantities, such as position and momentum, time and energy or amplitude and phase. Due to the Heisenberg uncertainty principle, any attempt to extract certain information from a quantum object would inevitably perturbitinanunpredictableway. This raises one question,"What is the precision limit in such quantum measurements?" The answer, standard quantum limit (SQL), has been obtained by Braginsky to figure out the fundamental quantum limits of displacement measurement in the context of gravitational wave detection. To circumvent the unavoidable quantum back-action from the priori measurement, quantum non-demolition measurement (QND) methods were introduced by Braginsky and Thorne. To surpass the SQL of the displacement measurement in an interferometer, one can measure only one quadrature of the mechanical motion while give up the information about the other canonically conjugated quadrature. Such measurements can be performed by periodic driving the mechanical oscillator, i.e. the back-action evading (BAE) measurement. Cavity optomechanics provides an ideal table-top platform for the testing of the quantum measurement theory. Mechanical oscillator is coupled to electromagnetic field via radiation pressure, which is enhanced by an optical micro-cavity. Over the last decade, laser cooling has enabled the preparation of mechanical oscillator in the ground state in both optical and microwave systems. BAE measurements of mechanical motion have been allowed in the microwave electromechanical systems, which led to the observations of mechanical squeezing and entanglement. However, despite the theoretical proposal almost 40 years ago, the sub-SQL measurements still remain elusive. This thesis reports our efforts approach the sub-SQL with a highly sideband-resolved silicon optomechanical crystal (OMC) in a 3He buffer gas environment at 2K. The OMC couples an optical mode at telecommunication wavelengths and a colocalized mechanical mode at GHz frequencies. The Helium3 buffer gas environment allows sufficient thermalization of the OMC despite the drastically decreased silicon thermal conductivity. We observe Floquet dynamics in motional sideband asymmetry measurement when employing multiple probe tones. The Floquet dynamics arises due to presence of Kerr-type nonlinearities and gives rise to an artificially modified motional sideband asymmetry, resulting from a synthetic gauge field among the Fourier modes. We demonstrate the first optical continuous two-tone backaction-evading measurement of a localized GHz frequency mechanical mode of silicon OMC close to the ground state by showing the transition from conventional sideband asymmetry to backaction-evading measurement. We discover a fundamental two-tone optomechanical instability and demonstrate its implications on the back-action evading measurement. Such instability imposes a fundamental limitation on other two-tone schemes, such as dissipative quantum mechanical squeezing. We demonstrate state-of-art laser sideband cooling of the mechanical motion to a mean thermal occupancy of 0.09 quantum, which is 7.4dB of the oscillator's zero-point energy and corresponds to 92% ground state probability. This also enables us to observe the dissipative mechanical squeezing below the zero-point motion for the first time with laser light.