Parametric coupling of optical and mechanical degrees of freedom forms the basis of many ultra-sensitive measurements of both force and mechanical displacement. An optical cavity with a mechanically compliant boundary enhances the optomechanical interaction, which gives rise to qualitatively new behavior which can modify the dynamics of the mechanical motion. As early as 1967, in a pioneering work, V. Braginsky analyzed theoretically the role of radiation pressure in the interferometric measurement process, but it has remained experimentally unexplored for many decades. Here, we use whispering-gallery mode optical microresonators to study these radiation pressure phenomena. Optical microresonators simultaneously host optical and mechanical modes, which are systematically analyzed and optimized to feature ultra-low mechanical dissipation, photon storage times exceeding the mechanical oscillation period (i.e. the "resolved-sideband regime") and large optomechanical coupling. In this manner, it is demonstrated for the first time that dynamical backaction can be employed to cool mechanical modes, i.e., to reduce their thermally excited random motion. Utilizing this novel technique together with cryogenic pre-cooling of the mechanical oscillator, the phonon occupation of mechanical radial-breathing modes could be reduced to (n) = 63 +/- 20 excitation quanta. The corresponding displacement fluctuations are monitored interferometrically with a sensitivity at the level of 1.10(-18) m/root Hz, which is below the standard quantum limit (SQL). This implies that the readout is already in principle sufficient to measure the quantum mechanical zero-point position fluctuations of the mechanical mode. Moreover, it is shown that optical measurement techniques employed here are operating in a near-ideal manner according to the principles of quantum measurement, displaying a backaction-imprecision product close to the quantum limit.