We report the study of the optical and mechanical properties of high-Q fused silica microtoroidal resonators at cryogenic temperatures (down to 1.6 K). A thermally induced optical multistability is observed and theoretically described, originating from the reverse thermally induced optical frequency shift. Moreover the influence of structural defect states (two level fluctuators) on their mechanical properties is observed and probed at an unprecedentedly achieved low phonon number. The resulting implications for cavity optomechanics and studies of mechanical decoherence are also discussed.