We developed an apparatus to couple a 50-mu m diameter whispering-gallery silica microtoroidal resonator in a helium-4 cryostat using a straight optical tapered-fiber at 1550 nm wavelength. On a top-loading probe specifically adapted for increased mechanical stability, we use a specifically-developed "cryotaper" to optically probe the cavity, allowing thus to record the calibrated mechanical spectrum of the optomechanical system at low temperatures. We then demonstrate excellent thermalization of a 63-MHz mechanical mode of a toroidal resonator down to the cryostat's base temperature of 1.65 K, thereby proving the viability of the cryogenic refrigeration via heat conduction through static low-pressure exchange gas. In the context of optomechanics, we therefore provide a versatile and powerful tool with state-of-the-art performances in optical coupling efficiency, mechanical stability, and cryogenic cooling. (C) 2013 AIP Publishing LLC. [http://dx.doi.org/10.1063/1.4801456]