We resolve the thermal motion of a high-stress silicon nitride nanobeam at frequencies far below its fundamental flexural resonance (3.4 MHz) using cavity-enhanced optical interferometry. Over two decades, the displacement spectrum is well-modeled by that of a damped harmonic oscillator driven by a thermal force, suggesting that the loss angle of the beam material is frequency-independent. The inferred loss angle at 3.4 MHz agrees well with the quality factor (Q) of the fundamental beam mode. In conjunction with Q measurements made on higher order flexural modes, and accounting for the mode dependence of stress-induced loss dilution, we find that the intrinsic (undiluted) loss angle of the beam changes by less than a factor of 2 between 50 kHz and 50 MHz. We discuss the impact of such “structural damping” on experiments in quantum optomechanics, in which the thermal force acting on a mechanical oscillator coupled to an optical cavity is overwhelmed by radiation pressure shot noise. As an illustration, we show that structural damping reduces the bandwidth of ponderomotive squeezing.