While quantum mechanics imposes a fundamental limit on the precision of interferometric measurementsof mechanical motion due to measurement backaction, the nonlinear nature of the coupling also leads toparametric instabilities that place practical limits on the sensitivity by limiting the power in theinterferometer. Such instabilities have been extensively studied in the context of gravitational wavedetectors, and their presence has recently been reported in Advanced LIGO. Here, we observeexperimentally and describe theoretically a new type of optomechanical instability that arises in two-tone backaction-evading (BAE) measurements, a protocol designed to overcome the standard quantumlimit. We demonstrate the effect in the optical domain with a photonic crystal nanobeam cavity and in themicrowave domain with a micromechanical oscillator coupled to a microwave resonator. In contrast to thewell-known parametric oscillatory instability that occurs in single-tone, blue-detuned pumping, and resultsfrom a two-mode squeezing interaction between the optical and mechanical modes, the parametricinstability in balanced two-tone optomechanics results from single-mode squeezing of the mechanical modein the presence of small detuning errors in the two pump frequencies. Counterintuitively, the instabilityoccurs even in the presence of perfectly balanced intracavity fields and can occur for both signs of detuningerrors. We find excellent quantitative agreement with our theoretical predictions. Since the constraints ontuning accuracy become stricter with increasing probe power, the instability imposes a fundamentallimitation on BAE measurements as well as other two-tone schemes, such as dissipative squeezing of opticaland microwave fields or of mechanical motion. In addition to identifying a new limitation in two-tone BAEmeasurements, the results also introduce a new type of nonlinear dynamics in cavity optomechanics.