In this paper we investigate the direct effect of resonator quality factor (Q) on the oscillator phase noise. We use 2-port contour mode resonators (CMRs), fabricated both with aluminum nitride (AlN) and aluminum scandium nitride (AlScN), as the frequency-determining element in the oscillator circuit. Over 70 oscillator configurations are tested using resonators with different Q and with different piezoelectric layer. The testing of so many devices is possible because, in our setup, interfacing the circuit to the resonator is streamlined using RF probes. Our results show that higher resonator Q yields better frequency stability of the oscillator, for both AlN and AlScN CMRs. Interestingly, the comparison between AlN-based oscillators and AlScN-based oscillators with equal Q shows that AlScN-based oscillators' Phase Noise is up 10 dBc/Hz better than the AlN oscillator at 1 kHz offset frequency, suggesting different intrinsic resonator flicker noise of the two piezoelectric layers. [2019-0200]