While at cryogenic temperatures, quantum dots generate single photons at a narrowband spectrum, temporal fluctuations in its peak position pose a significant challenge towards control and metrology protocols. Up to date, evidences of spectral fluctuations are by large measured indirectly. In this work, heralded spectroscopy is used to directly measure spectral fluctuations for single CsPbBr3 nanocrystals across 8 orders of magnitude in time, down to the microsecond. We find that spectral noise spans this entire temporal range with a weak power-law dependence and critically enhance with the excitation power, demonstrating a clear tradeoff between brightness and spectral stability. Further systematic studies of the parameters that determine the amplitude and time scale of spectral diffusion will aim to clarify its underlying mechanism with the outlook of synthetically overcoming this challenge.