We generalize the Kuramoto model for coupled phase oscillators by allowing the frequencies to drift in time according to Ornstein-Uhlenbeck dynamics. Such drifting frequencies were recently measured in cellular populations of circadian oscillator and inspired our work. Linear stability analysis of the Fokker-Planck equation for an infinite population is amenable to exact solution and we show that the incoherent state is unstable past a critical coupling strength K(c)(gamma,sigma(f)), where gamma is the inverse characteristic drifting time and sigma(f) the asymptotic frequency dispersion. Expectedly K(c)agrees with the noisy Kuramoto model in the large gamma (Schmolukowski) limit but increases slower as gamma decreases. Asymptotic expansion of the solution for gamma-->0 shows that the noiseless Kuramoto model with Gaussian frequency distribution is recovered in that limit. Thus varying a single parameter allows us to interpolate smoothly between two regimes: one dominated by the frequency dispersion and the other by phase diffusion.