We introduce the elliptical Ornstein-Uhlenbeck (OU) process, which is a generalisation of the well-known univariate OU process to bivariate time series. This process maps out elliptical stochastic oscillations over time in the complex plane, which are observed in many applications of coupled bivariate time series. The appeal of the model is that elliptical oscillations are generated using one simple first order stochastic differential equation (SDE), whereas alternative models require more complicated vectorised or higher order SDE representations. The second useful feature is that parameter estimation can be performed semi-parametrically in the frequency domain using the Whittle Likelihood. We determine properties of the model including the conditions for stationarity, and the geometrical structure of the elliptical oscillations. We demonstrate the utility of the model by measuring periodic and elliptical properties of Earth's polar motion.