This paper considers the problem of estimating the focus of expansion of optical flow fields from panoramic image sequences due to ego-motion of the camera. The focus of expansion provides a measurement of the direction of motion of the vehicle that is a key requirement for implementing obstacle avoidance algorithms. We propose a two stage approach to this problem. Firstly, external angular rotation measurements provided by an on-board inertial measurement unit are used to de-rotate the observed optic flow field. Then a robust statistical method is applied to provide an estimate of the focus of expansion as well as a selection of inlier data points associated with the hypothesis. This is followed by a least squares minimisation, utilising only the inlier data, that provides accurate estimates of residual angular rotation and focus of expansion of the flow. The least squares optimisation is solved using a geometric Newton algorithm. For the robust estimator we consider and compare RANSAC and a k-means algorithm. The approach in this paper does not require explicit features, and can be applied to patchy, noisy sparse optic flow fields. The approach is demonstrated in simulations and on video data obtained from an aerial robot equipped with panoramic cameras.