In this paper, we present a flexible new technique for single-viewpoint omnidirectional camera calibration. The proposed method only re-quires the camera to observe a planar pattern shown at a few different orienta-tions. Either the camera or the planar pattern can be freely moved. No a priori knowledge of the motion is required, nor a specific model of the omnidirec-tional sensor. The only assumption is that the image projection function can be described by a Taylor series expansion whose coefficients are estimated by solving a two-step least-squares linear minimization problem. To test the pro-posed technique, we calibrated a panoramic camera having a vision angle greater than 200° in the vertical direction, and we obtained very good results. To investigate the accuracy of the calibration, we also used the estimated omni-camera model in a structure from motion experiment. We obtained a 3D metric reconstruction of a scene from two highly distorted omnidirectional images by using image correspondences only. Compared with classical techniques, which rely on a specific parametric model of the omnidirectional camera, the pro-posed procedure is independent of the sensor, easy to use and flexible.