This paper addresses the reconstruction of high resolution omnidirectional images from a low resolution video acquired by an omnidirectional camera moving in a static scene. In order to exploit the additional information provided by the side images in the video sequence, the ego-motion of the camera must be accurately estimated in a first step. The reconstruction can then be modeled as a plenoptic sampling problem that has to encompass the change of viewpoint between each position of the omnidirectional sensor and the specific discretization of the real scene observed from each position. We formulate this problem as an ill-posed inverse problem that incorporates a regularization term based on a Total Variation (TV) prior. A graph variational formulation is used in order to ease the representation of omnidirectional data and to adapt the discretization of differential operators to the omnidirectional geometry. Experimental results on synthetic images demonstrate the relevance of this approach and its superiority compared to standard super-resolution using a single image.