We propose a Riesz transform approach to the demodulation of digital holograms. The Riesz transform is a higher-dimensional extension of the Hilbert transform and is steerable to a desired orientation. Accurate demodulation of the hologram requires a reliable methodology by which quadrature-phase functions (or simply, quadratures) can be constructed. The Riesz transform, by itself, does not yield quadratures. However, one can start with the Riesz transform and construct the so-called vortex operator by employing the notion of quasi-eigenfunctions, and this approach results in accurate quadratures. The key advantage of using the vortex operator is that it effectively handles nonplanar fringes (interference patterns) and has the ability to compensate for the local orientation. Therefore, this method results in aberration-free holographic imaging even in the case when the wavefronts are not planar. We calibrate the method by estimating the orientation from a reference hologram, measured with an empty field of view. Demodulation results on synthesized planar as well as nonplanar fringe patterns show that the accuracy of demodulation is high. We also perform validation on real experimental measurements of Caenorhabditis elegans acquired with a digital holographic microscope. (c) 2012 Optical Society of America