The effect of irregular hole shape on the spectrum and radiation losses of a photonic crystal waveguide is studied using Bloch-mode expansion. Deviations from a circular hole are characterized by a radius fluctuation amplitude and correlation angle. It is found that the parameter that determines the magnitude of the effect of disorder is the standard deviation of the hole areas. Hence, for a fixed amplitude of the radius fluctuation around the hole, those effects are strongly dependent on the correlation angle of the irregular shape, which suggests how to potentially improve the quality of photonic crystal structures. (C) 2012 Optical Society of America