We report on the effects of optical disorder on breaking the symmetry of the cavity modes of H-3 photonic crystal cavities incorporating site-controlled pyramidal quantum dots (QDs) as the internal light source. The high in-plane symmetry of the polarization states of the pyramidal QDs simplifies the analysis of the polarization states of the H-3 cavities. It is shown that the optical disorder induced by fabrication imperfections lifts the degeneracy of the two quadrupole cavity modes and tilts the elongation axes of the cavity mode patterns with respect to the ideal, hexagonal symmetry case. These results are useful for designing QD-cavity structures for polarization-entangled photon sources and few-QD lasers. (C) 2015 AIP Publishing LLC.