High symmetry epitaxial quantum dots (QDs) with three or more symmetry planes provide a very promising route for the generation of entangled photons for quantum information applications. The great challenge to fabricate nanoscopic high symmetry QDs is further complicated by the lack of structural characterization techniques able to resolve small symmetry breaking. In this work, we present an approach for identifying and analyzing the signatures of symmetry breaking in the optical spectra of QDs. Exciton complexes in InGaAs/AlGaAs QDs grown along the [111]B crystalline axis in inverted tetrahedral pyramids are studied by polarization resolved photoluminescence spectroscopy combined with lattice temperature dependence, excitation power dependence and temporal photon correlation measurements. By combining such a systematic experimental approach with a simple theoretical approach based on a point-group symmetry analysis of the polarized emission patterns of each exciton complex, we demonstrate that it is possible to achieve a strict and coherent identification of all the observable spectral patterns of numerous exciton complexes and a quantitative determination of the fine structure splittings of their quantum states. This analysis is found to be particularly powerful for selecting QDs with the highest degree of symmetry (C-3v and D-3h) for potential applications of these QDs as polarization entangled photon sources. We exhibit the optical spectra when evolving towards asymmetrical QDs, and show the higher sensitivity of certain exciton complexes to symmetry breaking.