The emission pattern of charged excitons in a semiconductor quantum dot (QD) is composed of a quadruplet of linearly polarized lines when a magnetic field is applied in a Voigt configuration. The orientation of the linear polarization of exciton emission is controlled by the orientation of the magnetic field in QDs with C-3v symmetry while for QDs with C-2v symmetry it is not. We demonstrate that the g factor of holes is very sensitive to the dot shape asymmetry but that of electrons is not. By comparing the effective g factors obtained for the neutral and charged excitons in the same quantum dot, we uncover the role of Coulomb correlations in these excitonic states. We show that the C-3v symmetry of pyramidal QDs makes them ideal candidates for implementing all-optical many-qubits gates based on electron spin as a quantum bit.