We investigate experimentally and theoretically the impact of valence band mixing and spectrum of confined states on the polarization of light emitted from or absorbed by GaAs/AlGaAs semiconductor quantum dots and quantum wires with tailored heterostructure potential. In particular, such nanostructures with parabolic-profile confinement potentials, realized by organometallic vapor phase epitaxy inside pyramidal pits, served as model systems for the study. Different degrees of linear polarization (DOLP) of emitted light, depending on the confinement potential profile, the specific excitonic transition, and the level of excitation, are observed. A theoretical model shows that, besides the impact of valence band mixing, the overlap of conduction and valence band wavefunctions as well as state occupation probability and broadening of transitions determine the DOLP. The conclusions are useful for the design of quantum light emitters with controlled polarization properties.