We report a systematic theoretical and experimental investigation of the polarization properties of columnar quantum dots (CQDs) with extremely large aspect ratio and high compositional contrast. A strong variation of the polarization of the dominant interband transitions as a function of the CQD composition and aspect ratio is predicted by the theory and observed experimentally. In optimized InAs CQDs grown on GaAs, dominant emission and lasing is obtained in the transverse-magnetic polarization, which shows the potential of this approach for polarization-independent amplifiers on GaAs.