First principles calculations are used to investigate electronic band structure and vibrational spectra of pnictogen-substituted ternary skutterudites. We compare the results with the prototypical binary composition CoSb3 to identify the effects of substitutions on the Sb site, and evaluate the potential of ternary skutterudites for thermoelectric applications. Electronic transport coefficients are computed within the Boltzmann transport formalism assuming a constant relaxation time, using a methodology based on maximally localized Wannier function interpolation. Our results point to a large sensitivity of the electronic transport coefficients to carrier concentration and to scattering mechanisms associated with the enhanced polarity. The ionic character of the bonds is used to explain the detrimental effect on the thermoelectric properties.