Band inversion, one of the key signatures of time-reversal invariant topological insulators (TIs), arises mostly due to the spin-orbit (SO) coupling. Here, based on ab initio density-functional calculations, we report a theoretical investigation of the SO-driven band inversion in isostructural bismuth and antimony chalcogenide TIs from the viewpoint of its interplay with the crystal-field effect. We calculate the SO-induced energy shift of states in the top valence and bottom conduction manifolds and reproduce this behavior using a simple one-atom model adjusted to incorporate the crystal-field effect. The crystal-field splitting is shown to compete with the SO coupling, that is, stronger crystal-field splitting leads to weaker SO band shift. We further show how both these effects can be controlled by changing the chemical composition, whereas the crystal-field splitting can be tuned by means of uniaxial strain. These results provide a practical guidance to the rational design of novel TIs as well as to controlling the properties of existing materials.