Oxygen diffusion plays an important role in grain growth and densification during the sintering of alumina ceramics and governs high-temperature processes such as creep. The atomistic mechanism for oxygen diffusion in alumina is, however, still debated; atomistic calculations not being able to match experimentally determined activation energies for oxygen vacancy diffusion. These calculations are, however, usually performed for perfectly pure crystals, whereas virtually every experimental alumina sample contains a significant fraction of impurity/dopants ions. In this study, we use atomistic defect cluster and nudged elastic band (NEB) calculations to model the effect of Mg impurities/dopants on defect binding energies and migration barriers. We find that oxygen vacancies can form energetically favorable clusters with Mg, which reduces the number of mobile species and leads to an additional 1.5 eV energy barrier for the detachment of a single vacancy from Mg. The migration barriers of diffusive jumps change such that an enhanced concentration of oxygen vacancies is expected around Mg ions. Mg impurities were also found to cause destabilization of certain vacancy configurations as well as enhanced vacancy-vacancy interaction.