The Bi/Ag(111), Pb/Ag(111), and Sb/Ag(111) surface alloys exhibit a two-dimensional band structure with a strongly enhanced Rashba-type spin splitting, which is in part attributed to the structural asymmetry resulting from an outward relaxation of the alloy atoms. In order to gain further insight into the spin splitting mechanism, we have experimentally determined the outward relaxation of the alloy atoms in these surface alloys using quantitative low-energy electron diffraction. The structure plays an important role in the size of the spin splitting as it dictates the potential landscape, the symmetry as well as the orbital character. Furthermore, we discuss the band ordering of the Pb/Ag(111) surface alloy as well as the reproducible formation of Sb/Ag(111) surface alloys with unfaulted (face-centered cubic) and faulted (hexagonally close-packed) top layer stacking.