We report on a combined experimental and theoretical investigation of the magnetic anisotropy of Fe and Co ultrathin layers on strongly polarizable metal substrates. Monolayer (ML) films of Co and Fe on Rh(111) have been investigated in situ by x-ray magnetic circular dichroism (XMCD), magneto-optic Kerr effect, and scanning tunneling microscopy. The experiments show that both magnetic adlayers exhibit ferromagnetic order and enhanced spin and orbital moments compared to the bulk metals. The easy magnetization axis of 1 ML Co was found to be in plane, in contrast to Co/Pt(111), and that of 1 ML Fe out of plane. The magnetic anisotropy energy (MAE) derived from the magnetization curves of the Fe and Co films is one order of magnitude larger than the respective bulk values. XMCD spectra measured at the Rh M2,3 edges evidence significant magnetic polarization of the Rh(111) surface with the induced magnetization closely following that of the overlayer during the reversal process. The easy axis of 1–3 ML Co/Rh(111) shows an oscillatory in-plane/out-of-plane behavior due to the competition between dipolar and crystalline MAE. We present a comprehensive theoretical treatment of the magnetic anisotropy of Fe and Co layers on Rh(111) and Pt(111) substrates. For free-standing hexagonally close-packed monolayers the MAE is in plane for Co and out of plane for Fe. The interaction with the substrate inverts the sign of the electronic contribution to the MAE, except for Fe/Rh(111), where the MAE is only strongly reduced. For Co/Rh(111), the dipolar contribution outweighs the band contribution, resulting in an in-plane MAE in agreement with experiment while for Co/Pt(111) the larger band contribution dominates, resulting in an out-of-plane MAE. For Fe films however, the calculations predict for both substrates an in-plane anisotropy in contradiction to the experiment. At least for Fe/Pt(111) comparison of theory and experiment suggests that the magnetic structure of the adlayer is more complex than the homogenous ferromagnetic order assumed in the calculations. The angular momentum and layer-resolved contributions of the overlayer and substrate to the MAE and orbital moment anisotropy are discussed with respect to the anisotropic hybridization of the 3d, 4d, and 5d electron states and vertical relaxation. The role of technically relevant parameters such as the thickness of the surface slab, density of k points in the Brillouin zone, and electron-density functionals is carefully analyzed.