We present a methodology to quantify polarization and electron affinity changes at interfaces by combining scanning tunneling spectroscopy, off-axis electron holography in transmission electron microscopy (TEM), and self-consistent calculations of the electrostatic potential and electron phase change. We use a precisely known grown-in doping structure to calibrate the surface potential of the TEM lamella and thereby achieve a quantitative analysis of electron phase changes measured by off-axis electron holography. Using this calibration, we deduce quantitatively polarization and electron affinity changes for Al0.06Ga0.94 N/GaN and In0.05Ga0.95N/Al0.06Ga0.94N interfaces. The latter interface reveals, as expected, biaxial relaxation as well as polarization and electron affinity changes. However, at the Al0.06Ga0.94N/GaN interface anomalous lattice relaxations and vanishing polarization and electron affinity changes occur, whose underlying physical origin is anticipated to be total energy minimization by the minimization of Coulomb interactions between the polarization-induced interface charges.