While plasmonic antennas composed of building blocks made of the same material have been thoroughly studied, recent investigations have highlighted the unique opportunities enabled by making compositionally asymmetric plasmonic systems. So far, mainly heterostructures composed of nanospheres and nanodiscs have been investigated, revealing opportunities for the design of Fano resonant nanostructures, directional scattering, sensing and catalytic applications. In this article, an improved fabrication method is reported that enables precise tuning of the heterodimer geometry, with interparticle distances made down to a few nanometers between Au–Ag and Au–Al nanoparticles. A wide range of mode energy detuning and coupling conditions are observed by near field hyperspectral imaging performed with electron energy loss spectroscopy, supported by full wave analysis numerical simulations. These results provide direct insights into the mode hybridization of plasmonic heterodimers, pointing out the influence of each dimer constituent in the overall electromagnetic response. By relating the coupling of nondipolar modes and plasmon–interband interaction with the dimer geometry, this work facilitates the development of plasmonic heterostructures with tailored responses, beyond the possibilities offered by homodimers.