The silencing of the second harmonic generation process from plasmonic nanostructures corresponds to the limited far-field second harmonic radiation despite the huge fundamental electric field enhancement in the interstice between two plasmonic nanoparticles forming a nanodimer. In this article, we report a comprehensive investigation of this effect using a surface integral equation method. Various geometries are considered, including nanoantennas with cylindrical and rectangular arms as well as nanodimers with surface defects. The existence of the silencing of the second harmonic generation from plasmonic nanogaps is first confirmed, and the problem of the origin of the second harmonic light from these plasmonic nanostructures is addressed in detail. Our results show that the distribution of the second harmonic sources, especially on the arm sides, plays a non-negligible role in the overall second harmonic emission. This contribution is induced by retardation effects at the pump wavelength and results in a dipolar second harmonic emission.